displaCy

{ "cells": [ { "cell_type": "markdown", "id": "c86ab743-4281-4362-88e1-e3c664e88d2c", "metadata": {}, "source": [ "# 06 Part of Speech tagging" ] }, { "cell_type": "markdown", "id": "ca8d69c5-600a-4008-99ab-5d57a0fa1000", "metadata": {}, "source": [ "## Coarse-grained Part-of-speech Tags\n", "\n", "Every token is assigned a POS Tag from the following list:\n", "\n", "| POS | DESCRIPTION | EXAMPLES |\n", "|--------|--------------------------------|-------------------------------------------------------|\n", "| ADJ | adjective | *big, old, green, incomprehensible, first* |\n", "| ADP | adposition | *in, to, during* |\n", "| ADV | adverb | *very, tomorrow, down, where, there* |\n", "| AUX | auxiliary | *is, has (done), will (do), should (do)* |\n", "| CONJ | conjunction | *and, or, but* |\n", "| CCONJ | coordinating conjunction | *and, or, but* |\n", "| DET | determiner | *a, an, the* |\n", "| INTJ | interjection | *psst, ouch, bravo, hello* |\n", "| NOUN | noun | *girl, cat, tree, air, beauty* |\n", "| NUM | numeral | *1, 2017, one, seventy-seven, IV, MMXIV* |\n", "| PART | particle | *s, not,* |\n", "| PRON | pronoun | *I, you, he, she, myself, themselves, somebody* |\n", "| PROPN | proper noun | *Mary, John, London, NATO, HBO* |\n", "| PUNCT | punctuation | *., (, ), ?* |\n", "| SCONJ | subordinating conjunction | *if, while, that* |\n", "| VERB | verb | *run, runs, running, eat, ate, eating* |\n", "| X | other | *sfpksdpsxmsa* |\n", "| SPACE | space | |\n" ] }, { "cell_type": "markdown", "id": "c3dc8b6f-28c3-4b09-8c90-c7fe90bd9a6a", "metadata": {}, "source": [ "___\n", "## Fine-grained Part-of-speech Tags\n", "Tokens are subsequently given a fine-grained tag as determined by morphology:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "

POS	Description	Fine-grained Tag	Description	Morphology
ADJ	adjective	AFX	affix	Hyph=yes
ADJ		JJ	adjective	Degree=pos
ADJ		JJR	adjective, comparative	Degree=comp
ADJ		JJS	adjective, superlative	Degree=sup
ADJ		PDT	predeterminer	AdjType=pdt PronType=prn
ADJ		PRP\\$	pronoun, possessive	PronType=prs Poss=yes
ADJ		WDT	wh-determiner	PronType=int rel
ADJ		WP\\$	wh-pronoun, possessive	Poss=yes PronType=int rel
ADP	adposition	IN	conjunction, subordinating or preposition
ADV	adverb	EX	existential there	AdvType=ex
ADV		RB	adverb	Degree=pos
ADV		RBR	adverb, comparative	Degree=comp
ADV		RBS	adverb, superlative	Degree=sup
ADV		WRB	wh-adverb	PronType=int rel
CONJ	conjunction	CC	conjunction, coordinating	ConjType=coor
DET	determiner	DT	determiner
INTJ	interjection	UH	interjection
NOUN	noun	NN	noun, singular or mass	Number=sing
NOUN		NNS	noun, plural	Number=plur
NOUN		WP	wh-pronoun, personal	PronType=int rel
NUM	numeral	CD	cardinal number	NumType=card
PART	particle	POS	possessive ending	Poss=yes
PART		RP	adverb, particle
PART		TO	infinitival to	PartType=inf VerbForm=inf
PRON	pronoun	PRP	pronoun, personal	PronType=prs
PROPN	proper noun	NNP	noun, proper singular	NounType=prop Number=sign
PROPN		NNPS	noun, proper plural	NounType=prop Number=plur
PUNCT	punctuation	-LRB-	left round bracket	PunctType=brck PunctSide=ini
PUNCT		-RRB-	right round bracket	PunctType=brck PunctSide=fin
PUNCT		,	punctuation mark, comma	PunctType=comm
PUNCT		:	punctuation mark, colon or ellipsis
PUNCT		.	punctuation mark, sentence closer	PunctType=peri
PUNCT		''	closing quotation mark	PunctType=quot PunctSide=fin
PUNCT		\"\"	closing quotation mark	PunctType=quot PunctSide=fin
PUNCT		``	opening quotation mark	PunctType=quot PunctSide=ini
PUNCT		HYPH	punctuation mark, hyphen	PunctType=dash
PUNCT		LS	list item marker	NumType=ord
PUNCT		NFP	superfluous punctuation
SYM	symbol	\n", "
SYM		\\$	symbol, currency	SymType=currency
SYM		SYM	symbol
VERB	verb	BES	auxiliary -be-
VERB		HVS	forms of -have-
VERB		MD	verb, modal auxiliary	VerbType=mod
VERB		VB	verb, base form	VerbForm=inf
VERB		VBD	verb, past tense	VerbForm=fin Tense=past
VERB		VBG	verb, gerund or present participle	VerbForm=part Tense=pres Aspect=prog
VERB		VBN	verb, past participle	VerbForm=part Tense=past Aspect=perf
VERB		VBP	verb, non-3rd person singular present	VerbForm=fin Tense=pres
VERB		VBZ	verb, 3rd person singular present	VerbForm=fin Tense=pres Number=sing Person=3
X	other	ADD	email
X		FW	foreign word	Foreign=yes
X		GW	additional word in multi-word expression
X		XX	unknown
SPACE	space	_SP	space
		NIL	missing tag

" ] }, { "cell_type": "markdown", "id": "2cda5b97-926e-4dc9-b03c-71db3aea3621", "metadata": {}, "source": [ "import spacy" ] }, { "cell_type": "code", "execution_count": 19, "id": "80b1125a-3647-4a53-ad11-5a2bed191237", "metadata": {}, "outputs": [], "source": [ "nlp = spacy.load('en_core_web_sm')" ] }, { "cell_type": "code", "execution_count": 3, "id": "6a5e4e16-4d1c-4e92-848f-1df10ded9927", "metadata": {}, "outputs": [], "source": [ "doc = nlp(u\"The quick brown fox jumped over the lazy dog's back\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "644272ca-886d-480c-b5ab-b236b75f1827", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The quick brown fox jumped over the lazy dog's back\n" ] } ], "source": [ "print(doc.text)" ] }, { "cell_type": "code", "execution_count": 6, "id": "5018cdf5-01e1-44f1-bf51-0b03f2edb5cd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "jumped VERB VBD\n" ] } ], "source": [ "print(doc[4].text, doc[4].pos_, doc[4].tag_)" ] }, { "cell_type": "code", "execution_count": 34, "id": "1b0f2918-5b3b-42c7-8b66-9456122ecaa1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The DET DT determiner\n", "quick ADJ JJ adjective (English), other noun-modifier (Chinese)\n", "brown ADJ JJ adjective (English), other noun-modifier (Chinese)\n", "fox NOUN NN noun, singular or mass\n", "jumped VERB VBD verb, past tense\n", "over ADP IN conjunction, subordinating or preposition\n", "the DET DT determiner\n", "lazy ADJ JJ adjective (English), other noun-modifier (Chinese)\n", "dog NOUN NN noun, singular or mass\n", "'s PART POS possessive ending\n", "back NOUN NN noun, singular or mass\n" ] } ], "source": [ "for token in doc:\n", " print(f\"{token.text} {token.pos_} {token.tag_} {spacy.explain(token.tag_)}\")" ] }, { "cell_type": "markdown", "id": "e5307d92-26e3-4c50-87f8-bcb6e756d927", "metadata": {}, "source": [ "Get the count for each POS in the sentence:" ] }, { "cell_type": "code", "execution_count": 35, "id": "12f09530-266d-4c27-b1e6-fbf7100bc807", "metadata": {}, "outputs": [], "source": [ "POS_counts = doc.count_by(spacy.attrs.POS)" ] }, { "cell_type": "code", "execution_count": 36, "id": "b4cd6ee2-a6e4-4b2a-a6b3-4378f510f977", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{90: 2, 84: 3, 92: 3, 100: 1, 85: 1, 94: 1}" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "POS_counts" ] }, { "cell_type": "markdown", "id": "8dc845b4-6114-4d48-b7e7-13a2ba41301b", "metadata": {}, "source": [ "There are 4 adjectives:" ] }, { "cell_type": "code", "execution_count": 37, "id": "ab145f11-73d5-463f-a2d3-27cb0f6be9d7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'ADJ'" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "doc.vocab[84].text" ] }, { "cell_type": "code", "execution_count": 45, "id": "6f748932-c5bb-47a6-b10e-80c8ce6ff3a6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 84. ADJ 3\n", " 85. ADP 1\n", " 90. DET 2\n", " 92. NOUN 3\n", " 94. PART 1\n", " 100. VERB 1\n" ] } ], "source": [ "for k, v in sorted(POS_counts.items()):\n", " print(f'{k:{5}}. {doc.vocab[k].text:{4}} {v}')" ] }, { "cell_type": "code", "execution_count": 46, "id": "d1e5d7a4-2d78-41ef-b077-891b0c82960a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 74. POS 1\n", "1292078113972184607. IN 1\n", "10554686591937588953. JJ 3\n", "15267657372422890137. DT 2\n", "15308085513773655218. NN 3\n", "17109001835818727656. VBD 1\n" ] } ], "source": [ "TAG_counts = doc.count_by(spacy.attrs.TAG)\n", "\n", "for k, v in sorted(TAG_counts.items()):\n", " print(f'{k:{5}}. {doc.vocab[k].text:{4}} {v}')" ] }, { "cell_type": "markdown", "id": "34bd20b1-27de-4946-b51b-6e75d2aa9b0f", "metadata": {}, "source": [ "Sintactic dependencies:" ] }, { "cell_type": "code", "execution_count": 47, "id": "0c2208f1-97ca-4199-a6fd-429148e36419", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 74. POS 1\n", "1292078113972184607. IN 1\n", "10554686591937588953. JJ 3\n", "15267657372422890137. DT 2\n", "15308085513773655218. NN 3\n", "17109001835818727656. VBD 1\n" ] } ], "source": [ "DEP_counts = doc.count_by(spacy.attrs.TAG)\n", "\n", "for k, v in sorted(DEP_counts.items()):\n", " print(f'{k:{5}}. {doc.vocab[k].text:{4}} {v}')" ] }, { "cell_type": "markdown", "id": "9d0806fc-a6f0-4510-992d-c0710ce3387e", "metadata": {}, "source": [ "## Visualize POS" ] }, { "cell_type": "code", "execution_count": 49, "id": "f7cb92a4-55a8-4547-8d8e-509df22486a8", "metadata": {}, "outputs": [], "source": [ "from spacy import displacy" ] }, { "cell_type": "markdown", "id": "fe88bd2f-c2d1-4008-8af5-ffeacb9ad77e", "metadata": {}, "source": [ "visualizing sintactic dependencies:" ] }, { "cell_type": "code", "execution_count": 52, "id": "16f015f2-c280-4ebd-8444-8d5121b2f8c6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "displacy.render(doc, style='dep', jupyter=True)" ] }, { "cell_type": "code", "execution_count": 53, "id": "66721ef4-5637-4caa-9e05-1d1c70377ba0", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "options = {\n", " 'distance': 110,\n", " 'compact': 'True',\n", " 'color': 'yellow',\n", " 'bg': '#09a3d5',\n", " 'font': 'Times'\n", "}\n", "displacy.render(doc, style='dep', jupyter=True, options=options)" ] }, { "cell_type": "code", "execution_count": 54, "id": "f8d5d1be-46a0-413b-b945-1665d6c2b509", "metadata": {}, "outputs": [], "source": [ "doc2 = nlp(u\"This is a sentence. This is another sentence, possibly longer than the other.\")" ] }, { "cell_type": "code", "execution_count": 55, "id": "6bc96704-61cf-41d8-a60e-c75daef33ff0", "metadata": {}, "outputs": [], "source": [ "spans = list(doc2.sents)" ] }, { "cell_type": "code", "execution_count": 56, "id": "7ae27fe5-d2d1-443f-b80a-e605fc643eab", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/lib/python3.11/site-packages/spacy/displacy/__init__.py:106: UserWarning: [W011] It looks like you're calling displacy.serve from within a Jupyter notebook or a similar environment. This likely means you're already running a local web server, so there's no need to make displaCy start another one. Instead, you should be able to replace displacy.serve with displacy.render to show the visualization.\n", " warnings.warn(Warnings.W011)\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " displaCy\n", " \n", "\n", " \n", "

\n", "\n", "\n", "\n", "

\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Using the 'dep' visualizer\n", "Serving on http://0.0.0.0:5000 ...\n", "\n", "Shutting down server on port 5000.\n" ] } ], "source": [ "displacy.serve(spans, style='dep',options={'distance': 110})" ] }, { "cell_type": "code", "execution_count": null, "id": "0dbb45b6-b047-489c-8017-0ae059719a9f", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }