displaCy

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 03 Spacy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Spacy is a high performance NLP library.\r\n", "It's possible to install spacy using command:\r\n", "```\r\n", "conda install -c conda-forge spacy\r\n", "```\r\n", "And is possible to install the en language pack using:\r\n", "```\r\n", "python -m spacy download en_core_web_sm\r\n", "```" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "import spacy\r\n", "nlp = spacy.load('en_core_web_sm')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tokenization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's possible to tokenize a string with the spacy as follow:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "doc = nlp(u'Tesla is looking at buying U.S. startup for $6 million')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "the document object contains an array of tokens. each Token has serveral attributes for example:\r\n", "- token.text => tokenized word or symbol\r\n", "- token.pos => number representing what part of the speech is (adverb, verb, noun, conjuction)\r\n", "- token.pos_ => textual representation of the part of the speech\r\n", "- token.dep_ => more information about token POS (sintactic dependency)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------------\n", "Tesla\n", "96\n", "PROPN\n", "nsubj\n", "--------------\n", "--------------\n", "is\n", "87\n", "AUX\n", "aux\n", "--------------\n", "--------------\n", "looking\n", "100\n", "VERB\n", "ROOT\n", "--------------\n", "--------------\n", "at\n", "85\n", "ADP\n", "prep\n", "--------------\n", "--------------\n", "buying\n", "100\n", "VERB\n", "pcomp\n", "--------------\n", "--------------\n", "U.S.\n", "96\n", "PROPN\n", "compound\n", "--------------\n", "--------------\n", "startup\n", "92\n", "NOUN\n", "dobj\n", "--------------\n", "--------------\n", "for\n", "85\n", "ADP\n", "prep\n", "--------------\n", "--------------\n", "$\n", "99\n", "SYM\n", "quantmod\n", "--------------\n", "--------------\n", "6\n", "93\n", "NUM\n", "compound\n", "--------------\n", "--------------\n", "million\n", "93\n", "NUM\n", "pobj\n", "--------------\n" ] } ], "source": [ "for token in doc:\r\n", " print('--------------')\r\n", " print(token.text)\r\n", " print(token.pos)\r\n", " print(token.pos_)\r\n", " print(token.dep_)\r\n", " print('--------------')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pipeline Object" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The pipeline object contains the set of operations that have been performed on the text. The nlp object exposes a pipeline attribute to give a look at operations that have been made:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('tok2vec', ),\n", " ('tagger', ),\n", " ('parser', ),\n", " ('attribute_ruler',\n", " ),\n", " ('lemmatizer', ),\n", " ('ner', )]" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nlp.pipeline" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nlp.pipe_names" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "doc2 = nlp(u'Tesla isn\\'t looking into startups anymore.')" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tesla PROPN nsubj\n", "is AUX aux\n", "n't PART neg\n", "looking VERB ROOT\n", "into ADP prep\n", "startups NOUN pobj\n", "anymore ADV advmod\n", ". PUNCT punct\n" ] } ], "source": [ "for token in doc2:\r\n", " print(token.text, token.pos_, token.dep_)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NNP\n", "Tesla\n", "Xxxxx\n" ] } ], "source": [ "print(doc2[0].tag_)\r\n", "print(doc2[0].lemma_)\r\n", "print(doc2[0].shape_)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This is the first sentence.\n", "This is another sentence.\n", "This is the last sentence\n" ] } ], "source": [ "doc3 = nlp(u'This is the first sentence. This is another sentence. This is the last sentence')\r\n", "for sentence in doc3.sents:\r\n", " print(sentence)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "doc3[6].is_sent_start" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tokenization\r\n", "\r\n", "Tokenization is commonly the first step of NLP and is the process of breaking up the original text into component pieces(tokens).\r\n", "\r\n", "\r\n", "Glossary:\r\n", "- Prefix: Character(s) at the beginning ($ ( \")\r\n", "- Suffix: Character(s) at the end (km ) , . !)\r\n", "- Infix: Character(s) in between (- -- / ...)\r\n", "- Exception: Special-case rule to split a string into several tokens or prevent a token from being split when punctuation rules are applied (Let's U.S.)\r\n", "\r\n", "\r\n", "It structured in the following steps:\r\n", "- split on whitespaces\r\n", "- split on prefixes\r\n", "- split on exceptions\r\n", "- split on suffixes\r\n", "- second split on exceptions\r\n", "\r\n", "\r\n", "Tokens are pieces of the originall text.\r\n", "\r\n", "\r\n", "Everything that is used to understand the meaning of the text is derived from tokens and their relationships to one another." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "import spacy\r\n", "\r\n", "nlp = spacy.load('en_core_web_sm')" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "mystring = '\"We\\'re moving to L.A.\"'" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "doc = nlp(mystring)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\"\n", "We\n", "'re\n", "moving\n", "to\n", "L.A.\n", "\"\n" ] } ], "source": [ "for token in doc:\r\n", " print(token.text)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "767" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# all registered tokens from en_core_web_sm\r\n", "len(doc.vocab)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "!Document object elements cannot be reassigned such as: doc[0] = 'x'" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "doc2 = nlp(u'Apple to build a Hong Kong factory for $6 million')" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Apple|to|build|a|Hong|Kong|factory|for|$|6|million|" ] } ], "source": [ "for token in doc2:\r\n", " print(token.text, end='|')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Spacy Entities\r\n", "\r\n", "Under the hood Spacy is able to recognized named entities in the given sentence." ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Apple\n", "ORG\n", "Companies, agencies, institutions, etc.\n", "\n", "\n", "Hong Kong\n", "GPE\n", "Countries, cities, states\n", "\n", "\n", "$6 million\n", "MONEY\n", "Monetary values, including unit\n", "\n", "\n" ] } ], "source": [ "for entity in doc2.ents:\r\n", " print(entity)\r\n", " # type of detected entity.\r\n", " print(entity.label_)\r\n", " # Spacy explanation for that label\r\n", " print(str(spacy.explain(entity.label_)))\r\n", " print('\\n')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Chunks" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "doc3 = nlp(u'Autonomous cars shift insurance liability toward manufacturers.')" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Autonomous cars\n", "insurance liability\n", "manufacturers\n" ] } ], "source": [ "for chunk in doc3.noun_chunks:\r\n", " print(chunk)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Displacy" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "from spacy import displacy\r\n", "\r\n", "doc = nlp(u\"Apple is going to build a U.K. Factory for $6 million\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Displacing syntactic dependencies:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "displacy.render(doc,style='dep',jupyter=True,options={'distance': 110})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Displacing entity recognition:" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

Over \n", "\n", " last year quarter\n", " DATE\n", "\n", " \n", "\n", " Apple\n", " ORG\n", "\n", " sold \n", "\n", " nearly 20 thousand\n", " CARDINAL\n", "\n", " \n", "\n", " iPods\n", " PRODUCT\n", "\n", " for a profit of \n", "\n", " $6 million\n", " MONEY\n", "\n", ".

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "doc = nlp(u\"Over last year quarter Apple sold nearly 20 thousand iPods for a profit of $6 million.\")\r\n", "\r\n", "displacy.render(doc,style=\"ent\",jupyter=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Serving displacy outside Jupyter notebook:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\loren\\anaconda3\\Lib\\site-packages\\spacy\\displacy\\__init__.py:108: UserWarning: [W011] It looks like you're calling displacy.serve from within a Jupyter notebook or a similar environment. This likely means you're already running a local web server, so there's no need to make displaCy start another one. Instead, you should be able to replace displacy.serve with displacy.render to show the visualization.\n", " warnings.warn(Warnings.W011)\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " displaCy\n", " \n", "\n", " \n", "

\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Using the 'dep' visualizer\n", "Serving on http://0.0.0.0:5000 ...\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "127.0.0.1 - - [30/Jul/2023 13:39:28] \"GET / HTTP/1.1\" 200 3395\n", "127.0.0.1 - - [30/Jul/2023 13:39:28] \"GET /favicon.ico HTTP/1.1\" 200 3395\n" ] } ], "source": [ "doc = nlp(u\"This is a sentence.\")\r\n", "\r\n", "displacy.serve(doc, style='dep')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stemming (Derivazione)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Stemming is a somewhat crude method for cataloging related words; it essentially chops off letters from the end until the stem is reached.\r\n", "Spacy hasn't a stemmer built-in because it aims to rely on lemmatization.\r\n", "\r\n", "\r\n", "Lemmatization is a more sophisticated methods to work with languages such as english that has several exceptions.\r\n", "\r\n", "\r\n", "One of the most common stemming tools is Porter's Algorithm (Martin Porter, 1980), it uses 5 phases of word reduction, each with its own set of mapping rules.\r\n", "It strives to reduce the original word to its stem (ex. SSES -> SS, caresses -> caress, ponies -> poni, relational -> relate).\r\n", "\r\n", "\r\n", "Since Spacy doesn't include a stemmer, NLTK is used as source in the following example:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import nltk\r\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from nltk.stem.porter import PorterStemmer" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "p_stemmer = PorterStemmer()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "run--------------run\n", "runner--------------runner\n", "ran--------------ran\n", "runs--------------run\n", "easily--------------easili\n", "fairly--------------fairli\n", "fairness--------------fair\n" ] } ], "source": [ "words = ['run', 'runner', 'ran', 'runs', 'easily', 'fairly', 'fairness']\r\n", "\r\n", "for word in words:\r\n", " print(word + '--------------' + p_stemmer.stem(word))" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from nltk.stem.snowball import SnowballStemmer" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "s_stemmer = SnowballStemmer(language='english')" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "run ------------ run\n", "runner ------------ runner\n", "ran ------------ ran\n", "runs ------------ run\n", "easily ------------ easili\n", "fairly ------------ fair\n", "fairness ------------ fair\n" ] } ], "source": [ "for word in words:\r\n", " print(f'{word} ------------ {s_stemmer.stem(word)}')" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "words = ['generous', 'generation', 'generously', 'generate']" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "generous ----- generous\n", "generation ----- generat\n", "generously ----- generous\n", "generate ----- generat\n" ] } ], "source": [ "for word in words:\r\n", " print(f'{word} ----- {s_stemmer.stem(word)}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Lemmatization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In contast to stemming, lemmatization looks beyond word reduction, and considers a language's full vocabolary to apply a morphological analysis to words (was => be, mice => mouse, meeting => meet)." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "import spacy\r\n", "\r\n", "nlp = spacy.load('en_core_web_sm')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "doc = nlp(u'I am runner running in a race because I love to run since i ran today')" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "I --- PRON --- 4690420944186131903 --- I \n", "am --- AUX --- 10382539506755952630 --- be \n", "runner --- NOUN --- 12640964157389618806 --- runner \n", "running --- VERB --- 12767647472892411841 --- run \n", "in --- ADP --- 3002984154512732771 --- in \n", "a --- DET --- 11901859001352538922 --- a \n", "race --- NOUN --- 8048469955494714898 --- race \n", "because --- SCONJ --- 16950148841647037698 --- because \n", "I --- PRON --- 4690420944186131903 --- I \n", "love --- VERB --- 3702023516439754181 --- love \n", "to --- PART --- 3791531372978436496 --- to \n", "run --- VERB --- 12767647472892411841 --- run \n", "since --- SCONJ --- 10066841407251338481 --- since \n", "i --- PRON --- 4690420944186131903 --- I \n", "ran --- VERB --- 12767647472892411841 --- run \n", "today --- NOUN --- 11042482332948150395 --- today \n" ] } ], "source": [ "for token in doc:\r\n", " print(f'{token.text:{12}} --- {token.pos_:{6}} --- {token.lemma:<{22}} --- {token.lemma_:<{20}}')" ] } ], "metadata": { "interpreter": { "hash": "7a862a13b35dbac2e51b8ddb35bec28b279ff8fc1fc5f914f468fce7c1ea1b96" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 4 }