{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "source": [ "___\n", "\n", "

\n", "___" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Parts of Speech Assessment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this assessment we'll be using the short story [The Tale of Peter Rabbit](https://en.wikipedia.org/wiki/The_Tale_of_Peter_Rabbit) by Beatrix Potter (1902).
The story is in the public domain; the text file was obtained from [Project Gutenberg](https://www.gutenberg.org/ebooks/14838.txt.utf-8)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# RUN THIS CELL to perform standard imports:\n", "import spacy\n", "nlp = spacy.load('en_core_web_sm')\n", "from spacy import displacy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**1. Create a Doc object from the file `peterrabbit.txt`**
\n", "> HINT: Use `with open('../TextFiles/peterrabbit.txt') as f:`" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "with open('./peterrabbit.txt') as f:\n", " doc = nlp(f.read())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**2. For every token in the third sentence, print the token text, the POS tag, the fine-grained TAG tag, and the description of the fine-grained tag.**" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "sent = list(doc.sents)[2]" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "They PRON PRP pronoun, personal \n", "lived VERB VBD verb, past tense \n", "with ADP IN conjunction, subordinating or preposition\n", "their PRON PRP$ pronoun, possessive \n", "Mother NOUN NN noun, singular or mass \n", "in ADP IN conjunction, subordinating or preposition\n", "a DET DT determiner \n", "sand NOUN NN noun, singular or mass \n", "- PUNCTHYPH punctuation mark, hyphen \n", "bank NOUN NN noun, singular or mass \n", ", PUNCT, punctuation mark, comma \n", "underneath ADP IN conjunction, subordinating or preposition\n", "the DET DT determiner \n", "root NOUN NN noun, singular or mass \n", "of ADP IN conjunction, subordinating or preposition\n", "a DET DT determiner \n", "\n", " SPACE_SP whitespace \n", "very ADV RB adverb \n", "big ADJ JJ adjective (English), other noun-modifier (Chinese)\n", "fir NOUN NN noun, singular or mass \n", "- PUNCTHYPH punctuation mark, hyphen \n", "tree NOUN NN noun, singular or mass \n", ". PUNCT. punctuation mark, sentence closer\n", "\n", "\n", " SPACE_SP whitespace \n" ] } ], "source": [ "for token in sent:\n", " print(f'{token.text:{15}}{token.pos_:{5}}{token.tag_:{5}}{spacy.explain(token.tag_):{30}}')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "They PRON PRP pronoun, personal\n", "lived VERB VBD verb, past tense\n", "with ADP IN conjunction, subordinating or preposition\n", "their ADJ PRP$ pronoun, possessive\n", "Mother PROPN NNP noun, proper singular\n", "in ADP IN conjunction, subordinating or preposition\n", "a DET DT determiner\n", "sand NOUN NN noun, singular or mass\n", "- PUNCT HYPH punctuation mark, hyphen\n", "bank NOUN NN noun, singular or mass\n", ", PUNCT , punctuation mark, comma\n", "underneath ADP IN conjunction, subordinating or preposition\n", "the DET DT determiner\n", "root NOUN NN noun, singular or mass\n", "of ADP IN conjunction, subordinating or preposition\n", "a DET DT determiner\n", "\n", " SPACE None\n", "very ADV RB adverb\n", "big ADJ JJ adjective\n", "fir NOUN NN noun, singular or mass\n", "- PUNCT HYPH punctuation mark, hyphen\n", "tree NOUN NN noun, singular or mass\n", ". PUNCT . punctuation mark, sentence closer\n", "\n", "\n", " SPACE _SP None\n" ] } ], "source": [ "# Enter your code here:\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**3. Provide a frequency list of POS tags from the entire document**" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "POS_counts = doc.count_by(spacy.attrs.POS)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 84. ADJ 53\n", " 85. ADP 125\n", " 86. ADV 63\n", " 87. AUX 49\n", " 89. CCONJ 61\n", " 90. DET 90\n", " 92. NOUN 172\n", " 93. NUM 9\n", " 94. PART 28\n", " 95. PRON 110\n", " 96. PROPN 74\n", " 97. PUNCT 171\n", " 98. SCONJ 19\n", " 100. VERB 135\n", " 103. SPACE 99\n" ] } ], "source": [ "for k, v in sorted(POS_counts.items()):\n", " print(f'{k:{5}}. {doc.vocab[k].text:{4}} {v}')" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "83. ADJ : 83\n", "84. ADP : 127\n", "85. ADV : 75\n", "88. CCONJ: 61\n", "89. DET : 90\n", "91. NOUN : 176\n", "92. NUM : 8\n", "93. PART : 36\n", "94. PRON : 72\n", "95. PROPN: 75\n", "96. PUNCT: 174\n", "99. VERB : 182\n", "102. SPACE: 99\n" ] } ], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**4. CHALLENGE: What percentage of tokens are nouns?**
\n", "HINT: the attribute ID for 'NOUN' is 91" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1258" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(doc)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "92" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "org = doc.vocab.strings[u\"NOUN\"]\n", "org" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "13.672496025437201" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "100*POS_counts[org]/len(doc)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "176/1258 = 13.99%\n" ] } ], "source": [ "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**5. Display the Dependency Parse for the third sentence**" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from spacy import displacy" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "displacy.render(doc, style='dep', jupyter=True)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**6. Show the first two named entities from Beatrix Potter's *The Tale of Peter Rabbit* **" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The Tale of Peter Rabbit - WORK_OF_ART - Titles of books, songs, etc.\n", "Beatrix Potter - PERSON - People, including fictional\n" ] } ], "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**7. How many sentences are contained in *The Tale of Peter Rabbit*?**" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "55" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(list(doc.sents))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "56" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**8. CHALLENGE: How many sentences contain named entities?**" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "35" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sents_count = 0\n", "\n", "for sent in doc.sents:\n", " if (len(sent.ents) > 0):\n", " sents_count+=1\n", "\n", "sents_count" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "49" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**9. CHALLENGE: Display the named entity visualization for `list_of_sents[0]` from the previous problem**" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", " The Tale of Peter Rabbit\n", " WORK_OF_ART\n", "\n", ", by \n", "\n", " Beatrix Potter\n", " PERSON\n", "\n", " (\n", "\n", " 1902\n", " DATE\n", "\n", ").

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "displacy.render(list(doc.sents)[0], style='ent')" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", " The Tale of Peter Rabbit\n", " WORK_OF_ART\n", "\n", ", by \n", "\n", " Beatrix Potter\n", " PERSON\n", "\n", " (\n", "\n", " 1902\n", " DATE\n", "\n", ").\n", "\n", "

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Great Job!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 4 }