{ "cells": [ { "cell_type": "markdown", "id": "669899c2-2ba7-46f7-90be-595ee564d5a2", "metadata": {}, "source": [ "## 15 Topic Modeling" ] }, { "cell_type": "markdown", "id": "e3332b9a-3c73-47cb-9ebb-0ba9a85ccc86", "metadata": {}, "source": [ "Topic modeling it is an unsupervised machine learning algorithm that allows to efficiently analyze large volumes of text by clustering documents into topics.\n", "\n", "It works with unlabeled data (historical data without the given prediction), it attempts to \"discover\" the labels.\n", "\n", "In the case of text data, it attempts to discover clusters of documents, grouped together by topic.\n", "\n", "The correct topic or the right answer aren't known, everything is known is that documents clustered together share similar topics ideas.\n", "\n", "It is up to the user to identify what these topics represent, there's no a good way to evaluate how good it did." ] }, { "cell_type": "markdown", "id": "788f8a13-4700-49a6-a77b-fc1998c585c6", "metadata": {}, "source": [ "## LDA (Latent Dirichlet Allocation)" ] }, { "cell_type": "markdown", "id": "091e8326-774a-4ac3-8dcf-b86e481be15a", "metadata": {}, "source": [ "LDA is based off a probability distribution named \"Dirichlet Distribution\".\n", "\n", "### Assumptions\n", "\n", "- Documents with similar topics use similar groups of words.\n", "- Latent topics can then be found by searching for groups of words that frequently occur together in documents across the corpus." ] }, { "cell_type": "markdown", "id": "bbe4f20f-2495-406c-8044-f0c8f0cb96c6", "metadata": {}, "source": [ "### How it works\n", "\n", "- Documents are probability distributions over latent topics.\n", "- By defining e.g. 5 topics, each document can be represented as the probability distribution of these topics over the corpus.\n", "- Topics themeselves are probability distribution over words.\n", "- LDA represents documents as mixtures of topics that split out words with certain probabilities" ] }, { "cell_type": "markdown", "id": "1f2a919f-25ca-4d69-9a45-d5670a28cf93", "metadata": {}, "source": [ "### Steps\n", "The final purpose is to get out the topic representation of each document and the words associated to each topic.\n", "\n", "- take a set of documents\n", "- Choose some fixed number of K topics to discover\n", "- Randomly assign each word in the document to one of the K topics(Initially obtained results won't make sense).\n", "- It will iterate over every word in every document for each topic to calculate:\n", "\n", "```\n", "p(topic(t) | document(d)) = the proportion of words in document(d) that are currently assigned to topic(t)\n", "```\n", "\n", "```\n", "p(word(w) | topic(t)) = the proportion of assignments to topic(t) over all documents that come form this word(w).\n", "```\n", "\n", "Reassign word(w) a new topic, where we choose topic(t) with probability:\n", "\n", "```\n", "p(topic(t) | document(d)) * p(word(w) | topic(t))\n", "```\n", "(it's essentially the probability that topic(t) generated word(w))\n", "\n", "\n", "After repeating these steps a large number of time, we eventually reach a roughly steady state where the assignments are acceptable." ] }, { "cell_type": "markdown", "id": "bf47b8ea-4ca4-421b-b969-0b2db364f5b4", "metadata": {}, "source": [ "### Output\n", "\n", "The algorithm will detect:\n", "- Document N assigned to Topic#4\n", "- Most common words (highest probability) for Topic#4: ['cat', 'vet', 'birds']\n", "\n", "It's up to the user to interpret these topics." ] }, { "cell_type": "markdown", "id": "5072cf9d-1cc6-4888-b863-6543723398af", "metadata": {}, "source": [ "## LDA with Python" ] }, { "cell_type": "code", "execution_count": 1, "id": "33883983-80d7-45e7-957a-d90908dfcee3", "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 4, "id": "60a1a93d-2a99-48fa-a545-187f85fdc6b7", "metadata": {}, "outputs": [], "source": [ "npr = pd.read_csv('npr.csv')" ] }, { "cell_type": "code", "execution_count": 5, "id": "04b1f1ee-fc48-410c-904f-70f7032b9080", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	Article
0	In the Washington of 2016, even when the polic...
1	Donald Trump has used Twitter — his prefe...
2	Donald Trump is unabashedly praising Russian...
3	Updated at 2:50 p. m. ET, Russian President Vl...
4	From photography, illustration and video, to d...

\n", "

" ], "text/plain": [ " Article\n", "0 In the Washington of 2016, even when the polic...\n", "1 Donald Trump has used Twitter — his prefe...\n", "2 Donald Trump is unabashedly praising Russian...\n", "3 Updated at 2:50 p. m. ET, Russian President Vl...\n", "4 From photography, illustration and video, to d..." ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# DF containing articles content\n", "npr.head()" ] }, { "cell_type": "code", "execution_count": 7, "id": "05059463-159b-4691-a918-43b6fab113e5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "11992" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(npr)" ] }, { "cell_type": "code", "execution_count": 9, "id": "70b66301-dd5d-4a7e-993d-fd116614a44e", "metadata": {}, "outputs": [], "source": [ "from sklearn.feature_extraction.text import CountVectorizer" ] }, { "cell_type": "code", "execution_count": 11, "id": "d31a5b30-9ee7-4872-8262-053ab682716f", "metadata": {}, "outputs": [], "source": [ "cv = CountVectorizer(\n", " # Max document frequency: discard words that show up 90% of times in the document\n", " max_df=0.9,\n", " # Min document frequency: discard words that aren't shown in at least 2 documents\n", " min_df=2,\n", " # remove stop_words for EN lang (replacement of spacy).\n", " stop_words='english'\n", ")\n", "# if a number between 0 and 1 is passed, it represent the frequency belong the document\n", "# if am int number is passed, it represent the frequency belong N documents" ] }, { "cell_type": "code", "execution_count": 12, "id": "6ae1f827-3825-4908-8460-0caf5ae0dfa1", "metadata": {}, "outputs": [], "source": [ "# document term metrics\n", "dtm = cv.fit_transform(npr['Article'])" ] }, { "cell_type": "code", "execution_count": 13, "id": "5a1399aa-678b-4165-b770-1d3fa262dca5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<11992x54777 sparse matrix of type ''\n", "\twith 3033388 stored elements in Compressed Sparse Row format>" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dtm" ] }, { "cell_type": "code", "execution_count": 14, "id": "ffec8103-77be-4ca6-bafb-f31ee404bb61", "metadata": {}, "outputs": [], "source": [ "from sklearn.decomposition import LatentDirichletAllocation" ] }, { "cell_type": "code", "execution_count": 15, "id": "9d1d1a4c-4c57-45d9-ad1e-9d269dd6dbf5", "metadata": {}, "outputs": [], "source": [ "LDA = LatentDirichletAllocation(\n", " # Number of topics\n", " n_components=7,\n", " random_state=42,\n", ")" ] }, { "cell_type": "code", "execution_count": 16, "id": "82117547-9877-4b26-9726-640103bca07a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

LatentDirichletAllocation(n_components=7, random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

" ], "text/plain": [ "LatentDirichletAllocation(n_components=7, random_state=42)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "LDA.fit(dtm)" ] }, { "cell_type": "code", "execution_count": 20, "id": "bade2ec9-07aa-47f6-a49a-966f9a5c8789", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "54777" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(cv.get_feature_names_out())" ] }, { "cell_type": "code", "execution_count": 21, "id": "15207a07-33c2-440d-917f-00635cb7b188", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "numpy.ndarray" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(cv.get_feature_names_out())" ] }, { "cell_type": "code", "execution_count": 22, "id": "89ab6fe4-1d5e-4efa-82a7-ee8fec317f14", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'bask'" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cv.get_feature_names_out()[5000]" ] }, { "cell_type": "code", "execution_count": 24, "id": "f50e023a-9be4-4fdb-a378-a37fcb9c93d6", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'honored'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import random\n", "\n", "rand_word_id = random.randint(0,54666)\n", "\n", "cv.get_feature_names_out()[rand_word_id]" ] }, { "cell_type": "code", "execution_count": 25, "id": "115a34dc-bf0f-47fb-a31b-e6c2433af426", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "7" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(LDA.components_)" ] }, { "cell_type": "code", "execution_count": 26, "id": "9e682046-9629-4681-b737-d9efb551b6bd", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "numpy.ndarray" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(LDA.components_)" ] }, { "cell_type": "code", "execution_count": 27, "id": "2cf5babe-85d7-4737-b0ee-8a819248ff3d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(7, 54777)" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "LDA.components_.shape" ] }, { "cell_type": "code", "execution_count": 28, "id": "1aba87ad-91f2-4070-9d45-f33795d505cf", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[8.64332806e+00, 2.38014333e+03, 1.42900522e-01, ...,\n", " 1.43006821e-01, 1.42902042e-01, 1.42861626e-01],\n", " [2.76191749e+01, 5.36394437e+02, 1.42857148e-01, ...,\n", " 1.42861973e-01, 1.42857147e-01, 1.42906875e-01],\n", " [7.22783888e+00, 8.24033986e+02, 1.42857148e-01, ...,\n", " 6.14236247e+00, 2.14061364e+00, 1.42923753e-01],\n", " ...,\n", " [3.11488651e+00, 3.50409655e+02, 1.42857147e-01, ...,\n", " 1.42859912e-01, 1.42857146e-01, 1.42866614e-01],\n", " [4.61486388e+01, 5.14408600e+01, 3.14281373e+00, ...,\n", " 1.43107628e-01, 1.43902481e-01, 2.14271779e+00],\n", " [4.93991422e-01, 4.18841042e+02, 1.42857151e-01, ...,\n", " 1.42857146e-01, 1.43760101e-01, 1.42866201e-01]])" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "LDA.components_" ] }, { "cell_type": "code", "execution_count": 29, "id": "0fc0f7fd-deb9-470a-85be-772e6b67b70a", "metadata": {}, "outputs": [], "source": [ "topic = LDA.components_[0]" ] }, { "cell_type": "code", "execution_count": 30, "id": "8d7153aa-d77d-430c-ade4-ebc59e5a527d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([ 2475, 18302, 35285, ..., 22673, 42561, 42993])" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# get the sorted locations of higher value topics,\n", "# index positions of high probability words for that topic\n", "topic.argsort()" ] }, { "cell_type": "code", "execution_count": 31, "id": "509700d8-dcb0-4aa8-a3c4-fca63e934fad", "metadata": {}, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "code", "execution_count": 32, "id": "9d542b49-5201-4176-b7e2-e62b116c72ca", "metadata": {}, "outputs": [], "source": [ "arr = np.array([10, 200, 1])" ] }, { "cell_type": "code", "execution_count": 33, "id": "15ca06b9-10a6-4d85-82d3-8f02ebf94714", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([2, 0, 1])" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "arr.argsort()" ] }, { "cell_type": "code", "execution_count": 38, "id": "2dd693ce-7583-47f0-ac2b-1d7890150a46", "metadata": {}, "outputs": [], "source": [ "# ARGSORT => index position of high probability words sorted DESC (least->greatest)\n", "top_ten_words = topic.argsort()[-20:]\n", "# 10 TOP PROBABILITY WORDS FOR TOPIC (10 GREATES VALUES)" ] }, { "cell_type": "markdown", "id": "0eec13db-4fc6-4bb7-b3e1-f3d65307d9ba", "metadata": {}, "source": [ "These words have the highest probability to appear in the first(taken) topic:" ] }, { "cell_type": "code", "execution_count": 39, "id": "22260d3a-775f-45d3-8d77-23baeb9747d3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "president\n", "state\n", "tax\n", "insurance\n", "trump\n", "companies\n", "money\n", "year\n", "federal\n", "000\n", "new\n", "percent\n", "government\n", "company\n", "million\n", "care\n", "people\n", "health\n", "said\n", "says\n" ] } ], "source": [ "for index in top_ten_words:\n", " print(cv.get_feature_names_out()[index])" ] }, { "cell_type": "code", "execution_count": 43, "id": "bd71540d-0af5-4788-9f70-3aefad5fa8f6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "THE TOP 15 WORDS FOR TOPIC #$0\n", "['companies', 'money', 'year', 'federal', '000', 'new', 'percent', 'government', 'company', 'million', 'care', 'people', 'health', 'said', 'says']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #$1\n", "['military', 'house', 'security', 'russia', 'government', 'npr', 'reports', 'says', 'news', 'people', 'told', 'police', 'president', 'trump', 'said']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #$2\n", "['way', 'world', 'family', 'home', 'day', 'time', 'water', 'city', 'new', 'years', 'food', 'just', 'people', 'like', 'says']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #$3\n", "['time', 'new', 'don', 'years', 'medical', 'disease', 'patients', 'just', 'children', 'study', 'like', 'women', 'health', 'people', 'says']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #$4\n", "['voters', 'vote', 'election', 'party', 'new', 'obama', 'court', 'republican', 'campaign', 'people', 'state', 'president', 'clinton', 'said', 'trump']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #$5\n", "['years', 'going', 've', 'life', 'don', 'new', 'way', 'music', 'really', 'time', 'know', 'think', 'people', 'just', 'like']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #$6\n", "['student', 'years', 'data', 'science', 'university', 'people', 'time', 'schools', 'just', 'education', 'new', 'like', 'students', 'school', 'says']\n", "\n", "\n" ] } ], "source": [ "for i,topic in enumerate(LDA.components_):\n", " print(f'THE TOP 15 WORDS FOR TOPIC #${i}')\n", " print([cv.get_feature_names_out()[index] for index in topic.argsort()[-15:]])\n", " print('\\n')" ] }, { "cell_type": "code", "execution_count": 44, "id": "a2d773c9-9de5-4594-b7da-cc06c9a67f12", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<11992x54777 sparse matrix of type ''\n", "\twith 3033388 stored elements in Compressed Sparse Row format>" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dtm" ] }, { "cell_type": "code", "execution_count": 45, "id": "9a45a8ef-2d5d-49c7-a990-96ce9d9efe2a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	Article
0	In the Washington of 2016, even when the polic...
1	Donald Trump has used Twitter — his prefe...
2	Donald Trump is unabashedly praising Russian...
3	Updated at 2:50 p. m. ET, Russian President Vl...
4	From photography, illustration and video, to d...
...	...
11987	The number of law enforcement officers shot an...
11988	Trump is busy these days with victory tours,...
11989	It’s always interesting for the Goats and Soda...
11990	The election of Donald Trump was a surprise to...
11991	Voters in the English city of Sunderland did s...

\n", "

11992 rows × 1 columns

\n", "

" ], "text/plain": [ " Article\n", "0 In the Washington of 2016, even when the polic...\n", "1 Donald Trump has used Twitter — his prefe...\n", "2 Donald Trump is unabashedly praising Russian...\n", "3 Updated at 2:50 p. m. ET, Russian President Vl...\n", "4 From photography, illustration and video, to d...\n", "... ...\n", "11987 The number of law enforcement officers shot an...\n", "11988 Trump is busy these days with victory tours,...\n", "11989 It’s always interesting for the Goats and Soda...\n", "11990 The election of Donald Trump was a surprise to...\n", "11991 Voters in the English city of Sunderland did s...\n", "\n", "[11992 rows x 1 columns]" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "npr" ] }, { "cell_type": "code", "execution_count": 46, "id": "e3e2f9e5-0128-47be-9bc2-148e4a6ad0ca", "metadata": {}, "outputs": [], "source": [ "topic_results = LDA.transform(dtm)" ] }, { "cell_type": "code", "execution_count": 47, "id": "1106542c-2ef6-40be-85d5-23f9f07d7336", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[1.61040465e-02, 6.83341493e-01, 2.25376318e-04, ...,\n", " 2.99652737e-01, 2.25479379e-04, 2.25497980e-04],\n", " [3.63424997e-02, 8.86130697e-01, 4.40751747e-04, ...,\n", " 7.57636804e-02, 4.40866779e-04, 4.40835574e-04],\n", " [3.28569485e-04, 6.96344889e-01, 3.28302105e-04, ...,\n", " 3.02012902e-01, 3.28724083e-04, 3.28352652e-04],\n", " ...,\n", " [1.44467964e-02, 1.60696622e-01, 1.73678310e-01, ...,\n", " 2.24636569e-02, 3.98728349e-04, 3.98359730e-04],\n", " [4.33560738e-04, 3.53196803e-02, 4.33022554e-04, ...,\n", " 9.62512640e-01, 4.33971991e-04, 4.33490254e-04],\n", " [3.98777533e-01, 2.54376049e-04, 3.59290659e-01, ...,\n", " 2.40914375e-01, 2.54445555e-04, 2.54253739e-04]])" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "topic_results" ] }, { "cell_type": "code", "execution_count": 48, "id": "bf9ffe22-8bae-4c65-a7f8-29134a8ca841", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(11992, 7)" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "topic_results.shape" ] }, { "cell_type": "code", "execution_count": 51, "id": "2a0c3d79-101d-460b-8438-d53f9d67cc1f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.02, 0.68, 0. , 0. , 0.3 , 0. , 0. ])" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Probabilities of a document beloging to a particular topic\n", "topic_results[0].round(2)" ] }, { "cell_type": "code", "execution_count": 52, "id": "36127070-01fa-4342-890b-f75f895757cb", "metadata": {}, "outputs": [], "source": [ "npr['Topic'] = topic_results.argmax(axis=1)" ] }, { "cell_type": "code", "execution_count": 53, "id": "32aac426-2fa7-4dd0-99c1-30c15280c8f5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	Article	Topic
0	In the Washington of 2016, even when the polic...	1
1	Donald Trump has used Twitter — his prefe...	1
2	Donald Trump is unabashedly praising Russian...	1
3	Updated at 2:50 p. m. ET, Russian President Vl...	1
4	From photography, illustration and video, to d...	2
...	...	...
11987	The number of law enforcement officers shot an...	1
11988	Trump is busy these days with victory tours,...	4
11989	It’s always interesting for the Goats and Soda...	3
11990	The election of Donald Trump was a surprise to...	4
11991	Voters in the English city of Sunderland did s...	0

\n", "

11992 rows × 2 columns

\n", "

" ], "text/plain": [ " Article Topic\n", "0 In the Washington of 2016, even when the polic... 1\n", "1 Donald Trump has used Twitter — his prefe... 1\n", "2 Donald Trump is unabashedly praising Russian... 1\n", "3 Updated at 2:50 p. m. ET, Russian President Vl... 1\n", "4 From photography, illustration and video, to d... 2\n", "... ... ...\n", "11987 The number of law enforcement officers shot an... 1\n", "11988 Trump is busy these days with victory tours,... 4\n", "11989 It’s always interesting for the Goats and Soda... 3\n", "11990 The election of Donald Trump was a surprise to... 4\n", "11991 Voters in the English city of Sunderland did s... 0\n", "\n", "[11992 rows x 2 columns]" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "npr" ] }, { "cell_type": "code", "execution_count": null, "id": "de4c920b-4d84-46d3-8ab6-2ff1c30e4b67", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }