{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "___\n", "\n", "

\n", "___" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "source": [ "# Topic Modeling Assessment Project\n", "\n", "Welcome to your Topic Modeling Assessment! For this project you will be working with a dataset of over 400,000 quora questions that have no labeled cateogry, and attempting to find 20 cateogries to assign these questions to. The .csv file of these text questions can be found underneath the Topic-Modeling folder.\n", "\n", "Remember you can always check the solutions notebook and video lecture for any questions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Task: Import pandas and read in the quora_questions.csv file." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "df = pd.read_csv('quora_questions.csv')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	Question
0	What is the step by step guide to invest in sh...
1	What is the story of Kohinoor (Koh-i-Noor) Dia...
2	How can I increase the speed of my internet co...
3	Why am I mentally very lonely? How can I solve...
4	Which one dissolve in water quikly sugar, salt...

\n", "

" ], "text/plain": [ " Question\n", "0 What is the step by step guide to invest in sh...\n", "1 What is the story of Kohinoor (Koh-i-Noor) Dia...\n", "2 How can I increase the speed of my internet co...\n", "3 Why am I mentally very lonely? How can I solve...\n", "4 Which one dissolve in water quikly sugar, salt..." ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Preprocessing\n", "\n", "#### Task: Use TF-IDF Vectorization to create a vectorized document term matrix. You may want to explore the max_df and min_df parameters." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from sklearn.feature_extraction.text import TfidfVectorizer" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "tfidf = TfidfVectorizer(\n", " max_df=0.95,\n", " min_df=2,\n", " stop_words='english'\n", ")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "dtm = tfidf.fit_transform(df['Question'])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<404289x38669 sparse matrix of type ''\n", "\twith 2002912 stored elements in Compressed Sparse Row format>" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dtm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Non-negative Matrix Factorization\n", "\n", "#### TASK: Using Scikit-Learn create an instance of NMF with 20 expected components. (Use random_state=42).." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from sklearn.decomposition import NMF" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "nmf_model = NMF(\n", " n_components=20,\n", " random_state=42\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/lib/python3.11/site-packages/sklearn/decomposition/_nmf.py:1710: ConvergenceWarning: Maximum number of iterations 200 reached. Increase it to improve convergence.\n", " warnings.warn(\n" ] }, { "data": { "text/html": [ "

NMF(n_components=20, random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

" ], "text/plain": [ "NMF(n_components=20, random_state=42)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nmf_model.fit(dtm)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TASK: Print our the top 15 most common words for each of the 20 topics." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The top 15 words for topic #$0\n", "['thing', 'read', 'place', 'visit', 'places', 'phone', 'buy', 'laptop', 'movie', 'ways', '2016', 'books', 'book', 'movies', 'best']\n", "\n", "\n", "The top 15 words for topic #$1\n", "['majors', 'recruit', 'sex', 'looking', 'differ', 'use', 'exist', 'really', 'compare', 'cost', 'long', 'feel', 'work', 'mean', 'does']\n", "\n", "\n", "The top 15 words for topic #$2\n", "['add', 'answered', 'needing', 'post', 'easily', 'improvement', 'delete', 'asked', 'google', 'answers', 'answer', 'ask', 'question', 'questions', 'quora']\n", "\n", "\n", "The top 15 words for topic #$3\n", "['using', 'website', 'investment', 'friends', 'black', 'internet', 'free', 'home', 'easy', 'youtube', 'ways', 'earn', 'online', 'make', 'money']\n", "\n", "\n", "The top 15 words for topic #$4\n", "['balance', 'earth', 'day', 'death', 'changed', 'live', 'want', 'change', 'moment', 'real', 'important', 'thing', 'meaning', 'purpose', 'life']\n", "\n", "\n", "The top 15 words for topic #$5\n", "['reservation', 'engineering', 'minister', 'president', 'company', 'china', 'business', 'country', 'olympics', 'available', 'job', 'spotify', 'war', 'pakistan', 'india']\n", "\n", "\n", "The top 15 words for topic #$6\n", "['beginners', 'online', 'english', 'book', 'did', 'hacking', 'want', 'python', 'languages', 'java', 'learning', 'start', 'language', 'programming', 'learn']\n", "\n", "\n", "The top 15 words for topic #$7\n", "['happen', 'presidency', 'think', 'presidential', '2016', 'vote', 'better', 'election', 'did', 'win', 'hillary', 'president', 'clinton', 'donald', 'trump']\n", "\n", "\n", "The top 15 words for topic #$8\n", "['russia', 'business', 'win', 'coming', 'countries', 'place', 'pakistan', 'happen', 'end', 'country', 'iii', 'start', 'did', 'war', 'world']\n", "\n", "\n", "The top 15 words for topic #$9\n", "['indian', 'companies', 'don', 'guy', 'men', 'culture', 'women', 'work', 'girls', 'live', 'girl', 'look', 'sex', 'feel', 'like']\n", "\n", "\n", "The top 15 words for topic #$10\n", "['ca', 'departments', 'positions', 'movies', 'songs', 'business', 'read', 'start', 'job', 'work', 'engineering', 'ways', 'bad', 'books', 'good']\n", "\n", "\n", "The top 15 words for topic #$11\n", "['money', 'modi', 'currency', 'economy', 'think', 'government', 'ban', 'banning', 'black', 'indian', 'rupee', 'rs', '1000', 'notes', '500']\n", "\n", "\n", "The top 15 words for topic #$12\n", "['blowing', 'resolutions', 'resolution', 'mind', 'likes', 'girl', '2017', 'year', 'don', 'employees', 'going', 'day', 'things', 'new', 'know']\n", "\n", "\n", "The top 15 words for topic #$13\n", "['aspects', 'fluent', 'skill', 'spoken', 'ways', 'language', 'fluently', 'speak', 'communication', 'pronunciation', 'speaking', 'writing', 'skills', 'improve', 'english']\n", "\n", "\n", "The top 15 words for topic #$14\n", "['diet', 'help', 'healthy', 'exercise', 'month', 'pounds', 'reduce', 'quickly', 'loss', 'fast', 'fat', 'ways', 'gain', 'lose', 'weight']\n", "\n", "\n", "The top 15 words for topic #$15\n", "['having', 'feel', 'long', 'spend', 'did', 'person', 'machine', 'movies', 'favorite', 'job', 'home', 'sex', 'possible', 'travel', 'time']\n", "\n", "\n", "The top 15 words for topic #$16\n", "['marriage', 'make', 'did', 'girlfriend', 'feel', 'tell', 'forget', 'really', 'friend', 'true', 'know', 'person', 'girl', 'fall', 'love']\n", "\n", "\n", "The top 15 words for topic #$17\n", "['easy', 'hack', 'prepare', 'quickest', 'facebook', 'increase', 'painless', 'instagram', 'account', 'best', 'commit', 'fastest', 'suicide', 'easiest', 'way']\n", "\n", "\n", "The top 15 words for topic #$18\n", "['web', 'java', 'scripting', 'phone', 'mechanical', 'better', 'job', 'use', 'account', 'data', 'software', 'science', 'computer', 'engineering', 'difference']\n", "\n", "\n", "The top 15 words for topic #$19\n", "['earth', 'blowing', 'stop', 'use', 'easily', 'mind', 'google', 'flat', 'questions', 'hate', 'believe', 'ask', 'don', 'think', 'people']\n", "\n", "\n" ] } ], "source": [ "for index,topic in enumerate(nmf_model.components_):\n", " print(f'The top 15 words for topic #${index}')\n", " print([tfidf.get_feature_names_out()[i] for i in topic.argsort()[-15:]])\n", " print('\\n')" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "THE TOP 15 WORDS FOR TOPIC #0\n", "['thing', 'read', 'place', 'visit', 'places', 'phone', 'buy', 'laptop', 'movie', 'ways', '2016', 'books', 'book', 'movies', 'best']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #1\n", "['majors', 'recruit', 'sex', 'looking', 'differ', 'use', 'exist', 'really', 'compare', 'cost', 'long', 'feel', 'work', 'mean', 'does']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #2\n", "['add', 'answered', 'needing', 'post', 'easily', 'improvement', 'delete', 'asked', 'google', 'answers', 'answer', 'ask', 'question', 'questions', 'quora']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #3\n", "['using', 'website', 'investment', 'friends', 'black', 'internet', 'free', 'home', 'easy', 'youtube', 'ways', 'earn', 'online', 'make', 'money']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #4\n", "['balance', 'earth', 'day', 'death', 'changed', 'live', 'want', 'change', 'moment', 'real', 'important', 'thing', 'meaning', 'purpose', 'life']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #5\n", "['reservation', 'engineering', 'minister', 'president', 'company', 'china', 'business', 'country', 'olympics', 'available', 'job', 'spotify', 'war', 'pakistan', 'india']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #6\n", "['beginners', 'online', 'english', 'book', 'did', 'hacking', 'want', 'python', 'languages', 'java', 'learning', 'start', 'language', 'programming', 'learn']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #7\n", "['happen', 'presidency', 'think', 'presidential', '2016', 'vote', 'better', 'election', 'did', 'win', 'hillary', 'president', 'clinton', 'donald', 'trump']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #8\n", "['russia', 'business', 'win', 'coming', 'countries', 'place', 'pakistan', 'happen', 'end', 'country', 'iii', 'start', 'did', 'war', 'world']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #9\n", "['indian', 'companies', 'don', 'guy', 'men', 'culture', 'women', 'work', 'girls', 'live', 'girl', 'look', 'sex', 'feel', 'like']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #10\n", "['ca', 'departments', 'positions', 'movies', 'songs', 'business', 'read', 'start', 'job', 'work', 'engineering', 'ways', 'bad', 'books', 'good']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #11\n", "['money', 'modi', 'currency', 'economy', 'think', 'government', 'ban', 'banning', 'black', 'indian', 'rupee', 'rs', '1000', 'notes', '500']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #12\n", "['blowing', 'resolutions', 'resolution', 'mind', 'likes', 'girl', '2017', 'year', 'don', 'employees', 'going', 'day', 'things', 'new', 'know']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #13\n", "['aspects', 'fluent', 'skill', 'spoken', 'ways', 'language', 'fluently', 'speak', 'communication', 'pronunciation', 'speaking', 'writing', 'skills', 'improve', 'english']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #14\n", "['diet', 'help', 'healthy', 'exercise', 'month', 'pounds', 'reduce', 'quickly', 'loss', 'fast', 'fat', 'ways', 'gain', 'lose', 'weight']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #15\n", "['having', 'feel', 'long', 'spend', 'did', 'person', 'machine', 'movies', 'favorite', 'job', 'home', 'sex', 'possible', 'travel', 'time']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #16\n", "['marriage', 'make', 'did', 'girlfriend', 'feel', 'tell', 'forget', 'really', 'friend', 'true', 'know', 'person', 'girl', 'fall', 'love']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #17\n", "['easy', 'hack', 'prepare', 'quickest', 'facebook', 'increase', 'painless', 'instagram', 'account', 'best', 'commit', 'fastest', 'suicide', 'easiest', 'way']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #18\n", "['web', 'java', 'scripting', 'phone', 'mechanical', 'better', 'job', 'use', 'account', 'data', 'software', 'science', 'computer', 'engineering', 'difference']\n", "\n", "\n", "THE TOP 15 WORDS FOR TOPIC #19\n", "['earth', 'blowing', 'stop', 'use', 'easily', 'mind', 'google', 'flat', 'questions', 'hate', 'believe', 'ask', 'don', 'think', 'people']\n", "\n", "\n" ] } ], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TASK: Add a new column to the original quora dataframe that labels each question into one of the 20 topic categories." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "topic_results = nmf_model.transform(dtm)\n", "df['Topic'] = topic_results.argmax(axis=1)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	Question	Topic
0	What is the step by step guide to invest in sh...	5
1	What is the story of Kohinoor (Koh-i-Noor) Dia...	16
2	How can I increase the speed of my internet co...	17
3	Why am I mentally very lonely? How can I solve...	11
4	Which one dissolve in water quikly sugar, salt...	14

\n", "

	Question
0	What is the step by step guide to invest in sh...
1	What is the story of Kohinoor (Koh-i-Noor) Dia...
2	How can I increase the speed of my internet co...
3	Why am I mentally very lonely? How can I solve...
4	Which one dissolve in water quikly sugar, salt...

\n", "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	Question	Topic
0	What is the step by step guide to invest in sh...	5
1	What is the story of Kohinoor (Koh-i-Noor) Dia...	16
2	How can I increase the speed of my internet co...	17
3	Why am I mentally very lonely? How can I solve...	11
4	Which one dissolve in water quikly sugar, salt...	14
5	Astrology: I am a Capricorn Sun Cap moon and c...	1
6	Should I buy tiago?	0
7	How can I be a good geologist?	10
8	When do you use シ instead of し?	19
9	Motorola (company): Can I hack my Charter Moto...	17

\n", "

" ], "text/plain": [ " Question Topic\n", "0 What is the step by step guide to invest in sh... 5\n", "1 What is the story of Kohinoor (Koh-i-Noor) Dia... 16\n", "2 How can I increase the speed of my internet co... 17\n", "3 Why am I mentally very lonely? How can I solve... 11\n", "4 Which one dissolve in water quikly sugar, salt... 14\n", "5 Astrology: I am a Capricorn Sun Cap moon and c... 1\n", "6 Should I buy tiago? 0\n", "7 How can I be a good geologist? 10\n", "8 When do you use シ instead of し? 19\n", "9 Motorola (company): Can I hack my Charter Moto... 17" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Great job!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 4 }