Design your own sentiment analysis tool

 .

While there are a lot of tools that will automatically give us a sentiment of a piece of text, we learned that they don't always agree! Let's design our own to see both how these tools work internally, along with how we can test them to see how well they might perform.

.

https://colab.research.google.com/github/littlecolumns/ds4j-notebooks/blob/master/sentiment-analysis-is-bad/notebooks/Designing%20your%20own%20sentiment%20analysis%20tool.ipynb

Read More

Install NRC Lexicon from within Jupyter Notebook

 .

1.Upgrade Python version (if needed)

import sys !{sys.executable} -m pip install --upgrade pip

Expected output:

Requirement already satisfied: pip in c:\portable\jupyterlab-portable-3.1.0-3.9\apps\lib\site-packages (21.2.1) Collecting pip Downloading pip-21.2.4-py3-none-any.whl (1.6 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 21.2.1 Uninstalling pip-21.2.1: Successfully uninstalled pip-21.2.1 Successfully installed pip-21.2.4

---

2.Install NRC Lexicon

!{sys.executable} -m pip install NRCLex

Expected Output

Collecting NRCLex Downloading NRCLex-3.0.0.tar.gz (396 kB) Collecting textblob Downloading textblob-0.15.3-py2.py3-none-any.whl (636 kB) Collecting nltk>=3.1 Downloading nltk-3.6.3-py3-none-any.whl (1.5 MB) Collecting joblib Downloading joblib-1.0.1-py3-none-any.whl (303 kB) Collecting regex Downloading regex-2021.8.28-cp39-cp39-win_amd64.whl (271 kB) Collecting click Downloading click-8.0.1-py3-none-any.whl (97 kB) Collecting tqdm Downloading tqdm-4.62.3-py2.py3-none-any.whl (76 kB) Requirement already satisfied: colorama in c:\portable\jupyterlab-portable-3.1.0-3.9\apps\lib\site-packages (from click->nltk>=3.1->textblob->NRCLex) (0.4.4) Using legacy 'setup.py install' for NRCLex, since package 'wheel' is not installed. Installing collected packages: tqdm, regex, joblib, click, nltk, textblob, NRCLex Running setup.py install for NRCLex: started Running setup.py install for NRCLex: finished with status 'done' Successfully installed NRCLex-3.0.0 click-8.0.1 joblib-1.0.1 nltk-3.6.3 regex-2021.8.28 textblob-0.15.3 tqdm-4.62.3

---

Note: Even after the installation of NRC Lexicon, MissingCorpusError may occur while running programs. So it is recommended to also install textblob.download_corpora.

3.Install textblob.download_corpora (Recommended)

import sys !{sys.executable} -m textblob.download_corpora

Expected Output

[nltk_data] Downloading package brown to C:\Portable\JupyterLab- [nltk_data] Portable-3.1.0-3.9\apps\nltk_data... [nltk_data] Unzipping corpora\brown.zip. [nltk_data] Downloading package punkt to C:\Portable\JupyterLab- [nltk_data] Portable-3.1.0-3.9\apps\nltk_data... [nltk_data] Unzipping tokenizers\punkt.zip. [nltk_data] Downloading package wordnet to C:\Portable\JupyterLab- [nltk_data] Portable-3.1.0-3.9\apps\nltk_data... [nltk_data] Unzipping corpora\wordnet.zip. [nltk_data] Downloading package averaged_perceptron_tagger to [nltk_data] C:\Portable\JupyterLab- [nltk_data] Portable-3.1.0-3.9\apps\nltk_data... [nltk_data] Unzipping taggers\averaged_perceptron_tagger.zip. [nltk_data] Downloading package conll2000 to C:\Portable\JupyterLab- [nltk_data] Portable-3.1.0-3.9\apps\nltk_data... [nltk_data] Unzipping corpora\conll2000.zip. [nltk_data] Downloading package movie_reviews to [nltk_data] C:\Portable\JupyterLab- [nltk_data] Portable-3.1.0-3.9\apps\nltk_data... [nltk_data] Unzipping corpora\movie_reviews.zip.

Read More

Text Analysis Tutorial at investigate.ai

 

.

Text analysis

1. Types of text analysis
2. Counting words
   1. Simple word counting
      • Getting started
      • Counting words in a book
      • Secret tricks with Counter
        • Only extracting some words with regular expressions
        • Most common verbs
        • Comparing top words
      • Review
   2. Counting words across many documents
   3. Segmenting words in East Asian languages
   4. Project: Caixin museums
   5. Using scikit-learn vectorizers with East Asian languages
3. Advanced word analysis
   1. Upgraded word counts with TF-IDF
   2. Multi-word phrases and n-grams
   3. Standardizing text with stemming and lemmatization
   4. Using TF-IDF with Chinese text
4. Sentiment analysis
   1. Comparing sentiment analysis tools
   2. Design your own sentiment analyzer
   3. Improving your tool
   4. NRC Emotional Lexicon
   5. Project: UpShot State of the Union
   6. Project: NYT Trump tweets
5. Documents to text
   1. Converting documents to text (English)
   2. Converting documents to text (non-English)
6. Concepts, people and places
   1. Extracting topics from documents
   2. Choosing the right number of topics
   3. Topic models with Gensim
   4. Topic models vs clustering
   5. Entity recognition
   6. Intro to word embeddings
   7. Conceptual document similarity
   8. Comparing documents in different languages

Putting things in categories automatically

1. Introduction to Classification
2. Techniques
   1. Evaluating classifiers
   2. Categorical features
   3. Classifiers with text
   4. Correcting for imbalanced datasets
3. Projects
   1. BuzzFeed: Spy planes
   2. WaPo chat: App reviews
   3. NYT: Faulty airbag search
   4. LA Times: crime classifier

How X affects Y

1. Finding relationships with regression
2. Linear Regression
   1. Linear regression (Quickstart)
   2. Linear regression for humans
   3. Putting regression to use
   4. Evaluating regressions
   5. Associated Press: Life expectancy and unemployment
3. Logistic Regression
   1. Logistic regression (Quickstart)
   2. Logistic regression for humans
   3. More complex logistic regressions
   4. Evaluating logistic regressions
   5. Boston Globe: Speeding tickets
   6. APM Reports: Jury selection

Python data science reference

1. Introduction
2. Vectorizing
3. Text Analysis
4. Regression
5. Classification

All Projects

1. Project Summaries
2. NYT: Takata airbag search
   1. Summary
   2. A simplistic reproduction of the NYT's research using logistic regression
   3. A decision-tree reproduction of the NYT's research
   4. Combining a text vectorizer and a classifier to track down suspicious complaints
3. LA Times: Crime classification
   1. Summary
   2. Predicting downgraded assaults with machine learning
   3. Taking a closer look at our classifier and its misclassifications
   4. Trying out and combining different classifiers
4. Caixin: Museum names
   1. Summary
   2. Chinese museum dataset cleanup
   3. Chinese museums per capita analysis
   4. Counting words in Chinese museum names
5. WaPo: Random chat app safety
   1. Summary
   2. Scrape and combine app store reviews
   3. Build a classifier to detect reviews about bad behavior
6. AJC: Doctors and sex abuse
7. The UpShot: Trump speeches
   1. Summary
   2. An introduction to the NRC Emotional Lexicon
   3. Reproducing The UpShot's Trump State of the Union visualization
8. USA Today: Model legislation
   1. Summary
   2. Downloading one million pieces of legislation from LegiScan
   3. Taking a million pieces of legislation from a CSV and inserting them into Postgres
   4. Download Word, PDF and HTML content and process it into text with Tika
   5. Import content into Solr for advanced text searching
   6. Checking for legislative text reuse using Python, Solr, and ngrams
   7. Checking for legislative text reuse using Python, Solr, and simple text search
   8. Search for model legislation in over one million bills using Postgres and Solr
   9. Using topic modeling to categorize legislation
9. FCC comment bots
10. Bloomberg: Democratic Candidate Tweets
    1. Summary
    2. Downloading all 2019 tweets from Democratic presidential candidates
    3. Using topic modeling to analyze presidential candidate tweets
    4. Assigning categories to tweets using keyword matching
    5. Building streamgraphs from categorized and dated datasets
11. NYT: Trump tweets
12. AP: Life expectancy
    1. Summary
    2. Simple logistic regression using statsmodels (formula version)
    3. Simple logistic regression using statsmodels (dataframes version)
13. FiveThirtyEight: P-values
14. Milwaukee Journal-Sentinel: Potholes
    1. Summary
    2. Pothole geographic analysis and linear regression, complete walkthrough
    3. Pothole demographics linear regression, no spatial analysis
15. Dallas Morning News: Cheating schools
    1. Summary
    2. Finding outliers with standard deviation and regression
    3. Finding outliers with regression residuals (short version)
    4. Reproducing the graphics from The Dallas Morning News piece
16. Tampa Bay Times: Failure factories
    1. Summary
    2. Linear regression on Florida schools, complete walkthrough
    3. Linear regression on Florida schools, no cleaning
17. Car accidents and car weight
    1. Summary
    2. Feature selection and engineering
    3. Combine Excel files across multiple sheets and save as CSV files
    4. Create make model weights csv
    5. Find car data from VINs
    6. Combine VINs and weights
    7. Clean combine and filter data
18. ProPublica: Opportunity Gap
19. Boston Globe: Ticketing bias
    1. Summary
    2. Logistic regression for speeding tickets
20. Stanford: Open Policing Data
21. BuzzFeed: Surveillance planes
    1. Summary
    2. Feature engineering - BuzzFeed spy planes
    3. Drawing flight paths on maps with cartopy
    4. Finding surveillance planes using random forests
22. Reveal: Mortgage lending bias
    1. Summary
    2. Cleaning and combining data for the Reveal Mortgage Analysis
    3. Wild formulas in statsmodels using Patsy (short version)
    4. Reveal Mortgage Analysis - Logistic Regression using statsmodels formulas
    5. Reveal Mortgage Analysis - Logistic Regression
23. APM Reports: Jury selection bias
    1. Summary
    2. Combining and cleaning the initial dataset
    3. Picking what matters and what doesn't in a regression
    4. Analyzing data using statsmodels formulas
    5. Alternative techniques with statsmodels formulas
24. Reuters: Asylum denials
    1. Summary
    2. Preparing the EOIR immigration court data for analysis
    3. How nationality and judges affect your chance of asylum in immigration court
25. ProPublica: Presidential pardons
26. ProPublica: Criminal sentencing
    1. Summary
    2. Breaking down machine bias
27. data.world: The FOIA Predictor

. Read More