How to use NLTK to analyze the sentiment of tweets

In the courses, we learned a lot of basic idea of how to learn the language and some basic algorithm used to model the idea. These field interested me a lot and I think it’s also very exciting to implement these in my own computer and let computer to analyze our own mind.

Motivation

Twitter is a popular microblogging service where users create status messages (called “tweets”). These tweets sometimes express opinions about different topics. I propose to build an automatic sentiment (positive or neutral or negative) extractor from a tweet. This is very useful because it allows feedback to be aggregated without manual intervention.

Basic Idea

Before the procedure, I have already get the data from the twitter by using REST API given by the twitter ( the detail is in the past blog) 

By using the data we have already crawled from twitter, we could implement our idea easily. In order to build a sentiment analyzer, first we need to equip ourselves with the right tools and methods. Machine learning is one such tool where people have developed various methods to classify. Classifiers may or may not need training data. In particular, we will deal with the following machine learning classifiers, namely, Naive Bayes Classifier, Maximum Entropy Classifier and Support Vector Machines…... All of these classifiers require training data and hence these methods fall under the category of supervised classification. The Naïve Bayes Classifier is the practical Classifier to classify the data. So I will discuss the implementation only by Naïve Bayes Classifier.

Supervised Classification

Implementation Details:

Trianing Sets:

The classifiers need to be trained and to do that, we need to list manually classified tweets. Let's start with 3 positive, 3 neutral and 3 negative tweets.

Positive tweets:

1. Neutral tweets:@PrincessSuperC Hey Cici sweetheart! Just wanted to let u know I luv u! OH! and will the mixtape drop soon? FANTASY RIDE MAY 5TH!!!!  

2. @Msdebramaye I heard about that contest! Congrats girl!! 

Neutral tweets:

1. Do you Share More #jokes #quotes #music #photos or #news #articles on #Facebook or #Twitter?

2. Good night #Twitter and #TheLegionoftheFallen.  5:45am cimes awfully early!

Negative tweets:

1. Disappointing day. Attended a car boot sale to raise some funds for the sanctuary, made a total of 88p after the entry fee - sigh

2. Just had some bloodwork done. My arm hurts

As you can see from above, the tweets can have some valuable info about it's sentiment and rest of the words may not really help in determining the sentiment. Therefore, it makes sense to preprocess the tweets.

Implement Naive Bayes Classifier

Before classify these data, we need to modify these data first to remove some redundant information and get the keywords of these twitter.

Tweet:

Neutral tweets:@PrincessSuperC Hey Cici sweetheart! Just wanted to let u know I luv u! OH! and will the mixtape drop soon? FANTASY RIDE MAY 5TH!!!!  

Key word extraction:

'hey', 'cici', 'luv', 'mixtape', 'drop', 'soon', 'fantasy', 'ride'

Now it’s time to take a look at Natural Language Toolkit (NLTK) and implement the first two classifiers namely Naive Bayes and Maximum Entropy.

To explain how a Naive Bayes Classifier works is beyond the scope of this post, having said so, it’s pretty easy to understand. Refer to the Wikipedia article and read the example to understand how it works. At this point, I have a training set, so all we need to do is instantiate a classifier and classify test tweets. 

Input Tweet:

Congrats @ravikiranj, i heard you wrote a new tech post on sentiment analysis

The output of the program:

positive