How to use NLTK to analyze the sentiment of tweets

In the courses, we learned a lot of basic idea of how to learn the language and some basic algorithm used to model the idea. These field interested me a lot and I think it’s also very exciting to implement these in my own computer and let computer to analyze our own mind.

Motivation

Twitter is a popular microblogging service where users create status messages (called “tweets”). These tweets sometimes express opinions about different topics. I propose to build an automatic sentiment (positive or neutral or negative) extractor from a tweet. This is very useful because it allows feedback to be aggregated without manual intervention.

Basic Idea

Before the procedure, I have already get the data from the twitter by using REST API given by the twitter ( the detail is in the past blog) 

By using the data we have already crawled from twitter, we could implement our idea easily. In order to build a sentiment analyzer, first we need to equip ourselves with the right tools and methods. Machine learning is one such tool where people have developed various methods to classify. Classifiers may or may not need training data. In particular, we will deal with the following machine learning classifiers, namely, Naive Bayes Classifier, Maximum Entropy Classifier and Support Vector Machines…... All of these classifiers require training data and hence these methods fall under the category of supervised classification. The Naïve Bayes Classifier is the practical Classifier to classify the data. So I will discuss the implementation only by Naïve Bayes Classifier.

Supervised Classification

Implementation Details:

Trianing Sets:

The classifiers need to be trained and to do that, we need to list manually classified tweets. Let's start with 3 positive, 3 neutral and 3 negative tweets.

Positive tweets:

1. Neutral tweets:@PrincessSuperC Hey Cici sweetheart! Just wanted to let u know I luv u! OH! and will the mixtape drop soon? FANTASY RIDE MAY 5TH!!!!  

2. @Msdebramaye I heard about that contest! Congrats girl!! 

Neutral tweets:

1. Do you Share More #jokes #quotes #music #photos or #news #articles on #Facebook or #Twitter?

2. Good night #Twitter and #TheLegionoftheFallen.  5:45am cimes awfully early!

Negative tweets:

1. Disappointing day. Attended a car boot sale to raise some funds for the sanctuary, made a total of 88p after the entry fee - sigh

2. Just had some bloodwork done. My arm hurts

As you can see from above, the tweets can have some valuable info about it's sentiment and rest of the words may not really help in determining the sentiment. Therefore, it makes sense to preprocess the tweets.

Implement Naive Bayes Classifier

Before classify these data, we need to modify these data first to remove some redundant information and get the keywords of these twitter.

Tweet:

Neutral tweets:@PrincessSuperC Hey Cici sweetheart! Just wanted to let u know I luv u! OH! and will the mixtape drop soon? FANTASY RIDE MAY 5TH!!!!  

Key word extraction:

'hey', 'cici', 'luv', 'mixtape', 'drop', 'soon', 'fantasy', 'ride'

Now it’s time to take a look at Natural Language Toolkit (NLTK) and implement the first two classifiers namely Naive Bayes and Maximum Entropy.

To explain how a Naive Bayes Classifier works is beyond the scope of this post, having said so, it’s pretty easy to understand. Refer to the Wikipedia article and read the example to understand how it works. At this point, I have a training set, so all we need to do is instantiate a classifier and classify test tweets. 

Input Tweet:

Congrats @ravikiranj, i heard you wrote a new tech post on sentiment analysis

The output of the program:

positive

Implement in social Digging data from twitter

Main Idea:

As we learn from the course, I find it is very interesting to these field and I find maybe we could also get our own data and try to implement these technology we learned to analyze our mind by using data we pushed on the social media .

Before we make our mind to handle these technical strategy get the information and learn from the social network. The first important thing is to get data from the server. We could not do anything unless we could get the data we published on the Internet.

Twitter is a good platform which not only have a lot of user and data but also have a great support for developers to get these data from their service. Twitter is a massive social networking site tuned towards fast communication. More than 140 million active users publish over 400 million 140-character “Tweets” every day. Twitter’s speed and ease of publication have made it an important communication medium for people from all walks of life. Twitter has played a prominent role in socio-political events, such as the Arab Spring and the Occupy Wall Street movement. Twitter has also been used to post damage reports and disaster preparedness information during large natural disasters, such as the Hurricane Sandy. So that we could implement our idea based on these data. Twitter provide many API which let us to send our request and get almost every data we want from the server. It’s a good platform for me to take a first step in the social network analysis and collect data is also the basic of analysis.

Using OAuth to get the authority:

OAuth is an open standard to authorization. OAuth provides client applications a ‘secure delegated access' to server resources on behalf of a resource owner. It specifies a process for resource owners to authorize third-party access to their server resources without sharing their credentials. Designed specifically to work with Hypertext Transfer Protocol (HTTP), OAuth essentially allows access tokens to be issued to third-party clients by an authorization server, with the approval of the resource owner, or end-user. The client then uses the access token to access the protected resources hosted by the resource server. OAuth is commonly used as a way for web surfers to log into third party web sites using their Google, Facebook or Twitter accounts, without worrying about their access credentials being compromised.

Therefore, OAuth is the first step to get an authorization to use the Rest API to get the data.

Using REST API to collect the data:

The REST APIs provides programmatic access to read and write Twitter data. Author a new Tweet, read author profile and follower data, and more. The REST API identifies Twitter applications and users using OAuth, responses are available in JSON.

For example: one of the API we could use

GET statues/usr_timeline

Returns a collection of the most recent Tweets posted by the user indicated by the screen_name or user_id parameters.

User timelines belonging to protected users may only be requested when the authenticated user either “owns” the timeline or is an approved follower of the owner.

The timeline returned is the equivalent of the one seen when you view a user’s profile on twitter.com.

Test--get Avril Lavignes's tweets from twitter by using these tech:

Next step:

1. Get more detail about twitter API and get more info and then try to mining the useful data from it.
2. Learning some necessary knowledge about JS to handle the JSON data.
3. Try to learn some NLP algorithm in Python and take some practice
4. Plan further and get read some API documents from other platform such as Facebook or Weibo.

The Growth of Social Network in Our Social Life

The growth of social media in my life

In the past 20-30 years, our life changed a lot. Especially the appearance of social media make our life changed a lot. Nowadays, everything is connected: people, information, events and places, all the more so with the advent of online social media. Being connected to the world around us has never been more easy and accessible than it is today. But it did not start out that way. After the Internet made it possible to reach people around the globe with merely a click of a button, social network exploded into one of the biggest industries of our time.

There are too many small steps progress of the social media that I can see during my life.

1.       The appearance of phone & mobile: I think the advent of phone make the first link of people far away from each other so that they can use this media to build a connection to each other. And when I was in junior high school, our classmates use mobile to communicate with each other frequently and it’s also the only way we can contact our friends outside.

2.       The appearance of Internet and the advent of BBS and ICQ P2P…: When I was a kid and do not have any knowledge of social science. A very small number of family have Internet and some adult around me using BBS to share their idea and communicate. A lot of people who have same hobby or interest get together in BBS and the friends could also use ICQ to send message.  And I get my first QQ account when I was 12 years old and I find it is a very efficient way to contact our friend. With the Internet comes into everybody’s home, more and more of us use QQ, E-mail, And also P2P to share our documents on the network.

3.       The social network website comes into our life: When I was an undergraduate student, many of us using Renren to share their information on the website. It also provide many online service. It provide the service consists of a presentation of the user, our social links, and a variety of additional services which make a great change of our lifestyle. Almost all of us share or push and communicate on this platform in our spare time. We could know, no matter how far our friends are and how long we have not connect with, the information and the news of them. So everything connected by the social network.

4.       Nowadays, The social media become more and more powerful and could do more things for us, not only provide the basic service for us, but also read our mind and try to understand our needs and wants. There comes some recommendation on the website for us by using Machine Learning and Data Mining technology to analyze our behavior on the network, such as, the online marketing website like Amazon and Taobao could recommend some product we need by analyze our needs before and there are also some recommendation for friends and also movies or sports…to us by learning our tweets on the social media.

The Tech We Learned

One the courses, we get more knowledge about how the provider served for us and how the computer could read our mind on the social network. The NLP is the important procedure in analyze our behavior and it could be used to analyze the tweets we pushed on the social media and also some condition and statues of us. These awesome learning algorithms build the powerful social network and analyze our mind. There are some basic idea we could get from the courses and learned some basic supervised learning function to understand the sentiment of the sentence by using the dictionary to classify the key words and try to learn their emotion. There must be more great and functional algorithm we have not learn but I believe when we go more deep during the courses and reach more tech detail in the social network, it must become more interesting and fascinating. And these is also very useful in our future academic life and work life.

The First Post of My Blogger.com & My opinion in Social Network

The past 20-30 years is the period which saw the rapid development of global technology, especially the huge progress of Information technology. I deeply feel the pulse of knowledge information-the main melody of the development of the world. Not only Facebook, Twitter…. emerging and swiftly developing, But also how social media’s great influence to our life.

These years, a ‘storm’ sweeps all over the world which is called ‘Social Network Site’ is also one type of social media. In China, There are also several different social network platform appear these years such as Renren and Weibo… And there are also more and more people engaged in the network and the number is still increasing at incredible speed.

Apparently, Social media plays an important role in people’s life nowadays. It gives us a platform that we could stay in touch with our friends in a straightforward way and help us to find more friends and build the relationship in a magical way that we never image before. We could easily find the people not only the friend we haven’t seen for ages, but also who have the same hobby or lifestyle with us, even though we don’t know each other before and more and more people build relationship in this way by the social media high technology. We could also see many different recommendation for us on the network which is about shopping on Taobao or Amazon or focus on push some great movie or stars on Douban or Google+. It’s also a machine analysis of our behavior in social network in our friend relationship and even the Weibo/Twitter we posted before. These huge data in the network help us learn our self and also give us a resources that we could use to manage or find small clues about our life.

The course gave us an efficient way to approach these tech. The first three courses let us know some basic conception in social media and give us a broad view to these technology. Social media is not one but multiple disciplinary field, Such as data mining, machine learning, artificial intelligence, Hadoop and even some human analysis technology…are been used in the social media analysis. NLP is one of them and it’s used to analyze the text that people posted on the network. Modem NLP algorithms are based on machine learning. We learned some basic supervised learning algorithm such as tf-idf, Naïve Bayes and unsupervised learning algorithm such as K-means.  

Although there is not much about coding on the class but the conception also help us to understand the basic theory in NLP and its very important to us to apply these algorithm and go deep in the future. 

Reference:

[1]Social Media - Wikipedia

http://en.wikipedia.org/wiki/Social_media

[2]Mining Online Data Across Social Networks - Jure Leskovec, PHD

https://www.youtube.com/watch?v=IBov98-LIbI