TweetSentiment – Twitter activity VS market activity

Stefano Buliani — Thu, 05 Nov 2009 18:30:47 +0000

Perhaps I’m bored. Maybe I’m just too single.
Anyway. A couple of days ago I was looking at StockTwits again and saw a considerable amount of activity. There and then I decided that it would have been pretty cool to check whether there was any correlation between the activity about a stock on Twitter and the actual trading volume on the market.

I set off to build a small system to do just that. A couple of groovy scripts to collect the data and save it plus some JavaScript and HTML to display it.

So here I am a couple of days later talking about TweetSentiment.

Collecting the data I needed was pretty trivial thanks to the fact that StockTwits asks all its members to tweet the symbols they are trading preceded by a dollar sign ($).
My first task was to build a list of securities I was interested in. I decided to go to Covestor.com and pick the most traded securities list. Once that list was ready all I had to do was call the Twitter Search APIs for each stock and store the results.
Groovy made these tasks incredibly simple.

def output =newURL("http://search.twitter.com/search.atom?q=${"\$"+curSecurity.symbol}&rpp=100").getText()
def parsedXml = new XmlParser().parseText(output)
parsedXml.entry.each() { curEntry -&gt;

…

I’m not doing much with the tweets. At the moment I just count them and look for “buy” and “sell”. I’m looking into smarter ways of analysing the text and look for positive/negative opinions. If you have any suggestion on this please do leave a comment.

Next in my TODO list was getting some market data. For that I’m using Yahoo finance asking for previous closing price and average volume for each security. Once again, piece of cake with Groovy. Same thing as before just different URL.
Yahoo finance actually has quite a simple interface to let you grab the data. Check out this page on Cliff’s Notes.

Now I had all the data I needed. Only thing left was to build some sort of interface to look at it. I wanted to display all of the data on a chart to be able to make sense of it as quickly as possible.
My server (i.e. the machine running this blog) with its puny power could have never handled the traffic I get while generating charts. Therefore what I decided to do is to export all the data in text files (JSON format) and do all of the chart generation with JavaScript.

I browsed the web for a bit looking for charting components for jQuery. Couldn’t find anything I was happy with. Not functionality-wise but aesthetically. Or maybe I just felt like playing around with JS for some time – unfortunately as it often happens with JS I spent the best part of the last 2 days working on it.

I decided to use a library called Raphaël. This library makes working with vector graphics incredibly simple and the results just look amazing. I know that they are in the process of building a charting library on top of it however… well I have no excuse now other than I wanted to work with JS.

What I did is build a very simple jQuery component called SAPchart. It only draws line charts but it’s quite simple and I think the results look pretty good.

You can download the source here. It requires jQuery, Raphaël, and raphael path methods.

// Constructor. Possible options are:
// showGrid: true
// width: 800
// height: 250
// legendWidth: 200
// showLabels: true
// showDots: true

$("#holder").SAPchart({legendWidth: 800});
// Give the library a list of labels for the x axis

$("#holder").setLabels(theLabels);
// Add to the chart as many series as you want – provided the length of the series is

// the same of the labels (quite unsophisticated but serves its purpose)

// array of values, unique id of the series, legend name, additional options)

$("#holder").addSeries(theTweets, "tweet", "Twitter Activity", { "color": [.6, 1, .75] });

$("#holder").addSeries(theVolumes, "volume", "Average Daily Volume", { "color": [.2, 1, .75] });
// Draws the chart in your html element

$("#holder").draw()
// This will return an HTML table containing the legend of all the series you specified

// the boolean parameter tells me whether you want your legend to be horizontal (default vertical)

$("#holder").getLegend(true).appendTo($("#legend"));

I’m now wondering whether I should keep working on it and improve it.

Anyway TweetSentiment is now available here. I’m planning to keep it running and collecting data for a while. I’d say you need at least 3/4 months worth of data before you can make any useful observations.

Bookmark It

SAPessi » Twitter

TweetSentiment – Twitter activity VS market activity