Nov 18

When I started developing TweetSentiment I decided that the interface should have as little text as possible. Most of the information I was interested in could be displayed graphically, with a chart.

So I looked at all the options available for chart generation.

  1. Backend code to generate a static image (JFreeChart or PHP)
  2. Flash object to draw a chart retrieving the data from a URL
  3. Draw charts in JavaScript directly on the client’s browser

I’m not a huge fan of option one. Primarily because TweetSentiment is hosted on a tiny linux box which would not be able to handle the load for the traffic the site gets, also because it’s a static image – it’s just not very funky – no interaction possible.

Option two would certainly create spectacular looking charts but I have almost no experience with flash and I wasn’t about to start learning a new language/technology. Plus I’m not into browser plugins if I can avoid them.

JavaScript is a language I’m familiar with and I remember seeing some cool-looking charts generated with Dojo. Unfortunately for TweetSentiment I have used jquery since the most important thing for me there was DOM manipulation (and jquery is just better for that).
I then started shopping around for jQuery plugins to generate charts. There are a few around but none of them impressed me. They just weren’t as good looking as I’d hoped nor they were interactive.

By coincidence I stumbled on RaphaelJS. A JavaScript library to draw Scalable Vector Graphics directly from JavaScript based on jQuery. I tested the samples on the website with a few browsers and I was happy to discover it worked just fine with all of them.

Scalable Vector Graphics (SVG) is a family of specifications of an XML-based file format for describing two-dimensional vector graphics, both static and dynamic (i.e. interactive or animated).

The SVG specification is an open standard that has been under development by the World Wide Web Consortium (W3C) since 1999.

I also discovered that there is a charting library built on top of RaphaelJS, which is exactly what I was looking for. However, being a geek, I decided to go ahead and try to develop something on my own. You know, just for kicks.

As I delved deeper into RaphaelJS I found the library to be incredibly powerful. It’s a shame that the documentation provided on the website lets it down a bit.
The most powerful bit is the ability to extend objects and attach new functions to them. Something scarcely mentioned in the available documentation.

For example if you need to use curved lines (paths as SVG calls them) you can just defined a default function you can then call from your code simply by adding it to the el “object” in RaphaelJS

Raphael.el.curveTo = function () {
  var args = Array.prototype.splice.call(arguments, 0, arguments.length),
  d = [0, 0, 0, 0, "s", 0, "c"][args.length] || "";
  this.isAbsolute && (d = d.toUpperCase());
  this._last = {x: args[args.length - 2], y: args[args.length - 1]};
  return this.attr({path: this.attrs.path + d + args});
};

Another very useful function I found in one of their samples is the andClose() This is used to close a polygon you have started drawing with paths. No matter where you got to it will reconnect to the initial point.

Raphael.el.andClose = function () {
  return this.attr({path: this.attrs.path + "z"});
};

This can then be used this way using chaining.

RaphaelJSElement.lineTo(x, opts.height - bottomgutter).andClose();

I’m still developing the chart library I used in TweetSentiment and I’m planning to publish it here with some documentation under MIT licence.

Tagged with:
Nov 05

Perhaps I’m bored. Maybe I’m just too single.
Anyway. A couple of days ago I was looking at StockTwits again and saw a considerable amount of activity. There and then I decided that it would have been pretty cool to check whether there was any correlation between the activity about a stock on Twitter and the actual trading volume on the market.

I set off to build a small system to do just that. A couple of groovy scripts to collect the data and save it plus some JavaScript and HTML to display it.

So here I am a couple of days later talking about TweetSentiment.

Collecting the data I needed was pretty trivial thanks to the fact that StockTwits asks all its members to tweet the symbols they are trading preceded by a dollar sign ($).
My first task was to build a list of securities I was interested in. I decided to go to Covestor.com and pick the most traded securities list.  Once that list was ready all I had to do was call the Twitter Search APIs for each stock and store the results.
Groovy made these tasks incredibly simple.

def output = new URL("http://search.twitter.com/search.atom?q=${"\$"+curSecurity.symbol}&rpp=100").getText()

def parsedXml = new XmlParser().parseText(output)

parsedXml.entry.each() { curEntry ->

I’m not doing much with the tweets. At the moment I just count them and look for “buy” and “sell”. I’m looking into smarter ways of analysing the text and look for positive/negative opinions. If you have any suggestion on this please do leave a comment.

Next in my TODO list was getting some market data. For that I’m using Yahoo finance asking for previous closing price and average volume for each security. Once again, piece of cake with Groovy. Same thing as before just different URL.
Yahoo finance actually has quite a simple interface to let you grab the data. Check out this page on Cliff’s Notes.

Now I had all the data I needed. Only thing left was to build some sort of interface to look at it. I wanted to display all of the data on a chart to be able to make sense of it as quickly as possible.
My server (i.e. the machine running this blog) with its puny power could have never handled the traffic I get while generating charts. Therefore what I decided to do is to export all the data in text files (JSON format) and do all of the chart generation with JavaScript.

I browsed the web for a bit looking for charting components for jQuery. Couldn’t find anything I was happy with. Not functionality-wise but aesthetically. Or maybe I just felt like playing around with JS for some time – unfortunately as it often happens with JS I spent the best part of the last 2 days working on it.

I decided to use a library called Raphaël. This library makes working with vector graphics incredibly simple and the results just look amazing. I know that they are in the process of building a charting library on top of it however… well I have no excuse now other than I wanted to work with JS.

What I did is build a very simple jQuery component called SAPchart. It only draws line charts but it’s quite simple and I think the results look pretty good.

You can download the source here. It requires jQuery, Raphaël, and raphael path methods.

// Constructor. Possible options are:
// showGrid: true
// width: 800
// height: 250
// legendWidth: 200
// showLabels: true
// showDots: true
$("#holder").SAPchart({legendWidth: 800});

// Give the library a list of labels for the x axis
$("#holder").setLabels(theLabels);

// Add to the chart as many series as you want – provided the length of the series is
// the same of the labels (quite unsophisticated but serves its purpose)
// array of values, unique id of the series, legend name, additional options)
$("#holder").addSeries(theTweets, "tweet", "Twitter Activity", { "color": [.6, 1, .75] });
$("#holder").addSeries(theVolumes, "volume", "Average Daily Volume", { "color": [.2, 1, .75] });

// Draws the chart in your html element
$("#holder").draw()

// This will return an HTML table containing the legend of all the series you specified
// the boolean parameter tells me whether you want your legend to be horizontal (default vertical)
$("#holder").getLegend(true).appendTo($("#legend"));

I’m now wondering whether I should keep working on it and improve it.

Anyway TweetSentiment is now available here. I’m planning to keep it running and collecting data for a while. I’d say you need at least 3/4 months worth of data before you can make any useful observations.

Tagged with:
preload preload preload