Mar 09

I love calculating driving directions on Google Maps and then drag the blue line marking my directions to change the route. Everything is updated automatically on the page and the directions are re-calculated to go through the new point I defined.

I’ve been playing around with the Google Maps APIs recently and imagine my disappointment when I found out that Google does not allow directions calculated through the APIs to be dragged around.

Not put off by this I decided to try and replicate the directions-dragging myself. How hard can it be?
As it turns out. Very, at least if you want to make it look as smooth as Google’s own solution.

When generating directions Google Maps adds a GPolyline element overlay to your map. You can set the returned line to be editable but this makes an awful lot of vertices appear on it, which makes reading your directions quite hard. Even so, once you dragged one of this vertices around you are not editing the route but just changing the shape of the line.

Mine is not a complete solution and I’m interested in feedback and ideas on how to improve it.

First off calculate your directions:

map = new google.maps.Map2(document.getElementById(‘map_canvas’));
var wayPoints = [];
wayPoints.push(startPoint.getLatLng());
wayPoints.push(endPoint.getLatLng());

var myDir = new google.maps.Directions(map)
myDir.loadFromWaypoints(wayPoints, { travelMode: G_TRAVEL_MODE_DRIVING });

This will calculate your driving directions and plot a GPolyline on your map. The line is easily accessible in the GDirections object once the directions are calculated. To intercept this I have decided to use the addoverlay event on the GMap object. This event is triggered every time something is plotted over the map (says on the tin).

google.maps.Event.addListener(myDir, "addoverlay", function() {
var dirLine = myDir.getPolyline(); // Get the polyline from the directions object
});

At this point we can ask the APIs to make the GPolyline editable. This will make the vertices appear on the line and make then draggable. As I said before this only changes the shape of the line and doesn’t actually affect your directions object. Luckily the GPolyline comes with a nifty event called lineupdated.
This is triggered once the user has finished dragging a vertex. By intercepting this we can look through the vertices and know what’s been changed on the line and where the vertex has been moved to.
In order to do this we must also know the previous position of the vertices (latitude and longitude) to be able to compare the old and the new “edited” line.
Another challenge is the fact that the GDirections object can accept only so many waypoints (25 if I’m not wrong). Which means we’ll have to add only the vertex that has changed to the directions and not all of them.

// In the addoverlay event also save the original vertices of the line
var origLine = [];
for (var i = 0; i < dirLine.getVertexCount(); i++) {
origLine.push(dirLine.getVertex(i));
}
// DONE saving vertices

// Now intercept the lineupdated event and add the new waypoints
google.maps.Event.addListener(dirLine, "lineupdated", function() {
routePoints = [];
for (var i = 0; i < dirLine.getVertexCount(); i++) {
var savedPoint = origLine[i];
if (!savedPoint || (savedPoint.lat() != dirLine.getVertex(i).lat() && savedPoint.lng() != dirLine.getVertex(i).lng())) {
routePoints.push(dirLine.getVertex(i));
}
}

// Now we remove the previous directions and recalculate the route
map.removeOverlay(dirLine)
calcRoute();
});

This works quite well but does not look as smooth as Google’s solution.
The problem is that while you are dragging a vertex only that bit of the GPolyline moves and the rest stays in its original position. which makes the shape of your directions quite awkward while you are dragging. Unforunately the GPolyline does not come with a “startdragging” event, otherwise we could just recalculate the route every few seconds while the vertex is being dragged.

This is not the most elegant of solutions but it does the job.

Tagged with:
Nov 18

When I started developing TweetSentiment I decided that the interface should have as little text as possible. Most of the information I was interested in could be displayed graphically, with a chart.

So I looked at all the options available for chart generation.

  1. Backend code to generate a static image (JFreeChart or PHP)
  2. Flash object to draw a chart retrieving the data from a URL
  3. Draw charts in JavaScript directly on the client’s browser

I’m not a huge fan of option one. Primarily because TweetSentiment is hosted on a tiny linux box which would not be able to handle the load for the traffic the site gets, also because it’s a static image – it’s just not very funky – no interaction possible.

Option two would certainly create spectacular looking charts but I have almost no experience with flash and I wasn’t about to start learning a new language/technology. Plus I’m not into browser plugins if I can avoid them.

JavaScript is a language I’m familiar with and I remember seeing some cool-looking charts generated with Dojo. Unfortunately for TweetSentiment I have used jquery since the most important thing for me there was DOM manipulation (and jquery is just better for that).
I then started shopping around for jQuery plugins to generate charts. There are a few around but none of them impressed me. They just weren’t as good looking as I’d hoped nor they were interactive.

By coincidence I stumbled on RaphaelJS. A JavaScript library to draw Scalable Vector Graphics directly from JavaScript based on jQuery. I tested the samples on the website with a few browsers and I was happy to discover it worked just fine with all of them.

Scalable Vector Graphics (SVG) is a family of specifications of an XML-based file format for describing two-dimensional vector graphics, both static and dynamic (i.e. interactive or animated).

The SVG specification is an open standard that has been under development by the World Wide Web Consortium (W3C) since 1999.

I also discovered that there is a charting library built on top of RaphaelJS, which is exactly what I was looking for. However, being a geek, I decided to go ahead and try to develop something on my own. You know, just for kicks.

As I delved deeper into RaphaelJS I found the library to be incredibly powerful. It’s a shame that the documentation provided on the website lets it down a bit.
The most powerful bit is the ability to extend objects and attach new functions to them. Something scarcely mentioned in the available documentation.

For example if you need to use curved lines (paths as SVG calls them) you can just defined a default function you can then call from your code simply by adding it to the el “object” in RaphaelJS

Raphael.el.curveTo = function () {
  var args = Array.prototype.splice.call(arguments, 0, arguments.length),
  d = [0, 0, 0, 0, "s", 0, "c"][args.length] || "";
  this.isAbsolute && (d = d.toUpperCase());
  this._last = {x: args[args.length - 2], y: args[args.length - 1]};
  return this.attr({path: this.attrs.path + d + args});
};

Another very useful function I found in one of their samples is the andClose() This is used to close a polygon you have started drawing with paths. No matter where you got to it will reconnect to the initial point.

Raphael.el.andClose = function () {
  return this.attr({path: this.attrs.path + "z"});
};

This can then be used this way using chaining.

RaphaelJSElement.lineTo(x, opts.height - bottomgutter).andClose();

I’m still developing the chart library I used in TweetSentiment and I’m planning to publish it here with some documentation under MIT licence.

Tagged with:
Nov 05

Perhaps I’m bored. Maybe I’m just too single.
Anyway. A couple of days ago I was looking at StockTwits again and saw a considerable amount of activity. There and then I decided that it would have been pretty cool to check whether there was any correlation between the activity about a stock on Twitter and the actual trading volume on the market.

I set off to build a small system to do just that. A couple of groovy scripts to collect the data and save it plus some JavaScript and HTML to display it.

So here I am a couple of days later talking about TweetSentiment.

Collecting the data I needed was pretty trivial thanks to the fact that StockTwits asks all its members to tweet the symbols they are trading preceded by a dollar sign ($).
My first task was to build a list of securities I was interested in. I decided to go to Covestor.com and pick the most traded securities list.  Once that list was ready all I had to do was call the Twitter Search APIs for each stock and store the results.
Groovy made these tasks incredibly simple.

def output = new URL("http://search.twitter.com/search.atom?q=${"\$"+curSecurity.symbol}&rpp=100").getText()

def parsedXml = new XmlParser().parseText(output)

parsedXml.entry.each() { curEntry ->

I’m not doing much with the tweets. At the moment I just count them and look for “buy” and “sell”. I’m looking into smarter ways of analysing the text and look for positive/negative opinions. If you have any suggestion on this please do leave a comment.

Next in my TODO list was getting some market data. For that I’m using Yahoo finance asking for previous closing price and average volume for each security. Once again, piece of cake with Groovy. Same thing as before just different URL.
Yahoo finance actually has quite a simple interface to let you grab the data. Check out this page on Cliff’s Notes.

Now I had all the data I needed. Only thing left was to build some sort of interface to look at it. I wanted to display all of the data on a chart to be able to make sense of it as quickly as possible.
My server (i.e. the machine running this blog) with its puny power could have never handled the traffic I get while generating charts. Therefore what I decided to do is to export all the data in text files (JSON format) and do all of the chart generation with JavaScript.

I browsed the web for a bit looking for charting components for jQuery. Couldn’t find anything I was happy with. Not functionality-wise but aesthetically. Or maybe I just felt like playing around with JS for some time – unfortunately as it often happens with JS I spent the best part of the last 2 days working on it.

I decided to use a library called Raphaël. This library makes working with vector graphics incredibly simple and the results just look amazing. I know that they are in the process of building a charting library on top of it however… well I have no excuse now other than I wanted to work with JS.

What I did is build a very simple jQuery component called SAPchart. It only draws line charts but it’s quite simple and I think the results look pretty good.

You can download the source here. It requires jQuery, Raphaël, and raphael path methods.

// Constructor. Possible options are:
// showGrid: true
// width: 800
// height: 250
// legendWidth: 200
// showLabels: true
// showDots: true
$("#holder").SAPchart({legendWidth: 800});

// Give the library a list of labels for the x axis
$("#holder").setLabels(theLabels);

// Add to the chart as many series as you want – provided the length of the series is
// the same of the labels (quite unsophisticated but serves its purpose)
// array of values, unique id of the series, legend name, additional options)
$("#holder").addSeries(theTweets, "tweet", "Twitter Activity", { "color": [.6, 1, .75] });
$("#holder").addSeries(theVolumes, "volume", "Average Daily Volume", { "color": [.2, 1, .75] });

// Draws the chart in your html element
$("#holder").draw()

// This will return an HTML table containing the legend of all the series you specified
// the boolean parameter tells me whether you want your legend to be horizontal (default vertical)
$("#holder").getLegend(true).appendTo($("#legend"));

I’m now wondering whether I should keep working on it and improve it.

Anyway TweetSentiment is now available here. I’m planning to keep it running and collecting data for a while. I’d say you need at least 3/4 months worth of data before you can make any useful observations.

Tagged with:
Nov 02

I have been interviewing quite a few people recently for a position as Java developer.
Every single CV came with either a Struts or Spring mention. If not both. Oh yes, I have even discussed systems where both frameworks were used at the same time.

Of course it’s your system, you are free to use whichever technology you prefer. However, you must also be able to explain why you picked that particular framework. And “It wasn’t me who picked it” is not a good answer. If you had a better idea why didn’t you question the decision?

Perhaps my problem is not exactly with MVC frameworks but with the people out there who use them without thinking things through. These are the sort of people who go around telling you that they used both Spring and Struts because it’s an “Architectural pattern”. The truth is that they haven’t thought about their architecture at all. They just thought that by using an MVC framework the problem would solve itself.

Yesterday I was reading a post by Albert Wenger at Union Square Ventures about Loose Coupling. I’m not going to repeat here what Albert already explained quite thoroughly. I will just say I agree with everything he said. Especially with this paragraph:

So how does loose coupling enable scaling and innovation?  Scaling bottlenecks tend to move around as a site or service grows.  With loose coupling it is often possible to isolate a bottleneck and prevent it from slowing down everything else.

This also points to how loose coupling enables innovation.  Different teams can more easily be in charge of their own part and make changes to it more quickly.  Entirely new parts can be added more easily too (a great example here is Facebook adding chat).

MVC frameworks themselves do not prevent loose coupling as an architectural decision. The problem is that most of the people who use them are the sort of over-engeneering maniacs who try to build the perfect system in one big blob of code, and inescapably, fail.
So here’s my point. Using an MVC should not be considered an architectural decision. Architecture is at a much higher level.

Most architecture design nowadays show systems split horizontally.

TWING MVC structure

This may allow your application to support higher load, but doesn’t give you the ability to isolate bottlenecks in your application. I believe what we’ll see in the future is architecture diagrams that are split vertically with every single component (or task your app has to perform) being completely independent from the rest, and feel free to use the MVC of your choice in each component. Just make sure you know why you decided to use it.

Tagged with:
Nov 23

Hi All,

During Movember (the month formerly known as November) I’ll be growin a Mo (slang for Moustache). That’s right I’m bringing the Mo back to raise funds for The Prostate Cancer Charity because I’m passionate about men’s health and the fight against prostate cancer. Why…

  • Prostate cancer is now the most common cancer diagnosed in men in the U.K. with at least one man dying every hour from the disease.
  • Every year about 35,000 men in the U.K. are diagnosed with prostate cancer and about 10,000 men die from the disease.
  • One man in 11 will be diagnosed with prostate cancer in their lifetime in the U.K.

To sponsor my Mo (moustache) and fight against prostate cancer please go to http://www.movember.com/uk/donate, enter my registration number which is 120922 and your credit card details. Or you can sponsor me by cheque made payable to “The Prostate Cancer Charity” clearly marking the donation as being for my Registration Number: 120922. Please mail cheques to: The Prostate Cancer Charity, ATT: Movember, First Floor, Cambridge House, 100 Cambridge Grove, Hammersmith, London W6 0LE.

All donations are made directly to The Prostate Cancer Charity which will use the money to fund high quality research into the causes, treatment and impact of prostate cancer and to provide support and information to men and their families.

Movember culminates at the end of the month at the Gala Partés. These glamorous and groomed events will see Tom Selleck and Borat look-a-likes battle it out for their chance to take home the prestigious Man of Movember title. If you would like to be part of this great night you’ll need to purchase a Gala Parté ticket.
Thanks for your support

Stefano

Movember - Sponsor Me

Tagged with:
Nov 13

After updating to Mac OS X Leopard I started experiencing some annoying lags and delays with my NFS mounts.
What happened was that Eclipse kept crashing whenever I tried to open a file from a project folder mounted with NFS. After a quick search on Google I realized that I wasn’t going to get any help on this one.

When I updated my laptop we also moved our development environment to a new server so up until now I wasn’t entirely sure whether to blame Leopard’s NFS client or the new server’s configuration.

After comparing the configuration of our old and new server, which turned out to be exactly the same, I started frowning at my laptop.
Leopard’s NFS client is not entirely to blame though, Eclipse IDE is equally at fault.

The crashes looked suspicious and definitely weren’t caused by Java since Eclipse seemed to work great with purely local projects.
First thing we have to consider is that NFS handles file locking in a fairly chaotic way. This, especially with Eclipse trying to access all the files inside a project folder, might cause your mount to be extremely unstable, so much so that the entire Eclipse IDE crashes waiting for the network mount to react.

NFS and file locking is a peculiar issue, e.g. the Linux 2.4 NFS client pretends to lock files but actually doesn’t, Linux 2.6 does it right, but the implementation has some nasty side effects: locks are held using .nfsXXX files that pollute the directories and prevent operations on it you could do on a local fs.

Eclipse’s problem is clearly that the network is not responding quickly enough and I could think only of 2 possible solutions. Which in the end solved my problem.

1. Increase the maximum number of read/write operations your NFS client can perform

This can be achieved either with the command nfsiod -n <number of threads> (deprecated in Leopard, don’t know about earlier versions of MacOS X) or by modifying NFS’ configuration file in /etc/nfs.conf adding the following line – nfs.client.nfsiod_thread_max=<max number of threads>.
If I remember correctly the nfsiod parameter should be set to 4 by default and even a small increase (8) should do the trick. Be careful not to play too much with this parameter or you’ll end up with an overloaded NFS client crashing you local network.

Going from 4 to 8 nfsiod threads resulted in write speeds going from 700k/sec to over 3.2mb/sec!

For additional information check the NFS Performance HOW-TO.

2. Configure your mounts either from fstab or from the command line to use local lock-files

The option we’re looking for is locallocks. The main drawback here is that your server is not going to be aware of which files you are editing so I wouldn’t recommend going this way if you’re working on an heavily trafficked/edited share.
To use this option in a mount command just run sudo mount_nfs -o locallocks servername:folder localfolder.

Either one of these solutions should do the trick. I have used both on my laptop since I’m the only one working on the NFS share in question and it worked like magic.

Nov 09

I have just spent an entire week writing JavaScript code. Am i feeling suicidal? Not really, at least not just yet.

I know exactly how frustrating JavaScript can be, especially when you’re dynamically rewriting big chunks of you pages’ HTML code.
This problem is particularly evident now that your code needs to be tested thoroughly with N different browser, who, obviously, behave in N completely different ways.

Using a framework like Prototype can make your life considerably easier here, especially if you use it the right way. Because if you’re not you’ll be left with just a somewhat shorter version of document.getElementById without any actual benefit. You’ll still have to write tons of lines of code.

I have found just this morning a couple of articles linked on Ajaxian discussing Prototype programming best practices. They solved most of my problems.

Oct 28

Thanks to all Covestor members who took the time to try the widget and sent me lots of feedback.

Working on this widget has been incredibly useful because it made me realize some of the “shortcomings” of the data export APIs.

This new release includes:

  1. Minor bug fixes in the feed validation
  2. Better look and feel (I hope)
  3. Feed filter as on your Covestor page
  4. Changed updated frequency to range between 1 and 20 minutes rather than a few seconds

As usual download it here or from the link in the right navigation bar. My previous post contains a more detailed installation guide.

Covestor widget

Oct 27

I have finally received my copy of Leopard and have spent the entire day playing with DashCode.

 

 

 

My first widget leverages Covestor data export functionality and displays the live feed you normally see on “My Summary”

Covestor widget

To use it simply install it (by double clicking on the widget file) then activate the data export from your account settings.

Covestor Data Export

 

 Covestor Data Export

Once the feed is enabled all you have to do is copy the “Live Feed” XML url from the link on the page and paste it in the widget configuration.
As you can see there’s an additional configuration parameter which allows you to specify how often you want the widget to check for updates on Covestor.

his is just a first beta release so don’t expect anything fancy. This widget is compatible with MacOS X 10.4.3 or newer.

Click here to download the widget or use the link on the right menu (under Evangelion interpretation). Feedback and suggestions are more than welcome!

Oct 03

Finally, TechCrunch just posted an article about Covestor, you can read it here.

Needless to say after months of hard work – and those of you who have worked in startups know exactly how frustrating it can be – we get some of the recognition a project like this deserves. I am absolutely elated and very proud to be part of it.

Quote from one of the comments on TechCrunch:

I just signed up for the service and am, in short, stunned. It blows away all the competitors in this space by guaranteeing trust (all the trades you see are real) and reducing friction (you don’t have to re-enter any of your trades, ever).

This is the first investment-related site that will actually gain my attention on a regular basis.

All in all a very exciting day!

Stefano

preload preload preload