The new version of MapKit in iOS 4 supports polylines, finally. MKPolyline to be precise.

I have been working with the Google Maps API for a while and now I’m learning Objective-C and writing an iPhone app ro Route.ly

Since the best way to send around the net the data to draw a polyline on a map is in encoded format I decided to write a small Objective-C function to decode the encoded string. Since I couldn’t find this anywhere on the net I decided to publish the code here.

You’ll notice that at the beginning of the function I replace “\\\\” with “\\”. This is because I use the encoded data both in web pages with javascript and my iPhone app. In javascript “\\” would be considered an escape and would therefore break my array of points. This is not the case in Objective-C so I switch back to the normal “\\”.

Waypoints is a simple model class I’ve written with two NSString properties: lat and lng.

Bookmark It

Brilliant video posted by Fred Wilson on his blog.

If you can spare half an hour definitely watch it!

Bookmark It

I love calculating driving directions on Google Maps and then drag the blue line marking my directions to change the route. Everything is updated automatically on the page and the directions are re-calculated to go through the new point I defined.

I’ve been playing around with the Google Maps APIs recently and imagine my disappointment when I found out that Google does not allow directions calculated through the APIs to be dragged around.

Not put off by this I decided to try and replicate the directions-dragging myself. How hard can it be?
As it turns out. Very, at least if you want to make it look as smooth as Google’s own solution.

When generating directions Google Maps adds a GPolyline element overlay to your map. You can set the returned line to be editable but this makes an awful lot of vertices appear on it, which makes reading your directions quite hard. Even so, once you dragged one of this vertices around you are not editing the route but just changing the shape of the line.

Mine is not a complete solution and I’m interested in feedback and ideas on how to improve it.

First off calculate your directions:

This will calculate your driving directions and plot a GPolyline on your map. The line is easily accessible in the GDirections object once the directions are calculated. To intercept this I have decided to use the addoverlay event on the GMap object. This event is triggered every time something is plotted over the map (says on the tin).

At this point we can ask the APIs to make the GPolyline editable. This will make the vertices appear on the line and make then draggable. As I said before this only changes the shape of the line and doesn’t actually affect your directions object. Luckily the GPolyline comes with a nifty event called lineupdated.
This is triggered once the user has finished dragging a vertex. By intercepting this we can look through the vertices and know what’s been changed on the line and where the vertex has been moved to.
In order to do this we must also know the previous position of the vertices (latitude and longitude) to be able to compare the old and the new “edited” line.
Another challenge is the fact that the GDirections object can accept only so many waypoints (25 if I’m not wrong). Which means we’ll have to add only the vertex that has changed to the directions and not all of them.

This works quite well but does not look as smooth as Google’s solution.
The problem is that while you are dragging a vertex only that bit of the GPolyline moves and the rest stays in its original position. which makes the shape of your directions quite awkward while you are dragging. Unforunately the GPolyline does not come with a “startdragging” event, otherwise we could just recalculate the route every few seconds while the vertex is being dragged.

This is not the most elegant of solutions but it does the job.

Bookmark It

While skimming through my RSS feed I stumbled upon an interesting article on Ajaxian about server side JavaScript programming.

I really, really enjoy writing JavaScript code so I decided to read through the article and take a look at the node.js library linked there.

Node is a framework to build server-side event-driven JavaScript applications. Developed in C++ on top of Google’s V8 JavaScript engine and accompanied by a set of JavaScript libraries Node seems to make building distributed (over a network), fast applications a piece of cake even for inexperienced developers.
Event-driven here really is the keyword because it represent a big change in the way applications are built (and architected in your brain).

Node is similar in design to and influenced by systems like Ruby’s Event Machine or Python’s Twisted. Node takes the event model a bit further—it presents the event loop as a language construct instead of as a library. In other systems there is always a blocking call to start the event-loop. Typically one defines behavior through callbacks at the beginning of a script and at the end starts a server through a blocking call like

EventMachine::run()

. In Node there is no such start-the-event-loop call. Node simply enters the event loop after executing the input script.

The example below (taken from Node’s website) will make everything clear… hopefully.

Can you see the beauty?!

I have only one worry. This is an open-source effort. The community behind it on Google groups is just 181 members strong (so far). What if node.js suddenly stops being the cool thing and the community disappears.

As much as I love writing JavaScript and I can really see the value in what they are building I’ll still wait until there are 100 Google Groups for node.js and a trillion members in each before using it in anything close to a production system.

Having said that I’m going back to writing silly node.js apps now. Well done to all the developers involved in the project and keep it up!

Bookmark It

When I started developing TweetSentiment I decided that the interface should have as little text as possible. Most of the information I was interested in could be displayed graphically, with a chart.

So I looked at all the options available for chart generation.

1. Backend code to generate a static image (JFreeChart or PHP)
2. Flash object to draw a chart retrieving the data from a URL
3. Draw charts in JavaScript directly on the client’s browser

I’m not a huge fan of option one. Primarily because TweetSentiment is hosted on a tiny linux box which would not be able to handle the load for the traffic the site gets, also because it’s a static image – it’s just not very funky – no interaction possible.

Option two would certainly create spectacular looking charts but I have almost no experience with flash and I wasn’t about to start learning a new language/technology. Plus I’m not into browser plugins if I can avoid them.

JavaScript is a language I’m familiar with and I remember seeing some cool-looking charts generated with Dojo. Unfortunately for TweetSentiment I have used jquery since the most important thing for me there was DOM manipulation (and jquery is just better for that).
I then started shopping around for jQuery plugins to generate charts. There are a few around but none of them impressed me. They just weren’t as good looking as I’d hoped nor they were interactive.

By coincidence I stumbled on RaphaelJS. A JavaScript library to draw Scalable Vector Graphics directly from JavaScript based on jQuery. I tested the samples on the website with a few browsers and I was happy to discover it worked just fine with all of them.

Scalable Vector Graphics (SVG) is a family of specifications of an XML-based file format for describing two-dimensional vector graphics, both static and dynamic (i.e. interactive or animated).

The SVG specification is an open standard that has been under development by the World Wide Web Consortium (W3C) since 1999.

I also discovered that there is a charting library built on top of RaphaelJS, which is exactly what I was looking for. However, being a geek, I decided to go ahead and try to develop something on my own. You know, just for kicks.

As I delved deeper into RaphaelJS I found the library to be incredibly powerful. It’s a shame that the documentation provided on the website lets it down a bit.
The most powerful bit is the ability to extend objects and attach new functions to them. Something scarcely mentioned in the available documentation.

For example if you need to use curved lines (paths as SVG calls them) you can just defined a default function you can then call from your code simply by adding it to the el “object” in RaphaelJS

Another very useful function I found in one of their samples is the andClose() This is used to close a polygon you have started drawing with paths. No matter where you got to it will reconnect to the initial point.

This can then be used this way using chaining.

I’m still developing the chart library I used in TweetSentiment and I’m planning to publish it here with some documentation under MIT licence.

Bookmark It

I have just stumbled on an article on Slashdot about Google Go. A programming language which was initially developed internally and it’s now been open-sourced.

Over the past few months I have been working with Erlang on multi-process distributed systems. Concurrency in Go is based on the same model Erlang uses: Hoare’s Communicating Sequential Processes, or CSP.

Go’s concurrency primitives derive from a different part of the family tree whose main contribution is the powerful notion of channels as first class objects.

The thing I appreciate the most of Erlang is how easy it is to distribute – even over a network – and monitor processes. After skimming through Go’s documentation I couldn’t help but notice the lack of integrated network-distribution of processes or goroutines, as they are called.

This for me means that while Go, exactly like Erlang and Occam, simplifies the approach to multi-threaded programming (at least from a developer’s point of view), it doesn’t go all the way and leaves developers to deal with distribution of processes over a network on their own.

This is a perfectly acceptable design choice in the name of flexibility, however it creates a whole new bunch of use cases for the developers to deal with, such as sending channels/channels-data over the network).

What I don’t like of Erlang is its handling of strings, which makes it unsuitable for a number of application that could benefit greatly from its multi-processing capabilities. Very few people would recommend Erlang as a high performance string manipulation language. To quote Sendmail’s case study in implementing their Sendmail load balancing “Client Daemon” in Erlang:

…But Erlang’s treatment of strings as lists of bytes is as elegant as it is impractical. The factor-of-eight storage expansion of text, as well as the copying that occurs during message-passing, cripples Erlang for all but the most performance-insensitive text-processing applications.

Go totally wins on this.

If you now asked me to pick one to use. I’d still go for Erlang.
Functional languages and CSP are changing the way big companies approach the development of business critical applications. In the last year alone I have seen the number of positions advertised for Erlang developers in the financial sector multiply.
Go, if it continues on this path definitely has a chance of making it there. Open-sourcing the language is the right decision to give it the exposure to real-life scenarios that tend to accelerate the growth and adoption rate.

Bookmark It

Perhaps I’m bored. Maybe I’m just too single.
Anyway. A couple of days ago I was looking at StockTwits again and saw a considerable amount of activity. There and then I decided that it would have been pretty cool to check whether there was any correlation between the activity about a stock on Twitter and the actual trading volume on the market.

I set off to build a small system to do just that. A couple of groovy scripts to collect the data and save it plus some JavaScript and HTML to display it.

So here I am a couple of days later talking about TweetSentiment.

Collecting the data I needed was pretty trivial thanks to the fact that StockTwits asks all its members to tweet the symbols they are trading preceded by a dollar sign ($).
My first task was to build a list of securities I was interested in. I decided to go to Covestor.com and pick the most traded securities list.  Once that list was ready all I had to do was call the Twitter Search APIs for each stock and store the results.
Groovy made these tasks incredibly simple.

I’m not doing much with the tweets. At the moment I just count them and look for “buy” and “sell”. I’m looking into smarter ways of analysing the text and look for positive/negative opinions. If you have any suggestion on this please do leave a comment.

Next in my TODO list was getting some market data. For that I’m using Yahoo finance asking for previous closing price and average volume for each security. Once again, piece of cake with Groovy. Same thing as before just different URL.
Yahoo finance actually has quite a simple interface to let you grab the data. Check out this page on Cliff’s Notes.

Now I had all the data I needed. Only thing left was to build some sort of interface to look at it. I wanted to display all of the data on a chart to be able to make sense of it as quickly as possible.
My server (i.e. the machine running this blog) with its puny power could have never handled the traffic I get while generating charts. Therefore what I decided to do is to export all the data in text files (JSON format) and do all of the chart generation with JavaScript.

I browsed the web for a bit looking for charting components for jQuery. Couldn’t find anything I was happy with. Not functionality-wise but aesthetically. Or maybe I just felt like playing around with JS for some time – unfortunately as it often happens with JS I spent the best part of the last 2 days working on it.

I decided to use a library called Raphaël. This library makes working with vector graphics incredibly simple and the results just look amazing. I know that they are in the process of building a charting library on top of it however… well I have no excuse now other than I wanted to work with JS.

What I did is build a very simple jQuery component called SAPchart. It only draws line charts but it’s quite simple and I think the results look pretty good.

You can download the source here. It requires jQuery, Raphaël, and raphael path methods.

I’m now wondering whether I should keep working on it and improve it.

Anyway TweetSentiment is now available here. I’m planning to keep it running and collecting data for a while. I’d say you need at least 3/4 months worth of data before you can make any useful observations.

Bookmark It

If you Google for this you’ll find that the easiest way to provide a full audit of everything that happened in your database is to create a duplicate of each table you need history and insert a copy of the record you are changing in there for every update.

For example if you have a table called client

You will also create a table called client_history

You’ll then attach an onUpdate trigger to the client table inserting all the data in your history table. This solution gives you the ability to keep track of everything that happens in your database, when it happened and who-done-it.
What this solution doesn’t deal with is schema changes. Also querying the state of a record at a point in time to join it with other tables might be a bit tricky.

Let’s start from the first point. Schema changes.
The most obvious and simple solution is to put in place a proper process for all database schema changes. i.e. whoever touches the schema is also in charge of updating the history table structure and the trigger on the main table. Doable but a bit too prone to human error if you ask me.

What I generally do is use a small function/stored procedure to create the history on a table. This procedure is in charge of both creating the history table the first time it’s called and updating the structure if the main table has changed.
In this example I’m going to use PostgreSQL. First because it’s a database I’m familiar with and more importantly being open-srouce you can just download it and try this yourself.

My example here is a bit verbose but it serves its purpose. There are many database-specific instructions you could use to extract a table structure or perform some of the simple tasks of this function. However, what I’m trying to demonstrate is the concept and not PostgreSQL trickery.
Most relational databases store the schema information in internal tables (pg_attribute, pg_class and pg_namespace in this case) so that’s what we are going to use to read the structure of our original table.

You’ll notice this function only adds fields to the history table. That’s because even if we remove a field from the original table we still want to keep track of it in our history.

Now on to querying the history to retrieve the state of a record at a specific date.

This is a bit tricky. At the moment as far as I know only Oracle has the ability return a “virtual table” from a function. PostgreSQL can return a RECORD variable – which is great to use inside functions but once outside it loses its structure and turns into a comma separated list of values – or a %ROWTYPE you can define.
This unfortunately means that you’d have to create a custom function for each history table to returns a SET OF client – in our case.

What we are trying to achieve is something like what this query does.

Basically the next record in the history after the time specified.

At this point so far as I can see we have two options. Either create a function that returns a specific type of ROW, or write a more generic history function to return only the id of the history record we need and not the whole row (If you look at the history creation function we are putting a histid column in there).

You could use this function in your queries to get the history record id out like this.

I’m sure you can figure out the rest.

Be careful this is very slow on large data sets. If you are planning to work on millions of records then you should consider building a history lookup function for each table either defining data types for your RECORD or using OUT variables.

Bookmark It

I have been interviewing quite a few people recently for a position as Java developer.
Every single CV came with either a Struts or Spring mention. If not both. Oh yes, I have even discussed systems where both frameworks were used at the same time.

Of course it’s your system, you are free to use whichever technology you prefer. However, you must also be able to explain why you picked that particular framework. And “It wasn’t me who picked it” is not a good answer. If you had a better idea why didn’t you question the decision?

Perhaps my problem is not exactly with MVC frameworks but with the people out there who use them without thinking things through. These are the sort of people who go around telling you that they used both Spring and Struts because it’s an “Architectural pattern”. The truth is that they haven’t thought about their architecture at all. They just thought that by using an MVC framework the problem would solve itself.

Yesterday I was reading a post by Albert Wenger at Union Square Ventures about Loose Coupling. I’m not going to repeat here what Albert already explained quite thoroughly. I will just say I agree with everything he said. Especially with this paragraph:

So how does loose coupling enable scaling and innovation?  Scaling bottlenecks tend to move around as a site or service grows.  With loose coupling it is often possible to isolate a bottleneck and prevent it from slowing down everything else.
…
This also points to how loose coupling enables innovation.  Different teams can more easily be in charge of their own part and make changes to it more quickly.  Entirely new parts can be added more easily too (a great example here is Facebook adding chat).

MVC frameworks themselves do not prevent loose coupling as an architectural decision. The problem is that most of the people who use them are the sort of over-engeneering maniacs who try to build the perfect system in one big blob of code, and inescapably, fail.
So here’s my point. Using an MVC should not be considered an architectural decision. Architecture is at a much higher level.

Most architecture design nowadays show systems split horizontally.

This may allow your application to support higher load, but doesn’t give you the ability to isolate bottlenecks in your application. I believe what we’ll see in the future is architecture diagrams that are split vertically with every single component (or task your app has to perform) being completely independent from the rest, and feel free to use the MVC of your choice in each component. Just make sure you know why you decided to use it.

Bookmark It

You may have read recently that Khronos is implementing something called WebGL. The objective of the project is to expose all of OpenGL ES calls to javascript. Thus allowing hardware accellerated 3d graphics within a browser.
Google has also been working on an alternative, called O3D.

Let’s first talk about the technical differencies between the two projects.

O3D and WebGL while both trying to bring accellerated 3D graphics to the web have taken two fairly different courses. As I mentioned in the introduction to this post WebGL’s plan is just to expose to JavaScript OpenGL ES 2.0 APIs. Whereas Google’s solution is based on a browser plugin.

If we think about this we’ll soon realise that WebGL depends entirely on JavaScript. JavaScript, as of today, is a fairly slow language. This point was made in a discussion thread on the O3D project website.

WebGL, being 100% dependent on JavaScript to do an application’s scene graph, is going to have serious problems drawing more than a few pieces of geometry at 60hz except in very special cases or on very fast machines. This means WebGL requires JavaScript to:

*) do all parent-child matrix calculations for a transform graph.

*) all culling calculations (bounding box to frustum or other)

*) all sorting calculations for dealing with transparent objects.

*) all animation calculations.

As an example the kitty demo in O3D is doing linear interpolations on 2710 floats to animate 170 transforms. The point is not that the artist that created the kitty should probably not have used 170 bones. Rather the point is it seems unlikely that JavaScript
will be able to do that anytime soon and if it can then just add more than one kitty to pass its limits.

Also we have to keep in mind that not all hardware supports OpenGL ES.

O3D, By virtue of being a browser plugin written in C++, so an additional (hopefully fast) abstraction layer on top of the GPU, allowed Google to define a new set of APIs to expose to JavaScript and keep us (the JavaScript developers) away from the hardware. O3D will take care of the interaction with either DirectX or OpenGL.

Furthermore Google has open-sourced O3D through its Google Code website. Which means we can all have a look at their code and participate in the project. This resulted in a lot of documentation being available. For a full overview of how O3D works check out the technical overview on the O3D project page.

Do you think that this is the making of a new “Standards war”? Both Google and Khronos are adamant that they are not competing. However I believe that ultimately only one project will come out as a standard. As the complexity of 3D web applications increases it is not feasible to write code for both “APIs”. The only question for me at this point is who will come out on top.

To answer this I would look at the audience of the two projects. OpenGL has been out in the wild for a long while and many developers of videogames, or general graphic application, are already familiar with the APIs and the way it works, therefore it would probably make sense for them to embrace WebGL.
Nonetheless O3D still stands a chance. For a very simple reason. It’s the web we’re talking about.

Frankly I can’t see myself playing a big videogame like fallout in a browser window anytime soon. These APIs will be used to enrich web application. Some examples are already coming out using O3D. Have a look at this Home Designer. Can’t you already see IKEA using it.
My point here is that we’re not likely to see game developers switch to the web. We’re much more likely to see web developers start working on games or application involving 3d graphics, and this is where Google wins.

O3D extends application JavaScript code with an API for 3D graphics. It uses standard JavaScript event processing and callback methods.

As a web developer I can keep writing JavaScript code as I’m used to without having to change the way I think to how a game developer does.

What do you think?

Bookmark It