One More Time, From The Top

Last time I’ve demonstrated a way to create mosaics from images. I wanted to do a little bit of image processing using javascript and HTML5’s new canvas abilities. As I mentioned, this was only the first of an idea I had about turning social experience into a beautiful, expressive and personal graphic.

Mosaics, Revisited

Canvas is an extremely powerful concept, allowing web developers to create more expressive web pages. The web will soon be filled with rich user experience and we will be dancing like it’s 1995. This has been a long haul, but we are finally WINNING. I just hope we have better designers on the web than we did on the desktop. I never (ever!) want to see the Word 6.0 toolbar sprawl on a web page.

My favorite web sites focus on content. They try to let everything else blend into the background. On the best sites content is king. Google, YouTube and Facebook all use a clean, white background, but there are other examples for beautiful design. TheSixtyOne is both artfully designed, while the functionality just melts away into an incredible user experience.

So, I took the mosaic idea one step further, creating a (very simple) tiling jQuery plugin that allows to recreate a large image from small images. Here’s a demo.

Original Image

Canvas (mouse over to see the small images)

The code is available here. It is by no means complete. You will notice I cheat terribly, as I overlay the miniature images on the colors of the background photo. I’m doing this for two reasons: first, getting a good color metric working for images of significant size isn’t as easy as I hoped it would be (and performance is atrocious), and second, I expect that the selection of photos I will have to use in a real life scenario to be too small for a good match to be found for all the pixels in a photo. All in all, this looks good enough, and was much simpler to code.

During my work, I’ve discovered a few issues I’ll go into the next section. If you have advice, I’d love to hear it. I’d much prefer not going the server-client route and keep this a client-only project.

Performance, To Heck With Thou

Javascript is slow. I didn’t expect it to be so. I knew it was considered relatively slow, however there has been a lot of improvement in javascript performance in the browsers over the past years. It is still not enough.

I don’t know if you noticed, but while creating the mosaic above, your browser gobbled up a large chunk of memory. My laptop’s cooling fan whirred loudly as I waited for the picture to be created. The experience sucks. For serious number crunching and data manipulation, I suppose we still need to fall back to good old server side coding.
Or maybe I write bad code. Care to take a look and throw me a line?

Not that it matters. As much as I could hope to fix the performance issues, there are still a limitation I can see no way around.

Hacker, To Hack With Thou

For some silly reason, security limitations on web developers are much stricter than they are for any other type of development. This could be because client-server architectures are by nature more vulnerable, or maybe it’s because it is so darn easy to steal data using a rouge <script> tag. The architects of the internet over at EMCA, that set up the specs for HTML5 and JS, decided to put precautions against every obscure scenario of possible security risk. Thank you guys. I love you, but you are bringing me down.

While for security as a concept, in this particular case it has been quite limiting. The only way to analyze images in JS is by turning them into giant bitmaps using the canvas. Basically, I draw the image on the canvas, then I read it out as an array of RGB tuples. Then I analyze the photo and decide where to place it against the larger image. The problem arises when I try and analyze images that are not from my domain (or my JS domain).

In many sites we use resources from other sites. We just embed them using an img tag, and never think about it. Apparently, in order to prevent a hacker from taking a snapshot of your password on a canvas and sending the canvas somewhere else, a canvas can only be read from programmatically if it is safe, where safe is defined as: only been manipulated by objects from the same domain.

Short version: this technique could never work with photos that are not served from the same domain, which makes it impossible for me to use them in any social networking context. It may still be possible to use this technique in a browser extension, but I don’t think I want to go down that path. I want a write-once-run-anywhere experience, if possible.

This may still be a learning experience, though. Can anyone recommend a good image processing library for Ruby or Scala? If not, I’ll probably default back to Python. Or maybe I’ll do the extension after all.