On Websites & APIs

A few weeks back, I attended Startup Weekend in Israel. Startup Weekend is a gathering of people of all sorts – coders, designers, marketers and the like – that join forces for one intensive weekend to create something out of nothing. While most groups spent most of their time discussing business plans and polishing presentations (which was a disappointment for some of the more talented developers in the bunch), our team spent almost all of our time developing a new internet service.

What’s In A Service?

Developing a new service in 48 hours is not an simple task, especially for a group of ten people who have only just met and have widely varying skill sets. We wanted to take one commercial area, which we felt was badly served by existing sites, and revamping it. Creating, in two days, the open source seed that could later be used to take over the category. In Israel, the worst served segment is of classified ads, so that is what we were aiming for.

Here’s a list of what we had in mind:

  • Insert a new item (for example, a car or a cellphone)
  • Different displays and properties for each kind of classified ad (cars are different than cellphones)
  • Filter many items (according to properties of the item)
  • Search
  • User registration and login (via existing services, such as Facebook Connect)

To make things more difficult, we had ambitious goals about creating the front end of the service as well:

  • Develop not only a website, but also mobile applications (mainly, iPhone and Android).
  • Translate to several languages.
  • Test several, completely different,  design concepts and user interfaces. We really wanted to make a radically better website. Aside from the graphic design, we had several ideas about comments and Facebook integration that were pretty cool. We also had ideas about mixing UI concepts from price comparison sites as well as classified ads sites.
  • Use real ads, taken from competing sites (which, in Israel, is probably legal since classified ads are considered too utilitarian to be protected by copyright law).

The Open Website API

Building several clients to the service required extreme separation of responsibilities. In most existing frameworks (it’s slightly better in Ruby on Rails than in Django), are built on three layers: the data model, the controller or view and the template.

The data model defines what data objects are used in the system. We’ve built a simple generic model for a classified ad, and a set of meta-models that define the properties expected for each type of item.

The controller or view is responsible for fetching the data required for this action. For example, in order to create a new ad, I need the list of expected properties for this type of item (a car). The controller is responsible of all the heavy lifting (such as fetching data, validating input correctness, etc.). We created the controllers for our main actions, quite simply.

The template defines the way the page layout. It is responsible for rendering the way the website looks like (the HTML). This is what we found most cumbersome. First, each template is tied to a URL. This meant that we couldn’t have two HTML clients without splitting a lot of code. Then, the mobile client needed an API, which required a third branch of controllers and templates to render the objects for the iPhone.

This just wouldn’t do. Instead, we decided to create the API once, and use it for everything else. That means that on the server side we only had to develop the API, and not even a single HTML page. Then, we could have written as many clients as we wanted, each with a completely different flow, look and feel. The web clients were just html with some ajax, that didn’t have to be on the server. The iPhone app was just as simple to develop. No code was duplicated. I was surprised at how fast we have iterated ideas.

What’s it good for:

  • Complete API – by default, there’s an API for everything. There isn’t anything on the website that can’t be easily implemented on any other client. It makes the website very open for developers, without requiring any special treatment.
  • Complete separation of functionality from design – the server side was responsible for authentication, data validation and simple access paths to the data. The client is responsible for the flow, and user interface. Each client can be completely different, tailored for the device it is used on (think: web vs. mobile). There were hardly any limitations on what a client could do, because the API was so basic.
  • DRY code – write once, use everywhere
  • Third party friendly – having an API is important for another reason. Think of the Twitter API and how it helped create the Twitter ecosystem (something they are fighting today). For an open source website, this is an only an advantage. You want as many people plugging in and creating something new on top of it. Over time, the best ideas will merge, and community will benefit from the competition, while not wasting resources duplicating the data layer.

Cons:

  • Slower loading times – pre-rendered HTML will always be faster to load. Do people care that gmail takes a few seconds to load, every time you open it? Not really, because it’s so useful. And with smart client side caching and some clever ajax pre-loading, you can cut this time down significantly.
  • Command & Control issues – does the project include a client? If so, which one? How do you chose which client is “official”, or best? If not, does it mean you need two packages to install the website? How do you manage a list of clients? Where’s the data? Is it free and portable?
  • Security and tampering – when you have a very open API, you are vulnerable. There’s a fine line between being open, and being so open that you endanger the data integrity.

Rapid Development, Distributed Development

There are two special cases where this method can show itself to be especially useful.

The Lean Startup:

A startup in its most early stages is an organization trying to build an unknown solution for an unknown problem. It is a team of people, trying to find both a business problem and a solution to such a problem. This is called the product/market fit. The lean startup mentality dictates that the early stages of the startup should focus on learning.

Having an open website API is a great way to learn. First, it’s a great way for A/B testing and iteration of ideas. Second, there’s a real chance for serendipity. Your users will create clients for themselves, thus telling you what they need, and why they love your service.

The Open Source Website:

We’ve all heard about open source code projects, but an open source website is a much rarer creature. I’ve talked a bit about the reasons why it’s so difficult last year. The open website API liberates the project in several ways. The biggest pain point of the open source API is the data. A website without data is useless. By having the data in one central place, there’s great opportunity for innovation on the client side, having several open source clients developed with ease. In any other way, you wouldn’t be able to fork the website’s look and feel without copying all the data as well (think: Wikipedia).

I’d love to hear what you think. What other pros and cons are there? Would you want to see more open source websites?

Don’t Worry, Be Happy

I Worry, Therefor I Am

Our brain is a simulator for experience. Try to imagine the taste of liver and onion ice-cream. I can guess the face of disgust you’ve just made. Have you ever tried any? You haven’t. Yet you know it is a terrible idea. You can simulate the taste in your mind.

We don’t drive the simulator*. It just buzzes in our heads. In many situations, it breaks down. So we skip ahead to the first thing that comes to mind. And we convince ourselves we were so clever.

*In Soviet Russia, the simulator drives you.

The BigCo Fallacy

After a few years at BigCo, I’ve managed to earn the title of a free electron. As a free agent, untied to any particular project, I was allowed to choose what I want spend my time on. That’s a sweet job if ever you can get any.

Anyway, convinced I can single handedly code BigCo into greatness, I’ve decided to build a framework for data visualization. Clients were really hot after this type of feature, yet almost none of our products had any visualization. It just wasn’t on the road-map for any of the upcoming releases. If no one has time to develop this, I thought, I should do it. For four months I coded at my desk, testing and documenting a generic framework for a feature nobody asked for. In order to increase chances of adoption, I focused on ease of integration. If it’ll be easy enough to use, wouldn’t they just add it to the next release as a bonus? And it really was easy. You could literally integrate with it and get it working in ten minutes or so.

So, I started showing it to people. A lot of them were very excited. One project manager listed visualization as a feature, until the project missed a milestone and the feature was cut out. With a couple product managers I’ve built demos for use cases, but couldn’t get them to document their excitement as features. This happened a few times and, finally, I abandoned the framework.

A few months ago, I got a phone call from someone in BigCo. Two years after I wrote it, my little framework finally made it to production.

The moral of the story is that we can’t predict the future. We waste a lot of mental capacity worrying about the future, but usually we’re really bad at predicting it.

The Pursuit Of Happiness

There is nothing stopping us from being happy, if we choose to be. Happiness is not a game of outcomes. It’s a game of process. Apparently, there are two types of happiness:

  • Natural Happiness, when we get what we wanted and
  • Synthetic Happiness, when we don’t.

Our society tries to convince us Natural Happiness is more real, but in fact research shows Synthetic Happiness is just as real and enduring. This short talk by Dan Gilbert, changed my perspective about life. It caused me to completely change how I look at decisions.

The Influence Of Decision Making

A few tips for predicting the future and staying sane:

  • You are overestimating – in many cases, the difference between outcomes of a decision are smaller than you imagine. Your mind is geared for life-and-death situations. Run-or-fight. Modern life, not so much.
  • Limit choices – the more options you have, the more likely it is that you’ll stall the decision. Having more options doesn’t make you happy. It makes you frustrated. Since outcomes are not critical for your (psychological) well being, limit choices as often and as soon as you can.
  • Decisions are temporary (but action speaks volume) – once you have made up your mind, don’t second guess. Act. When you wait, you learn nothing. When you act, you learn plenty. Correct your course when new information comes in.
  • Focus on what doesn’t change – the long run is what matters. 37 signals said it best: focus on what doesn’t change. You may be unhappy with your job in half a year, but at no point will you ever be sorry for not having cancer. Focus on the constants (work out, eat healthier, earn enough but don’t fret about optimizations).
  • Don’t worry, be happy

Bonus Track

Bobby McFerrin, singer of “Don’t Worry, Be Happy”, demonstrates the power of expectation.

How To Read Code

Scientifically Tested Code Reading Skills

I’ve run across a paper [pdf] researching how we read code. The article describes an eye tracking device that can identify what people were focusing on when they were given a piece of code. Each subject was given six snippets of code, each with a built-in error, and was asked to analyze the code and find the error. Here’s a sample from the article:

On the left, a small snippet of code that should sum numbers for 1 to a given maximum. On the right, the eye movement of the subject as he reads the code. Notice how the eye first scans the function headers, then briefly scans the function’s body and lastly focuses on where the problem is most likely to appear in real code (the loop).

Reading Code is Like Reading the Talmud

This reminded me of a very old post I’ve read at Joel Spolsky‘s blog. Joel quotes Seth Gordon which says:

The following Talmud-reading tactics are, I think, also useful for code-reading:

  1. Work in pairs, thinking out loud to one another.
  2. Argue. If your partner says “this means X”, and you either don’t understand why or you have another opinion, demand an explanation.
  3. Sometimes, when dealing with a chunk of text, it’s easier to figure out the middle *after* you understand what’s on both ends. Therefore, if a fragment of text has you stumped, try skipping over it and seeing if you can come back to it later. (But you still have to come back to it eventually.)
  4. Read the text both “inside” and “outside”. An inside reading translates the text into English (or whatever your native language is) phrase-by-phrase; an outside reading translates a larger chunk into an idiomatic paragraph. If you only read inside, you can miss the forest for the trees; if you only read outside, you can fool yourself by making broad guesses and not verifying them with details.

Seth describes two types of code. Some code is easier to read and understand first, and is the basis to the rest of the program (Type I). The second type of code is dependent of other code, and so is easier to understand in context (Type II). This fits with what I look for when I need to learn a new piece of code:

  1. Class and interface dependency: look for the base classes, which depend on no other class. These, usually, are small and simple objects, with the purpose of holding a little bit of state. They may do some calculations, but are mostly built around an internal storage and a bunch of get-set functions. These are classes of Type I. Skim through them now.
    After you are familiar with the basic building blocks, you can look for the manager classes. Most of the time there will be  a bunch of very big, hairy classes that are the core engine of the software. These are classes of Type II. Make sure you understand, from the function names, what each of these managers is supposed to do. After that, you can delve in. Manager objects usually only make sense when they interact with each other, so you have to have an idea about how all of them are designed before going into any of them too deeply. (Tip: The class called Manager is not always the manager. The class called from the main() usually is).
  2. Assume a lot: Assume that function and variable names matter. If you would have written the code, what would the GetNextIndex do? Assume it does exactly that. If a function has a name that sounds obvious enough, skip it. You’ll get back to it if you ever need to use it. Why waste your time on something you don’t need right now? There’s only this much detail you can keep in mind at this point. If you dig too deep, you’ll start forgetting things you’ve already read. Focus on purpose and structure, not on implementation.
  3. Learn by debugging: Unit tests matter, but real code matters more. If you really want to understand how the code works, the fastest way is to use it. Fire up your IDE and write a quick and dirty demo of what you wanted to use the code for. Chances are your code is bust. I’ll be surprised if you can get it to work on the first go. That’s OK. Now, start debugging. You’ll learn the inner workings of the code (and its bugs) much faster, and you’ll only focus on the parts that matter to your task.

Writing Readable Code Matters

  • Function names matter: we’ve known this for a long time, but now we know that we scan function names first. A good name, and you don’t need to focus on the implementation all that much.
  • Variable names matter: again, nothing new. Notice, though, how the eye goes back to the variable definition and assignments. We constantly look for reminders of the state of the variable.
  • Loops are the source of all evil: well, that’s not entirely true, but they are worse than all other statements. Notice how figuring out the loop is the most time consuming bit of the code. It’s where most pitfalls are (Oh, the index should be smaller OR EQUAL. Right. I forgot). Commenting can help some, but just keeping the conditions simple and readable is key. Break complexity down around the loop.
  • Short code is easier to process, so get your refactoring tools out, everybody, it’s time to split these functions.

My favorite list of tips for readable code is uncle Bob’s guide to Clean Code: A Handbook of Agile Software Craftsmanship. Even if on some occasions it gets tedious and overzealous, it is still a great read for the practical coder, dealing with many issues of writing readable code and refactoring.

Scientific Side-note

The research certainly doesn’t prove much about the way we read code, as it was only tested on 5 (that’s right – five) people. To me it seems like more could have been done.

I mean, going through the lengths of creating a really awesome eye tracking device, carefully calibrated so the machine can tell which line of code is being inspected at any moment, and then test it only on 5 people? Is getting more people into the test room so much hard work? I’d love to be a test subject and learn about my code reading habits. Next time, pick me.

The Woes Of Internet Identity

There’s no identity on the Internet

You can be an anonymous comment troll, claiming every single video on YouTube is the worst piece of film you have ever-ever seen (even worse than Batman & Robin!). How can anyone tell you are posting the same comment on every video? Or that you are, in fact, both sideshowkid1982 and hornyteen19?

Even worse, you can impersonate another person. You can buy a domain cleverly named after Steve Jobs, Bill Gates, Bill Clinton or your next door neighbor, where you can cause all sorts of grief. Anything from stealing some searches and making a few bucks on ads, all the way to impersonating and dominating their public image (easier when your target isn’t famous). Next, open a fake Facebook profile, maybe throw a couple of nasty comments in this or that site posing as them, and presto, you have a PR disaster waiting to happen.

For those who don’t know us, our reputation is what comes up in Google when they type in our name.

Claiming Assets

Since there’s no escrow service for identity, there’s a problem claiming assets on the internet. I can’t prove I really am me. There are several people with my name walking on planet earth. Are they mad at me for stealing their identity? For being first on Google?

It’s one of those hard problems, that no one seems to know how to handle. Technorati, the blog index, wants me to publish a post with a unique gibberish key they provide, such as this one:

H4MAMUUKAUD3

Does this prove I am the writer of this blog? Somewhat. Does this prove I am who I claim to be? Hardly.

In Cyberspace No One Can Hear You Scream

OpenId and other Authentication Services are a short step in the direction of creating a single sign-on to the Internet. It’s a way of connecting your Google Profile and your StackOverflow account. Supposedly, it leaves password security to a relatively small and trusted group of identity providers, and I have to memorize fewer username-password combinations.

I have several problems with this solution:

  1. There’s a real threat to OpenId. A clever hacker finds a way to reset the password on your OpenId provider, and your done. Dead to the Internet. All you street cred is gone (and worse – is your credit card number saved on any of these sites? You may lose real money, too). For a solution to be viable it has to be extremely secure. I’m talking fingerprint-DNA-retina-scanner secure. Something your country will be proud to use instead of its current identification system.
  2. Each of the big Internet companies is an identity provider of its own.You have to remember which identity you have used for any particular site. Did I use my Facebook account or my Google account for Posterous? Can’t remember. A solution will require that these profiles be merged or linked together.
  3. It’s opt-in. Anyone can comment on this blog claiming that they “love this post, it is great, post more on this subject, buy generic Viagra”. Thank you. I value your input. I want to find out where you live and send you flowers. But I can’t. Because you don’t have to tell me who you are, and I have no way of requiring you to, even if I wanted to.
  4. There’s no claiming of assets. When I buy a house, I register it under my name. When I buy a car, I register it under my name. When I buy a website, I type in a bunch of possibly-fake details on some obscure site. This means that I have to prove again and again that this blog is mine. I’ve had to repeat the process of insert-this-fugly-random-key-somewhere-in-a-blog-post-so-all-your-readers-will-get-an-rss-update-of-a-fugly-random-generated-key-and-hate-you about seven time for different services (mostly analytics and spam related). It is tiresome. And fugly. Please stop.

What About My Privacy

Convenience comes with a cost. Having my biological markers in some database somewhere can be disastrous if the database is broken into. Someone can steal my DNA. They can then pretend to me and frame me for a crime I didn’t commit. Or patent my DNA and sue me for infringement. Or any other idea taken from a really bad 80’s sci-fi movie.

Seriously, though, I’ll be screwed.

Until we really understand what identity is, and how to protect it, memorizing a bunch of passwords isn’t that bad. And you will just have to get these randomly generated keys every once in a while. Don’t blame me. Blame society.

Virtual Reality, Not What We Have Imagined

Computer interfaces have stayed the same for very long. Virtual, remote, locked in a glass screen and interacted with via a mouse cursor. This kept the 3D world separate from the digitized world. The dream of fully immersive virtual worlds is still years away.

Computers are changing. The new mobile devices, such as the iPhone, are portable computers fully aware of their location in the physical world. They can act as a portal into a virtual layer of information on top of the real one. A virtual, digital space that we can all tap into, pull and push information from and to it.

The revolution starts with applications on the mobile phone. Applications such as Layar. Layar has a whole slew of geo-located layers of information, that will hover in front of you whenever you need them. Here’s a sample of a real-estate information layer, used to find a house to rent:

Not Good Enough

This is all fun and games, but not really useful. I don’t want to have to walk with my eyes on my device at all time. I want to act naturally in the real world, and I want the information available to me at all times.

I want a pair of goggles with built-in LCD. I want a glove with a controller in it. I want it to know where I’m looking at. In his very compelling book “Daemon“, Daniel Suarez* envisioned a D-space, a digital virtual layer of information to which everybody is connected. It is the ultimate blend of reality and virtual.

The technology is still far for perfect, but it is on it’s way. Minority Report style controlling gloves just turned into a reality, thanks to some very clever fellows at Oblong. These are the guys who designed the original Minority Report gloves for the movie, and in the past few years have developed the system for real. They claim it will be commercially available for every computer within five years.

Last Piece In The Puzzle

The only missing piece right now are the goggles. Creating a display system that is both lightweight and can accurately tell where your head is directed is still far away. Devices are available, but they seem cumbersome. They require too much power, too much equipment. They’re too expensive, too big, too obtrusive.

I mean, if you were to wear these goggles all the time, you would be more in the virtual world than in the real.

Wearing them you’d be stuck in the virtual world. There is no way to anyone talking to you would feel comfortable.

“Is he listening to me, or is he updating his Facebook status right now?”

No. It has to be much more subtle, much less intrusive. I want to be able to look the other person in the eye, and have him look back at me. What I want will look exactly like seeing glasses or even contacts. Most of the time, the display will be minimal. When going to the grocery store, I can review my shopping list. When buying a book, I can read reviews. Most of the time, I need additional display, not alternative reality. It would be nice if the display was good enough to read the news on the bus. Most of the time, however, it is the real world I want to dominate. I’d just like to have the Internet available at my fingertips to put an extra edge on reality.

OK. Daydream is over. Back to work. Where did I put my phone, anyway? Can somebody give me a call so I can hear where it’s ringing?

* Read it. No, seriously. Read it.