Humans to the rescue

Images like the one above are familiar to any of us who have ever used webmail, Ticketmaster, or any other web service that wants to prevent automated spammers and scalpers from exploiting their systems. The distorted, fuzzy letters don't provide a challenge to humans, but are indecipherable to the most sophisticated computer algorithms. Our genius is facilitated by our "invariant" perceptual abilities; that is, we can recognize objects, faces, and letters independent of rotation, translation, and scale.

However, these CAPTCHAs (Completely Automated Public Turing Test to Tell Computers and Humans Apart) were designed with a wonderfully clever ulterior motive. In addition to preventing rogue bots from devastating our virtual lives, CAPTCHAs like the one above are actually exploiting you, the human, and the invariance of your human perception, to help digitize the world. The words presented in these CAPTCHAs are pulled from the book-scanning project of the Internet Archive, which aims to scan millions of public-domain books and put them online for free. One word of the CAPTCHA is known to the computer, and is used to verify your humanness, while the other was indecipherable to the Archive's scanners. When you type in that word, you're actually translating the image into text for the Archive.

There's a fantastic article in Wired Magazine about this type of "human computation," "the art of using massive groups of networked human minds to solve problems that computers cannot." The article profiles the work of Luis von Ahn, who designs clever ways to harness the powerful brains of bored web surfers to solve computing problems (e.g. judging random pictures as "pretty," tagging images and audioclips, etc.)

From Wired Magazine:

If people could so easily recognize pictures of letters and numbers, could [they] use this ability to identify and label the vast number of images on the Web?

...

The way to do it, he realized, was as a game. It would pull images off the Web, then randomly pair two players from around the world. They would be shown the same images, then each would type in as many words as they could to describe those images, hoping to hit upon the same ones as their anonymous partner. They'd get 50 points for each match, and two and a half minutes to earn as many points as possible. Von Ahn suspected that whenever the players agreed on a word — "meadow" to describe a tree-lined clearing, for example — they would be choosing a highly accurate label for the picture.

Von Ahn cobbled the game together in a week — "crappy, totally terrible code," he admits — and threw it online. He dubbed it The ESP Game and emailed the URL to a few friends. Within days it was Slashdotted, whereupon his server nearly crashed under the load of new players. Astonished, von Ahn watched for the next four months as 13,000 players produced 1.3 million labels for some 300,000 images — with a few hardcore fans clocking more than 50 hours of play. "It's like crack," as one player complained in an email to von Ahn. The labels his players generated were far more accurate than what other image-search technologies produced. Most search engines are limited to sniffing out words associated with a picture, such as the name given to the image, words in the page around it, or links pointing to it. That's inherently imprecise: When von Ahn recently searched for "dog" on Google, a third of the pictures showed no dogs at all. When he queried the ESP database, almost all the results contained canines. Better yet, players often generated labels that were subtle and nuanced. A search for "funny" found a picture of Ronald McDonald being hauled away by police and one of Queen Elizabeth picking her nose.

Even the DHS wants to employ your brainpower as you procrastinate on the web:

This spring, von Ahn got a call from the Department of Homeland Security. He went to Washington to meet with DHS officials, and together they devised a game in which people are challenged to find dangerous objects in images of x-rayed baggage. The pictures would be fed from airport scanners, and players would act as a second set of eyes for overtaxed security employees. If enough players noticed something amiss, an alert would be triggered.

Von Ahn's other games that capitalize on human superiority (supposedly available at Games with a Purpose "in July," but as of now the site isn't running yet) include:

1) Matchin' Players are shown the same pair of images, then each tries to pick the one they'll both agree is more attractive. Creates a database of images searchable by aesthetic value, a task no algorithm can perform.

2) Babble Two English-speaking players are shown a sentence in a foreign language that neither of them speak. A list of possible English meanings appears below each word. Players try to agree upon a set of English words that forms the most coherent sentence. Translates foreign text into English without requiring anyone fluent in both languages.

3) InTune Players listen to the same audioclip and then try to come up with the same phrase to characterize it. Tags sounds with searchable descriptive text.

4) Squigl Two players are shown the same picture and a word describing an element within the image (e.g., a picture of a dog and the word "leash"). They each draw a border around the element. Produces a set of pictures with their internal components tagged — terrific for very specific image searches.

5) Verbosity One player is given a word, and the other tries to guess that word by completing phrases such as "It is near a ____" or "It is a type of ____." The first player answers "true" or "false" but can't use the word itself. Creates a database of commonsense knowledge describing the objects.

Link to the full Wired article.