December 2009

Happy Holidays and all the best for 2010. It's going to be an exciting year!!

The deluge of data and analysis on Twitter is continuing to roll. By the time the 10,000 conference and peer-reviewed papers get published in the next 6-8 months, they'll all have been scooped by the folks who are doing public analysis for other audiences.

First, there's a paper by Mor Naaman, Jeffrey Boase, Chih-Hui Lai called "Is it Really About Me? Message Content in Social
Awareness Streams" (PDF). Among a variety of interesting and nuanced findings, the authors show evidence that about 80% of Twitter users are "meformers" – people whose Tweets are mostly about themselves. Only 20% were in the "informer" category – people who share information about other topics.

Surprisingly, TechCrunch has lately been the source of seemingly high quality data about Twitter. In addition to Geoff Cook's great guest post on Why Teens Don't Tweet, back in October Robert Moore posted a huge amount of data and longitudinal analysis.

(Click for a Larger Image.)

And, just today, TechCrunch has news from Comscore that Twitter's growth has basically flattened out for both international and US users, this despite a recent push of new features and new languages. In September I predicted the demise of Twitter, and this seems to be the first stage. 2009 was definitely their year – arguably no technology was more popular, more widely talked about than Twitter was this year. 2010 will be the year of soul searching for Twitter, where the new-ness wears off, new features don't gain the expected traction, and the company continues to look for a reliable business model. If Twitter has a big future, it's going to be as a messaging platform that underlies more interesting services.

This is truly one of the strangest things I've ever seen. And I watch a lot of football! Looks like the snap took the O-line by surprise…

Well, yesterday was the day, and in less than 9 hours DARPA crowned a winner: The MIT Red Balloon Challenge Team. I spent a good part of the day stuck to Twitter watching the contest develop and trying to read between the lines (mostly unsuccessfully). So after all the build up and a day of searching, what have we learned from the DARPA Network Challenge?

  1. DARPA Made it Easy Take a look at the map of the balloon locations. Notice anything? Huge swathes of the country with no balloons? Yup.

    (Click for a larger image.)

    Remember when I flippantly dismissed the Twitter-only strategy by pointing to the NY Times infographic showing how few Tweets there are in so much of the country? Well, apparently DARPA knew that too, and decided to make the challenge very, very easy. The balloons were in major parks in major cities, almost entirely on the Atlantic and Pacific seaboards. I mean, for peets sake they put a balloon in Union Square, San Francisco!

  2. How did MIT win? Well, first let me tell you how they did not win. MIT's victory had absolutely nothing to do with the ridiculous reverse pyramid scheme they were using to hand out money to tipsters. I am willing to stake a fortune on that one. Here's the graphic they use to try and explain it:

    MIT's Reverse Pyramid Scheme
    (Click for a larger image.)

    I'm hoping the folks over at MIT aren't patting themselves on the back for their brilliant use of monetary incentives.

    So how did MIT win? Well, you can find the answer to that question on Slashdot, Techcrunch, CNet News, the Washington Post, and… Get the picture? Shocking. Well known educational institution gets huge amount of publicity, attracts more tipsters, wins scavenger hunt.

  3. Where was the Innovation? Unclear. I didn't see any interesting innovation in incentives. I didn't see many creative uses of technology. Most teams set up websites with simple web forms for submitting tips. Army of Eyes had an iPhone app. for that same purpose, and they did quite well I think. But it still boiled down to a simple form for submitting tips. I mean, the FBI has been doing the same thing with a phone number and a few operators for years. This is a nice reminder that, although the internet and social media provides some fascinating new contexts for interaction, what we do with it in social interaction, organizing is fundamentally the same as before.

    MIT's team says they were using some algorithm to verify balloon tips. I'm not sure what that means. I know that they were keeping track of all DARPA-related posts during the challenge. They may have been looking for posts such as these, which I found earlier on in the day…

    DARPA Tweet 2
    (Click for a larger image.)

    Of course, none of those actually referred to a balloon in the challenge (as best as I can tell), but looking for those types of messages was more the sort of strategy I was expecting.

  4. Fin I can't help but feel that the whole thing was a bit of a disappointment. The way DARPA decided to place the balloons meant that teams could win without any secret sauce. I was excited to see what people would do to find the balloons on a random stretch of I80 in Nebraska or whatever.

    I'm hoping that this is all part of DARPA's strategy, and the next thing they'll do give people a chance to really organize, make the challenge really hard. I think they've got the right idea that there's innovation to be done in this space. But Twitter is such a blip on the radar. To really tackle nationwide emergencies, and to effectively harness the power of networked media and the internet we need to learn to integrate new technologies with old organizational tools. We need to look at lessons from MoveOn and Obama – people who arguably did networked organizing, combined the power of new and old media better than anyone ever has – and we should see how we can use those lessons for a more directed goal.

    I'd love to work on that problem. DARPA – bring it!

DARPA posted a poll on their Facebook page. Here are the results as of about 10:30AM PDT on Dec. 4th:


Note sure how many people have voted. My vote was 'Never.' Although it occurs to me now that I don't quite know their definition of solved. If they mean when will someone find at least 5 balloons and get the prize, well, that should happen in less than 24 hours. Maybe even less than 6. But I'm betting no one will find all 10.

The Wall Street Journal recently made news by publishing some results by Spanish researcher Felipe Ortega. Ortega crunches some numbers and finds a decline in the number of Wikipedia editors. The folks at Wikimedia decided to hit back (subtly) a few days later, basically arguing that Ortega is counting the wrong thing.


One of the issues the WSJ piece brings up is the question of whether the decline is the result of a lack of new material. In other words, some people argue that, with over 3 million articles in the English Wikipedia alone, it's hard to find new stuff to write about. Is this the case?

Guardian columnist Mark Graham thinks not. In a recent column, Wikipedia's Known Unknowns, he takes a look at geographical distribution of Wikipedia articles that have geotags:


Graham sees this as evidence that there's plenty left to write about. But I think he's missed the point. It's true, Wikipedia has not yet covered the entire domain of human knowledge. There are many places on the globe that aren't well documented yet. But that's exactly the point. The people who live in those places aren't well represented on Wikipedia (yet). And the people in heavy Wikipedia-using countries don't often go to those places.

Graham's map essentially shows that this is, in fact, a big challenge for Wikipedia. With 3 million articles, Wikipedia has largely covered the easy stuff. General knowledge and popular culture are comparatively well represented, and so is geographical knowledge in the parts of the world where Wikipedia is very popular. So the barrier is now much higher for someone who comes to Wikipedia looking for something to write about. Increasingly, that person needs to have some kind of relatively specialized knowledge, to have been somewhere relatively unique, and then has to feel able and willing to share that knowledge. Well, that's a high barrier to entry for a lot of casual users, and I think it's at least a part of the reason why Wikipedia's editor numbers have plateaued.

So, actually, the question isn't whether Wikipedia is running out of new material. It's not. The question is: who knows (and will write about) the material that isn't on Wikipedia yet?

Well, the day of reckoning is almost upon us – Dec 5th, DARPA Network Challenge Day! I've previously blogged about this here, and make sure to take a look at the comments because there's some good stuff there.

Anyway, I was looking again at the rules recently, and noticed some things I didn't notice before. The prize is to the 1st team that submits the most correct balloon locations. The balloons will only be aloft during the daylight hours of Dec 5th, unless there are weather difficulties, in which case the balloons will go up on the 6th or later. It's unclear whether they'll delay all the balloons, or whether it means that depending on the weather there could be balloons up on different days in different parts of the country (more bad news for teams that are thinking of driving around looking!!!). But teams have until Dec. 14th to submit winning entries. And they've revised the accuracy of the location to be within 1 mile (huge!). To me, this suggests that DARPA thinks the challenge will be won algorithmically. I think they might be right.

On Dec. 5th, I'm guessing we'll hear about the locations of 3-6 of the balloons. There may be a team here or there that has some private information, but I suspect most of the locations will be known by all teams who are paying attention. (Of course, only one of them will be first to submit…) But then the real fun begins. Starting at nightfall, the balloons are gone, but we can still find them. How? Well, I don't know. How much media is uploaded to the cloud each day? How many pictures, videos, that might show a balloon in the background, even if the photographer never noticed it? Of course, that media would have to be geotagged. It's possible to infer the location of balloons after the fact with some accuracy, especially now that we only need to be right within 1 mile. After all, teams will have more than a week to look, and then even to go out to these locations and pace off distances if they choose.

So, what do you think? Is this how the challenge will be solved?

Altruistic by Nature
(Click for a larger view.)

An interesting NY Times article got me thinking about one of the big questions that anyone who studies cooperation eventually has to deal with: is pro-social behavior innate? Is the natural inclination of human beings to cooperate, to be altruistic, to consider other people's outcomes and act accordingly? Or are we wired to be selfish, and to look out for our own interests at the expense of others?

Well, obviously it's a complex question. One of the things that makes it so interesting is the presence of drastically opposing views. Much of economics is driven by the idea that human beings are innately rational, self-interested, maximizing people. Economic models traditionally assume that we all look out for number one, and we do it as often and to the highest degree that we can. From one evolutionary view this makes sense. Survival requires that we look out for our own welfare. Only the strong survive, right?

Well, not really. As soon as human beings became social creatures, self-interest may have given way to cooperation in our genetic code. By ourselves self-interest can lead to survival. As soon as social groupings became advantageous, though, cooperative behavior may have been the best strategy. The aforementioned NY Times article, We May Be Born With the Urge to Help reviews some research showing evidence of helping behavior even in very small children. This is important because if we see evidence of this in kids so young that they haven't had a chance to learn or experience social norms of helping or to develop situational awareness, then that suggests that we're hard-wired to be altruistic. Right?

Well, like any absolute assertion in a complex situation, in this case I think the answer is inevitably yes and no, right and wrong, both and neither. As tempting as it is to make blanket claims about "the way we are," in any social phenomenon we have to consider the power of situations. I believe there are probably genetic bases for both types of behaviors: self-interest and pro-sociality. But it's most likely the situation, and patterns of situations over time, that elicit one type of behavior or the other in any given case. An idea gaining credence in many circles is epigenetics. The idea is basically that exposure to our environments can inhibit or suppress the expression of certain genes. So, whereas our genetic code was once thought to be static, epigenetics suggests it can adapt, and in doing so sort of smash nature and nurture into one. The upshot of epigenetics when it comes to the self-interested/pro-social question is pretty ground-shaking too. What if we're hardwired for both self-interest and pro-sociality when we're born, but as we develop it's not just social norms that influence our behavior, but also the environmentally-influenced expression of certain genetic traits? How's that for complicating things?

Obviously we're just scratching the surface here. And full disclosure, I'm no expert on the biological stuff. For a review of the research on altruism (which, it has to be noted, is only one type of pro-social behavior), I recommend:

Piliavin, Jane Allyn, and Hong-Wen Charng. 1990. “Altruism: A Review of Recent Theory and Research.” Annual Review of Sociology 16:27-65.

What's great about this review is that it debunks the all-too-common assertion that what looks like altruism is actually self-interest in disguise. Economists who looked at the world and saw how common pro-social behavior was had to come up with some explanation, after all. Piliavin and colleagues synthesize the literature masterfully from a social psychology point of view.

Recent Nobel prize winner Elinor Ostrom does something similar from a primarily economic point of view:

Ostrom, Elinor. 1998. “A Behavioral Approach to the Rational Choice Theory of Collective Action: Presidential Address, American Political Science Association, 1997.” The American Political Science Review 92:1-22.