Yahoo!’s new reputation design patterns got me thinking - what makes a reputation? When I browsed through the 9 design patterns lumped under the title of ‘reputation’, my first thought was that these are interesting and valuable, but they are not reputation elements.

But then, step back. A reputation system is a substitute for personal experience. It provides you with the information you need to make a determination about someone (something?) else without having had to go to all the trouble of getting to know them. Traditionally, that determination has been about interpersonal trust. eBay’s reputation systems is the best example. I don’t know anyone who’s selling blidgets on eBay, so I don’t know how likely they are to cheat me. eBay has found a formal way to represent the likelihood that I’m dealing with a seller who will meet my expectations.

So, the reputation design patterns aren’t like that. They’re not about trust, at least not directly. But, they are signifiers that help me know someone better. Just like my score on eBay represents something about my behavior, so do achievement badges, rankings, etc. They encapsulate information about the type or volume of my participation in different ways. And this information, in turn, may help me figure out something more about trust. Certainly, in an indirect way at least, these things act as elements of a reputation because they substitute for my personal experience with someone else’s contributions over time. If I’d been there to see what they did myself, I wouldn’t need the badge or the level information.

Still, if we take this view, is my age reputation information? My address? After all, that information saves you the trouble of having to be around to count the years, or having to travel to my town to check where i live. (I’m going off the deep end now!)

More importantly, why does any of this matter? Who cares whether it counts as a reputation. Well, there’s a bit of truth in that - maybe reputation is in the eye of the beholder, at least in practice. But only sort of. When it comes to design patterns, I think the important thing is to realize what badges or points are good for. So, certainly my badges and levels help others figure out how to assess my contributions when they don’t know any better. But they also work as incentives that make me feel valued, like my contributions count, like I’m making progress, and like people think of me as an expert. They give me a goal or a quota to shoot for, or a status marker that tempers my insecurity.

I think Yahoo! gets this. But the patterns kind of mix up the part that’s for you (the reputation) and the part that’s for me (the incentive). If the point is to understand your users deeply, design incentive mechanisms with them in mind, then breaking those two apart is essential, and there’s a lot of good work to be done there. Anyway, as I said, I’m somewhat conflicted. Comments welcome (as always)!

Through it’s Developer Network, Yahoo! has just released a nice set of design patterns for reputation systems. I have some issues with some of the language and patterns, but overall, I think they put together a really great typology.

My biggest beef is with the most ‘meta’ pattern that they call the ‘Competitive Spectrum‘. I understand the desire to simplify, but in my view, these 5 things are not really on the same spectrum at all. I think the ‘combative’ type is off in a corner of its own - a corner that really doesn’t exist much on the web. As for the other four, I can’t make out what the axis is that they vary on - overall level of competition doesn’t make sense to me. Yahoo! seems to realize the confusion themselves, as they include a variety of caveats in their description of the spectrum.

I really agree with Bryce Glass (one of the patterns’ creators), who points out that these patterns are pretty ubiquitous now, and so simply pointing them out isn’t enough. It’s how they’re used - or more specifically how intelligently they’re used - that will make them powerful. I think Yahoo! still has some work to do to provide best practices for implementing these patterns intelligently. Obviously, given my interests, I’d like to see them look at some of the underlying social psychological processes, and use them to make some informed recommendations. Also, I think designers really need an accessible way to understand the ‘corruption effect of extrinsic motivation’ (or, as economists call it, ‘crowding out’). I would argue that in many contexts when incentives like the ones Yahoo! lays out don’t work as expected, the corruption effect is a big reason why. But, all in all, its a great start. (Note that my opinion is in no way influenced by the fact that I’m an intern at Yahoo! this summer… heh)

(Thanks to Ben for the tip!)

Duncan Riley’s recent post at TechCrunch is a little contradictory, on its face, about the fate of the new Digg-clone Ximmy. Ximmy is Digg-style social voting with the twist that they pay people for making pro-social (or at least pro-Ximmy) contributions. First Duncan says “Will Ximmy steal away top Digg and Reddit users looking for pocket money? probably not” but then he ultimately concludes:

I’d bet that sites such as Ximmy (although perhaps not Ximmy itself) will win the hearts and minds of a decent portion of the market, after all, if we’re going to spend time building value for these sorts of sites, it’s not much to ask in return that we should be compensated for our time, even in a small way.

Now, there certainly is a subtle distinction there between what could happen to extremely dedicated Digg users and what could happen to the rest of the rabble. I’m just not sure that Duncan is making it. I’m going to make a slightly different prediction that sort of splits Duncan’s down the middle. First, I agree - Ximmy will not steal away top Digg-ers, any more than Knol will steal away top Wikipedians (see my previous post on this subject). If we were to prioritize the incentives that motivate these people, monetary incentives are far down the list, and much less powerful than robust social psychological incentives like rational zealotry (i.e. fierce belief in the cause), reputation, status, and group belonging. Greenbacks just can’t compete.

However, Ximmy and sites like it will never be a large part of the market in the long term for at least two reasons.

1. Cash recruits the wrong kind of content, the wrong kind of users. Paying users to submit and promote stories may indeed promote a certain amount of contribution - but what kind of contribution? Ximmy chooses to offer a comparatively large payoff when a submitted story gets promoted to the front page - undoubtedly a move that’s intended to encourage high-quality postings. However, experience with Digg has shown that quality often has little to do with what gets promoted to the front page. Some have argued that a small cabal of powerful users can effectively get any story they want promoted. Other analyses have shown that stories in certain categories are much more likely to be promoted overall. Case in point - stories with the word ‘Linux’ in the title are almost 10 times more likely to be promoted.
   

   So, gaming the system is possible - that’s not news. The story here is that when you pay people, you make it about the money - social norms, ideology, community-orientation can all take the back seat. Amongst several options, you encourage gaming the system. So, though a caring, thoughtful, internally motivated user might view the payment as an incentive for quality, most users will just see it as a way to make money. So, cash will promote the wrong kind of users and the wrong kind of contributions. Quality will suffer, promoted stories will be old-news, garbage, or within a few narrow, stereotyped categories like Linux and lolcats. Critical mass will never arrive for the Ximmy community, it will fold right quick.

2. The economics won’t work out. This is a corrollary of reason 1, in a way. (Keep in mind, I’ve not done the reasearch necessary to back up this claim, but one could speculate, for example, about what would happen it Diff were paying users Ximmy-style, if the right stats. were available.) So, now Ximmy is paying plenty of people, but the content is garbage. They’re drawing the wrong kind of users who are motivated by the money. It becomes something of a closed system - a relatively small number of people who submit and promote each other’s stories. Since Ximmy doesn’t provide anything new content-wise over Digg or Reddit, its postings are lower quality and its community is smaller, they’re having a hard time getting pageviews. How does this business model work out? It’s a simple equation if we had the right inputs. How much is each of Ximmy’s ‘points’ worth in dollar value? How many do they give out each day? Take that amount, plus overhead, plus at least 40% to make it a sustainable, profitable business. This scenario seems very unlikely to me. Is this a funded start-up? I hope not. I assume the guys at Ximmy tried to work this out themselves, and if they got funding, convinced some VC’s that it would work. But if they did, they were operating on a false assumption - that cash would stand in for other motivations.

So, anyway, I expect that Ximmy will fail, and soon. I would guess that most of the user-generated content site-clones that are popping up will fail in the same spectacular way when they try cash-based models. I’m sure I sound like a broken record by now, but Duncan got the most important part of online participation wrong. He thinks that ‘it’s not much to ask in return that we should be compensated for our time.’ But people are compensated. It just turns out that the narrow-minded view of online participation is that anything we can’t fit on a spreadsheet or assign a dollar value to can’t count. But I think cash is a weak and fickle tool when compared to the powerful social and psychological incentives that drive people on the web, and I think the next year will prove it.

Blogs have been buzzing lately about the introduction of a new(-ish) platform from Google called Knol. Check out the announcement about Knol on Google’s blog. The idea is that Knol is a cross between Wikipedia and one of a few systems (e.g. Squidoo, Mahalo, Hubpages) that give users tools for creating portal pages on specific topics. The hope is that people will take ownership of particular topics, style themselves as experts, summarize and edit all the content that’s out there in the cloud for everyone’s benefit. Most of these sites have thus far tried to motivate users through a combination of reputation and community involvement. And let’s not forget the value of the knowledge base itself.

Google is tackling issues of motivation and quality with reputational and social networking tools, just like everyone else. But with Knol, they also seem to be stepping in with the increasingly classic Google move: add cash. How else is a deep-pocketed late-comer supposed to make a dent in the market? The strategy is no-doubt driven by Google’s bevy of economists who argue: when a rational person has the choice between doing something just for the warm-fuzzies, or for warm-fuzzies and cash, that person will go for the cash.

I’m surprised, however, that for all the talent they have on staff, no one around there has told them how dangerous this idea is. It turns out there’s all sorts of evidence that when you add monetary payments (or, more generally extrinsic incentives), all kinds of unexpected things can happen. Motivation can be reduced, quality can dip, resentment brewed. I recommend the good folks at Google get started by reading Not Just for the Money by Bruno Frey (an economist!) and The Hidden Costs of Reward edited by Lepper and Greene.

Of course, all of this will depend on just how carefully Google designs their system. One of the most fascinating areas of my research is understanding how the minutiae of user interaction and design elements can influence social psychological motivations. Crowding out (when extrinsic incentives push out intrinsic ones instead of adding to them) can in some cases be crowding in when the context is right. We know so very little about this stuff right now, at least in a scientific sense.

Ultimately, there’s a pretty fundamental divide here. Wikipedia is the 10-ton gorilla of knowledge sharing, and they’ve gotten this far without paying people a cent. Google is betting that Knol will be able to leech away contributors from Wikipedia. Michael Arrington over at TechCrunch seems to agree. And they may be right. But, I worry, to Google’s own peril. Who are those users who will abandon Wikipedia to feed from the Google cash trough? Are they they the invested, high quality, knowledgeable contributors that Google would need to build a respectable knowledge repository? Doubtful. But it may be presumptuous of me to assume that Google cares about the quality of their Knol content. Maybe sheer volume is enough. It’s their own property, so they can promote it in their search results all they want, and if the eyeballs and ad revenues are there, maybe Google is happy. But then let’s not fool ourselves by calling it a ‘knowledge repository’ when it’s really just another ad vehicle.

Why do I keep writing about Wikipedia? Mostly it’s because Wikipedia is emblematic of the sorts of online collective action that I study. And I just love that all their dirty laundry is falling out. Actually, I don’t intend that to be mean - more that Wikipedia has, up until now, been driven by a sort of utopian ideal that masks a lot of the turbulence and reality under the surface. And we shouldn’t be surprised - this is what happens when people try to cooperate, online or not. It an everyday part of social systems.

Anyway, The Register is reporting on a new scandal in which - shocklingly - it turns out that Wikipedia is not as open and democratic as everyone thought - or hoped. Turns out a select group of upper-level administrators have been using private mailing lists to coordinate responses (read: attacks) on users they think are trying to undermine their power, among other things. My reaction is similar to when I hear about a White House scandal - it’s good to get a tiny window into the machinations of people in power, but we don’t even know the half of it.

An interesting question is, would Wikipedia be able to survive without the ‘rings of hegemony‘ that have now apparently sprung up? Would the challenges of coordinating the efforts of millions towards a stated goal (uber-open-online encyclopedia) overwhelm any truly democratic, egalitarian (dare I say socialist?) efforts? I have opinions, of course, but if anyone is out there, I’m interested in yours.

I think this recent clash between Wikipedia editors and webcomic creators is just the tip of the iceberg - we’re going to see more frequent and intense clashes in the near future. Read the short article linked above to get the gist, but basically: a few Wikipedia users (one in particular) went around deleting many pages related to webcomics, saying they did not meet Wikipedia’s notability guidelines. The webcomics community is angry, refusing to help with Wikipedia’s fundraising efforts, trying to raise the profile of the issue.

As an observer, I love that this battle is happening. On the one hand we have editors who feel that it’s their duty to police the boundaries of Wikipedia, and delete content they feel does not meet the notability guideline. On the other we have a community of content producers who, irrespective of any other measure of notability or popularity, perceive the intentional deletion of articles about their webcomics as a slight.

And rightly so. It is a slight. Someone with more power than most - in this case the user Dragonfiend - has applied an arbitrary interpretation of the notability guideline. Editors do this all the time, right? Well, it wouldn’t be so problematic in this case except for two factors - and these are two fascinating factors that I think will continue to haunt and define Wikipedia in the years to come.

First, Wikipedia is wrestling with its openness as it grows. It wants to be democratic - indeed it has built its brand upon it. It’s “the free encyclopedia that anyone can edit”. (Actually, right now that slogan strikes me as a little duplicitous. *Almost* anyone can edit, and someone may end up denying your edits in the end.) And yet the challenges of immense popularity have made people like Dragonfiend necessary. As the size of the encyclopedia increases, the problems of order and coordination also increasing. In Wikipedia’s case this seems to have required increased activity by self-appointed editors who wield the power of interpreting Wikipedia’s policies as they see fit. Check out, for example, this recent paper that documents the increasing use of Wikipedia’s Talk pages to handle emerging problems of coordination. The arbitrariness of these interpretations, as well as the mere existence of people who, like judges and editors, can single-handedly influence the system, is clashing with Wikipedia’s democratic, open, community-based ideals.

None of this would be as big a problem as it has been, however, if it weren’t for the second factor I want to mention. This debate is a lens that magnifies the social and cultural position of prestige and authority that Wikipedia has come to occupy among certain stakeholders. The webcomics community, for example, probably wouldn’t be so upset if their articles were not included in the latest edition of Britannica. That encyclopedia just has a different position as a sociocultural icon. Similarly, many (but not all) chemists, philosophers, and literature critics (and the like) might not be as upset if they were left out of Wikipedia (and indeed they are under-represented there), but would be aghast at being left out of Britannica.

To generalize (in a slightly unfair way): In 2007 the web is a platform for community to coalesce around their hobbies and interests, to push across geographic boundaries and form dense networks of content creation and sharing. For these sorts of web-enabled, tech-savvy folks, Wikipedia was supposed to be a safe haven where they could participate and be included on the same footing as everyone else. After all, these were the exact people who embraced the new model that Wikipedia represented when it was just taking off. And now that they’re being pushed to the fringes as Wikipedia is forced (or chooses) to take on the character of a traditional edited publication, they’re pissed. Who can blame them?