I just read through Oded Nov's paper from Communications of the ACM:
Nov, O. (2007). What motivates Wikipedians? Commun. ACM, 50(11), 60-64. (link)
Two things occur to me. First, Nov explains away the potential influence of social desirability in about two sentences, but I'm not buying it. When you ask people why they do something, there's a huge number of social factors that are going to come into play. In the case of Wikipedia I also think there are likely to be lots of soft and implicit attitudes. Soft attitudes are expressions that don't reflect beliefs, but rather answers to questions someone might not have thought about previously. For example, if I asked you "How do you feel about Kobe Bryant elbowing Ron Artest in the neck last night?", you might respond by saying it's abhorrent. If I took that at face value, I'd be ignoring the fact that many people don't know about basketball, don't know who Bryant or Artest are, don't know the contest, or don't care. Unconscious attitudes, on the other hand, are attitudes that we hold and act on but can't express. To me, neither of these things makes survey research of this type invalid – I do similar surveys myself! But they're important issues, too often left out of discussions.
The second issue, maybe more important, is about scope. There are a fair number of studies now about motivations for contributing to various online collective actions. But they almost always focus on people who contribute a lot. However, these papers, like Nov's, usually don't make that distinction. They make claims about motivations for all contributors. In reality, the motivations of casual or infrequent contributors are likely to be very, very different. Harder to study, though! On the one hand, by studying the heavy contributors we capture motivations for majority of the work that gets done, but we do that at the expense of attention to the vast majority of people who contribute.
In sum: Social desirability, soft attitudes, etc. need more consideration when we talk about motivation. Studies that focus on heavy contributors should say as much, and more studies should look at casual contributors' motivations.
