Thu 3 Mar 2005
As an anthropologist (and a post-modernist) it has become second nature to me to take every piece of scholarship with a grain of salt. In a way I have become a Marxist – always 'looking for the lie.' Leaving aside the fact that I truly believe that no research can ever be truly objective, in doing qualitative research I have become very comfortable with biases and confounding factors. Granted, the goal of good anthropology is rarely to make generalizations and representative statements, but instead to capture the richness and depth of culture and human experience. Still I feel as though any paper worth its salt – anthropological or otherwise – should include a limitations section.
I don't view a limitations section as an inoculation to criticism or refutation. More I think it's an acknowledgment of the realities of research. I am skeptical of any researcher who, having set out a plan for research which meets all known criteria for validity and reliability, unquestioningly assumes that he/she has carried out that plan without bias or judgment of any kind. That kind of attitude is so naïve, and far too common, especially in the more quantitative fields. Knowing that a researcher is aware of the contextual factors that can influence his or her work lends credibility in my view.
Why don't more papers in the ICT space do this? I don't see limitations sections often included in CHI papers, ACM papers, or very often in any other popular and prestigious forum in our field. I'm not sure how to explain this omission – it can't be that all these smart people honestly think that their research is totally unbiased and without limitations or confounding factors that are worth mentioning. (Having said this, I also tend to think that focusing on biases or imperfections, especially in the statistical sense, is a convenient way to discredit or disregard research that one happens to disagree with.)
I don't want to speculate as to why this really is – I just want to say that I think there should be a requirement for a limitations section in most of these forums. Especially as the perceived utility and legitimacy of qualitative research is growing, and more and more technologists are becoming aware of the importance of context (in every sense of the word), I would think there'd be more of an emphasis on both the positive and the negative aspects of research.