Thu 3 Mar 2005
As an anthropologist (and a post-modernist) it has become second nature to me to take every piece of scholarship with a grain of salt. In a way I have become a Marxist – always ‘looking for the lie.’ Leaving aside the fact that I truly believe that no research can ever be truly objective, in doing qualitative research I have become very comfortable with biases and confounding factors. Granted, the goal of good anthropology is rarely to make generalizations and representative statements, but instead to capture the richness and depth of culture and human experience. Still I feel as though any paper worth its salt – anthropological or otherwise – should include a limitations section.
I don’t view a limitations section as an inoculation to criticism or refutation. More I think it’s an acknowledgment of the realities of research. I am skeptical of any researcher who, having set out a plan for research which meets all known criteria for validity and reliability, unquestioningly assumes that he/she has carried out that plan without bias or judgment of any kind. That kind of attitude is so naïve, and far too common, especially in the more quantitative fields. Knowing that a researcher is aware of the contextual factors that can influence his or her work lends credibility in my view.
Why don’t more papers in the ICT space do this? I don’t see limitations sections often included in CHI papers, ACM papers, or very often in any other popular and prestigious forum in our field. I’m not sure how to explain this omission – it can’t be that all these smart people honestly think that their research is totally unbiased and without limitations or confounding factors that are worth mentioning. (Having said this, I also tend to think that focusing on biases or imperfections, especially in the statistical sense, is a convenient way to discredit or disregard research that one happens to disagree with.)
I don’t want to speculate as to why this really is – I just want to say that I think there should be a requirement for a limitations section in most of these forums. Especially as the perceived utility and legitimacy of qualitative research is growing, and more and more technologists are becoming aware of the importance of context (in every sense of the word), I would think there’d be more of an emphasis on both the positive and the negative aspects of research.
March 13th, 2005 at 1:45 pm
I wonder if some of it has to do with the pressure of getting your papers published and the expectations of reviewers. I wholeheartedly agree with you that it would be great for more researchers to openly acknowledge the potential sources of bias and limitations in their research, however, if the number one criteria is that the paper is accepted then people don’t want to be focusing too heavily on anything that could negatively affect the reading of their paper in comparison to others that don’t highlight any limitations (even though no doubt there will be some). Of course I’ve no idea if this is the case or whether it’s just not become standard practice or whether people genuinely aren’t aware enough of these factors - but it does seem that it would be good to encourage sections on ‘Limitations’ as a strength of the paper rather than a weakness.