Thu 23 Oct 2008
I find Sam Wang's polling analysis at PEC to be smart and informed, and his commentary to be pretty interesting. But his confidence scares me. Wang seems to be doing all the right things. He's making his data and his models freely available. He's keeping his models purposely simple in order to avoid the chance of a mistake. And he's not fiddling with his model as time goes on. All good things.
But… Wang thought his model was right in 2004 until it turned out it wasn't. And people probably didn't think much of the problems in polling methodology until the Dewey / Truman fiasco, and then they did. What don't we know now? I think the data problem is the more dangerous one. As a commenter on PEC points out, polling is a 'black art', and though we gain a lot by doing a meta-analysis, we're still at the mercy of the underlying numbers. Another commenter points out that for Obama, a win is not a win. We want to crush McCain. Doing so sets a completely different political landscape than a narrow victory would.
Anyway, I don't suggest that Wang should be doing anything differently on the statistics. But, and I'm sure he hates the constant comparisons to Nate Silver at 538, where Nate just says that McCain is in trouble, Wang advises that Presidential race is in the bag, and we should forget about it and start focusing on Senate races. A little more temperance would make the commentary much more palatable for me.

Jon Krosnick keynoted a big e-voting conference this summer and spoke about primacy effects (whereby listing something first in a questionnaire results in a bias towards that result) and how ballot rotation can eliminate these problems. One funny revelation that he gave was that the exit pollsters *don't rotate candidates* at all. This leads to an obvious bias that some smart states have been dealing with for a while by permuting the order of candidate listing. Anyway, this is all to say that there are so many effects that *we know about* that people aren't taking into consideration, which leads me to being cautious about all of the damn thing.
Yo, yer link to 538 is wrong… should be: http://www.fivethirtyeight.com/
Thanks, Joe!