About those biased polls . . .

November 7, 2012 22:30 pm

by Bill Dawers · 22 comments

As I write this, Obama has 303 electoral votes and leads by 2.4 percent (50.4 to 48) in the popular vote. Florida seems extremely likely to be called officially for Obama at some point, which will put him at 332.

In the final run of his model that aggregates polling data and makes certain assumptions based on history, Nate Silver at FiveThirtyEight showed Obama the clear favorite in states totaling 303 electoral votes and a tiny favorite to win Florida, the only true tossup. The model predicted the final popular vote margin would be 2.5 (50.8 to 48.3).

So the polls were right. Not all of them, of course. But viewed collectively, the polls (and Silver) pretty much nailed it — just like in 2008. Independent polling companies and analysts like Silver — the best known but certainly not the only statistician aggregating data — have a vested interest in getting it right. Their livelihoods depend on turning out quality products. In spite of the inherent difficulties of heavy cell phone use and low response rates, the pollsters are figuring out how to generate solid data. (FWIW, on Monday on my own blog, I predicted Obama would get 332.)

Given margins of error, any single poll of a moderately close state could be wrong even if the poll was conducted under ideal conditions. But we don’t have to rely on individual polls or pollsters in presidential elections. In the final 33 polls of Ohio, Romney led in two, two others were tied, and Obama led 29 times. In 12 of those polls, Obama’s lead was 4 points or more. As it turned out, Romney did better in Ohio than that mass of polling indicated, but there was almost no chance of a different outcome given Obama’s consistent leads.

As a numbers guy, I’ll confess to being puzzled by the reluctance of so many Romney supporters to accept the polls generally and Silver’s thoughtful, detailed work specifically. He’s awfully transparent about how he works. Some people were clearly tricked by those big names Gallup and Rasmussen, which right up till their tight polls on the final days, showed leads — sometimes solid ones — for Romney. But, as Silver patiently explained, Gallup’s likely voter polls had a history of implausible volatility and Rasmussen had a recent history of having a Republican lean.

The Gallup likely voter model is especially problematic. An article in Slate almost a year ago detailed an interesting phenomenon: about 55 percent of registered voters in 2008 who told pollsters they would not vote actually did vote. And not all those who said they would vote did vote.

It was also interesting over the last few months to hear the exasperated assertions that any poll that showed more Democrats going to the polls than Republicans must be skewed. These arguments missed the simple fact that pollsters weren’t adjusting their results to match a specific preordained party identification ratio. Party identification was a question in the polls, and party ID is fluid. Plus we have a long history of more voters identifying themselves as Democrats than as Republicans, even in strongly Republican years. Silver wrote a long post back in September explaining that over the years the polls have not shown any consistent bias to the right or to the left. In that same post as well as in others, he clearly documented that state polls tend to be more accurate than national ones.

But it seemed like the more patient Silver was in his explanations, the more he was attacked for things he wasn’t doing.

I hope that work like Silver’s will force the punditocracy to be more thorough, responsible, and mathematical when making predictions. For example, consider that George Will predicted Romney would get 321 electoral votes, including Minnesota, where Obama had led in every single poll, by about 8 points on average. Obama took the state by about 8 points.

xdog November 8, 2012 at 7:21 am

“I’ll confess to being puzzled by the reluctance of so many Romney supporters to accept the polls”

Come on, no mystery here. Everyone, left and right, creates a world where wish fulfillment becomes reality. For the many gopers who see polling as the latest effort of the liberal scientific media to cram its preconceived notions of ‘truth’ down patriots’ throats, denial was/is even more appealing.

D_in_ATL November 8, 2012 at 9:36 am

“Is this just math you do as a republican to make yourself feel better?”
-Megen Kelly to Karl Rove


Engineer November 9, 2012 at 9:57 am

Yeah, I saw that on the Daily Show. I got a good laugh out of that.

Noway November 8, 2012 at 11:43 am

Help me out, Bill. Who is this Silver guy? If I’m already supposed to know, I apologise. Can you tell me why this guy is so politically news right now?

griftdrift November 8, 2012 at 12:28 pm

Nate Silver runs the NY Times 538 blog ( he started it then they bought it)

In 2008 he called 49 of 50 states. Two nights ago, he called all 50.

He’s a statistician.

And possibly a witch.

Bill Dawers November 8, 2012 at 12:58 pm

Ha. He just might be a witch.

Charlie November 8, 2012 at 1:13 pm

Well, after the real results matched his projections, I think we can say he didn’t sink. Looks like he floats.

Bill Dawers November 8, 2012 at 12:54 pm

There’s lots of background about him and the controversies out there on the web, but here’s a succinct piece at Mashable: http://mashable.com/2012/11/07/nate-silver-wins/

Silver called all the states but Indiana in 2008 using his model that inputs polling data. He called the 2008 popular vote margin to the tenth of a point. He expanded that model considerably in 2012 to include economic data and additional historical trend information. He showed Obama with a narrow lead throughout this campaign — with probabilities of an Obama win as high as 90+% and as low as the high 50%s — but in recent weeks especially he was pilloried as biased and trivial (see Michael Gerson http://www.washingtonpost.com/opinions/michael-gerson-the-trouble-with-obamas-silver-lining/2012/11/05/6b1058fe-276d-11e2-b2a0-ae18d6159439_story.html). Even the Public Editor of the NYT (where his blog is based) called his relationship to the paper in question and spoke rather dismissively of his work in a blog post that also contains some good background on the attacks on him: http://publiceditor.blogs.nytimes.com/2012/11/01/under-attack-nate-silver-picks-the-wrong-defense/ Maybe it’s because I’m a columnist, a numbers-cruncher, and a college journalism instructor, but the various issues seem really important regarding the news media’s future as its “horse race” mentality is challenged by data analysis. I explored those issues a bit here: http://www.billdawers.com/2012/11/03/nyts-public-editor-throws-nat-silver-under-the-bus-what-does-it-mean/

I hope that helps.

griftdrift November 8, 2012 at 1:11 pm

Bill, I just finished his book. I highly recommend it. Especially for numbers guys, like you.

Bill Dawers November 8, 2012 at 1:46 pm

I really need to pick it up. Maybe that will be my holiday read . . . Thanks for the recommendation.

Noway November 8, 2012 at 2:47 pm

Helps a ton, Bill, thanks!

Noway November 8, 2012 at 12:32 pm

LOL! Kind of like the nerd who used stats to help the Oakland A’s in Moneyball? So this Silver guy is somebody, regardless of our political persuasion, we better listen to?

griftdrift November 8, 2012 at 12:33 pm

Exactly right on both counts.

Noway November 8, 2012 at 12:36 pm

I remember when Zogby used to be that good. Wonder what ever happened to him?

griftdrift November 8, 2012 at 12:52 pm

Online polls. Bad samples. Bad fundamentals

Buzz Brockway November 8, 2012 at 4:21 pm

Good article Bill. I admit I became skeptical of Silver’s model and of many of the polls. I assumed Republican turnout would match Democratic turnout this year making assumption that turnout would mirror 2008 incorrect. It looks like my assumption was wrong and thus the polls were generally correct.

As best I can tell Silver’s model relies on the assumption that State polls will be accurate. As long as that’s the case he’ll be accurate.

griftdrift November 8, 2012 at 4:33 pm

That’s the core of his data, but it’s a little more than that. He weights the polls based on past performance (using data from the past three decades), weights the state polls for impact (for example, Ohio polls had a much higher impact than Georgia polls) and also adds other fundamentals such as economic performance.

The idea is simple. The execution is complex.

Bill Dawers November 8, 2012 at 5:57 pm

And let me add a little more to the preference for state polls. I don’t know off the top of my head the typical number of respondents in the national polls, but let’s just say 1000. Of those 1000, there will be only a few hundred who live in swing states. But a poll just of Ohio could have just as many respondents — all from Ohio. So in the last few weeks of a campaign, state polls could literally be counting tens of thousands of voters in a particular state. If the various polling firms active in that state have any sound methodology at all, then an aggregation of those polls is going to have a very small margin of error.

It’s worth noting that Silver and his team are now analyzing results from Tuesday. He has written a fascinating post today about the margins in various states. The results suggest that Romney might have needed a 2 point popular vote edge to win the electoral college. And I was surprised to see that, after North Carolina, Georgia was the next closest state that Romney won. http://fivethirtyeight.blogs.nytimes.com/2012/11/08/as-nation-and-parties-change-republicans-are-at-an-electoral-college-disadvantage/

sunkawakan November 8, 2012 at 6:28 pm

Silver’s great. Uses real statistical models and everything. He also was a baseball handicapper…perhaps Chip Rogers could sell Silver’s picks?

The Last Democrat in Georgia November 8, 2012 at 6:32 pm

“He also was a baseball handicapper…perhaps Chip Rogers could sell Silver’s picks?”

…Don’t give him any ideas.

Romegaguy November 8, 2012 at 9:13 pm

Speaking of biased polls, has anybody heard from Matt Towery or his wig?

The Last Democrat in Georgia November 9, 2012 at 5:09 am

The last I heard, Towery had gone missing and there was an A.P.B. put out for him and his wig was wanted as an accomplice.

Comments on this entry are closed.