Are Pollsters Taking the Cheap Way Out?

July 24, 2014 14:11 pm

by Tim Darnell · 17 comments

With so many polls wrong about Tuesday’s GOP runoff, one political scientist may have an explanation.

“I don’t think any of the polls in the primary or runoff were live interviewer calls,” said Alan Abramowitz of Emory University. He told Newsradio 106.7 on Thursday that, “Live interviewer calls, where you can call both landline and cell phone lines, is generally the preferred method of doing these polls, but its more expensive.”

Abramowitz also says its a big challenge to get people to answer their phones when they see a number they don’t recognize.

{ 17 comments… read them below or add one }

southernpol July 24, 2014 at 2:22 pm

I received a live interview call on June 25th. It took 25 minutes.

It had 8 questions about Kingston (2 negative) and 3 about Perdue (all negative). Also asked fav/unfav of Michelle Nunn, US Chamber of Commerce, The NRA, and Herschel Walker.

Something most everyone dissing the polls in this race is missing is that no one predicted the drop in turnout in the 1st district. The 1st had more early votes than the 11th, but ended up having 11,000 fewer election-day votes. If the 1st matched the 11th in votes in the runoff, I don’t know if Kingston wins but it would have been a razor-thin margin one way or the other.

Ralph July 24, 2014 at 2:37 pm

Polling simply does not work for low turnout elections mostly because “likely” voters can not be pinned down accurately. Furthermore, so many voters refusing to participate in polls (like me) makes those bogus plus or minus accuracy and confidence numbers a joke.

And I lie to exit pollsters. I wonder how many others lie to pollsters just for the fun of it.

ChuckEaton July 24, 2014 at 3:10 pm

I think the low turnout of GOP runoffs are difficult to poll – especially one in a completely different month than the prior history of runoffs. That being said, I believe both campaigns had internal polling that showed a tighter race. Perhaps those polls were based on “live interviewers.”

Monica July 24, 2014 at 3:50 pm

There was a poll that came out maybe a month ago where they polled all sorts of local races but left this one out – I don’t recall what they said exactly about it, but it was something along the lines of it being impossible to poll.

Also, a pollster I know in Atlanta didn’t touch the GOP runoff, and he said that the turnout was simply impossible to predict. First time we had a 9-week long runoff, first time the election was mid-July when a lot of people take their last vacations before the school starts, he said that predicting metro areas was too difficult. Incidentally, it was the Savannah metro area that didn’t turn out and probably did Kingston in. So basically, it was always just a big guess for pollsters, but something tells me they were leaning Kingston to come out with the numbers they did…

Ralph July 24, 2014 at 3:30 pm

I don’t see how internal polling can be any more accurate, but might be more honest as most pollsters and/ or those who buy them have their own biases and agendas. When a majority (probably a HUGE majority) of people duck pollsters, those few who partake are probably not very representative of the electorate, except probably more accurate in the higher turnout elections since “likely” voters are easier to determine. At least we all know the reported accuracy has no correlation to reality because competing polls are all over the place with similar claims of accuracy.

Chris Huttman July 24, 2014 at 4:07 pm

The difference between internal polling and media polling is the difference between the guy who is the GM for the Atlanta Falcons and a guy on ESPN who just spouts on about what he thinks.

An internal pollster won’t get rehired (Kingston’s guy probably after the Cantor mishap) if he doesn’t get it right or close to right. A media pollster – I refer you to ESPN and the wisdom of their on air talent when they give prediction about who will win. Guys who have gone 0-for are back on the next week making predictions again.

Now keep in mind a few things – what the public sees of an internal poll might not be the full story. They could see a ballot test that came after a barrage of positive and negative questions, for example, or was after a descriptive paragraph or two was read about the candidate that paid for it.

Another thing to keep in mind – judging a pollster in the aggregate on accuracy is fine, but statistically being within the margin of error should be acceptable. In the case of the runoff, we had a 51-49 victory for Perdue. Why are people saying the polls were off? Few reasons – on the one hand no/very few released polls predicted that Perdue would win. With that final result, more than half should have been showing Perdue winning. Just compare to Virginia or Florida Presidential election in 2012, similarly close races and at least half the polls predicted Obama would win. Other problem – the margin. Most polls, Kingston led by 5+. If someone had put Kingston up 1 and then he lost by 2, I’d say that was a pretty good poll. They could have perfectly modeled the turnout and just been off within the margin of error. It happens.

But 5+ for the wrong guy – the polls were wrong. The question is why? And I think the most likely conclusion is the agenda/assumptions that went in. Many Republicans who comment here/Facebook never took Perdue seriously. Not when he announced his candidacy. Not when he stroked his first big check to himself. Not when he went on TV first. Not when early polls showed him jumping in the lead in the primary. Not when he came in first in the primary. There was almost an assumption/preference that someone else would win. I’ll just say that if you’re trying to bake a chocolate cake, you don’t usually end up with a carrot one by accident.

ChuckEaton July 25, 2014 at 12:10 pm

Good analysis. Plus, kudos to you; in a post, on election day, you predicted the race would be tighter than the public polls had predicted.

Mark Rountree July 24, 2014 at 4:40 pm

The most accurate polling operations are actually those who employ IVR (“recorded”) technology. There are many reasons for this, but a major one is that the recording is identical in all cases, with zero banter or discussion between caller and voter, etc.

There are many groups which report the most accurate polling operations after national elections, and recorded calls have pretty consistently held their own, or been more accurate than, live dial polling.

That said, there are limitations in my view, including in-depth or multi-level policy questions. It sometimes helps to have the assistance of a live interviewer, especially for those voters who hearing impaired, older voters, or in some cases those with lower degrees of education.

More important than “live vs recorded” is proper weighting of demographics, fair and clearly written questions and quality lists.

Chris Huttman July 24, 2014 at 5:31 pm

All good points. In the millions of calls I’ve completed, the results have mostly been right on. I also attribute a lot of value to proper weight, question composition, list construction etc.

KD_fiscal conservative July 25, 2014 at 9:36 am

Great points from both of our resident pollsters. In some ways polling is more of an art than a perfect science. How much you weight the various cohorts(groups) is often based on intuition and previous elections which may or may not correlate to future results. For example, even if you gauged that 80% of district 1 will support Kingston…how do you guage how many of those folk will actually turn out? This is especially damaging to poll results(and pollster reputations) when the turnout for one group is significantly different from another.

My question to either Mark or Chris is how do one construct a solid list? Trial and error?

Will Durant July 25, 2014 at 10:57 pm

How do you support your statement of IVR superiority? An IVR poll?

Granted, this is not a scientific sample but only 1 in 3 households in my neighborhood still have conventional land lines. Those that still have them skew to being older. Typically the more technologically adept the more likely they are to have just cell phones and/or VOIP. Does this skew the results even with the weighting of demographics?

How reliable is the IVR technology in determining whether the participant is just messing with you? Is that really only +/-4%? Even when I had a land line I used to giving false responses in retaliation for the irritation for having a machine interrupt my dinner.

sandsage July 24, 2014 at 4:46 pm

I keep reading about how District 1 in general and Chatham County in particular didn’t turn out to vote, and that’s what did Kingston in. First read it yesterday in Jim Galloway’s column, and now here in these comments. I haven’t looked at the numbers for the entire district but it’s not true about Chatham. Here are our numbers: Tuesday – 17,586 R votes total, 15,064 for Kingston, 2,522 for Perdue. And here are the numbers for May 20th: 17,865 R votes total, 13,992 for Kingston, 1,779 for Perdue, the rest to the others. We only fell off from the original primary in the spring to the run-off in July by less than 300 votes. And Kingston’s total in Chatham went UP by 1,000 votes.

Also about turn-out, fivethirtyeight did an analysis of whether it was turnout that threw off the polling, and said no. Even with higher turnout in Kingston’s counties, it still would have gone to Perdue though with a smaller margin – 0.8%.

Mark Rountree July 24, 2014 at 11:32 pm

Your numbers are correct about Chatham. I was the author of those comments Mr. Galloway/Pol Insider was referring to,by the way.

However, the rest of the First Congressional District turned out in much lower numbers than Chatham. The drop off was significant in non-Chatham First District counties — collectively down 28% if I remember correctly.

The analysis that Jim Galloway was referring to didn’t limit the conversation to the First District. The bigger picture was that Metro Atlanta (where Perdue was stronger in May) drop-off was -11%, while the rest of the state (where Kingston was stronger) was down 26%.

sandsage July 24, 2014 at 5:03 pm

Just for fun decided to check on some of our surrounding counties and saw it really varies widely by county:
Effingham July – 2810 Kingston, 474 Perdue (3284 total)
Effingham May – 3221 Kingston, 408 Perdue, 640 other (4269 total)
Bryan July – 1923 Kingston, 353 Perdue (2276 total)
Bryan May – 1884 Kingston, 252 Perdue, 280 other (2316 total)
Liberty July – 1041 Kingston, 168 Perdue (1209 total)
Liberty May – 1132 Kingston, 110 Perdue, 167 other (1409 total)
McIntosh July – 852 Kingston, 227 Perdue (1079 total)
McIntosh May – 1520 Kingston, 202 Perdue, 243 other (1965 total)

FranInAtlanta July 24, 2014 at 5:35 pm

Would love to know what happened in McIntosh County.

Ralph July 24, 2014 at 7:00 pm

My guess is that the folks in McIntosh County (and others) didn’t see enough difference between Kingston and Perdue to bother to vote in the runoff – almost like me. At least I was very relieved that Broun and Handel were out of the runoff. I was surprised that the runoff turnout was in double digits.

FranInAtlanta July 24, 2014 at 11:13 pm

I knew I wouldn’t have trouble voting for either in November. When some early rumors said the Dems were going to vote for Broun to try to put Nunn in and hurt the Repubs nationally, I was wondering what I would do if he won.