Time to Wonk Out: AJC Poll Edition

May 15, 2014 10:30 am

by Eric The Younger · 3 comments

Let’s wonk out for a bit. The AJC commissioned a poll to look into the Senate race as well as the Governor’s race. This is hardly unusual at this point in the election cycle, however what makes this one worth looking into is how different it is from other polls that have been conducted and released. This (as well as some promptings from friends) got me curious and gave me a reason to put to good use all those methodology classes I took. Consider this the level 10 geek alert, though I will try to break some things down for the adventurous and statistics disinclined.

Let’s talk about methodology first. The sample size was a total of 1302 individuals, 290 of which are non-registered voters.  The sample was drawn from random number dial and not from known voters. The poll was conducted from 5-8 May and  the MoE is +/- 4. This is a Frequentist approach rather than a Bayesian approach. For the difference between the two, go here.

In terms of valid methodology, the random number dial can work, though it is probably better suited for a general election in a presidential year where you will have the highest turnout. A better sample would have come from a known population of registered voters who have voted in recent primaries/elections. Think of this like shaving with a straight razor whereas the former is an electric razor. Still getting the job done, but one is more exact.

The next thing to talk about is the nature of the sample. Included are  32% Democrats, 28% Independents, and 27% Republicans. I find this is odd for a state that elected Mitt Romney 53% to 45% in 2012 and Johnny Isakson 58% to 39% in 2010. My gut reaction is to question the accuracy of the sample. It seems odd that Georgia midterm voters would include a 5 point advantage to Democrats – and a one point advantage to Independents. This sample bias would definitely be in favor of the Democrats. Here are the full cross tabs.

Now this is where things get interesting. Let’s look at individual question results. Just for kicks, let’s start with the Governor’s race. Here we see that Governor Deal has a 4 point lead on Jason Carter. Going back to the MoE, that means that this is a statistical tie. While other polls have shown that Deal isn’t as far ahead of Carter as one would expect, none of them -to my knowledge-  have shown a statistical tie. This could be attributed to the sample bias.

Now for the Senate race there are questions to see how  each republican candidate would fare against Michelle Nunn.  Nunn fares better than the four candidates that actually have records to run with (Handel, Kingston, Broun and Gingrey) but not against Perdue among registered likely voters. This also seems kinda odd to me given how Republican Georgia generally is.

Potential flaws: First there is the sample bias. The methodology is sound and valid, however the sample does not appear to be representative of the population of Georgia. Admittedly, Georgia voters do not have to register by party, so any estimates of party ID are simply a guestimate to estimate. These are also self identifying party ID affiliations, which can also cause some flux.

Including the non-registered voters in the sample also helps to skew the poll away from an accurate reflection of a Georgia primary. It’s like asking the people who were invited to the wedding and didn’t go how good was the catering. It is possible that they could know how good the catering was, but you’d be better off asking someone that was there.

A second potential, and I would be tempted to say likely, flaw is a confirmation bias. The AJC has not given a lot of ink to the various Republican candidates. On the flip side, the AJC has also given more ink to Nunn and not any of her Democratic challengers. This information diet could influence the sample more towards the Democrat side of the spectrum because there is only one name, while the sheer number of Republicans are causing a pan-Republican candidate to be seen by the respondent.

While the methodology used would be appropriate in some cases, it would appear to me that this poll is rife with problems. Some are definitely nitpicking, but others are more significant and cause me to doubt the accuracy of the results.

{ 3 comments… read them below or add one }

Doug Deal May 15, 2014 at 11:26 am

Another point that leads to error in a general election poll during primary season is that when there are multiple candidates on one side and really only one viable candidate on the other, supporters of the opponents of a candidate may convince themselves in the heat of a primary campaign that they will never vote for another one of the candidates if their guy doesn’t win the primary. Plus, Republicans are running negative ads against each other, which will cease once there is a winner.

So it is like pitting an army fighting among itself against one that is more or less unified. To mess with the numbers in a poll, it only takes a few percent here or there jumping ship to the other side or replying as undecided.

When things get serious after the primaries are decided, I suspect the GOP will coast on an 6-8 point advantage the rest of the way.

Will Durant May 15, 2014 at 11:55 am

I am a certified geek but primarily relating to the Stone Age of mainframe computing, though I once supported a staff of top ranked statisticians their formulas may as well have called for eye of newt or chicken lips for all I knew. That being said, how do you justify a sampling using 22% non-registered voters to predict the outcome of an election that requires the voters to be registered?

Chris Huttman May 16, 2014 at 12:31 am

Simple. The people who said they weren’t registered weren’t included in the results that include vote totals for candidates.

And I’ve said it before, I’ll say it 1,000,000 times again, a good poll of Georgia will typically include more Democrats than Republicans because Georgia’s independents lean pretty heavily Republican.

Consider that 35% of the state is non-white and so you aren’t likely to find many Republicans in those ranks. Probably like 65% Democratic, 5% Republican and 30% independent from those ranks. Now consider whites – we know from exit polls that about 20% of white voters give their votes to Democrats even in the worst years for Democrats (they live in places like Decatur and Athens). So let’s assume that whites are 20% Democrats, 30% independents and 50% Republicans.

Add the two together by white/non-white and you get:
35.75 Democrats
34.25 Republicans
30.00 Independents.

Now also imagine that all of the non-white independents vote for Democrats and all of the white independents vote for Republicans. What do you get? 54.75 Republican, 45.25 Democrat.

Why is this so hard for people to understand? Democrats typically have a small advantage, it does not go up for Republicans as some argue as a general election gets closer. It’s pretty constant, the demographics explain it, and Republicans are still favored in a general election.