Let’s wonk out for a bit. The AJC commissioned a poll to look into the Senate race as well as the Governor’s race. This is hardly unusual at this point in the election cycle, however what makes this one worth looking into is how different it is from other polls that have been conducted and released. This (as well as some promptings from friends) got me curious and gave me a reason to put to good use all those methodology classes I took. Consider this the level 10 geek alert, though I will try to break some things down for the adventurous and statistics disinclined.
Let’s talk about methodology first. The sample size was a total of 1302 individuals, 290 of which are non-registered voters. The sample was drawn from random number dial and not from known voters. The poll was conducted from 5-8 May and the MoE is +/- 4. This is a Frequentist approach rather than a Bayesian approach. For the difference between the two, go here.
In terms of valid methodology, the random number dial can work, though it is probably better suited for a general election in a presidential year where you will have the highest turnout. A better sample would have come from a known population of registered voters who have voted in recent primaries/elections. Think of this like shaving with a straight razor whereas the former is an electric razor. Still getting the job done, but one is more exact.
The next thing to talk about is the nature of the sample. Included are 32% Democrats, 28% Independents, and 27% Republicans. I find this is odd for a state that elected Mitt Romney 53% to 45% in 2012 and Johnny Isakson 58% to 39% in 2010. My gut reaction is to question the accuracy of the sample. It seems odd that Georgia midterm voters would include a 5 point advantage to Democrats – and a one point advantage to Independents. This sample bias would definitely be in favor of the Democrats. Here are the full cross tabs.
Now this is where things get interesting. Let’s look at individual question results. Just for kicks, let’s start with the Governor’s race. Here we see that Governor Deal has a 4 point lead on Jason Carter. Going back to the MoE, that means that this is a statistical tie. While other polls have shown that Deal isn’t as far ahead of Carter as one would expect, none of them -to my knowledge- have shown a statistical tie. This could be attributed to the sample bias.
Now for the Senate race there are questions to see how each republican candidate would fare against Michelle Nunn. Nunn fares better than the four candidates that actually have records to run with (Handel, Kingston, Broun and Gingrey) but not against Perdue among registered likely voters. This also seems kinda odd to me given how Republican Georgia generally is.
Potential flaws: First there is the sample bias. The methodology is sound and valid, however the sample does not appear to be representative of the population of Georgia. Admittedly, Georgia voters do not have to register by party, so any estimates of party ID are simply a guestimate to estimate. These are also self identifying party ID affiliations, which can also cause some flux.
Including the non-registered voters in the sample also helps to skew the poll away from an accurate reflection of a Georgia primary. It’s like asking the people who were invited to the wedding and didn’t go how good was the catering. It is possible that they could know how good the catering was, but you’d be better off asking someone that was there.
A second potential, and I would be tempted to say likely, flaw is a confirmation bias. The AJC has not given a lot of ink to the various Republican candidates. On the flip side, the AJC has also given more ink to Nunn and not any of her Democratic challengers. This information diet could influence the sample more towards the Democrat side of the spectrum because there is only one name, while the sheer number of Republicans are causing a pan-Republican candidate to be seen by the respondent.
While the methodology used would be appropriate in some cases, it would appear to me that this poll is rife with problems. Some are definitely nitpicking, but others are more significant and cause me to doubt the accuracy of the results.