April 30, 2020
As pollster for more than 30 years, one of the more common questions I get is whether there is a difference between “likely voter” and other types of polls?
Many public and private polling firms use a series of qualifying likely voter questions such as the Perry-Gallup method which asks a series of seven questions and then ranks the voter’s likelihood based on their responses.
Other firms, like my self, use voter registration files that lists the voter’s past voting history in different types of elections, such presidential or local races. And then there are pollsters who interview any adult who answers the phone.
Pew Research studied this issue a few years ago and found that qualifying question techniques were not nearly accurate as voter registered files, which was no surprise to me. The reason is simple: if a person voted in the last election he or she has an 85% probability of voting in the next election.
So if you are a political pollster, who regular surveys for candidates, the choice is simple: use a voter registration list and randomly select respondents that have voted in at least the last election.
Public pollsters that work for news organizations, on the other hand, use one of three different types of modes. Some use the “all adults” samples, which is any adult in the household who says they will vote in the election. Others prefer only registered voters and still others use what are classified as “likely voters.”
My interest here is whether there are any significant differences between the three different modes. In other words, do the different sample types actually produce different results that are statistically significant?
The general consensus suggests that likely voter surveys favor Republican candidates, and that among all-voters they favor the Democratic candidates. With my curiosity at full throttle, I decided to test these assumptions using Analysis of the Variance (ANOVA) which is a statistical method to test whether the differences between two or more groups are real or only a random occurrence.
I collected 28 different national public surveys done over the past two months that contained the Biden vs. Trump presidential question and designated the type of sample used: all adults (AD), registered voters (RV) or, likely voters (LV). Below is the ANOVA report and voter percent for each Biden and Trump.
|1 (All Adults)||Mean||43.00||35.50%|
|2 (Registered Votes)||Mean||48.38||41.00%|
|3 (Likely Voters)||Mean||52.13||44.38%|
|Sum of Squares||df||Mean Square||F||Significant|
Table 1 displays the three modes: all adults (1), registered voters (2) and likely voters (3). With all adults (mode 1), Biden has a 7.5% lead over Trump. Registered voters (Mode 2) shows Biden with a 7.4% advantage and with likely voters (Mode 3) Biden has a 7.75% lead.
All very close. Table 2 shows the statistical analysis, which is highly significant at the <.001 level, meaning the three modes are statistically different.
What is missing here are the undecided voters, but we can calculate it by adding the two candidate’s combined percentages and subtracting it from 100. The all adults poll has an undecided of 21.5%. The registered voter survey has an undecided of 10.6% and finally, the likely voter poll has an undecided vote of only 3.5%.
Although this experiment did not produce a political bias of one mode over another as some have suggested, it does reveal that voter modes do significantly effect the number of undecided voters.
All of this makes perfect sense, since the political knowledge of each group should be reflected in there past voting behavior. Likely voters should know more about the candidates than just registered voters, and even more so than non-voters. In other words, the likely voter survey is more likely to be accurate than the other two modes.
So if it’s a political event you are trying to predict, this analysis recommends the likely voter survey over the other two options. After all, what’s the point of doing a political survey if it doesn’t accurately tell you who’s winning!