Recent polls in seven key states show a surprisingly tight presidential race: 124 of the last 321 polls in those states – almost 39% – show a margin of 1 percentage point or less.
In fact, state polls show not only a surprisingly tight race, but an improbably tight one. Even in a truly equal election, the randomness inherent in voting will produce more varied and less clustered results—unless state polls and polling averages are artificially close due to decisions made by pollsters.
The results of the survey depend on the opinion of the voters and the decisions of the voters. Decisions about how to draw polls to match the expected composition of the electorate the results of the survey can carry up to 8 points. This is true even if pollsters make perfectly reasonable decisions about how to measure survey data, as pollsters are forced to consider new methods and ideas for reducing and weighting response rates after the 2016 and 2020 polls.
But the fact that so many polls report the same differences and results raises a troubling possibility: some pollsters are adjusting in such similar ways that these choices cause the results to converge, potentially creating an illusion of certainty — or that some pollsters may even be guiding their own results. they look after the results of others (ie, “herding”).
If so, the artificial similarity of the polls may create a false impression that may not materialize on polling day. A very close election awaits us. But at the same time, there is a chance that one or the other candidate could sweep every state and win the presidency with some ease, at least compared to the evenly balanced picture in the polls.
What should we see in a world of random perfect voting?
In a perfect world for polling—a researcher’s paradise where every voter can be contacted and every contacted voter responds—we can use math to calculate how much variation there must be because voters are randomly selected to participate in a survey.
If a race in this world were truly tied 50%-50%, polls wouldn’t all produce 50%-50 split results. Imagine that pollsters in this world conducted 100 identical surveys among 863 randomly selected voters (this is the average sample size of this year’s state polls). The results of 95 of these polls would show the candidates receiving 46.7% to 53.3% support – although in this imaginary world we know the race is actually tied at 50%. The other five polls will show candidates earning something larger or smaller outside of that range.
This variation is known as the “margin of error” in polling, which is how much a random selection of voters who always respond can affect the poll’s score for a candidate.
Because each candidate’s support varies randomly, these polls predict margins in close races ranging from -6.6 to +6.6 for 95 out of 100 polls (with larger margins for the other five).
It’s important to note that the range of margins we can expect in a close race (and in a perfect voting world) is larger than the margins in swing states in 2020. Even under ideal voting conditionsit is difficult, if not impossible, for a poll to be very informative about who is leading a tight race. And that’s certainly a lower bound for what we should see in the messier real world, where polls differ in how respondents are selected, contacted, and measured by what voters believe will result in 2024.
We can also calculate what proportion of the 863-person poll we should expect to show different margins in a truly tied race. Rounded to the nearest percentage point, about 11% of polls in a tied race should show a tie.
This means that due to randomness and the margin of error, almost 9 out of 10 polls of a tied race should not actually show a tied vote.
About 32% of polls should have a difference of 1 point or closer, 55% should have a difference of 2 points or closer, and 69% should have a difference of 3 points or closer. Even in a 50-50 race, about 10% of the polls should have a margin of more than 5 points due to random chance – about the same percentage that shows a (rounded) tie!
With enough polls, the predicted margin should also resemble a “bell curve” normal distribution—with similar numbers of polls showing both candidates leading.
What do we see in swing state polls?
Actual swing state polls show less variation than the benchmarks we would expect in a perfect voting world. Out of 321 polls in seven swing states, only 9 polls (3%) report a difference of more than 5 points. Even if every race were tied—they’re not—we’d still expect, by chance, to see 32 of the 321 polls by more than 5 points.
Visualizing how reported voting margins compare to what we would expect in a perfect voting world strongly suggests a “drift” of state voting margins that fluctuate around statewide voting averages. Of those 321 state polls, 69 (21%) report a clear tie, and 124 (39%) report a difference of 1 percentage point or less. Both of these numbers are roughly twice what we would expect in a perfect voting world where the only source of variability is the random selection of responding voters.
Pennsylvania is perhaps the most disturbed state. Of the 59 polls there, exactly 20 (34%) show an exact tie, and 26 (44%) show a difference of 1 point or less. 5-point margin due to randomness, we see only 2 of 59 Pennsylvania polls (3.3%) with a margin of more than 5 points.
Even in places where the polls are not as tightly knit, as in Arizona, Michigan and Wisconsin, there are still far more polls around the polling average than we would expect, and too few polls by large margins.
What’s going on?
The concentrated margins we see in the flash state polls probably reflect one of two possibilities.
One possibility is that pollsters may sometimes correct a survey result that seems “odd” to them by selecting a weighting scheme that produces results closer to those of other surveys. Risk-averse respondents have strong incentives to do so. Unless the pollster conducts many polls and is confident that the effects of randomness are moderated, pollsters may fear the reputational and financial costs of getting the wrong result due to randomness, as pollsters are judged on voting accuracy.
A risk-averse pollster with a 5-point lead in what they perceive to be a tie may choose to “adjust” the poll results to something closer to what other polls show, so that their outside opinion will negatively affect their reputation relative to their competitors.
Another possibility is that some of the tools pollsters use in 2024 to address 2020 polling problems, such as weighting by partisanship, past voting, or other factors, may correct for differences and reduce variation in reported poll results. The effect of such decisions is subtle but important because it means that the similarity of the polls is driven by the decisions of the pollsters, not the voters.
And if those assumptions are wrong, something that won’t be known until after the election, the risk of a potentially large polling error increases as the variation across polls decreases.
Why is this important?
The fact that so many swing state polls report similarly close margins is a problem because it raises questions about whether the voting is tied in those races because of voters or pollsters. Will 2024 be close to 2020 because our politics are stable, or will the 2024 polls only look like 2020 results because of decisions made by state pollsters? The fact that the polls appear more tightly packed than we might expect in a perfect voting world raises serious questions about the second scenario.
The reported polls and polling averages create a consensus that the race will be very tight and we will see a similar result in 2020. Maybe that’s true. It would be great to have a survey in 2024 to successfully address the concerns of 2016 and 2020.
However, the fact that the surveys all report such similar margins does not make it more likely that these margins are representative of the final result. In fact, it raises the possibility that the results of the election could be unexpectedly different from what the state polling team and polling averages suggest.