When You Know How Polls Work, You Realize that the Early Polls Don’t Matter
Despite four criminal indictments with 91 felony charges, recent surveys show Donald Trump leading Joe Biden narrowly. Or by as much as 10 points. Or trailing by a point or two. Or down by five.
This array of results has left many Democrats and some Republicans anxious about their candidate’s prospects. They should take a breath and get back to work because early polls are worthless at predicting winners of presidential contests, let alone their margins of victory.
We can assess the early horserace polls for the last 12 presidential elections with the metric statisticians use to evaluate the accuracy of surveys versus the outcome—the early leader’s projected margin, whether that candidate won the election, and the winner’s margin of victory.
The presidential victor trailed in pre-convention polls in seven of the 12 elections, with those polls missing the mark by an average of 18.9 percentage points. In two other races, the election winners led in early polls, but the gap between their early leads and their margins of victory averaged 10.6 percentage points. The voters’ verdicts closely resembled early surveys in only three of the 12 presidential contests from 1976 to 2020.
Pollsters always acknowledge a “margin of error.” September’s The Economist/YouGov survey, which has Biden up five, has a margin of error of about three points, so it actually shows a Biden lead ranging from 2 to 8 percentage points. Meanwhile, the recent ABC News/Washington Post poll, pegging Trump’s lead at a whopping 10 points, has a four-point margin of error, meaning Trump could lead by as much as 14 or as little as 6. The margin of error is mainly a cover for the historical evidence that there simply is no statistical correlation between early polls and who wins the election.
The 1980 contest between Jimmy Carter and Ronald Reagan produced the most wildly inaccurate early polling, with the December 1979 Gallup poll showing Carter ahead by 24 percentage points in a race he lost by 9.5 points—a stupefying gap of 33.5 percentage points. Subsequent early polling wasn’t much better. Walter Mondale had a narrow 2-point edge on Ronald Reagan in a September 1983 Washington Post/ABC survey, which had no bearing on Reagan’s eventual 18.2-point popular vote victory, a 20.2-point shift.
In May 1988, Michael Dukakis led George H. W. Bush by 16 points in Gallup polling. Yet by the following November, Dukakis lost to Bush by 7.7 percentage points, a swing of 23.7 points in six months.
Bush had an illusionary early lead over Bill Clinton of 19 points in the March 1992 Gallup survey and then lost by 5.5 points—a 24.5-point gap.
John Kerry led George W. Bush by 12 in the February 2004 Gallup poll before the incumbent gained 14.4 ground points to win by 2.4. Mitt Romney led Barack Obama by 4 points in an August 2011 Washington Post/ABC News poll and then lost to Obama by 3.9 points, a shift of 7.9 points. 2016 was another 7.9-point change when Hillary Clinton led Donald Trump by 10 points in the September 2015 CNN poll and then narrowly won the popular vote by 2.1 points—but not enough to secure the Electoral College
The winners in 2000 and 2020 led in early polls, but the gaps between their margins in those surveys and the election results were significant. George W. Bush led Al Gore by 15 percentage points in the November 1999 Gallup poll, and while Bush won the 2000 presidential election thanks to a still-disputed Supreme Court ruling, he lost the popular vote by 0.5 points for a gap of 15.5 percentage points. Most recently, Biden led Trump by 10 percentage points in the November 2019 CNN survey and won in 2020 by 4.4 points, an early polling miss of 5.6 percentage points.
In only three elections since 1976, did the early polls get the winner right with a winning margin that differed little from early polling. Carter led Gerald Ford by 4 percentage points in the March 1976 Gallup survey and won the election later that year by the same margin (although Carter led Ford by 13 points in a May 1976 Gallup poll). In the 1996 race, Bill Clinton led Bob Dole by 10 points in the November 1995 The New York Times/CNN poll and won the election by 8.5 percentage points, a gap of 1.5 percentage points. And in the 2008 election, Obama led John McCain by 4 percentage points in the November 2007 Gallup survey and won a year later by 7.2 points, a modest 3.2-point difference.
But those three elections share no common factors such as incumbency or economic conditions to distinguish them from the other nine elections. Overall, the relationship between early polls and the final results of presidential elections appears to be random, and for good reasons.
Some people change their minds between early surveys and Election Day. For example, in 2016, the USC Dornsife/LA Times Presidential Election Poll embarked on a novel project in which the same 3,000 people were asked for their voting preference on a regular basis; Trump and Clinton exchanged leads several times over the course of the survey, with Trump gaining momentum at the end.
Sometimes, significant developments do intervene to change voters’ support. In 2008 polling, McCain caught up to Obama in early September, but Obama pulled away for good after the financial crisis detonated, and McCain responded awkwardly.
Moreover, we should not presume early polls are meaningful at all when many factors badly undermine their prospects for accuracy or meaning, as Michael Podhorzer discussed in the Washington Monthly last week.
The greatest challenge involves creating a demographic sample for early surveys that will reasonably correspond to the demographics of the people who end up voting a year later. Most pollsters rely heavily on the demographics of the voters in the previous election, but the composition of the voting electorate changes from election to election. In 2016, for example, turnout by young and rural voters increased, and African American turnout declined, developments that were recognized only after the election. The sample of likely voters is also unstable since as many as 40 percent of respondents change their intention to vote or not at least once during a campaign.
Geography can be tricky because swing state respondents are typically underrepresented in early national polls. Polls with small samples also struggle to measure support for third-party candidates, especially with small samples, and who generally attract scant support but nevertheless can change the outcome in close elections.
Political scientists also argue that the sponsorship of election polling by media outlets has contributed to the problem. Polling and media are businesses, so pollsters have to work with fixed, limited budgets that dictate rapid turnarounds and samples as small as 500.
More and more people also refuse to respond to pollsters, and non-responders can warp the data. Analysts also have found that political polls are sensitive to differences in nonresponse rates on weekdays and weekends, an issue hard to reconcile with media schedules. Most media sponsors also prefer a clean story with small numbers of undecided voters, so how a pollster deals with those who initially say they’re unsure will affect the results. The order of the questions also can lead to large errors. When a survey begins with questions about recent events or policy preferences—especially when most people are not yet focused on an election a year away—their feelings about the events or policies can figure prominently in how they respond to which candidate they prefer.
Finally, new technologies have reduced the accuracy of many early surveys. Some pollsters now rely on automated “interactive voice response” (IVR) technologies instead of the telephone questioning by real humans that used to be the standard. The problem is that many people find it easier to hang up or provide whimsical responses to IVR software. Internet polling is even worse since it relies on an opt-in system that often results in unreliable samples. But IVR and internet opt-in surveys are cheaper for pollsters and media outlets that pay for them. Happily, some pollsters continue to use human telephone questioning, including the Quinnipiac University polls, Gallup, and the Siena surveys sponsored by The New York Times.
All of these reasons contribute to early polling results that bear no relationship to an election’s outcome. So, both Democrats and Republicans can put aside recent horserace polls. They should also remember the historic record when surveys a few weeks or months from now show President Biden or ex-President Trump surging.
This essay appeared originally in Washington Monthly.