Blog Post Politics

WINS 2020 Election Roundtable Part 6 – What Went Wrong With Political Polling In 2020, And How Can It Be Fixed?

This discussion is one segment of a multi-part series centered around the 2020 Presidential Election.

Interviews have been transcribed from a digital roundtable that took place on 11/19 that included panelists, Democratic pollsters, and WINS team members Bernard Whitman and Matt McDermott.  The roundtable was moderated by Scott Kotchko.

SCOTT:  2020 was a bad year for campaign polling.  Republican and Democratic campaign pollsters missed the mark on understanding what they thought voters were going to do, and how they were going to vote.  Media pollsters even more so.

And so it was a bad year for the pundit prognosticators like Nate Silver.

But, interestingly, it was also a terrible year for the poli-sci prognosticators who don’t use polling in their models, like Rachael Bitecofer. (Note: The link to Prof. Bitecofer’s final Senate forecast has been removed from the Niskanen Center’s website ¯\_(ツ)_/¯)

So what happened with political polling this year?  Is it fixable? What should people know or understand about it?

MATT:   I am increasingly over time convinced that the polling error that exists and there certainly is an error, is solely correlated to Donald Trump’s name on the ballot.

If you look sort of over the last decade of polling, there is not a systematic bias against polling against Republicans.  It just doesn’t exist. It does exist in two election environments, 2016 and 2020. And the only thing similar between those two election cycles is Donald Trump was on the ballot. If you actually look at polling in the 2018 mid-term, democrats actually were under polled in that election. And the bias in that mid-term was actually favored democrats, not republicans.

And so, the two cycles we have, where there was a systematic issue, were the two elections Donald Trump was on the ballot. The question then becomes why is that? And you can see it, particularly this year, but there were shades of this in 2016 as well. There is no doubt, even if he’s a loser, that Donald Trump can turn out disaffiliated, disaffected, non-college, white girl voters in a way that no candidate has been able to do in the modern era.

You look at going back to one of your earlier points Scott, which is true. Is that I think Democrats have always said, turn out for the sake of turnout is good for the party. That merely by turning people out, Democrats can win the tide and win the majority of votes. What you saw in this election is, turnout on the margins can help Democrats.

On the flip side, turnout benefits Republicans as well. And Republicans got a huge surge of voters this year, from disaffected white voters who typically don’t vote in elections.

What seems to be happening in polling when trump is on the ballot, is that, that surge of disaffected white girl voters is not picked up in polling and he’s able to pad his margins. I think the question for me though is, does that polling error exist without Donald J. Trump on the ballot? And I think sort of, after this election cycle, we can confidently say that regardless of where this country heads in the next four years, Donald Trump will never be on the ballot again, anywhere in this country.

I hate to sort of be tongue in cheek about it, but the solution is just…well… Donald Trump doesn’t run again in election , right?

If that is the fundamental problem, which it increasingly seems like it is, then either it gets fixed by Donald Trump not running again or republicans figure out how to repeat the same thing Trump has been able to do.

And I’m firmly of the belief that they’re going to fail in doing that.

BERNARD:  I do think there is something anomalous about Donald Trump. Look, the Republicans in the 2016 primary couldn’t figure him out. We couldn’t figure him out in the 2016. We barely could figure him out in 2020. Thank god we did. I think he does bring some anomalies to the equation and they get particularly hard to identify who is actually going to show up to the polls.

To get a little bit more into the weeds about it, I do think it’s more of a trump issue than we might even imagine. So where did pollsters mess up? How do we of fail to adjust for Trump being on the ballot?

In general I think pollsters probably weighted data too much this cycle. Weighting is the process of adjusting a sample to ensure you have enough of different groups of people.

I think the turnout models also didn’t really sort of figure out who is actually going to turn out. I think we relied too much on past as prologue. We relied too much on past voter history to predict who’s going to vote this time, which means when we had a 14% increase in turnout or whatever it was, with so many people mailing in the ballots, we sort of didn’t figure out who was actually going to turn out. We relied too much on past history.

And we relied too much on saying, we didn’t have enough of these people in this poll. And the last time, it was – we had more educated people or more uneducated people or whatever it was. I think we adjusted, we fiddled, we played with the data too much. We didn’t let it just fall as it is, au natural, which is what Ann Seltzer did in Iowa, although Iowa is not a diverse state and therefore easier to poll.

But I think it’s a combination of Trump being anomalous, too much emphasis on past turnout, models to predict future turnout of this election, and then too much reliance on waiting all combined. In some states, we got almost exactly right. The national polls were actually correct with Hilary Clinton. They were wildly off in 2020, and in some states, pollsters were wrong.

SCOTT:  From my perspective I would say that the problems with political polling in 2020 come down to a few interrelated things.

Look the single biggest issue and dirty secret in the political polling world is that non-response bias in phone polling has been a problem for 30 years and is now getting worse.

This refers to the fact that (thanks to telemarketers starting in the 1980s) many of us don’t want to answer a call from a random number and take 20 minutes to chat with a stranger.  And so the people who do, don’t necessarily look like the people who vote in elections.

You can fix this, to some extent, as Bernard said, by weighting.  But in other regards you didn’t really need to fix it, because there wasn’t really a partisan split in not wanting to be bothered with random phone calls.

That seems to have changed in the COVID-19 and Trump era.  My theory is that response rates for Democrats, particularly professional-class suburban Democrats who were more likely to be able to work from home during the pandemic, and more likely to comply with stay at home orders went up this cycle.

On the flip-side, I would venture to guess that response-rates to polling among Republicans, particularly white men, decreased.  And I’d guess that some smart academics in the next few months will study why that happened and point to growing lack of institutional faith in government, politics, and shared civic duty during the Trump Era.

And just to be clear, this is totally different than the “Shy Trump Voter” theory which claimed that there were lots of Trump supporters who were not comfortable admitting it to other people outside of the voting booth.  That theory was always total bullshit, and the Trump boat parades and car caravans of 2020 prove it.  What I’m talking about here is that the woman with 10 Biden/Harris signs on her lawn was more than happy to talk to a pollster, while the guy with 10 MAGA signs on his lawn hung up.

The last piece of the puzzle, in my conjecture, is that I think pollsters (and the media and most of America) fell into this demographic trap of thinking about voters, especially white men, along the lines of working-class vs. college-educated.  That’s always been a false dichotomy, and to better understand this spectrum I’d suggest reading Patrick Wyman’s excellent piece American Gentry.

Anyway, I’d guess many pollsters over-weighted Trump support among working-class white men to correct for 2016 mistakes, and undercounted Trump support among more affluent and/or college educated white men.

MATT:   Yes.  On the topic of lack of institutional faith and civic duty, there is no amount of waiting, modeling, tweaking around the edges that can be done if fundamentally there’s a cohort of people in this country who, because of the discord and distrust that have been sowed into our institutions are refusing to participate in a poll, right? That is a very different problem, and I tend to think that was a bit part of the problem in political polling in 2020.

SCOTT:  The last thing I’d like to say on this subject, because I get a lot of questions about this from friends, family, clients.  What people need to understand about political polling is that despite what happened this year, polling over the last 2 decades has been far, far more accurate than it was for the 50 years prior in predicting outcomes of local, state, and presidential elections.

Even if the quality and accuracy varies, I think it is good for the country and our democracy that we have more polling, more publicly available polling, and more people like Nate Silver and Nate Cohn who aggregate polling as journalism.  Because what national political media coverage looks like otherwise is pretty atrocious.

It may not always be perfect, but without polling insights, all we have are the Chris Cillizzas or Salena Zitos of the world, and “journalists” who go to a diner in the Milwaukee suburbs, talk to three people (two of whom are Republican operatives) and then file a story on what Swing voters in America are thinking.

Political polling data may not always be 100% accurate, but at least it brings some semblance of science and objectives facts into the equation.

BERNARD:  And the other thing I’d add is that we sort of, a lot of people forget, when we say the margin of error, people don’t hear that. They don’t realize the margin of error, even on a large poll, with a margin of error, might be like 2.5-3%. It’s plus or minus.

It means the result could be up to six points off of what we’re actually saying. People just sort of ignore that, they forget it. And then they go, why were you guys off? It said X and it was actually X plus or minus three or four.

Well, sorry folks, that’s called margin of error.

MATT:   I’ve always argued that the issue itself is not with polling, but in how it’s consumed.

At the end of the day, all three of us are well aware of this. There are a number of different reasons you do polling. There are a number of different things you get out of it. And by and large, the way in which polling is consumed today through a political lens, nationally, is topline numbers.

Who’s up? Who’s down? What does the ballot look like?

The reality is polling actually provides you a lot more information. It provides information about what issues people care about.

What are people prioritizing in their lives? Why are people making decisions in the household that they do?

All of that is vastly, in my opinion at least, vastly more important in terms of what you’re getting out of polling. And if all we’re doing is looking at, within a margin of a point or two, what is a congressional race or a senate race or presidential race looks like, you’re sort of missing the forest for the trees.


Continue to Part 7: Does Scott Owe Hillary Clinton An Apology?