As in 2016, the final results of the presidential election had Donald Trump massively over performing when compared to the expectations based on the results of almost all the major polling outfits. There will be a temptation to beat up on the pollsters who got it so wrong and we can expect Nate Silver and others to be relentless grilled as to why the polls were so off. The ire aimed at them may be less this time because the final outcome of a likely Biden win is looking like it will be correct, unlike in 2016 where the prediction was of a Clinton win. But there will, and should be, much soul-searching as to why the polls were so off again.
Although they were burned by the experience of 2016, before this election many of the pollsters said that it would not happen this time because they had adjusted their samples to include more of the voters who were likely to vote for Trump and had perhaps been undersampled the previous time. Clearly that change did not not make much of a difference.
I think the problem is that Trump has the ability to fire up voters to come out in droves and vote for him and that these voters are also less likely to reveal their intent to pollsters. In other words, the ‘shy Trump’ voter who is enthusiastic about him but is unwilling to reveal that to pollsters is a real phenomenon. As evidence, look at the mid-term elections in 2018 when Trump was not on the ballot. The pollsters predicted pretty accurately the outcomes of the House and Senate races. This time he was again on the ballot and all those who turned up purely because of him also ended up voting for Republicans in down ballot races, thus negating the rosy predictions for Democratic gains in the House and Senate races as well.
While it is true that the Republican party has long been creating a fertile ground for a racist bigot like Trump to take it over, he brings to the role something that I think may not have been anticipated, and that is that a grifter and lying sociopath would be able to create a cult-like devotion among his followers that would allow them to ignore and even gleefully embrace the worst elements of human nature.
Trump is a black swan and statistical models are just not designed to easily incorporate black swans. I am not a statistician but I don’t know if pollsters can take into account the effects of singular phenomena like this. If they had realized after the 2016 polling debacle that he was sui generis, one thing they might have done is make a purely ad hoc adjustment to the polls, by simply taking the difference between the polls and actual results in the 2016 election and modifying this year’s poll results accordingly. That may have got them closer in their predictions to the final outcome. But that would be ugly guesswork and professional statisticians might well be hesitant to do something so crude.
In future posts I will analyze what the results may mean for both Democratic and Republican parties. I think there is room for hope. The Democrats should not over-react to their underperformance in the Senate and House races and start blaming each other as they did today but wait until a more sober analysis is done.
What should we do to win the senate in 2022?
The moderate democrats were blaming progressive democrats for the losses in house seats. Blaming it on “social” this and “socialized” that. Yet Stacey Abrams may be the person most responsible for turning Georgia blue (purple) this election. She and her supporters helped register 800,000 new voters in Georgia. Perhaps it’s not the progressives that should be blamed for any house losses.
No progressive seats were lost in the house. Only DiNOs. The should take heed, but will they?
@Myra Greenwood
Do what Abolitionists, Free Soilers and progressive Whigs did in 1854 and get busy creating a brand new party that elected Abraham Lincoln six years later.
It isn’t the pollsters who are the problem it is how the news brings the polls.
For example 538 puts Biden winning at 7/8 with 326 electors to be the most likely. Then the news comes around: “BIDEN WILL WIN WITH 326 ELECTORS!!!!!!!!!!!!” or the pollsters come out with there is a 54% chance that Florida will go to Biden, the news goes “FLORIDA WILL VOTE FOR BIDEN !!!1!@!!!!!”
Then when Florida goes to Trump it isn’t the news that is wrong but the pollsters while 54% is about as close to a coin toss you can get due to uncertainty.
When Biden doesn’t get the landslide (immediately) it is the pollsters who are wrong not the way the news was brought. When it looked like Biden would barely get to 270 electors it wasn’t the way the news brought the polls that was wrong but the pollsters.
The whole problem is that if it is on TV it has to fit into a 20 second segment between the latest shooting and something embarrassing that a famous person did. If it in a newspaper you get an inch, maybe two inch for the latest poll. Online you get a link to go look for yourself which most people won’t do. They can’t even get their talking/writing heads out of the way to explain in a bit more nuanced way that for example 538 ran thousands of simulations and that most of those, being X percent, give the result that Biden will win with 326 electors but that the results give an 1 in 8 chance that Trump wins.
It is same with the house and senate. If you take time to dig through the numbers you get something way more nuanced then the news gives but you need to do it yourself instead of being able to get an informative digest through what ever news source you use.
And even worse/better the polls might not be wrong. As someone succinctly put it: “The numbers don’t matter, what matters is how many of those people are motivated enough to get and vote instead of answering who they’d vote for.”
Mano,
I think the more apt comparison would be to The Mule who threw a monkey wrench in the psychohistory of Isaac Asimov’s hero Hari Seldon.
Also, @Tadas No. 2: That’s true, you’re absolutely correct.
hyphenman, you read my mind. The Mule was where my thoughts went too.
I keep reading that phrase as “sus generis.” Which aptly translates roughly to “pig-kind.”
.
@Who Cares: Democrats in Disarray! — NYT, et al.
hyphenman @4:
We can only hope.
Interesting. But where my mind went to four years ago was Nehemiah Scudder!
@6 hyphenman
I was mentally objecting to Mano’s use of the term “Black Swan”. I think you have hit it. A Black Swan is not an unusual event that we know can occur but rather something that is outside our theoretical knowledge. All swans in Europe and North America are white. Therefore all swans are white until one suddenly arrives in Australia.
I think your comparison to the Mule is a perfect example of a Black Swan. I’ve seen references to the Covid-19 pandemic as a Black Swan but it’s not. We have had lots of epidemics and pandemics, Covid-19 just another one.
@ 10 Steve Morrison
To be honest I didn’t think of that 4 years ago but I have been watching the US for some time and it definitely is worrying. I think the idea came at least 20 years ago.