A Short Critique of the 538 Electoral Model

As of 8/17/20, 538's state of the 2020 presidential race.

I want to give credit where credit is due.  Nate Silver was among the first to stress the importance of taking the average of polls to get the state of the race. In 2016, his website, fivethirtyeight.com, was one of the only election models to give Trump a significant chance, about 29%, of winning the election.  It's also worth noting that those who don't actually follow polls and modeling closely continue to falsely accuse Silver of saying that Hillary was a lock to win.  But to many in the press, 538 is seen as the gold standard of political modeling.

Fivethirtyeight has published their 2020 model, and, as you can see in the snapshot above, Biden starts the model roughly where Hillary left off, with about a 72% chance of winning, with the caveat that this does not include the possibility that Trump steals the election outright.  In their "now-cast" (which they don't publicize as much), Biden has about a 93% chance of winning if the election were held tomorrow.

This differs substantially from my own projection, which gives Biden a ~99% chance of winning:

Why the difference? Well, I have a couple of thoughts on that.

1) Secret Sauce - Let me give you one example, the popular vote margin. They currently estimate that Biden will win by 7 points, but their own polling average currently has Biden up by 8.4. That's because they use all sorts of other information in their modeling, including economic data, complex models for incumbency, assumptions about how far we are out, and so on. The problem is that all of the presidential modeling data is based on just a handful of elections, a dozen or so in the modern polling era, which means that at some level they are simply over-fitting to the data.

When you put too many variables into your model, it may look fancier, but those variables become less and less well justified.  Indeed, a simple interpretation of history – that undecided voters break for the challenger – would suggest that Biden's current lead is an underestimate of the final margin.

Furthermore,  their model apparently is pretty easy to fool, as I note in my post-mortem from 2018. Based on early results, 538 decided all was lost:

while others (including mine) never budged.

The point is, when your algorithm becomes too complicated, it's not always clear what you're really predicting.

2) Overestimating uncertainty - Besides random polling errors, there's the possibility of all polls being off by a certain amount. That number was about 3% in 2016, which is why people claim the polls were "wrong." To give you a sense of how much polls are typically off, here are the polling errors from about 2 decades of House races:

While individual polls can be off by a lot, in any given year, polls tend to be very accurate.  The average of polls are typically off by only about 2 points!  For senate and presidential races, you can do a similar estimate, and the number, as I noted, is closer to 3 points.

3% is a very small error, but 538 seems to assume a much larger number, 4 or 5%. This is a big deal because it allows them to be "right" without taking a real chance. 

Think of it this way: imagine an election where candidate A is leading candidate B by 10 points in the polls. Normally, we'd say that A is more or less a lock to win (realistically, > 99%). But if I assume the possibility of very large polling errors (of 10%), I'd still say that A is more likely to win (thus getting credit for being "right" when they do), but I'd predict a ~15% chance of B winning as well, so I get credit for being "right" (or at least giving a non-negligible chance) even if B wins. See? I get credit either way!
538's model is fine overall, but every tick upwards or downwards is going to be endlessly debated. I'm suggesting to you now that it's far less scientific or rigorously justified than you might guess at first inspection.

But in some sense, 538's sin is even worse than this. Because (looking at objection #1), if your complicated model really does add additional information besides polls, you'd expect the scatter in your errors (and thus your uncertainty in predicting the future) to be even smaller than what you'd naively guess from polls alone.

Take everything with a grain of salt.  But 538 isn't the only game in town, and there's the very real fear that Republicans will try everything to steal the election.  But the silver lining is that 72% seems to be a lower limit to Biden's chances at  this time.