538, polls, and betting markets aren't competitors and you shouldn't frame it as such. Polls-> 538 -> markets are a stack. They are built on top of each other and add additional information at each layer. Each layer *should* be better than the lower levels. If you want to compare, you should compare value-add. I suspect the markets are the least valuable. And calibration across states in one election is not sufficient for any judgment due to correlated errors across states. Markets would be very very bad without polls or models.
Two key points i would add: 1) you are only looking at ex-post calibration for a single election with a strong common component (GOP outperformed polls). In other elections (eg 2012) Dems outpeformed polls. 538 has done analysis across all their predictions in many cycles and shown they are at least unconditionally well calibrated. 2) I've bet on differences between 538 and betting markets in 538s direction for three straight cycles and its been profitable each time. Eg 538 was more bullish on Trump than betting markets in 16. Small sample of course but still.
So i think your addendum was the most crucial part. Betting markets have a strong pro-Trump bias, and when the polls miss in his favor, they look smart. However, this clearly isnt a calculated/accurate/correcting pro-Trump/red bias, as the outstanding markets in certain 2020 states show at the moment. When the polls don't miss towards R's, as in 2018, markets end up being way off.
My strongest point against markets--- they failed to understand the very predictable "red mirage" and so vastly overestimated Trump's odds in the live betting on the night of the election. At 10pm, Biden was +450, massively mispriced. I was able to make a huge profit off Biden +300 at around 10:30 because bettors were seemingly not aware of the effect of mail in ballots and took Trump leads in MI and PA at face value. Huge mistake that a more in-tune political observer could pick up quite easily. I then made significant profits in GA and WI thanks to bettors who couldnt understand/extrapolate outstanding vote totals. The betting markets are not sophisticated, and while they are interesting to compare to the models, I think one needs to evaluate the efficacy of each's accuracy over many more election cycles.
538, polls, and betting markets aren't competitors and you shouldn't frame it as such. Polls-> 538 -> markets are a stack. They are built on top of each other and add additional information at each layer. Each layer *should* be better than the lower levels. If you want to compare, you should compare value-add. I suspect the markets are the least valuable. And calibration across states in one election is not sufficient for any judgment due to correlated errors across states. Markets would be very very bad without polls or models.
Two key points i would add: 1) you are only looking at ex-post calibration for a single election with a strong common component (GOP outperformed polls). In other elections (eg 2012) Dems outpeformed polls. 538 has done analysis across all their predictions in many cycles and shown they are at least unconditionally well calibrated. 2) I've bet on differences between 538 and betting markets in 538s direction for three straight cycles and its been profitable each time. Eg 538 was more bullish on Trump than betting markets in 16. Small sample of course but still.
Can you share your dataset of predictions from each source with the actual outcome by state?
So i think your addendum was the most crucial part. Betting markets have a strong pro-Trump bias, and when the polls miss in his favor, they look smart. However, this clearly isnt a calculated/accurate/correcting pro-Trump/red bias, as the outstanding markets in certain 2020 states show at the moment. When the polls don't miss towards R's, as in 2018, markets end up being way off.
My strongest point against markets--- they failed to understand the very predictable "red mirage" and so vastly overestimated Trump's odds in the live betting on the night of the election. At 10pm, Biden was +450, massively mispriced. I was able to make a huge profit off Biden +300 at around 10:30 because bettors were seemingly not aware of the effect of mail in ballots and took Trump leads in MI and PA at face value. Huge mistake that a more in-tune political observer could pick up quite easily. I then made significant profits in GA and WI thanks to bettors who couldnt understand/extrapolate outstanding vote totals. The betting markets are not sophisticated, and while they are interesting to compare to the models, I think one needs to evaluate the efficacy of each's accuracy over many more election cycles.
You compare the performances based upon where things stood on Nov 2. How do they compare, say, 3 weeks before election day? Or 2 months before?