Aggies2009 said:
ABATTBQ11 said:
P.U.T.U said:
538 is wrong a lot more than they are right, and they are wrong A LOT
Do you understand how forecasts and probability work?
I'm not him, but why don't you explain it?
If a meteorologist says there's a 70% chance of rain and it doesn't rain,
he's not necessarily a bad forecaster.. If it rains
every time he says there's a 70% chance, he
is a bad forecaster. The difference is, and how forecasting is ultimately measured, is in how well your forecasts match up with outcomes over time. The meteorologist is good (actually more like fantastic) if his 70% predictions result in rain 70% of the time, his 30% predictions result in rain 30% of the time, and so forth.
In the case of his 70% forecast, if it doesn't rain,
he isn't wrong per se. There's still a 30% chance it does not rain, and even if he gives you a 70% chance, he's not saying it will rain or that there is a guarantee even though the probability of rain is greater than the probability of no rain. You may interpret it that way, but that isn't the meaning of the forecast.
If you do view his forecasts in binary or classification terms, either it will or it will not rain, he is "wrong" quite a bit. Let's say he makes 100 forecasts, 50 times he predicts a 70% chance and the other 50 he predicts a 30% chance. He is an excellent forecaster, and those predictions are accurate over the 100 forecasts. 70% of the 70% predictions have rain, and 30% of the 30% predictions have rain. That said, you view those 30% no rain and 30% rain outcomes as "wrong" because the lower probability turned out to be correct. So he's "wrong" on 30 forecasts out of 100, but that doesn't make him a bad forecaster.
Dice are another example. There are known and calculable probabilities for rolling dice. If you roll 2 dice, I say you have an 83% chance of not rolling a double, and you roll a double six,
I'm not wrong. The math says so, and your rolling of a double does not disprove that fact. You still had a 17% chance of rolling a double. Rolling three doubles in a row has about a .5% chance of happening, but even at those low odds it still has a pretty good chance of happening if you roll long enough.
When making point or regression forecasts, you include a confidence interval (think the probability cone for hurricane tracks). The actual value of the outcome should fall within the interval for the given confidence percentage (95% of the time for a 95% interval). As long as that remains true, you aren't wrong on the 5% of instances when the actual value falls outside the interval. If it falls outside more or less often over a long enough time, then and only then are you "wrong" and do you need to revisit your forecast. If it falls outside too much, then what you think the variance between outcomes is is probably too low. If it falls within too much, then the variance is likely smaller and you can tighten the interval to make the forecast more useful.
When it comes to 538's political forecasts, they make probability estimates based on many different polls and they're historical accuracy on vote totals to determine election outcomes. These have confidence intervals. They actually do pretty well at this, and the votes fall in line with what you would statistically expect with a frequency you would statistically expect based on those confidence intervals.
Lots here line to say they whiffed on the 2016 election because Trump won and they only gave him a 28% chance, but that is analysis of wrong. Trumps chances were based on the probabilities for individual states, and he needed to win in several tight state races where he wasn't favored to win the EC. Conditionally,
those individual state race odds multiply if they all need to happen together, so his odds were actually low. However, he still had chances to win each one, and he did. In each case, the outcome was within the confidence interval and margin of error.
Some people here will call this "hedging," but it is not. Hedging involves making a second bet or prediction opposite another to limit exposure or to secure winnings based on new/different odds. Forecasting or predicting a range of outcomes is just statistics. The measure of how good you are is how often your ranges are correct in regards to how confident you are. 538 is actually pretty good about this.