Brier score 1

If we are serious about measuring and improving, forecasts must have clearly defined terms and timelines. They must use numbers. In addition, one more thing is essential: we must have many forecasts. We cannot rerun history so we cannot judge one probabilistic forecast—but everything changes when we have many probabilistic forecasts. If a meteorologist says there is a 70% chance of rain tomorrow, that forecast cannot be judged, but if she predicts the weather tomorrow, and the day after, and the day after that, for months, her forecasts can be tabulated and her track record determined. If her forecasting is perfect, rain happens 70% of the time when she says there is 70% chance of rain, 30% of the time when she says there is 30% chance of rain, and so on. This is calibration and can be plotted on a simple chart. The diagonal line on this chart captures perfect calibration.

1 Picture-perfect calibration
If the meteorologist's curve is far above the line, she is under-confident—so things she says are 20% likely actually happen 50% of the time. If her curve is far under the line, she is overconfident—so things she says are 80% likely actually happen only 50% of the time. This method works well for weather forecasts because there is new weather every day and forecasts stack up fast.

2 Two ways to be miscalibrated: underconfident (over the line) and overconfident (under the line)
It works less well for events like presidential elections because it would take centuries—undisturbed by wars, plagues, and other shocks that perturb the true underlying causes—to pile up enough forecasts to make the statistics work. A little creativity can help. We could focus on the state level in presidential elections, for example, which would give us fifty results per election, not one. But we still have a problem. The many forecasts required for calibration calculations make it impractical to judge forecasts about rare events, and even with common events it means we must be patient data collectors—and cautious data interpreters. Important as calibration is, it is not the whole story because "perfect calibration" is not what we think of when we imagine perfect forecasting accuracy. Perfection is godlike omniscience. It is saying "this will happen" and it does, or "this won't happen" and it does not. The technical term for this is "resolution."

3 Well-calibrated but cowardly (top); well-calibrated and brave (bottom)
The two figures above show how calibration and resolution capture distinct facets of good judgment. The figure on top represents perfect calibration but poor resolution. It is perfect calibration because when the forecaster says there is a 40% chance something will happen, it happens 40% of the time, and when she says there is a 60% chance something will happen, it happens 60% of the time. However, it is poor resolution because the forecaster never strays out of the minor-shades-of-maybe zone between 40% and 60%. The figure on the bottom represents superb calibration and resolution. Again, the calibration is superb because forecasts happen as often as they should—40% calls happen 40% of the time. But this time the forecaster is much more decisive and does a great job of assigning high probabilities to things that happen and low probabilities to things that don't. When we combine calibration and resolution, we get a scoring system that fully captures our sense of what good forecasters should do.
Someone who says there is a 70% chance of X should do fairly well if X happens. But someone who says there is a 90% chance of X should do better. And someone bold enough to correctly predict X with 100% confidence gets top marks. But hubris must be punished. The forecaster who says X is a slam-dunk should take a big hit if X does not happen. How big a hit is debatable, but it is reasonable to think of it in betting terms. If I say it is 80% likely that the Yankees will beat the Dodgers, and I am willing to put a bet on it, I am offering you 4 to 1 odds. If you take my bet, and put $100 on it, you will pay me $100 if the Yankees win and I will pay you $400 if the Yankees lose. However, if I say the probability of a Yankees victory is 90%, I have upped the odds to 9 to 1. If I say a win is 95% likely, I have put the odds at 19 to 1. That is extreme. If you agree to bet $100, I will owe you $1,900 if the Yankees lose. Our scoring system for forecasting should capture that pain.
Glenn W. Brier developed the math behind this system in 1950; hence, results are called Brier scores. In effect, Brier scores measure the distance between what you forecast and what actually happened. Brier scores are like golf scores: lower is better. Perfection is 0. A hedged fifty-fifty call, or random guessing in the aggregate, will produce a Brier score of 0.5. A forecast that is wrong to the greatest possible extent—saying there is a 100% chance that something will happen and it doesn't, every time—scores a disastrous 2.0, as far from The Truth as it is possible to get.