A trader can be lucky once and calibrated over hundreds of forecasts. Prediction markets need a scorecard that punishes overconfidence and rewards honesty. The Brier score is the standard metric for probabilistic forecasts—and cousins appear in oracle games and research platforms where proper scoring incentivizes truthful reports.
Definition
For a binary event, submit forecast probability f (your p at forecast time). Outcome o is 1 if YES, 0 if NO. Brier score BS = (f − o)²—lower is better. Perfect is 0; certainty on the wrong side approaches 1.
If you forecast 90% and YES happens, BS = (0.90 − 1)² = 0.01—excellent. Same 90% when NO happens: (0.90 − 0)² = 0.81—disaster. Forecasting 55% whether YES or NO lands near 0.20–0.30. Saying 50% when unsure scores 0.25—mediocre but not catastrophic. Saying 95% and missing destroys leaderboards.
Multi-outcome extension
Categorical markets use multiclass Brier: sum over categories i of (f_i − o_i)², where o_i = 1 for the realized outcome and 0 elsewhere. Your vector f should sum to 1.
You assign A=0.50, B=0.30, C=0.20 and B wins: BS = 0.25 + 0.49 + 0.04 = 0.78. Sharper A=0.10, B=0.75, C=0.15 yields 0.01 + 0.0625 + 0.0225 = 0.095—much better. Later modules on categorical contracts use the same scoring logic.
Brier versus P&L
Brier answers calibration: were your stated probabilities honest? Trading P&L answers whether you bought low and sold high. You can beat the market on Brier yet lose money if prices were efficient and you lacked edge; you can win money with awful Brier if you were lucky on a few 95% calls.
Treat market c at trade time as a forecast and score (c − o)² to benchmark the crowd. Edge without calibration: buying YES at 30¢ when true rate is 40% can profit while saying 95% wrecks Brier even if you occasionally win.
Election night sequence
Forecast “wins Pennsylvania” at T−30d with f = 0.52 (market c = 0.48), update T−1d to f = 0.58 (c = 0.55), outcome YES. Your Brier contributions: (0.52−1)² = 0.2304 and (0.58−1)² = 0.1764; average ≈ 0.2034. Market at T−1d: (0.55−1)² = 0.2025—you barely beat the market on calibration that day despite being directionally right. Trailing a moving tape flatters scores; pre-register forecasts.
Proper scoring and honesty
A rule is proper if truthful reporting optimizes expected score (minimize Brier). Platforms care because oracle voters and research panels face incentives to avoid “100% unless sure.” Improper rules reward bold lies or hedged wording.
Calibration versus resolution
Calibration: bucket all your 70% claims and check how often YES occurs—ideal buckets hit ~70%. If 40 forecasts in the 0.6–0.7 bucket hit only 55%, you are overconfident.
Resolution (sharpness): always saying 50% yields Brier ~0.25 but tells the market nothing. Good forecasters are calibrated and sharp.
Over 100 events, mean Brier 0.19 versus always using market c at 0.21 versus always 0.50 at 0.25 versus always 0.95 YES at perhaps 0.38 when base rate is 40%—beating market Brier by 0.02 over 100 events is strong if rules and samples match.
Brier and trading decisions
If you beat market Brier for 50+ events, trust your p more cautiously for Kelly sizing. If you lose consistently, shrink toward c or stop discretionary bets. Great Brier with negative P&L means prices were efficient—you lacked edge, not skill. Bad Brier with positive P&L is luck; do not scale.
Loop: trade → log f and c at entry → resolve → update Brier dashboard → adjust Kelly fraction.
Hygiene
Timestamp every f—no retro edits. Store market c and venue. Score only resolved contracts with clear oracle outcomes. Segment by category (politics, macro, sports). Plot reliability diagrams quarterly; fix overconfidence buckets first. Compare to baselines “always market c” and “always 50%.” Do not optimize Brier by never forecasting—sharpness matters for edge.
Pitfalls and reputation
Hindsight edits destroy learning. Single-event bragging proves nothing. Ignoring base rates on rare props inflates f. Mixing play-money and real pools blends different participant pools.
Metaculus and Good Judgment Open publish Brier or related scores. Crypto oracles sometimes weight reporter reputation by historical accuracy. Kalshi traders often need private journals before scaling Kelly.
Market Brier as benchmark
Score (c − o)² at entry for the venue mid or your fill. If your personal Brier beats market Brier over 100+ matched events with the same resolution rules, you may have information beyond price. If you lose while P&L is green, you were paid for risk or luck, not calibration—do not confuse the two when scaling size.
Sharpness without arrogance
Moving f from 50% to 70% when evidence warrants it improves Brier when right and hurts when wrong. Never moving off 50% is safe on the scoreboard but useless for trading. The art is moving with likelihoods, not with adrenaline after a red candle on the chart.
Segment your scorecard
Politics, macro, crypto, and sports draw different participant pools. A 0.17 Brier on Fed cuts does not license 0.17 confidence on celebrity awards. Segment logs; fix the worst bucket first—usually overconfidence in the 0.7–0.9 range.
Reliability diagram in words
Plot every forecast in the 0.6–0.7 bucket. If 40 events and 28 resolved YES, hit rate is 70%—well calibrated in that bucket. If only 18 resolved YES, you were overconfident by ten points on average in that band. Fix the bucket, not the last trade. Leaderboards that show one lucky 95% call hide this structure.
Play money versus real
Manifold and play-money pools teach mechanism without ruin; calibration mixes meme traders and serious forecasters. Compare Brier only within comparable universes. A great play-money score does not license full Kelly on a regulated USD book without a fresh sample.
Team and public forecasts
When a desk publishes a single f, agree internally before the timestamp—otherwise Brier rewards whoever talked loudest after the fact. Public Metaculus comments create reputation; private Kalshi journals create P&L. Same scoring math, different incentives to honesty.
Mean Brier over time
Rolling mean BS over the last 50 resolves smooths luck. A bad week is not a broken model; a bad year might be. Pair mean Brier with mean P&L and with average edge at entry to see whether you are calibrated but unprofitable (efficient prices) or profitable but uncalibrated (luck).
Worst single forecast
One 99% miss can dominate mean Brier for months. Policy: cap published f below 0.95 unless resolution is effectively certain and oracle risk is priced. The log score chapter explains why tails dominate scoring; Brier punishes them too, just less dramatically.
What comes next
Next: logarithmic scoring rule—why LMSR and many AMMs care about log utility, not squared error alone.