Findings: what 12,232 settled forecasts taught us
Every conclusion below is recomputed live from the arena's own database — the same stored forecasts, the same market prices captured at forecast time, the same resolved outcomes. Nothing is hand-tuned; reload after the next settlement and the numbers update.
0.1162
Brier of 80% market + 20% models (backtest)
0.1139
Brier of the market price alone
+2.1%
Current gap to the market (models not adding value)
1. Cheap LLMs share a skepticism bias
All six models — from four countries and five companies — systematically under-predict YES. Events they collectively priced in the teens resolved YES roughly a third of the time. This is not one model's quirk; it is a shared prior, which means averaging more models does not wash it out.
Colored bar: average predicted P(YES). Gray bar: share of those markets that resolved YES.
Amber: all model forecasts pooled. Gray: the market price. Dashed: perfect calibration. Models saying "15%" were wrong often enough that those events landed near 34%.
2. The superforecasting playbook backfires on LLMs
With human forecasting teams, the classic move is to extremize the pooled forecast — push it away from 0.5, because averaging washes out legitimate confidence. LLMs are the opposite: already overconfident, so extremizing makes them strictly worse and shrinking toward 0.5 helps. Received wisdom from human crowds does not transfer unchanged to silicon ones.
3. Models can't beat the market, and the blend no longer improves it
Head-to-head, the market price wins: when the model consensus diverges from the price, the market is right about 70% of the time. But "usually wrong" still carries signal. Blending roughly 20% of the model consensus into the market price (in log-odds space) beat the market on the first few hundred settled forecasts. It no longer does: over the full record the blend now scores 0.0023 Brier worse than the price alone. The Market × Models forecaster still runs the rule live, unchanged, because retiring a rule the moment it stops flattering us would be exactly the mistake this page warns about — and its out-of-sample record is reported below either way.
The vertical dashed line marks w = 0.8 — the blend the live Market × Models forecaster uses.
Amber: model consensus was closer to the outcome. Gray: the market price was. Betting against the market on divergence alone is a losing rule — the edge only shows up as a small, systematic tilt.
Live out-of-sample scorecard
Since going live, Market × Models has settled 1614 forecasts with a Brier of 0.1072 vs the market's 0.1034 on the same markets. The backtest edge has not shown up yet — which is itself the honest result.
4. What we tried and rejected
The obvious fix for a systematic bias is to learn a correction from history (Platt recalibration: fit a line through past logit-forecasts vs outcomes, apply it to new ones). In-sample it looks great. Fit on the earlier half of the data and tested on the later half, it made forecasts worse — the bias isn't stable across news regimes, so yesterday's correction over-corrects today. That is why the leaderboard carries no "calibrated" forecaster: it failed the honest test.
| Period | Market-rounds | Market Brier | Model pool | Market × Models |
|---|---|---|---|---|
| Earlier half | 1085 | 0.1096 | 0.1641 | 0.1126 |
| Later half | 1085 | 0.1182 | 0.1466 | 0.1199 |
Forecast difficulty swings hard between news regimes (compare the market columns), but the blend's edge over the market is the more stable pattern — which is why it, and not a learned bias correction, went live.
Why this matters. A prediction market is already an aggregation machine — money turns opinions into a price. The interesting question was never "can a $0.001 LLM call out-trade Polymarket?" (it can't), but "does it know anything the market hasn't priced?" The data says: a little, reliably enough to measure. The leaderboard now tracks that claim in real time, and the methodology page explains every metric.