Findings: what 4,868 settled forecasts taught us
Every conclusion below is recomputed live from the arena's own database — the same stored forecasts, the same market prices captured at forecast time, the same resolved outcomes. Nothing is hand-tuned; reload after the next settlement and the numbers update.
0.1229
Brier of 80% market + 20% models (backtest)
0.1209
Brier of the market price alone
+1.7%
Current gap to the market (models not adding value)
1. Cheap LLMs share a skepticism bias
All six models — from four countries and five companies — systematically under-predict YES. Events they collectively priced in the teens resolved YES roughly a third of the time. This is not one model's quirk; it is a shared prior, which means averaging more models does not wash it out.
Colored bar: average predicted P(YES). Gray bar: share of those markets that resolved YES.
Amber: all model forecasts pooled. Gray: the market price. Dashed: perfect calibration. Models saying "15%" were wrong often enough that those events landed near 34%.
2. The superforecasting playbook backfires on LLMs
With human forecasting teams, the classic move is to extremize the pooled forecast — push it away from 0.5, because averaging washes out legitimate confidence. LLMs are the opposite: already overconfident, so extremizing makes them strictly worse and shrinking toward 0.5 helps. Received wisdom from human crowds does not transfer unchanged to silicon ones.
3. Models can't beat the market — but they can improve it
Head-to-head, the market price wins: when the model consensus diverges from the price, the market is right about 70% of the time. But "usually wrong" still carries signal. Blending roughly 20% of the model consensus into the market price (in log-odds space) produced a better Brier score than the market alone — the Market × Models forecaster on the leaderboard runs exactly this rule, live, as an out-of-sample test.
The vertical dashed line marks w = 0.8 — the blend the live Market × Models forecaster uses.
Amber: model consensus was closer to the outcome. Gray: the market price was. Betting against the market on divergence alone is a losing rule — the edge only shows up as a small, systematic tilt.
Live out-of-sample scorecard
Since going live, Market × Models has settled 280 forecasts with a Brier of 0.0844 vs the market's 0.0745 on the same markets. The backtest edge has not shown up yet — which is itself the honest result.
4. What we tried and rejected
The obvious fix for a systematic bias is to learn a correction from history (Platt recalibration: fit a line through past logit-forecasts vs outcomes, apply it to new ones). In-sample it looks great. Fit on the earlier half of the data and tested on the later half, it made forecasts worse — the bias isn't stable across news regimes, so yesterday's correction over-corrects today. That is why the leaderboard carries no "calibrated" forecaster: it failed the honest test.
| Period | Market-rounds | Market Brier | Model pool | Market × Models |
|---|---|---|---|---|
| Earlier half | 418 | 0.1659 | 0.2203 | 0.1687 |
| Later half | 418 | 0.0759 | 0.1313 | 0.0771 |
Forecast difficulty swings hard between news regimes (compare the market columns), but the blend's edge over the market is the more stable pattern — which is why it, and not a learned bias correction, went live.
Why this matters. A prediction market is already an aggregation machine — money turns opinions into a price. The interesting question was never "can a $0.001 LLM call out-trade Polymarket?" (it can't), but "does it know anything the market hasn't priced?" The data says: a little, reliably enough to measure. The leaderboard now tracks that claim in real time, and the methodology page explains every metric.