The Data: 23,921 forecasts, and what they cost to collect

Two of these numbers deserve attention before any chart. The first is 2,170 scored market-rounds — not forecasts, market-rounds. Six models answering the same question are not six independent observations about forecasting skill; they are one question, answered six ways. Every comparison on this page uses the market-round as the unit, which is why the sample looks smaller here than the leaderboard's forecast counts suggest, and why it should.

The second is 27% YES. That is the base rate of the sample, and it is the number a lazy forecaster would exploit. Any claim that a model "knows something" has to beat a constant forecast of 27%, not a coin flip.

Forecast difficulty is not evenly spread across topics. The hardest slice for the market itself was gaming (crowd Brier 0.6097 over 11 market-rounds), and the models' contribution varies just as much: the blend helped most on trump (+0.0819 Brier vs. the price) and hurt most on crypto (-0.0314).

Resist reading a strategy into that. The typical bucket here holds about 34 market-rounds, so the gap between the best and worst topic is comfortably inside what chance produces — the honest use of this chart is to notice heterogeneity, not to pick a category to trade. Aggregate numbers on the other pages are averages over slices that behave quite differently, and that is worth knowing before you trust a single headline Brier.

A caveat this chart exposes about the arena itself. The row labels are Polymarket's own tags, and a single news cycle fragments across many of them — a Middle East escalation shows up separately as middle east, iran, strait of hormuz, geopolitics, and the names of individual figures. The round builder caps markets at three per category, but that cap compares tag strings, so eight differently-tagged markets about one week's events sail straight through it. The topic diversity in this dataset is therefore lower than the 103 distinct tags suggest, and the effective sample size is correspondingly smaller than 2,170.

Categories with fewer than 8 scored market-rounds are pooled into a single "other" row rather than dropped, so the counts still sum to the full sample.

Intuition says a question resolving tomorrow should be easier than one resolving in a month, and mostly it is — but the arena's selection gate suppresses the effect on purpose. A market only enters a round while it is still trading between 5¢ and 95¢, so the near-dated questions that survive filtering are precisely the ones that stayed genuinely uncertain right up to the deadline. The easy short-horizon markets never appear in this chart because they were already priced at 97¢ when the round was built.

When the six models cluster within 2.0 points of each other, the market's own Brier on those questions is 0.0954. When they scatter by 27.0 points, it is 0.1331. The models are detecting hard questions — and detecting them without ever seeing the price, which means the signal is genuinely independent of the market's own uncertainty.

This one has a practical payoff that does not require the models to be good. A confidence signal that costs no extra API call — you already made the six calls — and correlates with difficulty is usable for triage: flag the wide-spread questions for a human, or for a more expensive model, and leave the tight ones alone.

The unglamorous section, and the one most likely to be useful if you are building something similar. Three things stand out. Qwen3 235B is the least reliable of the roster at 69.3% valid responses, which matters more than it looks: a model that fails 5% of the time is scored on a slightly different — and not randomly different — set of markets than its peers. DeepSeek V4 Flash has the longest median latency at 2.6s, which is what sets the concurrency ceiling for the whole round.

And the cost structure is the genuinely surprising part: inference is nearly free compared to retrieval. At roughly $0.005 per web search versus fractions of a cent for the tokens, the search plugin dominates the bill for every model on the roster. If you wanted to run this ten times cheaper, you would not switch models — you would batch or cache the search.

This is the section the rest of the site should be read through. A Brier difference of a few thousandths is easy to report and hard to believe, so here every headline gap is resampled 2,000 times to ask what range of values is consistent with the data.

An interval that clears zero is a real effect at this sample size; one that straddles zero is not yet distinguishable from luck. As of this page load, 4 of 4 sit below zero with the whole interval on the losing side — measurably worse than simply quoting the market price. The least-bad of the four is Market × Models rule (80/20 logit blend) at -0.0023.

That is worth stating plainly rather than burying: on the full settled record, these aggregation rules do not beat the market price. Anchoring hard on the price and nudging it 20% toward the model consensus gets closest — the blend loses by a small fraction of what the raw model pool loses, which is the same ordering the Findings page describes. But "closest to the market" is not "better than the market," and an interval that sits entirely below zero is the data declining to support the stronger claim.

The bottom row is the one that counts as evidence. The three above it replay a rule across the same history that was used to pick the 0.8 weight, so they are flattered by construction. The live row contains only forecasts made after the weight was fixed: 1,614 market-rounds, an observed advantage of -0.0038, and an interval that excludes zero. At this sample size that is a real effect, in whichever direction the sign points.

The pairing is what makes these intervals as tight as they are. Comparing two overall Brier scores would drown the signal in the variation between easy and hard markets; comparing per-market differences removes that variation entirely, because both forecasters faced the identical question.

Brier scores only grade a forecaster once the market resolves, which is slow and wastes most of the data. There is a sharper test borrowed from finance: instead of asking whether the model is accurate, ask whether it is early. If a model disagrees with the price today, does the price move toward the model tomorrow? A forecaster with genuine information should lead the market it is trading against, and this test yields an observation every round instead of one per resolution.

Run naively, it works. Pooled across all 1,885 round-to-round transitions, the model consensus predicts the direction of the next price move with a slope of 0.0145, and the interval clears zero. That is a headline: cheap LLMs anticipate prediction-market moves.

It is not true. It is a bug in this codebase, measured.

The arena refreshes market prices by re-syncing Polymarket's top markets by volume before each round. A market that is still being forecast but has dropped out of that top slice keeps whatever price it was last given — so 64% of round-to-round transitions show no price change at all, not because the market was quiet but because nobody asked it.

When that stale snapshot eventually refreshes, it jumps to catch up with reality. A model that just ran a live web search "predicts" that jump for free — it is not forecasting the market, it is forecasting our own cache. Splitting the sample on that single distinction separates the two stories cleanly: 0.0223 when the price was stale (interval clears zero), and 0.0025 when it was fresh (indistinguishable from zero). All of the apparent edge lives in the stale rows. Against a market price we actually observed moving, the models lead by nothing.

The same conclusion arrives from a direction that does not depend on price refresh at all. Regressing the eventual outcome on the model's disagreement with the price gives a slope of -0.0796 (95% CI [-0.263, 0.166], 195 markets). If disagreement carried information about the truth, that slope would be positive, and it would tell you how far to move from the price toward the models — in other words, the optimal blend weight. The point estimate is negative, but the interval straddles zero, so the honest reading is "no detectable information" rather than "negative information." Either way it gives no support to a positive weight on the models, which is the same verdict the confidence intervals above deliver — reached without using a single Brier score.

Note the sample here: 2,167 observations, but only 195 markets, and observations inside one market share an outcome. The bootstrap resamples whole markets for exactly that reason, which is why the interval is as wide as it is. The effective sample size is the market count, not the row count.

Why this section exists. The result is null, so it wins no argument and appears on no leaderboard. But the near-miss is the most useful thing in this article: a plausible test, a clean significant result, a satisfying story — and an artifact of our own infrastructure. The only reason it was caught is that the confound suggested an obvious falsification test, and the test was run before the headline was written.

It also flags a live defect. If 64% of forecasts are scored against a price that is at least a round old, then "the crowd" on the leaderboard is not quite the market's current opinion — it is a slightly delayed one, and a delayed price is a weaker forecaster than a live one. That handicaps the baseline the models are already losing to, which means the models' deficit is, if anything, understated.

Every claim on this site rests on one assumption: the six models never see the market price, so their forecasts are independent of it. The prompt does withhold the price. But the models also run a live web search, and Exa indexes Polymarket, Kalshi, and every site that quotes them. A model can simply read the number it was never told.

It happens constantly. 72% of valid blind forecasts (9,065 of 12,636) name a prediction venue or cite market-implied odds in their own stated reasoning. The detector is deliberately narrow — it matches "Polymarket", "implied probability", "betting odds" and similar, but not "stock market" or "market cap" — so this is a floor, not an estimate.

That table is the part worth sitting with. The intuitive worry is that leakage inflates the models — that they look competitive only because they are copying. The data says the effect is real and larger than the headline suggests: on rounds where no model cited a market, the ensemble loses to the crowd by -0.1339, against -0.0210 where every model cited one. The pooled number reported everywhere else on this site sits between those, because most rounds are mixed.

One honest confound: the no-leakage rounds are harder for everyone — the crowd's own Brier is worse there too, because a market nobody writes about is a market nobody has analysed. But the models degrade far more steeply than the crowd does across those strata, which is what you would expect if part of their apparent accuracy had been borrowed rather than earned.

What this does and does not overturn. It does not rescue the models — the direction of every finding on this site survives, and the honest version of "cheap LLMs lose to the market" is worse than the number we have been reporting. What it does undermine is the word independent. The ensemble argument on The Lesson assumes six forecasts that err in unrelated ways; forecasts partly anchored to a shared price are correlated by construction, and averaging cannot diversify away an error they all inherited from the same source.

The fix is not obvious. Removing web search would make the models genuinely blind and much worse, and would stop testing anything interesting. Excluding prediction-market domains from the search plugin is the narrow repair, and it is the one worth trying — but it changes the data-generating process, so it splits the dataset in two and every comparison across the boundary becomes suspect. That is a decision about the experiment, not a bug fix.