How this arena works — a walkthrough

Almost every well-known way of scoring a language model is a fixed set of questions with fixed answers — MMLU, HumanEval, GSM8K. They share a structural weakness: once a benchmark exists on the public internet, it eventually gets scraped into the next model's training data. After that, a high score is ambiguous. Did the model reason its way there, or did it remember the answer key? You cannot tell from the score alone, and the ambiguity gets worse every year.

A question about next month has no answer key. Nobody knows whether a ceasefire will hold or a bill will pass, so there is nothing to memorize. To do well, a forecaster has to actually combine what it knows about how the world usually works with what it can find out about this particular situation — and then commit to a number.

Prediction markets give us those questions with two useful properties attached. They resolve unambiguously against a public rule, and they come with a price, which is itself a forecast made by people with money at stake. That price is our benchmark: not a leaderboard of models against each other, but a standing bet against the crowd.

Twice a day a scheduled job picks twelve markets and asks every forecaster for a probability. Most of the engineering is in the filtering: which markets are worth asking about at all.

The selection gate is doing real work. Sports and daily-weather markets are excluded on purpose — not because they are uninteresting, but because a same-day game resolves in hours and a web search just fetches the score. That is lookup, not forecasting. The horizon window (one to forty-five days) is the same idea from both ends: at least a day out so the outcome is genuinely undetermined, and inside six weeks so the forecast-to-score feedback loop actually closes while the project is running.

The price filter matters too. A market trading at 97¢ tells you almost nothing about a forecaster — everyone says 97%, everyone scores well, and the comparison is noise. Keeping to the 5¢–95¢ band means every market in the sample still has something to be right or wrong about.

Nine rows get written for every market: six model forecasts, plus three computed ones. The design decision that everything else depends on is which of them can see the price.

If a model could see that a market trades at 62¢, the cheapest way to get a good score would be to answer "62%" and stop thinking. It would look well-calibrated. The ensemble of six such models would look well-calibrated too — and would have measured nothing except the models' ability to read a number off a page. Blinding them is what makes the six forecasts independent of the crowd, and independence is the precondition for the entire ensemble argument in Part 7.

The three computed forecasters are not models and cost nothing to run:

• Ensemble — the plain average of whichever model forecasts came back valid. No weighting, no tuning.
• Market × Models (hybrid) — the one forecaster that does see the price. It starts from the price and nudges it 20% of the way toward the model consensus. This is the live test of "do the models add anything?"
• The Crowd — the price itself, scored exactly like any other forecaster. This is the bar.

This is the first objection almost everyone raises, and it is a good one. The answer has two halves, and keeping them separate is most of the insight.

Pretraining supplies the prior. A model that learned nothing after its cutoff still knows how the world tends to behave: how often announced mergers actually close, how central banks respond to inflation prints, how long a coalition government survives a scandal, how frequently a ceasefire announced in week one is still holding in week four. That is base-rate knowledge, and it does not go stale the way facts do. The system prompt asks for it explicitly — "think in terms of base rates, then adjust for specific evidence."

Retrieval supplies the current state. Every call runs a web search first and the results land in the context window before the model answers. The weights are frozen; the context is not. So the model is not remembering the answer, it is reading today's news and applying an old, still-valid sense of how such stories usually end.

Where this genuinely breaks down is worth naming, because it is a real limitation and not a hypothetical one. With only four search results, a model can carry a stale premise — who holds an office, whether a company still exists independently — straight into its forecast, and nothing in the response will flag that it did. And if search silently returns nothing, the model answers from memory alone without saying so.

You cannot grade a single probability. "30% chance of rain" is not wrong when it rains. But you can grade a collection of probabilities, and the tool for it is embarrassingly simple: take the distance between what you said and what happened, and square it.

Squaring is the whole trick. It makes the rule proper — a piece of jargon with a concrete meaning: your expected score is best when you report your honest belief. There is no way to game a Brier score by shading your answers. If you truly think something is 70% likely, saying 85% to look decisive raises your expected penalty. Saying 55% to play it safe does too.

Two reference points make the numbers legible. A forecaster that always says 50% scores 0.25 — that is the "knows nothing" line. A perfect oracle scores 0. Real forecasters on genuinely uncertain questions land somewhere in between, and the differences that matter are small: a gap of 0.01 between two forecasters is a meaningful edge, not a rounding error.

The arena also tracks log loss, which is the same idea with a harsher temperament. It punishes confident mistakes far more severely — being certain and wrong costs infinitely much, which is why probabilities are clamped away from exactly 0 and 1 before scoring. And the headline number on the leaderboard is neither of these but a difference: skill vs. crowd = the crowd's Brier minus yours, on the same markets. Positive means you beat the market. Comparing on the identical resolved set is what makes it a fair fight; a forecaster that skipped the hard questions would otherwise look brilliant.

Calibration asks a narrow question: when you say 30%, does it happen about 30% of the time? Sort every forecast into confidence buckets, compare the stated probability to the realized rate, and plot it.

The trap is that calibration alone is cheap. Predict the base rate for everything and you are perfectly calibrated and completely uninformative. What you also need is resolution — the willingness to separate the likely from the unlikely and be right about which is which. The Brier score contains both, and the decomposition makes it explicit: Brier = reliability − resolution + uncertainty. Reliability is your distance from the diagonal above (lower is better), resolution is how much your forecasts actually spread outcomes apart (higher is better), and uncertainty is the irreducible difficulty of the questions themselves, identical for everyone.

That last term is why you should never compare Brier scores across different question sets. A forecaster working on easy questions posts better numbers than a better forecaster working on hard ones. It is also why every comparison in this project is restricted to a shared set of resolved markets.

This is the claim the project is built to test, and it is not obvious. Averaging six forecasts helps for a specific reason: each forecast is roughly signal plus error, and when you average, the signal reinforces while the errors partly cancel — but only to the extent the errors are independent.

That is why the roster is deliberately assembled from five companies across three regions rather than from six variants of the strongest model. Models trained on similar data with similar methods make similar mistakes, and correlated mistakes survive averaging intact. The error-correlation heatmap measures exactly this, and the ensemble-size curve on the same page shows how much each additional model is actually buying.

It also predicts a specific failure, which is what the data shows: all six models share a skepticism bias, systematically under-predicting YES. That bias is in the shared column, not the independent one, so the ensemble inherits it undiminished. No amount of averaging fixes a prior that everyone holds.

Probabilities are a bad scale for arithmetic on beliefs, because equal steps do not carry equal meaning near the ends.

Log-odds fixes this. In that space, evidence adds: Bayes' rule becomes "prior log-odds plus the weight of the evidence." It also has no boundary, so averaging cannot pull results toward the middle the way probability-space averaging does — which matters because a mean of raw probabilities is systematically less extreme than its inputs, making the pool under-confident by construction.

The hybrid forecaster uses this to answer the project's actual question. It takes the market price as its anchor, converts both the price and the model consensus to log-odds, and blends them 80/20. The weight is not arbitrary — it comes from sweeping every weight from 0 to 1 across the settled history and reading off where the curve bottoms out. If the models carried no information the market lacked, the best weight would be 1.0 (all market) and the hybrid would be pointless.

A tutorial that ends with the results is only half of one. The harder skill is knowing which numbers deserve weight, so here is the short list of things that should make you discount a finding on these pages.

• Small n. Brier differences of a few thousandths over a couple hundred markets are well inside noise. Every chart here reports its sample size; check it before you believe the ordering. The Data page puts confidence intervals on the headline gaps for exactly this reason.
• In-sample tuning. Any rule chosen by looking at outcomes will look good on those outcomes. Only the forward-looking version counts.
• Regime dependence. Forecast difficulty swings hard with the news cycle. A finding that holds in one half of the data and reverses in the other is a description of that period, not a fact about language models.
• Survivorship in the market set. These are liquid, mid-priced, non-sports markets on one platform. Nothing here generalizes automatically to illiquid markets, to near-certain ones, or to questions nobody is trading.
• Paper P&L. It ignores fees, slippage, and the fact that your order would move the price. It is a sanity check on the skill numbers, never a trading result.

There is a sixth item that deserves its own line, because it is the one that catches people who have already learned the other five: a significant result can be an artifact of your own plumbing. The arena ran a standard test for whether the models lead the market — do prices move toward a model that disagrees with them? — and got a clean, significant yes. It was false. The market prices in our database are only refreshed for markets that still rank highly by volume, so most forecasts are made against a slightly stale snapshot, and a model with live web search "predicts" the moment that snapshot catches up. Split the sample by whether the price was actually fresh and the entire effect evaporates. The write-up is on the Data page. No amount of statistical care would have caught that — only knowing how the data was produced.

The project has a second worked example of the same discipline: a learned bias correction that looked excellent in-sample made forecasts worse when fit on the earlier half of the data and tested on the later half. It was dropped, and the reason it was dropped is written up on the Findings page. That is the template — a result that fails the honest test is still a result.