About

Traditional LLM benchmarks leak into training data. Models can memorize answers to MMLU, HumanEval, and other static datasets, so scores drift further from genuine ability every time a benchmark is scraped into a training corpus.

Prediction markets test something that cannot be memorized: forecasting a future that has not happened yet. When a model estimates the probability that an event resolves YES next week, there is no answer key to recall — only reasoning, research, and calibration.

But this project asks a sharper question than “which model forecasts best?” It asks how much you gain by combining them. That is the entire point of the ensemble, and it only works if the individual forecasts are genuinely independent.

Inspired by Forecaster Arena.

1. We pull short-horizon binary markets from Polymarket — mostly those resolving within about a week — so forecasts settle quickly and the leaderboard reflects real outcomes, not open bets.
2. Each model is shown only the market's question and description and asked for one number: the probability of YES. Crucially it never sees the market price. A model that could peek at the price could just echo it, and an ensemble of price-echoers would teach us nothing.
3. The market price is scored as its own forecaster, “the Crowd.” A market at 62¢ is a 62% forecast. The crowd pools the money and opinions of many humans and bots, so it is a strong, hard-to-beat baseline.
4. The ensemble is simply the unweighted mean of the valid model probabilities for each market — no tuning, no weighting, zero extra API cost. If a model fails to return a usable forecast, it is left out of that market's average rather than coerced into a default.
5. Rounds run automatically twice daily (10:00 and 22:00 UTC). Settlement checks for resolved markets every few hours and computes each forecaster's scores. A fresh weekly cohort opens every Monday.

• Brier score — mean squared error between the forecast and the outcome. 0 is perfect; 0.25 is a coin flip. Lower is better.
• Log loss — punishes confident mistakes far more harshly than Brier, with probabilities clamped away from 0 and 1.
• Calibration (ECE) — when a forecaster says 70%, does it happen about 70% of the time? Visualized on each profile's calibration chart.
• Skill vs. crowd — the headline number: crowd Brier minus the forecaster's Brier on the same resolved markets. Positive means it beat the market.
• Paper P&L — a clearly-labeled secondary view that Kelly-stakes each forecaster's edge over the crowd. It is a sanity check on the skill numbers, never the headline.

Every forecaster — the six models, the ensemble, and the crowd — is scored on exactly the same shared set of resolved markets, so the comparison is apples-to-apples.

• Blind forecasts: hiding the market price is what makes the forecasts independent — and independence is what makes the ensemble meaningful.
• Failures are visible: a model that errors, times out, or returns an unparseable answer is marked failed and excluded from scoring, never coerced into a default probability. The leaderboard's reliability column is that valid-response rate.
• Plugins web search (not the :online suffix): models can research without silently swapping versions.
• temperature: 0 for every model, so runs are reproducible.
• Full audit trail: every forecast stores its prompt, raw response, cost, and score in the database.
• Budget cap: a hard limit on API spend; rounds stop automatically when it is reached.