Can AI beat the crowd?
Six LLMs make blind probability forecasts on live prediction markets — they never see the market price. We score each against the crowd (the market price itself) and test whether pooling them into an ensemble wins.
The verdict so far
Not yet — the crowd still leads the ensemble
Ensemble Brier
0.152
Crowd Brier
0.093
Markets
185
Best vs. Crowd
⚡ Market × Models
-0.045 Brier
Ensemble Skill
-0.059
Brier vs. crowd
Models Beating Crowd
0 / 6
on shared markets
Markets Resolved
304
998 still open
Skill vs. the Crowd
Brier improvement over the market price (positive = beat the crowd)
Skill leaderboard
Why the ensemble wins| # | Forecaster | Skill vs Crowd ▼ | Brier | Log Loss | ECE | Resolution | Forecasts | Reliability |
|---|---|---|---|---|---|---|---|---|
| 1 | 👥The Crowdbaseline | — | 0.111 | 0.358 | 0.048 | 0.106 | 3484 / 3810 | 100.0% |
| 2 | ⚡Market × Modelsensemble | -0.045 | 0.105 | 0.343 | 0.075 | 0.092 | 1670 / 3257 | 51.9% |
| 3 | 🎯Ensembleensemble | -0.083 | 0.153 | 0.469 | 0.049 | 0.052 | 2223 / 3810 | 58.9% |
| 4 | 🌀Mistral Small 3.2Mistral | -0.091 | 0.160 | 0.487 | 0.067 | 0.051 | 2096 / 3813 | 55.4% |
| 5 | 🔮DeepSeek V4 FlashDeepSeek | -0.091 | 0.162 | 0.545 | 0.081 | 0.044 | 1902 / 3810 | 50.3% |
| 6 | 💎Gemini 3.1 Flash LiteGoogle | -0.092 | 0.161 | 0.552 | 0.078 | 0.048 | 2214 / 3813 | 58.6% |
| 7 | 🌱Seed 1.6 FlashByteDance | -0.095 | 0.165 | 0.545 | 0.069 | 0.046 | 2219 / 3813 | 58.7% |
| 8 | 🐲Qwen3 235BAlibaba | -0.100 | 0.167 | 0.672 | 0.077 | 0.044 | 1868 / 3811 | 49.4% |
| 9 | 🧠GPT-4.1 MiniOpenAI | -0.102 | 0.171 | 0.544 | 0.076 | 0.042 | 2224 / 3812 | 58.9% |
Sorted by skill vs. the crowd. Lower Brier, log loss, and ECE are better; higher resolution means more informative forecasts. Reliability is the share of valid (non-errored) forecasts.