Can AI beat the crowd?

Six LLMs make blind probability forecasts on live prediction markets — they never see the market price. We score each against the crowd (the market price itself) and test whether pooling them into an ensemble wins.

The verdict so far

Not yet — the crowd still leads the ensemble

Ensemble Brier

0.168

Crowd Brier

0.101

Markets

Best vs. Crowd

⚡ Market × Models

-0.017 Brier

Ensemble Skill

-0.067

Brier vs. crowd

Models Beating Crowd

0 / 6

on shared markets

Markets Resolved

290 still open

Skill vs. the Crowd

Brier improvement over the market price (positive = beat the crowd)

Skill leaderboard

Why the ensemble wins

#	Forecaster	Skill vs Crowd ▼	Brier	Log Loss	ECE	Resolution	Forecasts	Reliability
1	👥The Crowdbaseline	—	0.125	0.394	0.071	0.091	675 / 1038	100.0%
2	⚡Market × Modelsensemble	-0.017	0.034	0.171	0.149	0.131	122 / 485	100.0%
3	🎯Ensembleensemble	-0.083	0.166	0.496	0.072	0.049	675 / 1038	100.0%
4	🌀Mistral Small 3.2Mistral	-0.091	0.174	0.513	0.111	0.051	672 / 1041	99.3%
5	🔮DeepSeek V4 FlashDeepSeek	-0.093	0.177	0.563	0.111	0.043	640 / 1038	93.6%
6	💎Gemini 3.1 Flash LiteGoogle	-0.094	0.177	0.546	0.119	0.054	675 / 1041	99.2%
7	🌱Seed 1.6 FlashByteDance	-0.098	0.181	0.530	0.103	0.047	678 / 1041	99.9%
8	🐲Qwen3 235BAlibaba	-0.098	0.181	0.678	0.116	0.043	619 / 1039	90.1%
9	🧠GPT-4.1 MiniOpenAI	-0.108	0.191	0.564	0.133	0.043	677 / 1040	100.0%

Sorted by skill vs. the crowd. Lower Brier, log loss, and ECE are better; higher resolution means more informative forecasts. Reliability is the share of valid (non-errored) forecasts.