Can AI beat the crowd?

Six LLMs make blind probability forecasts on live prediction markets — they never see the market price. We score each against the crowd (the market price itself) and test whether pooling them into an ensemble wins.

The verdict so far

Not yet — the crowd still leads the ensemble

Ensemble Brier

0.155

Crowd Brier

0.092

Markets

178

Best vs. Crowd

⚡ Market × Models

-0.047 Brier

Ensemble Skill

-0.063

Brier vs. crowd

Models Beating Crowd

0 / 6

on shared markets

Markets Resolved

208

773 still open

Skill vs. the Crowd

Brier improvement over the market price (positive = beat the crowd)

Skill leaderboard

Why the ensemble wins

#	Forecaster	Skill vs Crowd ▼	Brier	Log Loss	ECE	Resolution	Forecasts	Reliability
1	👥The Crowdbaseline	—	0.118	0.375	0.060	0.088	2269 / 2718	100.0%
2	⚡Market × Modelsensemble	-0.047	0.107	0.347	0.075	0.089	1614 / 2165	78.1%
3	🎯Ensembleensemble	-0.086	0.156	0.476	0.050	0.048	2167 / 2718	82.6%
4	🌀Mistral Small 3.2Mistral	-0.093	0.162	0.493	0.068	0.049	2055 / 2721	77.7%
5	🔮DeepSeek V4 FlashDeepSeek	-0.094	0.164	0.552	0.084	0.043	1863 / 2718	70.6%
6	💎Gemini 3.1 Flash LiteGoogle	-0.095	0.164	0.562	0.083	0.045	2158 / 2721	82.1%
7	🌱Seed 1.6 FlashByteDance	-0.098	0.168	0.553	0.071	0.043	2164 / 2721	82.3%
8	🐲Qwen3 235BAlibaba	-0.103	0.169	0.682	0.082	0.041	1824 / 2719	69.3%
9	🧠GPT-4.1 MiniOpenAI	-0.104	0.174	0.552	0.082	0.039	2168 / 2720	82.5%

Sorted by skill vs. the crowd. Lower Brier, log loss, and ECE are better; higher resolution means more informative forecasts. Reliability is the share of valid (non-errored) forecasts.