Benchmark noise, computed 2026-10-29: when is a lead on a question benchmark more than luck?
Arithmetic, not a measurement of any real model. For two models scoring a and b on the same n questions, the standard error of the gap is sqrt((a(1-a) + b(1-b)) / n); the 95% range for the true lead is the gap plus or minus 1.96 standard errors; the questions needed to show a lead d at 5% significance and 80% power are about 7.849 x (a(1-a) + b(1-b)) / d^2. A 1,000-run simulation of two identical models (each right 80% of the time) on 100 questions was run alongside to check the first figure.
What we found
- On 100 questions two equally good models at 80% differ by four points or more about half the time: the gap's standard error is 5.66 points.
- 82% against 78% on 100 questions: the standard error of the gap is 5.65 points and the 95% range for the true lead is -7.1 to +15.1 points, which includes zero.
- The same scores on 2,000 questions: standard error 1.26 points, range +1.5 to +6.5.
- Showing a true 4-point lead near 80% takes about 1,566 questions; a 1.5-point lead takes about 11,160.
- Rounded as the article quotes them: a standard error of about 5.7 points on 100 questions, four or more points apart in roughly half of the runs (53.5% of 1,000 simulated runs), and 95% of the runs within about 11 points.
- The 95% range for the true lead runs from about -7 to +15 points for 82% against 78% on 100 questions.
- To show a true 4-point lead near 80% (5% significance, 80% power) takes about 1,570 questions; a 1.5-point lead takes about 11,000.
- The formulas: the standard error of a gap is √((a(1-a) + b(1-b)) / n), and the questions needed are about (1.96 + 0.84)² × (a(1-a) + b(1-b)) / d² for a lead d.
- On a 1,000-question set a 4-point lead has a lower end of about +0.5 points; a 2-point lead there (81% against 79%) has range -1.5 to +5.5, so it would not clear it.
- a(1-a) is largest at 50%: each answer is most uncertain there, so each question tells you least about the gap, and a benchmark near 50% needs more questions than one near 95%.
- With 448 questions (GPQA) a 4-point lead has range -1.2 to +9.2; with 1,000 (GSM8K) it is +0.5 to +7.5.
The table
| Scores | Questions | Lead (points) | Standard error | 95% range for the true lead | Verdict |
|---|---|---|---|---|---|
| 82% vs 78% | 100 | +4 | 5.65 | -7.1 to +15.1 | Inside the noise |
| 82% vs 78% | 448 | +4 | 2.67 | -1.2 to +9.2 | Inside the noise |
| 82% vs 78% | 1,000 | +4 | 1.79 | +0.5 to +7.5 | Clears it |
| 82% vs 78% | 2,000 | +4 | 1.26 | +1.5 to +6.5 | Clears it |
| 81.5% vs 80% | 500 | +1.5 | 2.49 | -3.4 to +6.4 | Inside the noise |
What this does not show
Real benchmarks give both models the same questions and models miss the same hard ones, so true noise is somewhat smaller than this unpaired figure; a paired test (McNemar) uses only the questions where they disagree.