Ask the same model three times: does the answer change? The bench's repeat run, 2026-10-29
Our own measurement. Three OpenAI models (GPT-4.1, GPT-4.1 mini, GPT-4o mini) answered the bench's 25 checkable tasks three times each, once at temperature 0 and once at temperature 0.7: 450 answers in all (3 models x 25 tasks x 3 runs x 2 temperatures). Every answer was scored by a program (a number within tolerance, an exact word, JSON equal to the expected value, or a format rule). No system prompt. Runs: repeat3/2026-10-29 (temperature 0) and temp07/2026-10-29 (temperature 0.7), results.csv in each.
The finding in one line
Across 150 task-triplets (a model asked the same task three times), the scored verdict never changed: 0 flips at temperature 0 and 0 flips at temperature 0.7. What changed at 0.7 was the wording of free-form answers, which the scorer does not care about.
Scores, identical at both temperatures
| Model | Correct (temp 0) | Correct (temp 0.7) | Median seconds (0 / 0.7) | Cost per 100 tasks |
|---|---|---|---|---|
| GPT-4.1 | 96% | 96% | 0.78 / 0.77 | $0.0114 |
| GPT-4o mini | 92% | 92% | 0.87 / 0.72 | $0.0009 |
| GPT-4.1 mini | 88% | 88% | 0.81 / 0.74 | $0.0023 |
Costs at list prices (openai.com/api/pricing, read 2026-10-29); the whole 450-answer experiment cost $0.022 in API fees.
How often the answer text was identical across the three runs
| Model | Temperature 0 | Temperature 0.7 | What differed at 0.7 |
|---|---|---|---|
| GPT-4.1 | 25 of 25 tasks | 23 of 25 | the fruit list (Mango vs Banana), the no-letter-e sentence |
| GPT-4.1 mini | 25 of 25 | 24 of 25 | the no-letter-e sentence |
| GPT-4o mini | 25 of 25 | 23 of 25 | "4.0" vs "4" for a percentage; the no-letter-e sentence |
At temperature 0, all 75 task-triplets came back word for word the same. At 0.7, 7 of 75 differed in text, all in tasks with more than one acceptable answer (write a sentence, list three fruits) or in number formatting; none of the differences crossed the line between right and wrong.
The same three tasks were wrong every time, for every run, at both temperatures
| Task | Expected | What the models answered | Who got it wrong |
|---|---|---|---|
| How many days from 1 March 2024 to 1 March 2025? | 365 (29 Feb 2024 is before the span) | 366, all nine times | all three models |
| One sentence about the sea, at most 12 words, no letter e | a sentence without e | sentences with "waves", "tranquil", "under" (contain e), all six times | GPT-4.1 mini, GPT-4o mini |
| How many times does r appear in "strawberry"? | 3 | 2, all three times | GPT-4.1 mini |
So the errors are systematic, not sampling noise: a model that is wrong on a task at temperature 0 was wrong on it in every one of six attempts. Re-running a failed task, or lowering the temperature, would not have rescued any of them.
What this means for anyone comparing models
- On short checkable tasks, one run per task at temperature 0 gave the same ranking as three runs at either temperature. A leaderboard built from single runs of this kind of task is not noisy for these models; the risk is elsewhere (task choice, scoring).
- Temperature 0.7 is not "less accurate" here; it is less repeatable in wording only. For extraction and arithmetic, the number came back the same; for open sentences, it did not.
- "Try again" is not a strategy for a systematic error. All three models insist on 366 days for a span that contains no 29 February.
What this cannot tell
- 25 tasks, 3 models from one provider, one day. A harder task set (long reasoning, long documents) would likely show verdict flips.
- Temperature 0 at OpenAI is not guaranteed deterministic; here it happened to be, for 75 of 75 triplets.
- Latency is wall time from one PC in Europe and includes the network; the 0.7 runs were faster by 0.01 to 0.15 s, which is within what the network alone does from one minute to the next.
- Prices are list prices on the day; a reader should recheck them.