No verdict changes when asking models multiple times

When you ask a language model the same checkable question three times—even at a higher temperature—the verdict almost never flips. Our own measurement from October 2026 found zero changes in correctness across 150 task-triplets, covering three models and both deterministic and higher-variance settings [1]. If a model gets a short factual task right (or wrong) once, it does so every time; only the phrasing shifts slightly under less strict settings.

Verdict: No flips in 150 triple-asks, regardless of temperature

In October 2026, we ran 25 checkable tasks through three OpenAI models (GPT-4.1, GPT-4.1 mini, GPT-4o mini), each asked three times at both temperature 0 (deterministic) and temperature 0.7 (more random) [1]. The result: across 150 sets of three runs each (a "task-triplet"), the answer was scored right or wrong identically every time. Not a single verdict flipped between runs at either temperature [1].

So, for checkable tasks—anything that can be scored as a number, an exact word, or a structured format—asking again does not shake a different answer loose [1].

Wording can shift, but the verdict does not

At temperature 0, every task-triplet came back word for word the same in all three runs. At temperature 0.7, 7 out of 75 triplets (just under 10%) differed in the answer text, but always within the bounds of correctness [1]. For example, a fruit list might swap "Mango" for "Banana," or a sentence might be phrased differently, but both were still scored right.

These differences only appeared on tasks with more than one right answer or where format (like "4.0" vs "4") was flexible [1]. None changed the underlying verdict: correct stayed correct, incorrect stayed incorrect.

Systematic errors: Retry does not help

Three tasks were answered incorrectly every time, at both temperatures, by whichever model got them wrong: one by all three models, one by two, one by one [1]. Each error was systematic, not a random slip. For instance, all three models answered "366" to "How many days from 1 March 2024 to 1 March 2025?" even though the correct answer is 365 (since 29 February 2024 is before the interval) [1].

Re-asking the question—at the same or different temperature—never produced a correct answer for these cases [1]. For the kinds of systematic errors that show up in short factual tasks, "try again" is not a fix.

What this means for leaderboard readers and model testers

If you’re comparing models on short, checkable tasks, a single run at temperature 0 is enough to establish the model’s score [1]. Three runs at either temperature did not change the ranking or the verdicts. This means a leaderboard based on single deterministic runs is not noisy for these tasks; the real risk is in task selection and scoring, not random model behavior.

Temperature 0.7, often thought of as "less reliable," did not produce more wrong answers in this structured test—just more variety in phrasing [1]. For extractive or arithmetic questions, the number or value was identical across all runs.

What this bench cannot tell you

This test covered 25 short, checkable tasks, three OpenAI models, and a single day’s run [1]. It does not measure behavior on long-form, creative, or reasoning-heavy tasks, or across a broader range of models. More open-ended prompts or harder reasoning questions may well produce verdict flips between runs, especially at higher temperatures.

Temperature 0 is not absolutely guaranteed to be deterministic by OpenAI, though in this experiment, it was: every response to a given prompt was word-for-word identical across all three runs for each triplet [1]. Latency differences were within normal network noise, and prices reflect list rates as of the test date [1].

A worked example

Suppose you are building a mini leaderboard for three models on a set of 25 arithmetic and extraction tasks. You run each task three times per model at both temperature 0 and 0.7, scoring the answers programmatically (e.g., comparing to a known number or JSON output).

• Every run returns the same verdict for each model-task pair, so your leaderboard is identical whether you use the first, second, or third run.
• If a model makes a systematic mistake (“How many ‘r’s in ‘strawberry’?” answered as 2 instead of 3), repeating the task or adjusting the temperature does not fix it.
• On tasks with open answers (e.g., “Write a sentence about the sea without the letter ‘e’”), you may see different sentences at higher temperature, but if your scorer checks only for the absence of ‘e’, the verdict stays the same.

This shows that, for these tasks and models, your leaderboard is stable and retrying on failures is not productive.

Questions readers ask

Will this hold for longer or more complex tasks?

No source gives this number; what the sources do show is that for 25 short, checkable tasks, verdicts were stable [1]. For longer or creative tasks, or with more complex reasoning, verdict flips may occur.

Is temperature 0 always deterministic?

In this test, temperature 0 gave word-for-word identical responses every time [1]. However, OpenAI does not guarantee determinism at temperature 0 in all circumstances.

Can changing the prompt or scoring method change the result?

Yes. The stability found here applies to short, checkable tasks with programmatic scoring [1]. Changing the prompt, task type, or how you score answers can introduce new sources of variation.

How the numbers work

These figures were computed for this article, not estimated by its writer. Inputs come from the numbered sources or are assumptions, labelled as such; change an assumption and the results change with it.

*Corrected 2026-11-02: The sentence said all models failed all three tasks; the material's table says one task by all three, one by two, one by one.*

Sources

1. Our own data, 29 Oct 2026: Bench repeat run 2026-10-29: 3 models x 25 tasks x 3 runs at temperature 0 and 0.7