One date question, seven phrasings: the leap-year trap for AI models

Three OpenAI models each answered 366 days to the question "How many days from 1 March 2024 to 1 March 2025?"—a span that never includes 29 February, so the correct answer is 365—when the question was phrased plainly or with a leap-year hint [1]. When required to count month by month or check if 29 February lies in the span, most models got it right, but not always [1]. This is our own calculation, run on 29 October 2026, testing seven wordings across three models for a total of 63 runs [1].

What the models get right—and wrong—on seven wordings

The question is simple: How many days from 1 March 2024 to 1 March 2025? The correct answer is 365; 2024 is a leap year, but 29 February 2024 falls outside this interval [1]. Each model (GPT-4.1, GPT-4.1 mini, GPT-4o mini) received seven different phrasings, including plain, hinting, month-by-month, and code-based prompts. Each was run three times per phrasing, at temperature 0.

Here are the findings:

On the plain and hint versions—where the year "2024" is called out—all nine runs across three models gave 366, not 365 [1]. The hint about "2024 is a leap year" did not help; if anything, it made the reflex more certain [1].

When forced to count month by month or to check if 29 February is inside the interval, GPT-4.1 and GPT-4.1 mini got the answer right in almost every run (2/3 or 3/3) [1]. GPT-4o mini still failed half the time, showing less grasp of the rule [1].

When asked as a code calculation—specifically, "In Python, what does (date(2025,3,1) - date(2024,3,1)).days return?"—GPT-4.1 and GPT-4.1 mini each got 365 in every run [1].

The reflex versus the rule: why the models trip

This pattern shows that the models have the calendar rule somewhere in their training—but do not reach for it unless the question demands it [1]. When simply told the span is from "March 1, 2024 to March 1, 2025," all three models, every time, answered 366 [1]. Even being told "2024 is a leap year" did not help: the models doubled down on the reflex, not the logic [1].

The only exception is when the model is asked to count month by month or explicitly check for the leap day [1]. In those cases, GPT-4.1 and GPT-4.1 mini can access the needed step-by-step rule and answer 365 [1]. When prompted for the result of a code calculation, these models trust the code as an authority and answer 365 every time [1].

GPT-4o mini, however, failed even the control questions (where the correct answer is 366 because 29 February is inside the span), answering 365 both times [1]. This is not just a leap-year rule error; it is a case of no calendar rule at all, just a nudge from seeing "leap" or "2024" in the question [1].

The controls: where the leap day really is

The test included two control questions:

• From 1 March 2023 to 1 March 2024 (span includes 29 February 2024)
• From 1 June 2027 to 1 June 2028 (span includes 29 February 2028)

The correct answer to both is 366, as these periods include a leap day [1].

GPT-4.1 got both controls right every time, showing it does know the rule when the context pushes it to apply it [1]. GPT-4.1 mini got only one control right (the latter), and GPT-4o mini failed both, sticking to 365 even when the leap day is truly in the span [1]. This shows that for some models, the leap-year failure is not just about misplacing 29 February; it is a broader failure to apply the calendar's structure [1].

What this pattern cannot tell

This test gives a clear result for these three OpenAI models, on a single date question, with seven phrasings, on one day and one provider [1]. It does not show how other providers' models would answer, or how these models would do on different date spans [1]. The runs were at temperature 0, but the provider's output is not guaranteed fully deterministic, even so; in this case, every triplet agreed [1]. Also, phrasing the question to force step-by-step reasoning (counting months or checking for 29 February) uses more tokens and takes more time, but the test did not measure or price out that difference [1].

A correct answer under one phrasing does not guarantee the model will answer the next date question right, or that it has truly internalized the leap-year rule [1]. The knowledge is there, but the reflex is strong.

A worked example

Our measurement on 29 October 2026 put three OpenAI models—GPT-4.1, GPT-4.1 mini, GPT-4o mini—to the leap-year test: how many days from 1 March 2024 to 1 March 2025? The correct answer is 365, since 29 February 2024 is not in the span. Asked plainly, all three models answered 366 in every run, and adding the prompt “2024 is a leap year” made no difference: the reflex for 366 stayed [1].

When prompted to count days month by month, GPT-4.1 got the answer right in 2 out of 3 runs, and both mini models were correct in all 3 [1]. For the phrasing “Does the period contain a 29 February? Then say how many days it lasts,” GPT-4.1 and GPT-4.1 mini were correct in 2 or 3 runs, but GPT-4o mini defaulted to 366 every time [1]. Framing the question as code (“what does (date(2025,3,1) - date(2024,3,1)).days return?”), GPT-4.1 and GPT-4.1 mini gave 365 in all runs, while GPT-4o mini managed it only once [1].

The full bench (seven phrasings, three runs each) scored GPT-4.1 at 62% accuracy (100% on controls, 47% on traps), GPT-4.1 mini at 57% (50% controls, 60% traps), GPT-4o mini at 19% (0% controls, 27% traps) [1]. GPT-4o mini failed both controls: for spans that do include a leap day, it answered 365 instead of 366 in every run, showing it does not apply a

Questions readers ask

Why do the models get the plain question wrong so consistently?

All three models reflexively associate "2024" and "leap year" with 366 days, without checking if 29 February actually falls inside the span [1]. They do not retrieve the rule unless forced by the question's structure [1].

Does this mean the models cannot do date math?

No; when asked to count step by step or to compute using code, the models can get the right answer [1]. The problem is not a lack of knowledge, but a shortcut in how they answer plain date questions [1].

How can I get the right answer from a model?

Ask the model to show its work: either by counting the months, checking if 29 February is in the interval, or by asking for the code result [1]. For these models and this question, that is enough to override the reflex [1].

Sources

1. Our own data, 29 Oct 2026: Model bench leap set, 2026-10-29: one date question in seven phrasings, 3 models x 3 runs