Tool use, such as web search, has become a standard capability even in freely available large language models (LLMs). However, existing benchmarks evaluate temporal reasoning mainly in static, non-tool-using settings, which poorly reflect how LLMs perform temporal reasoning in practice. We introduce Time Puzzles, a constraint-based date inference task for evaluating iterative temporal reasoning with tools. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations and may admit one or multiple valid dates. The puzzles are algorithmically generated, enabling controlled and continual evaluation. Across 13 LLMs, even the best model (GPT-5) achieves only 55.3% accuracy without tools, despite using easily searchable facts. While web search improves performance, models perform substantially better when constraints are rewritten with explicit dates, removing the need for factual lookup. These results reveal a gap in reliable tool use for iterative temporal reasoning.
Figures & tables
Figure 1: Time Puzzles: a simple date inference task requiring iterative tool-aided temporal reasoning.
Figure 2: Average exact match accuracy across solution counts, with and without web search (only on solution counts 1, 3, 5). GPT-4.1/5 run live web search; for open-weight models we re-use the same cited GPT web results.
Model
EM (%)
JI (%)
F1 (%)
# Tks
GPT-5
55.33
58.62
59.52
4126
GPT-5-nano
15.83
19.37
20.31
5224
GPT-4.1
35.17
39.13
40.14
615
GPT-4.1-nano
9.33
13.53
14.84
1098
GPT-oss-20B
10.83
13.51
14.32
5704
DeepSeek-V3.2-R
36.67
39.33
40.08
7862
Table 1: Exact-match accuracy (EM), Jaccard index (JI), F1 score (F1), and output tokens (Tks), averaged over six solution counts, for the 600 generated puzzles with implicit (default) constraints in the tool-less setting.
Figure 3: Average EM accuracy (%) for GPT-5 (left) and GPT-4.1 (right) under different conditions across three solution counts (1, 3, and 5). Numbers annotated on bars are mean values, beside error bars. +Web enables web search. +CI enables Code Interpreter.
Type
Setting
G4.1-Nano
G5-Nano
G4.1
G5
Implicit
Default
9.3
15.8
35.2
55.3
+Reason Inst
8.8
14.2
25.3
47.3
New Runs
11.0
16.8
29.8
50.1
Explicit
Default
41.8
60.2
63.0
84.7
+Reason Inst
35.5
55.5
53.5
77.7
New Runs
42.7
60.2
63.7
80.5
Table 2: EM accuracy (%) averaged over six solution counts under different settings for both explicit and implicit constraint across the four GPT models. Default setting is based on Section 3 . “+Reason Inst” enhances the default setting with detailed step-by-step reasoning instruction. “New Runs” reruns the default setting with completely new puzzles generated from scratch.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Fact Name
Description
Level
ExplicitYearFact
Explicitly states the year (e.g., “The year is 2025”).
Year
DecadeFact
Specifies the decade (e.g., “The 1990s”).
Year
LeapYearFact
States that the year is a leap year.
Year
ChineseZodiacFact
Specifies the Chinese Zodiac animal.
Year
PersonAliveFact
States a famous person (e.g., Steve Jobs) was alive.
Year
USPresidentFact
States a specific US President was in office.
Year
Appendix
Table 3: Taxonomy of temporal facts used in generation, categorized by constraint granularity level.
Model
Variant used (snapshot / HF repo)
Params
Release date
Knowledge cutoff
GPT-5 OpenAI (2025a)
gpt-5-2025-08-07
Not disclosed
2025-08-07
2024-09-30
GPT-5-nano OpenAI (2025a)
gpt-5-nano-2025-08-07
Not disclosed
2025-08-07
2024-05-31
GPT-4.1 OpenAI (2025b)
gpt-4.1-2025-04-14
Not disclosed
2025-04-14
2024-06-01
GPT-4.1-nano OpenAI (2025b)
gpt-4.1-nano-2025-04-14
Not disclosed
2025-04-14
2024-06-01
GPT-oss-20B OpenAI et al. (2025)
openai/gpt-oss-20b
20.9B (3.6B active)
2025-08-05
2024-06
DeepSeek-V3.2-R DeepSeek-AI et al. (2025)
deepseek-reasoner
685B
2025-12-01
Not disclosed
Appendix
Table 4: Model variants and specifications used in our experiments. We identify OpenAI models by their dated snapshot IDs and open-weight models by their Hugging Face repository IDs. Release dates reflect snapshot timestamps or official provider documentation. Note that precise release and knowledge cutoff dates may vary slightly across sources.