Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs

Auss Abbood; Fernando Martínez-Plumed; José Hernández-Orallo; Nigel Collier

cs.CLOct 21, 2026

Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs

Authors: Auss Abbood, Fernando Martínez-Plumed, José Hernández-Orallo, Nigel Collier

Organizations: Language Technology Lab, University of Cambridge, UK · Valencian Research Institute for Artificial Intelligence (VRAIN) Universitat Politècnica de València, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK

Abstract

Temporal reasoning is widely assumed to decompose into two modes: temporal arithmetic (date and time calculations), and temporal semantics (understanding temporal and causal relations). This dichotomy shapes how benchmarks are designed, how models are trained, and how progress is measured, yet its empirical validity has never been directly tested. We provide two lines of evidence that challenge it. First, we test whether tasks labelled as arithmetic or semantic by the benchmarks support transferable gains within their purported mode. Finetuning on these subsets yields predominantly dataset-specific gains ( $+6.8$ to $+23.1$ percentage points), while transfer to a different dataset in the same mode ranges from $-3.2$ to $+1.1$ percentage points. Across the two benchmark families used in the transfer analysis, this pattern provides little evidence of coherent, transferable temporal abilities. Second, using cognitive-demand-based model evaluation, we examine which capabilities actually drive performance on these benchmarks. We find that quantitative and logical reasoning, the assumed proxies for temporal arithmetic and temporal semantics, are not the primary drivers; knowledge is. Together, these findings show that current temporal reasoning benchmarks do not cleanly realise the arithmetic-semantics distinction and instead largely reward models for remembering how time typically works.

Figures & tables

Dataset	Format	Temporal phenomena
TCP ( Ding et al., 2025 )	Free-form	Constraint-based planning
ToT Arithmetic ( Fatemi et al., 2025 )	Free-form	Addition, duration, scheduling, time zones
ToT Semantic ( Fatemi et al., 2025 )	Free-form	Event relationships, before/after, timelines
TRAM ( Wang and Zhao, 2024 )	Multiple-choice	Order, arithmetic, frequency, duration, NLI
TempReason ( Tan et al., 2023 )	Free-form	Temporal relation extraction
TimeQA ( Chen et al., 2021 )	Free-form	Time-sensitive question answering

Table 1: Overview of the nine temporal reasoning datasets from eight benchmarks. ToT Arithmetic and Semantic are counted separately.

Condition	Item filter			$n$
Control	$\mathrm{QLl}$	$=$	$0$	938
	$\mathrm{QLq}$	$=$	$0$	2,425
QLq-dominant	$\mathrm{QLq}$	$=$	$\mathrm{QLl}+1$	2,479
	$\mathrm{QLq}$	$=$	$\mathrm{QLl}+2$	247
QLl-dominant	$\mathrm{QLl}$	$=$	$\mathrm{QLq}+1$	1,826
	$\mathrm{QLl}$	$=$	$\mathrm{QLq}+2$	54

Table 2: Item-level operationalisation based on relative QLqand QLldemand.

Panel of Figure 1 — Figure 1: Joint distribution of QLqand QLldemand levels across all datasets.

Figure 2: PCA of the full 18-dimensional DeLeAnprofiles (PC1: 40.5%, PC2: 15.5%; 56.0% variance explained). TCP and ToT Semantic are annotated; the remaining seven datasets are shown in grey. TCP Long and Short share the TCP label.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

	Dimension (Broad)		Dimension (Specific)	Demand description
AS	Attention and Scan	AS	Attention and Scan	Focus on or locate specific elements within a given stream of information or environment in the whole process of solving a task.
CE	Comprehension and	CEc	Verbal Comprehension	Understand text, stories or the semantic content of other representations of ideas in different formats or modalities.
	Expression	CEe	Verbal Expression	Generate and articulate ideas, stories, or semantic content in different formats or modalities.
CL	Conceptualisation, Learning and Abstraction	CL	Conceptualisation, Learning and Abstraction	Build new concepts, engage in inductive and analogical reasoning, map relationships between domains, and generate abstractions from concrete examples.
MC	Metacognition and	MCr	Identifying Relevant Information	Recognise what information helps solve the task or does not, and how this recognition process unfolds as they work toward the solution.
	Critical Thinking	MCt	Critical Thinking Processes	Monitor or regulate multiple thought processes to answer the question effectively, ranging from simple recall to high-level critical thinking.

Appendix

Table 3: Overview of the DeLeAn demand dimensions, including broad cognitive and knowledge categories, specific sub-dimensions, and their semantic descriptions.

Dataset	Train	Validation	Test
ToT Arithmetic	1295	93	462
ToT Semantic	1959	141	700
TRAM Arithmetic	25499	2834	2700
TRAM Semantic	20121	2236	500

Appendix

Table 4: Train/validation/test cardinalities for the four dataset-mode combinations used in controlled transfer experiments, drawn from two benchmarks (ToT and TRAM).

Dataset	Train	Validation	Test
TRAM	2736	322	739
ToT Semantic	2042	208	544
ToT Arithmetic	1297	141	401
TimeQA	530	58	139
MenatQA	490	49	141
TempReason	455	55	147

Appendix

Table 5: DeLeAn-labelled dataset cardinalities (train/validation/test). TCP is split into Long and Short, yielding ten subsets from nine datasets.

Coarse task	Task	QLq	QLl	$\text{QLq}-\text{QLl}$
Arithmetic	Arithmetic	1.40	1.01	0.39
Arithmetic	Frequency	0.77	0.55	0.22
Arithmetic	Duration	0.86	0.66	0.20
Arithmetic	Ambiguity Resolution	1.19	1.07	0.12
—	Typical Time	0.10	0.25	-0.15
Semantic	Ordering	0.20	0.86	-0.66

Appendix

Table 6: Average QLqand QLlscores and their difference for TRAM tasks.

Figure 5: PCAof TRAM items coloured by their assigned mode. The two groups show substantial overlap, confirming that the arithmetic/semantic partition is not cleanly reflected in TRAM’s cognitive demand structure.

Question types	QLq	QLl	$\text{QLq}-\text{QLl}$
RelationDuration	2.04	1.90	0.13
Timeline	2.15	2.07	0.08
EventAtTimeT	1.76	1.94	-0.18
EventAtWhatTime	1.22	1.43	-0.21
NumberOfEventsInTimeInterval	2.58	2.85	-0.27
EventAtTimeOfOtherEvent	2.39	2.84	-0.45

Appendix

Table 7: Average QLqand QLlscores and their difference for ToTSemantic question types

Figure 7: Demand level counts per cognitive ability for the ten dataset subsets, with TCP Long and Short shown separately.

Dimension	Exact (%)	Within one (%)	Ordinal $\alpha$	$r_{\mathrm{WG}}$
KNc	27	45	-0.23	0.66
CEe	62	98	-0.01	0.96
SNs	99	100	0.06	1.00
MCu	42	85	0.34	0.90
CL	70	93	0.42	0.95
CEc	60	99	0.44	0.96

Appendix

Table 8: Inter-rater agreement between Gemini 3.1 Flash Lite and GPT-4o demand labels.

Dimension	Gemini	GPT-4o	GPT-4o $-$ Gemini
KNc	0.36	1.96	1.59
KNf	0.61	1.27	0.66
MCu	0.73	1.36	0.63
AT	1.67	2.19	0.53
AS	1.16	1.67	0.52
KNs	0.26	0.57	0.30

Appendix

Table 9: Mean demand level per dimension for Gemini 3.1 Flash Lite and GPT-4o annotations.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	2.10	0.05	0.25	0.21	0.28	0.39	0.11
AT	2.69	0.10	0.16	0.17	0.20	0.05	0.01
CEc	2.93	-0.30	0.41	-0.08	0.16	-0.34	-0.09
KNa	3.08	0.05	0.05	-0.07	-0.04	0.08	-0.12

Appendix

Table 10: Capability AUCchange (

\Delta

) for gemma-3-4b in the accuracy reward setting. Highest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	2.10	-0.16	-0.21	0.30	0.01	-0.47	-0.01
AT	2.69	-0.47	-0.44	0.03	0.00	-0.45	-0.02
CEc	2.93	-1.28	-0.27	-0.31	0.09	-0.64	-0.22
KNa	3.08	-1.05	-0.14	0.17	0.01	-0.24	-0.22

Appendix

Table 11: Capability AUCchange (

\Delta

) for gemma-3-4b in the one-minus-accuracy reward setting. Lowest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	3.19	0.17	-0.18	0.05	0.03	-0.08	0.17
AT	3.74	0.23	0.29	0.35	0.12	0.03	0.07
CEc	3.51	0.17	0.52	0.71	0.46	-0.29	0.04
KNa	3.25	0.10	0.10	-0.14	-0.00	0.15	-0.02

Appendix

Table 12: Capability AUCchange (

\Delta

) for gemma-3-12b in the accuracy reward setting. Highest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	3.19	0.09	-0.02	-0.24	-0.21	-0.39	-0.16
AT	3.74	-0.06	-0.13	-0.26	-0.09	-0.45	0.21
CEc	3.51	-1.24	-1.33	-1.27	-0.17	-1.62	-0.91
KNa	3.25	-1.46	-1.78	-1.42	-0.05	-1.71	-0.71

Appendix

Table 13: Capability AUCchange (

\Delta

) for gemma-3-12b in the one-minus-accuracy reward setting. Lowest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	1.79	0.09	0.35	0.19	0.32	0.28	0.12
AT	2.22	0.40	0.47	0.27	0.46	0.53	0.40
CEc	1.24	0.82	0.61	0.72	0.71	1.01	0.62
KNa	1.23	1.47	1.42	1.11	1.21	1.29	1.31

Appendix

Table 14: Capability AUCchange (

\Delta

) for Llama-3.2-3B in the accuracy reward setting. Highest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	1.79	-1.01	-0.86	-1.30	-1.78	-1.19	-0.99
AT	2.22	-1.37	-1.36	-1.63	-1.98	-1.54	-1.31
CEc	1.24	-1.06	-0.87	-0.84	-1.12	-0.87	-0.53
KNa	1.23	-1.00	-0.84	-0.43	-1.10	-0.88	-0.76

Appendix

Table 15: Capability AUCchange (

\Delta

) for Llama-3.2-3B in the one-minus-accuracy reward setting. Lowest values per condition are bold. “–” indicates missing values, e.g., where a fit was not possible.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	2.42	0.08	0.18	0.35	-0.01	0.37	0.21
AT	2.95	0.15	0.36	0.75	0.37	0.67	0.51
CEc	1.82	0.30	1.09	1.13	-0.16	0.68	1.09
KNa	1.62	0.78	0.96	0.80	-0.20	1.34	0.89

Appendix

Table 16: Capability AUCchange (

\Delta

) for Llama-3.1-8B in the accuracy reward setting. Highest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	2.42	-0.87	-0.77	-2.27	-2.12	-1.67	-1.70
AT	2.95	-1.38	-1.41	-2.76	-2.35	-1.88	-1.84
CEc	1.82	-1.19	-1.25	-1.56	-1.48	-1.26	-1.13
KNa	1.62	-1.34	-1.42	-1.57	-1.49	-1.32	-0.97

Appendix

Table 17: Capability AUCchange (

\Delta

) for Llama-3.1-8B in the one-minus-accuracy reward setting. Lowest values per condition are bold. “–” indicates missing values, e.g., where a fit was not possible.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	2.80	-0.05	0.08	0.20	0.20	0.07	-0.02
AT	3.38	0.02	0.16	0.22	0.25	0.16	-0.14
CEc	2.64	0.26	0.13	0.53	0.20	0.21	0.24
KNa	3.17	-0.27	-0.22	-0.24	-0.24	-0.06	-0.02

Appendix

Table 18: Capability AUCchange (

\Delta

) for Qwen3-4B in the accuracy reward setting. Highest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	2.80	-1.29	-1.29	-0.16	-0.42	-0.22	-0.24
AT	3.38	-1.67	-1.71	-0.29	-1.17	-0.39	-0.30
CEc	2.64	-1.66	-0.68	0.00	-0.04	0.17	-0.01
KNa	3.17	-2.61	-1.52	-0.40	-0.49	-0.14	-0.19

Appendix

Table 19: Capability AUCchange (

\Delta

) for Qwen3-4B in the one-minus-accuracy reward setting. Lowest values per condition are bold. “–” indicates missing values, e.g., where a fit was not possible.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	3.03	-0.08	-0.01	0.04	0.03	0.03	-0.03
AT	3.54	0.06	0.11	0.25	0.13	0.16	0.03
CEc	2.83	0.21	0.03	0.34	0.19	0.23	0.43
KNa	3.05	-0.31	-0.19	-0.32	0.05	-0.04	-0.20

Appendix

Table 20: Capability AUCchange (

\Delta

) for Qwen3-8B in the accuracy reward setting. Highest values per condition are bold.

		Capability AUC $\Delta$ vs. Baseline
	Baseline	QLl $=0$	QLq $=0$	QLl $>$ QLq		QLq $>$ QLl
				+1	+2	+1	+2
AS	3.03	-1.16	-0.26	-0.20	-1.05	-0.27	-0.34
AT	3.54	-1.83	-0.55	-0.26	-1.96	-0.31	-0.39
CEc	2.83	-1.70	0.05	0.14	-0.46	-0.48	-0.32
KNa	3.05	-2.41	-0.17	-0.26	-0.25	-0.17	-0.19

Appendix

Table 21: Capability AUCchange (

\Delta

) for Qwen3-8B in the one-minus-accuracy reward setting. Lowest values per condition are bold.

Reward	Train	Test	Baseline			$\Delta$ (pp)		Imp./Worse
Reward	Train	Test	Min	Median	Max	Mean	Median	Imp./Worse
Acc.	TRAM-Ari.	TRAM-Ari.	25.74	70.09	87.74	23.12	14.65	6/0
		TRAM-Sem.	10.00	61.40	70.60	11.13	1.90	5/0
		ToT-Ari.	15.37	50.22	72.08	0.83	1.73	4/2
		ToT-Sem.	27.86	42.50	55.86	0.86	1.07	4/2
	TRAM-Sem.	TRAM-Ari.	25.74	70.09	87.74	3.35	1.52	3/3

Appendix

Table 22: Model performance deltas relative to baseline (percentage points) for both reward settings. Includes the number of models which improved or worsened their performance after fine-tuning.

Dimension	$\boldsymbol{\Delta}$ AUC
AS	+0.01
AT	-0.02
CEc	+0.14
KNa	-0.04
KNc	+0.02
KNf	+0.08

Appendix

Table 23: Mean capability AUC

\Delta

versus baseline after GRPO on non-temporal QL-dominant data, averaged across models. Highest values are bold.

	Baseline	QL-dominant.
	Acc.	Acc.	$\boldsymbol{\Delta}$
Llama-3.2-3B	24.85	25.40	+0.55
Llama-3.1-8B	33.33	32.87	-0.46
gemma-3-4b	43.42	43.71	+0.29
gemma-3-12b	62.53	62.78	+0.25
Qwen3-4b	53.59	52.36	-1.23

Appendix

Table 24: Overall accuracy on the temporal mix (%) after GRPO on non-temporal QL-dominant training data.

\Delta

is percentage points versus each model’s baseline.

Temporal Reasoning in Language Models

Temporal reasoning in language models studies how models represent, order, update, and apply facts and events across time.

See all 98 papers

Measuring Iterative Temporal Reasoning with Time Puzzles
Jan 12, 2026Zhengxiang Wang, Zeyu DongLLM Evaluation Temporal Reasoning in Language Models
Department of Applied Math and Statistics Stony Brook University
What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?
Mar 19, 2026Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard +1Tokenization in Language Models Temporal Reasoning in Language Models
University of Aberdeen · Université Grenoble Alpes & CNRS
Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives
Oct 12, 2026Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim +9Temporal Reasoning in Language Models Relation Extraction
Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh

Jan 12, 2026J/K move · Enter open · S save

Measuring Iterative Temporal Reasoning with Time Puzzles

Zhengxiang Wang, Zeyu Dong

Department of Applied Math and Statistics Stony Brook University

LLM Evaluation Temporal Reasoning in Language Models Temporal Reasoning LLM Tool Use

Open paper AskPDF arXiv

Tool use, such as web search, has become a standard capability even in freely available large language models (LLMs). However, existing benchmarks evaluate temporal reasoning mainly in static, non-tool-using settings, which poorly reflect how LLMs perform temporal reasoning in practice. We introduce Time Puzzles, a constraint-based date inference task for evaluating iterative temporal reasoning with tools. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations and may admit one or multiple valid dates. The puzzles are algorithmically generated, enabling controlled and continual evaluation. Across 13 LLMs, even the best model (GPT-5) achieves only 55.3% accuracy without tools, despite using easily searchable facts. While web search improves performance, models perform substantially better when constraints are rewritten with explicit dates, removing the need for factual lookup. These results reveal a gap in reliable tool use for iterative temporal reasoning.

Abstract

Temporal Reasoning in Language Models

Explore similar work

Measuring Iterative Temporal Reasoning with Time Puzzles

What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives