Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Adib Sakhawat; Fardeen Sadab; Tamjid Hasan Fahim; Multazam Mahmud; Tahsan Ferdous Lihan; Md Isfak Iqbal Chowdhury; Nafis Ahnaf Jamil; Md. Atik Shahriar

cs.CLOct 12, 2026

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Authors: Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim, Multazam Mahmud, Tahsan Ferdous Lihan, Md Isfak Iqbal Chowdhury, Nafis Ahnaf Jamil, Md. Atik Shahriar,

Organizations: Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh

Abstract

Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.

Figures & tables

Figure 1: A model must return the relation for every event pair of a passage (schematic). Blue cells are scored against gold; orange cells must be their converses. \scriptsize1⃝ Malformed rows: one symbol too many and the whole row is lost, so the four models with non-trivial coverage answer only 34–60% of cells. \scriptsize2⃝ Converse violations: m above but

>

below, in 0.3–37.6% of pairs. \scriptsize3⃝ Consistent, still wrong: gold o read as

<

>

; overlaps stay below 0.07 F1.

Figure 2: Task formulation. (a) The input is a passage

T

and

n

events, each a verbatim span

E_{k}

T

. (b) The model returns a relation matrix

R=[r_{ij}]\in\mathcal{A}^{n\times n}

with exactly one relation for every ordered pair and a dash on the diagonal. (c) The 13 Allen relations between

E_{i}

(black) and

E_{j}

(blue). A well-formed answer is (d) converse consistent, so that

r_{ji}=r_{ij}^{-1}

(here

r_{12}=\texttt{m}

and

r_{21}=\texttt{mi}

), and (e) satisfiable: there are intervals

[s_{k},e_{k}]

with

s_{k}<e_{k}

that realise every relation, as the timeline does for the example matrix in (b).

	All 101,880 pairs (missing = wrong)						Answered cells
System	Accuracy [95% CI]	Macro-F1 [95% CI]	$\kappa$	F1 7	F1 4	Coverage	Accuracy	Macro-F1
DeepSeek	0.452 [0.396, 0.515]	0.270 [0.244, 0.300]	0.263	0.292	0.496	0.600	0.753	0.349
GPT-6-Luna	0.350 [0.301, 0.404]	0.215 [0.187, 0.246]	0.183	0.244	0.391	0.481	0.728	0.329
Gemini	0.328 [0.284, 0.379]	0.128 [0.113, 0.144]	0.105	0.185	0.325	0.577	0.569	0.179
Qwen	0.202 [0.169, 0.242]	0.119 [0.099, 0.141]	0.083	0.157	0.267	0.345	0.585	0.232
Mistral	0.001 [0.000, 0.002]	0.003 [0.002, 0.004]	0.000	0.006	0.012	0.024	0.049	0.024

Table 1: Agreement with the gold matrices on the upper triangle. F1 7 and F1 4 are macro-F1 after collapsing to 7 undirected relations and to 4 classes.

\kappa

treats a missing cell as its own category; since gold never abstains, that category appears in one margin only, depressing

\kappa

as the missing rate grows, so these values are not comparable with

\kappa

elsewhere. On each model’s answered cells, always before reaches 0.594 (DeepSeek), 0.620 (GPT-6-Luna), 0.612 (Gemini), 0.624 (Qwen) and 0.633 (Mistral) accuracy, and below 0.06 macro-F1.

Figure 3: Coverage (a) and accuracy on answered cells (b) by the number of events in the passage (152, 127, 48 and 33 passages per bin).

Figure 4: Per-relation F1 on all pairs (missing counted as wrong). The bottom line of each column label is the number of gold pairs with that relation.

System	Conv. viol.	Both dirs.	Sat. / complete	Triple viol.	Self- entail.
Gold	–	–	328/360	0.06	–
DeepSeek	0.3	39.9	212/242	0.12	0.9
GPT-6-Luna	6.4	27.0	138/222	2.83	12.0
Gemini	21.3	25.6	68/207	3.08	11.0
Qwen	37.6	12.8	19/114	6.76	21.3
Mistral	90.1	0.0	1/4	56.26	34.0

Table 2: Logical consistency (all values in %, except satisfiable networks among complete networks). Complete here means every cell of the upper triangle was answered, which is a weaker requirement than the complete-matrix parse status of Table 3 (every cell of both triangles); the counts in this column are therefore higher. Conv. viol.: converse violations among pairs with both cells answered. Both dirs.: share of all pairs whose two cells are both correct. Triple viol.: path-inconsistent triples among checked triples. Self-entail.: answered pairs that contradict a relation entailed by the model’s own answers for two other pairs.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Methodology pipeline. Everything in the dashed band was fixed in advance. Each model answers every passage, the parser turns each answer into an

n\times n

matrix with missing cells, and all analyses read the stored matrices: the upper triangle is scored against gold, the lower triangle serves as a converse test, and the endpoint graph gives the satisfiability check and a best-found consistent repair. Uncertainty comes from a passage-level bootstrap with Holm correction and GEE models clustered by passage.

Model	OpenRouter identifier	Complete / partial / failed (%)	Trunc. (%)	Completion tokens	Cost (USD)	Median latency (s)	Macro-F1 per USD
DeepSeek	deepseek/deepseek-v4.1-flash	66.7 / 29.7 / 3.6	3.3	7,461,014	4.025	123.2	0.067
GPT-6-Luna	openai/gpt-6-luna	56.9 / 40.8 / 2.2	0.3	1,933,773	1.007	44.0	0.214
Gemini	google/gemini-3.5-flash-lite	55.3 / 42.8 / 1.9	0.0	1,188,712	3.076	11.7	0.042
Qwen	qwen/qwen3.7-flash	28.6 / 58.6 / 12.8	0.8	5,290,213	0.694	87.5	0.171
Mistral	mistralai/mistral-nemo	0.6 / 35.0 / 64.4	11.7	1,028,362	0.040	10.2	0.075

Appendix

Table 3: Models, reliability and cost over the 360 passages. Reasoning effort was set to medium for all models except Mistral, which has no reasoning parameter; the output limit was 65,536 tokens (16,384 for Mistral). Temperature 0 and seed 20261006 were sent to every model that accepts them (GPT-6-Luna does not accept temperature). A call is complete if every cell of the full

n\times n

matrix was parsed, partial if at least one cell is missing and failed if no cell is usable. Table 2 instead counts networks whose upper triangle is complete, a weaker condition, so its counts are higher. Transport-level retries were needed for 19 (DeepSeek), 5 (GPT-6-Luna), 20 (Gemini), 18 (Qwen) and 30 (Mistral) calls.

Model	Usable	Short	Long	$\|\Delta\|=1$	$\|\Delta\|\leq 2$
DeepSeek	5,774	172	1,121	77.5	91.3
GPT-6-Luna	5,018	2,158	318	46.4	62.0
Gemini	5,379	421	1,825	66.3	77.2
Qwen	3,887	968	1,878	40.6	60.9
Mistral	360	2,791	2,378	11.5	20.6

Appendix

Table 4: Matrix rows emitted for the 7,627 events, by whether they have a usable length or too few (short) or too many (long) symbols. The last two columns give the percentage of malformed rows that miss the required length by one symbol or by at most two.

Hypothesis (fixed in advance)	Outcome
H1a: models beat the trivial baselines by a wide margin on macro-F1 but much less on accuracy.	Not supported as stated. Four models exceed the baselines on macro-F1, but under the pre-registered parser all fall below them in accuracy; under the recovery parser the best model is level with them (§ 5.2 ).
H1b: the rare relations s , f , o , m score lowest.	Partly supported: o and oi are lowest; s and m are not.
H2: converse violations concentrate in the non-symmetric fine relations ( s / si , f / fi , d / di ).	Partly supported: true for Gemini and Qwen, not for GPT-6-Luna; DeepSeek has almost no violations.
H3: offline repair raises accuracy slightly, but consistency is not the same as correctness.	Supported, with two qualifications: the accuracy gain comes almost entirely from filling missing cells, and a control that fills them with the majority label beats the solver for every model.
H4: most errors are near misses differing in one endpoint comparison.	Not supported: 23.8% to 33.4% of errors have severity 1.
H5: a large share of errors on distant pairs are composition-available.	Not supported: at most 4.4%, against 66–78% availability on cells the models get right, so the low rate is not an artifact of premise scarcity.

Appendix

Table 5: Pre-registered hypotheses and outcomes, based on the four models with non-trivial coverage. Holm correction applies within this family; analyses reported elsewhere in the paper that are not listed here are exploratory.

Model	All	Gold- consistent	Fully linear	Non- linear	Excl. 548, 550
DeepSeek	0.270	0.290	0.393	0.247	0.273
GPT-6-Luna	0.215	0.244	0.369	0.188	0.218
Gemini	0.128	0.143	0.226	0.110	0.129
Qwen	0.119	0.138	0.187	0.104	0.121
Mistral	0.003	0.003	0.003	0.002	0.003

Appendix

Table 6: Macro-F1 on subsets: all 360 passages, the 328 passages with satisfiable gold networks, the 90 fully linear and 270 non-linear passages, and all passages except two that repeat event wording.

Figure 7: (a) Macro-F1 by sentence distance between the two events. (b) Macro-F1 by passage non-linearity (90, 106, 57 and 107 passages), with the text-order baseline. (c) Accuracy by the position of the row in the generated matrix (missing cells counted as wrong).

Figure 8: Row-normalised confusion matrices (%) for the two strongest models, including missing cells. Values below 0.5% are omitted.

Figure 9: (a) Cohen’s $\kappa$ between systems, computed on the cells both systems answered (the gold annotation answers every cell). (b) Total cost against macro-F1.

$r$	Name	Condition	ss	se	es	ee
<	before	$x^{+}<y^{-}$	$<$	$<$	$<$	$<$
>	after	$y^{+}<x^{-}$	$>$	$>$	$>$	$>$
m	meets	$x^{+}=y^{-}$	$<$	$<$	$=$	$<$
mi	met-by	$y^{+}=x^{-}$	$>$	$=$	$>$	$>$
o	overlaps	$x^{-}<y^{-}<x^{+}<y^{+}$	$<$	$<$	$>$	$<$
oi	overlapped-by	$y^{-}<x^{-}<y^{+}<x^{+}$	$>$	$<$	$>$	$>$

Appendix

Table 7: The 13 basic relations for

X\ r\ Y

(listed in converse pairs), their defining conditions and endpoint signatures: the order of

x^{-}

versus

y^{-}

(ss),

x^{-}

versus

y^{+}

(se),

x^{+}

versus

y^{-}

(es) and

x^{+}

versus

y^{+}

(ee).

< m o s d

full

o s d

fi = f

di si oi mi >

< m o

< m o fi di

o fi di

o s d

conc

di si oi

di si oi mi >

o s d

fi = f

di si oi

di si oi mi >

< m o fi di

o fi di

conc

di si oi

di si oi mi >

< m o

< m o fi di

s = si

d f oi

Appendix

Table 8: Composition table: the cell in row

r_{1}

and column

r_{2}

lists the possible relations between

X

and

Z

given

X\ r_{1}\ Y

and

Y\ r_{2}\ Z

. full denotes all 13 relations and conc the nine relations in which the two intervals share time ( o , oi , s , si , d , di , f , fi , = ).

	Coverage		Accuracy		Macro-F1
Model	strict	rec.	strict	rec.	strict	rec.
DeepSeek	0.600	0.836	0.452	0.630	0.270	0.301
GPT-6-Luna	0.481	0.870	0.350	0.580	0.215	0.243
Gemini	0.577	0.979	0.328	0.535	0.128	0.150
Qwen	0.345	0.636	0.202	0.361	0.119	0.155
Mistral	0.024	0.116	0.001	0.039	0.003	0.014

Appendix

Table 9: Strict (pre-registered) versus recovery parsing of the same stored outputs. Accuracy and macro-F1 are over all 101,880 pairs with missing cells counted as wrong. Always before reaches 0.607 accuracy.

Figure 10: Accuracy after the best-found consistent repair, decomposed into raw accuracy, the change on cells the model answered, and correct labels in cells the model left missing. (i): upper triangle as evidence; (ii): both triangles.

\Delta

F1 is the change in macro-F1.

	Accuracy			Macro-F1
Model	raw	rep. (ii)	fill <	raw	rep. (ii)	fill <
DeepSeek	0.452	0.591	0.702	0.270	0.278	0.284
GPT-6-Luna	0.350	0.476	0.659	0.215	0.220	0.232
Gemini	0.328	0.417	0.582	0.128	0.132	0.142
Qwen	0.202	0.295	0.594	0.119	0.129	0.147

Appendix

Table 10: The best-found consistent repair (variant ii) against a control that fills every missing cell with < . The control is better everywhere.

Figure 11: Accuracy on answered before/after pairs, split by whether event list order and narration order agree with chronology (95% passage-bootstrap CIs). Cell sizes for DeepSeek: 34,447, 1,902, 2,068 and 8,412.

Type	Mentions	Per 100 words	Passages with $\geq$ 1
Absolute anchor	38	0.04	22 (6.1%)
Partial anchor	919	0.89	197 (54.7%)
Offset	494	0.48	246 (68.3%)
Deictic phrase	598	0.58	211 (58.6%)
Connective	2,488	2.42	358 (99.4%)
Past perfect	1,909	1.86	300 (83.3%)

Appendix

Table 11: Temporal expressions in the 360 passages (102,746 words), counted with regular expressions. Categories are defined in the text.

Temporal Reasoning in Language Models

Temporal reasoning in language models studies how models represent, order, update, and apply facts and events across time.

See all 92 papers

Jan 12, 2026

Measuring Iterative Temporal Reasoning with Time Puzzles

Tool use, such as web search, has become a standard capability even in freely available large language models (LLMs). However, existing benchmarks evaluate temporal reasoning mainly in static, non-tool-using settings, which poorly reflect how LLMs perform temporal reasoning in practice. We introduce Time Puzzles, a constraint-based date inference task for evaluating iterative temporal reasoning with tools. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations and may admit one or multiple valid dates. The puzzles are algorithmically generated, enabling controlled and continual evaluation. Across 13 LLMs, even the best model (GPT-5) achieves only 55.3% accuracy without tools, despite using easily searchable facts. While web search improves performance, models perform substantially better when constraints are rewritten with explicit dates, removing the need for factual lookup. These results reveal a gap in reliable tool use for iterative temporal reasoning.

Zhengxiang Wang, Zeyu Dong

Department of Applied Math and Statistics Stony Brook University

LLM Evaluation Temporal Reasoning in Language Models Temporal Reasoning LLM Tool Use

Ask PDF

Mar 19, 2026

What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?

We present MultiTempBench, a multilingual temporal reasoning benchmark spanning three tasks, date arithmetic, time zone conversion, and temporal relation extraction across five languages (English, German, Chinese, Arabic, and Hausa) and multiple calendar conventions (Gregorian, Hijri, and Chinese Lunar). MultiTempBench contains

15,000

examples built by translating

750

curated English questions and expanding each into controlled date-format variants. We evaluate 20 LLMs and introduce the multilingual Date Fragmentation Ratio (mDFR), calibrated with human severity ratings, together with geometric-probing analyses of internal temporal representations. We find tokenisation quality of temporal artefacts is a resource-dependent bottleneck: in low-resource languages and rarer calendar formats, fragmentation disrupts Year/Month/Day separation and accuracy collapses, while high-resource settings are often robust to digit-level splitting. Beyond tokenisation, crossed mixed-effects regression shows that temporal linearity is the strongest predictor of temporal reasoning in high-resource languages, whereas fragmentation is the stronger predictor in low-resource languages. Code is available at: https://github.com/gagan3012/mtb

Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard +1

University of Aberdeen · Université Grenoble Alpes & CNRS

Tokenization in Language Models Temporal Reasoning in Language Models Language Modeling Multilingual Language Model Evaluation Low-Resource Language Processing

Ask PDF

Oct 1, 2026

Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks

Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sensitivity, and unstable model behavior. Existing evaluations focus on in-domain performance, offering limited insight into reliability under distribution shifts. We evaluate multiple model configurations across families, architectures, and reasoning strategies over four dimensions of generalization, examining transfer from base performance, cross-dimensional correlations, and the effects of scale, architecture, and prompting. This provides a systematic study of how prompted LLMs generalize in time and event expression extraction tasks. We find that strong base-task performance generally predicts better generalization. However, this relationship weakens under substantial distribution shifts. Inductive prompting performs most consistently across domain shift, adversarial perturbations, compositionality, and length increase, while gains from scale, architecture, and deductive and abductive prompting strategies are uneven and dimension-specific. We conclude that LLM generalization in temporal extraction tasks cannot be predicted from any single dimension alone and cannot be reliably inferred from in-domain or single-dimension evaluations, highlighting the need for reasoning strategies that generalize across dimensions.

Fahmid Shahriar Iqbal, Ritam Dutt, Soumitra Das +2

University of North Texas · Carnegie Mellon University · University of Massachusetts Amherst +1

Language Model Generalization Temporal Reasoning in Language Models LLM Information Extraction LLM Evaluation Language Model Robustness Document Information Extraction

Ask PDF

Abstract

Temporal Reasoning in Language Models

Explore similar work

Measuring Iterative Temporal Reasoning with Time Puzzles

What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?

Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks