Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks

Fahmid Shahriar Iqbal; Ritam Dutt; Soumitra Das; Arnav Verma; Sagnik Ray Choudhury

cs.CLOct 1, 2026

Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks

Authors: Fahmid Shahriar Iqbal, Ritam Dutt, Soumitra Das, Arnav Verma, Sagnik Ray Choudhury

Organizations: University of North Texas · Carnegie Mellon University · University of Massachusetts Amherst · New York University

Abstract

Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sensitivity, and unstable model behavior. Existing evaluations focus on in-domain performance, offering limited insight into reliability under distribution shifts. We evaluate multiple model configurations across families, architectures, and reasoning strategies over four dimensions of generalization, examining transfer from base performance, cross-dimensional correlations, and the effects of scale, architecture, and prompting. This provides a systematic study of how prompted LLMs generalize in time and event expression extraction tasks. We find that strong base-task performance generally predicts better generalization. However, this relationship weakens under substantial distribution shifts. Inductive prompting performs most consistently across domain shift, adversarial perturbations, compositionality, and length increase, while gains from scale, architecture, and deductive and abductive prompting strategies are uneven and dimension-specific. We conclude that LLM generalization in temporal extraction tasks cannot be predicted from any single dimension alone and cannot be reliably inferred from in-domain or single-dimension evaluations, highlighting the need for reasoning strategies that generalize across dimensions.

Figures & tables

Figure 1: Generalization Dimensions.

Figure 2: Pipeline for evaluation across tasks, architectures, and reasoning strategies.

Figure 3: If

\mathcal{M}_{1}

is significantly better than

\mathcal{M}_{2}

on the base case, what is the probability that the advantage will hold for different generalization dimensions?

Figure 4: Generalization probabilities for eventx , split into inductive+ (left) and non-inductive (right) strategies.

Panel of Figure 5 — Figure 5: Spearman’s Rank Correlation $\rho$ between the source and the target datasets for different generalization dimensions: temporal and event expression extraction.

Figure 6: Extraction error analysis on T2C , pooled across all reasoning strategies.

Panel of Figure 7 — Figure 7: t-SNE projections of final-layer roberta-base token embeddings from the Base, T2C , and Voc datasets.

Figure 8: Percentage of models that perform significantly better , worse , or almost equal for reasoning strategy

i

vs.

j

( Hyp = i vs j ), summed over all generalization dimensions. The upper and lower panels show the results for timex ( T ) and eventx ( E ) tasks, respectively.

Figure 9: Percentage of Dense models that perform significantly better , worse , or almost equal for the corresponding MoE model in a particular dimension ( dim = x ), summed over all reasoning strategies.

Figure 10: Generalization performance of Mistral Dense models with varying parameter sizes.

Figure 11: The proportion of timex (upper panel) and eventx (lower panel) datasets where a fine-tuned PLM

m_{i}

is significantly better, worse, or almost equal to another model

m_{j}

( hyp =

m_{i}

m_{j}

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Base task performance in inductive+ vs. non-inductive strategies.

Panel of Figure 13 — Figure 13: $\rho$ across generalization dimensions for event expression extraction in inductive+ and non-inductive strategies.

Figure 14: Percentage of models that perform significantly better , worse , or almost equal for reasoning strategy

i

vs.

j

, expressed as Hyp = i vs j , in each dimension ( Dim = x ) for time expression extraction tasks . Each column corresponds to one dimension.

Figure 15: Percentage of models that perform significantly better , worse , or almost equal for reasoning strategy

i

vs.

j

, expressed as Hyp = i vs j , in each dimension ( Dim = x ) for event expression extraction tasks . Each column corresponds to one dimension.

Panel of Figure 16 — Figure 16: Inductive prompting template used for time expression extraction.

Panel of Figure 17 — Figure 17: Deductive prompting template used for time expression extraction.

Panel of Figure 18 — Figure 18: Abductive prompting template used for time expression extraction.

Panel of Figure 19 — Figure 19: Inductive, deductive, and abductive prompting template used for time expression extraction.

Rank	Base timex	T2C timex	Base eventx	T2C eventx
1	friday (25)	the time (9)	said (187)	admission (279)
2	tuesday (25)	the day (9)	talks (27)	discharge (237)
3	monday (24)	that time (5)	told (20)	showed (139)
4	wednesday (17)	the night (5)	take (20)	admitted (122)
5	thursday (17)	day of life two (5)	meeting (18)	transferred (116)
6	now (14)	one (4)	war (14)	revealed (74)

Appendix

Table 1: Top 10 most frequent time and event expressions across the Base ( Base ) and clinical ( T2C ) datasets.

Panel of Figure 20 — Figure 20: Example instances from the six dataset settings used in our time expression extraction experiments: base, adversarial, domain, domain-vocab, compositional, and structural. In the adversarial example, perturbed tokens are highlighted in red.

Panel of Figure 21 — Figure 21: Example instances from the six dataset settings used in our event extraction experiments: base, adversarial, domain, domain-vocab, compositional, and structural. In the adversarial example, perturbed tokens are highlighted in red.

Model	Reasoning	Base	T2C	Adv	Length	Adv_P	Voc
Llama 17B	D	0.75	0.83	0.78	0.75	0.75	0.73
	I	0.82	0.88	0.89	0.79	0.77	0.78
	A	0.83	0.86	0.84	0.79	0.81	0.84
	D+I	0.82	0.82	0.80	0.69	0.78	0.79
	D+A	0.81	0.82	0.78	0.79	0.83	0.84
	I+A	0.84	0.87	0.89	0.79	0.85	0.89

Appendix

Table 2: Time expression extraction results for MoE models under different reasoning strategies. Scores are reported using micro F1.

Model	Reasoning	Base	T2C	Adv	Length	Adv_P	Voc
Llama 17B	D	0.77	0.24	0.77	0.66	0.50	0.71
	I	0.82	0.23	0.82	0.66	0.51	0.70
	A	0.72	0.22	0.75	0.54	0.53	0.74
	D+I	0.80	0.24	0.77	0.60	0.50	0.70
	D+A	0.77	0.23	0.79	0.43	0.54	0.77
	I+A	0.82	0.26	0.83	0.60	0.56	0.81

Appendix

Table 3: Event expression extraction results for MoE models under different reasoning strategies. Scores are reported using micro F1.

Model	Reasoning	Base	T2C	Adv	Length	Adv_P	Voc
Llama 8B	D	0.67	0.79	0.66	0.59	0.70	0.67
	I	0.76	0.82	0.75	0.62	0.76	0.76
	A	0.72	0.83	0.71	0.30	0.76	0.72
	D+I	0.76	0.82	0.76	0.62	0.77	0.78
	D+A	0.71	0.83	0.70	0.30	0.77	0.71
	I+A	0.76	0.83	0.76	0.58	0.82	0.77

Appendix

Table 4: Time expression extraction results for dense models under different reasoning strategies. Scores are reported using micro F1.

Model	Reasoning	Base	T2C	Adv	Length	Adv_P	Voc
Llama 8B	D	0.55	0.35	0.52	0.35	0.46	0.54
	I	0.74	0.27	0.75	0.48	0.54	0.76
	A	0.54	0.29	0.57	0.33	0.49	0.55
	D+I	0.75	0.28	0.75	0.46	0.55	0.75
	D+A	0.51	0.21	0.48	0.08	0.34	0.50
	I+A	0.63	0.26	0.66	0.33	0.50	0.65

Appendix

Table 5: Event expression extraction results for dense models under different reasoning strategies. Scores are reported using micro F1.

Model	Reasoning	Base	Adv	T2C	Length
Mistral 3B	D	0.60	0.59	0.44	0.36
	I	0.68	0.64	0.44	0.46
	A	0.28	0.28	0.41	0.13
	D+I	0.67	0.65	0.35	0.41
	D+A	0.31	0.29	0.30	0.05
	I+A	0.39	0.37	0.32	0.22

Appendix

Table 6: Event expression extraction results for model size analysis. Scores are reported using micro F1.

Model	Reasoning	Base	Adv	T2C	Length
Mistral 3B	D	0.62	0.60	0.77	0.52
	I	0.78	0.77	0.79	0.68
	A	0.69	0.68	0.80	0.49
	D+I	0.77	0.77	0.78	0.67
	D+A	0.68	0.64	0.79	0.50
	I+A	0.73	0.73	0.82	0.63

Appendix

Table 7: Time expression extraction results for model size analysis. Scores are reported using micro F1.

Table 29

Time Expression Extraction (Fine-tuned Models)
Model	Base	T2C	Adv	Length	Voc	Adv_P
RoBERTa	0.93	0.69	0.92	0.88	0.93	0.79
T5	0.90	0.55	0.89	0.83	0.90	0.79
GPT-2	0.88	0.48	0.86	0.83	0.88	0.74
Event Expression Extraction (Fine-tuned Models)
Model	Base	T2C	Adv	Length	Voc	Adv_P

Appendix

Table 10: Performance of fine-tuned models on time and event expression extraction across multiple generalization dimensions. Scores are reported using micro F1.

Language Model Generalization

See all 23 papers

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives
Oct 12, 2026Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim +9Temporal Reasoning in Language Models Relation Extraction
Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh
Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning
May 14, 2026Chenlu Ding, Jiancan Wu, Yanchen Luo +3LLM Prompting LLM Fine-Tuning
University of Science and Technology of China · The Hong Kong Polytechnic University · University of Notre Dame
Forecasting With LLMs: Improved Generalization Through Feature Steering
Jun 25, 2026Humzah Merchant, Bradford LevyLook-Ahead Bias in Language Models LLM Interpretability
University of Chicago, Illinois, United States.

Oct 12, 2026J/K move · Enter open · S save

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim, Multazam Mahmud+8

Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh

Temporal Reasoning in Language Models Relation Extraction LLM Evaluation Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.

Abstract

Language Model Generalization

Explore similar work

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning

Forecasting With LLMs: Improved Generalization Through Feature Steering