Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sensitivity, and unstable model behavior. Existing evaluations focus on in-domain performance, offering limited insight into reliability under distribution shifts. We evaluate multiple model configurations across families, architectures, and reasoning strategies over four dimensions of generalization, examining transfer from base performance, cross-dimensional correlations, and the effects of scale, architecture, and prompting. This provides a systematic study of how prompted LLMs generalize in time and event expression extraction tasks. We find that strong base-task performance generally predicts better generalization. However, this relationship weakens under substantial distribution shifts. Inductive prompting performs most consistently across domain shift, adversarial perturbations, compositionality, and length increase, while gains from scale, architecture, and deductive and abductive prompting strategies are uneven and dimension-specific. We conclude that LLM generalization in temporal extraction tasks cannot be predicted from any single dimension alone and cannot be reliably inferred from in-domain or single-dimension evaluations, highlighting the need for reasoning strategies that generalize across dimensions.
Figures & tables
Figure 1: Generalization Dimensions.
Figure 2: Pipeline for evaluation across tasks, architectures, and reasoning strategies.
Figure 3: If M1 is significantly better than M2 on the base case, what is the probability that the advantage will hold for different generalization dimensions?
Figure 4: Generalization probabilities for eventx , split into inductive+ (left) and non-inductive (right) strategies.
Figure 5: Spearman’s Rank Correlation ρ between the source and the target datasets for different generalization dimensions: temporal and event expression extraction.
Figure 6: Extraction error analysis on T2C , pooled across all reasoning strategies.
Figure 7: t-SNE projections of final-layer roberta-base token embeddings from the Base, T2C , and Voc datasets.
Figure 8: Percentage of models that perform significantly better , worse , or almost equal for reasoning strategy i vs. j ( Hyp = i vs j ), summed over all generalization dimensions. The upper and lower panels show the results for timex ( T ) and eventx ( E ) tasks, respectively.
Figure 9: Percentage of Dense models that perform significantly better , worse , or almost equal for the corresponding MoE model in a particular dimension ( dim = x ), summed over all reasoning strategies.
Figure 10: Generalization performance of Mistral Dense models with varying parameter sizes.
Figure 11: The proportion of timex (upper panel) and eventx (lower panel) datasets where a fine-tuned PLM mi is significantly better, worse, or almost equal to another model mj ( hyp = mi vs mj ).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Base task performance in inductive+ vs. non-inductive strategies.
Figure 13: ρ across generalization dimensions for event expression extraction in inductive+ and non-inductive strategies.
Figure 14: Percentage of models that perform significantly better , worse , or almost equal for reasoning strategy i vs. j , expressed as Hyp = i vs j , in each dimension ( Dim = x ) for time expression extraction tasks . Each column corresponds to one dimension.
Figure 15: Percentage of models that perform significantly better , worse , or almost equal for reasoning strategy i vs. j , expressed as Hyp = i vs j , in each dimension ( Dim = x ) for event expression extraction tasks . Each column corresponds to one dimension.
Figure 16: Inductive prompting template used for time expression extraction.
Figure 17: Deductive prompting template used for time expression extraction.
Figure 18: Abductive prompting template used for time expression extraction.
Figure 19: Inductive, deductive, and abductive prompting template used for time expression extraction.
Rank
Base timex
T2C timex
Base eventx
T2C eventx
1
friday (25)
the time (9)
said (187)
admission (279)
2
tuesday (25)
the day (9)
talks (27)
discharge (237)
3
monday (24)
that time (5)
told (20)
showed (139)
4
wednesday (17)
the night (5)
take (20)
admitted (122)
5
thursday (17)
day of life two (5)
meeting (18)
transferred (116)
6
now (14)
one (4)
war (14)
revealed (74)
Appendix
Table 1: Top 10 most frequent time and event expressions across the Base ( Base ) and clinical ( T2C ) datasets.
Figure 20: Example instances from the six dataset settings used in our time expression extraction experiments: base, adversarial, domain, domain-vocab, compositional, and structural. In the adversarial example, perturbed tokens are highlighted in red.
Figure 21: Example instances from the six dataset settings used in our event extraction experiments: base, adversarial, domain, domain-vocab, compositional, and structural. In the adversarial example, perturbed tokens are highlighted in red.
Model
Reasoning
Base
T2C
Adv
Length
Adv_P
Voc
Llama 17B
D
0.75
0.83
0.78
0.75
0.75
0.73
I
0.82
0.88
0.89
0.79
0.77
0.78
A
0.83
0.86
0.84
0.79
0.81
0.84
D+I
0.82
0.82
0.80
0.69
0.78
0.79
D+A
0.81
0.82
0.78
0.79
0.83
0.84
I+A
0.84
0.87
0.89
0.79
0.85
0.89
Appendix
Table 2: Time expression extraction results for MoE models under different reasoning strategies. Scores are reported using micro F1.
Model
Reasoning
Base
T2C
Adv
Length
Adv_P
Voc
Llama 17B
D
0.77
0.24
0.77
0.66
0.50
0.71
I
0.82
0.23
0.82
0.66
0.51
0.70
A
0.72
0.22
0.75
0.54
0.53
0.74
D+I
0.80
0.24
0.77
0.60
0.50
0.70
D+A
0.77
0.23
0.79
0.43
0.54
0.77
I+A
0.82
0.26
0.83
0.60
0.56
0.81
Appendix
Table 3: Event expression extraction results for MoE models under different reasoning strategies. Scores are reported using micro F1.
Model
Reasoning
Base
T2C
Adv
Length
Adv_P
Voc
Llama 8B
D
0.67
0.79
0.66
0.59
0.70
0.67
I
0.76
0.82
0.75
0.62
0.76
0.76
A
0.72
0.83
0.71
0.30
0.76
0.72
D+I
0.76
0.82
0.76
0.62
0.77
0.78
D+A
0.71
0.83
0.70
0.30
0.77
0.71
I+A
0.76
0.83
0.76
0.58
0.82
0.77
Appendix
Table 4: Time expression extraction results for dense models under different reasoning strategies. Scores are reported using micro F1.
Model
Reasoning
Base
T2C
Adv
Length
Adv_P
Voc
Llama 8B
D
0.55
0.35
0.52
0.35
0.46
0.54
I
0.74
0.27
0.75
0.48
0.54
0.76
A
0.54
0.29
0.57
0.33
0.49
0.55
D+I
0.75
0.28
0.75
0.46
0.55
0.75
D+A
0.51
0.21
0.48
0.08
0.34
0.50
I+A
0.63
0.26
0.66
0.33
0.50
0.65
Appendix
Table 5: Event expression extraction results for dense models under different reasoning strategies. Scores are reported using micro F1.
Model
Reasoning
Base
Adv
T2C
Length
Mistral 3B
D
0.60
0.59
0.44
0.36
I
0.68
0.64
0.44
0.46
A
0.28
0.28
0.41
0.13
D+I
0.67
0.65
0.35
0.41
D+A
0.31
0.29
0.30
0.05
I+A
0.39
0.37
0.32
0.22
Appendix
Table 6: Event expression extraction results for model size analysis. Scores are reported using micro F1.
Model
Reasoning
Base
Adv
T2C
Length
Mistral 3B
D
0.62
0.60
0.77
0.52
I
0.78
0.77
0.79
0.68
A
0.69
0.68
0.80
0.49
D+I
0.77
0.77
0.78
0.67
D+A
0.68
0.64
0.79
0.50
I+A
0.73
0.73
0.82
0.63
Appendix
Table 7: Time expression extraction results for model size analysis. Scores are reported using micro F1.
Table 29
Time Expression Extraction (Fine-tuned Models)
Model
Base
T2C
Adv
Length
Voc
Adv_P
RoBERTa
0.93
0.69
0.92
0.88
0.93
0.79
T5
0.90
0.55
0.89
0.83
0.90
0.79
GPT-2
0.88
0.48
0.86
0.83
0.88
0.74
Event Expression Extraction (Fine-tuned Models)
Model
Base
T2C
Adv
Length
Voc
Adv_P
Appendix
Table 10: Performance of fine-tuned models on time and event expression extraction across multiple generalization dimensions. Scores are reported using micro F1.
Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.
Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim +9
Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh
Large language models (LLMs) often fail to reason under temporal cutoffs: when prompted to answer from the standpoint of an earlier time, they exploit knowledge that became available only later. We study this failure through the lens of ex-ante reasoning, where a model must rely exclusively on information knowable before a cutoff. Through a systematic analysis of prompt-level interventions, we find that temporal leakage is highly sensitive to cutoff formulation and instruction placement: explicit cutoff statements outperform implicit historical framings, and prefix constraints reduce leakage more effectively than suffix constraints. These findings indicate that prompting can steer models into a temporal frame, but does not endow them with the ability to verify whether a response is temporally admissible. We further argue that supervised fine-tuning is insufficient, since ex-ante correctness is not an intrinsic property of an answer, but a relation between the answer and the cutoff. To address this gap, we propose TCFT, a Temporal Critique Fine-Tuning framework that trains models to acquire cutoff-aware temporal verification. Given a query, a cutoff, and a candidate response, TCFT teaches the model to identify post-cutoff leakage, explain temporal boundary violations, and judge temporal admissibility. Experiments with Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct show that TCFT consistently outperforms prompting and SFT baselines, reducing average leakage by 41.89 and 37.79 percentage points, respectively.
Chenlu Ding, Jiancan Wu, Yanchen Luo +3
University of Science and Technology of China · The Hong Kong Polytechnic University · University of Notre Dame
Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations. We apply LLMs to a variety of forecasting tasks and inspect their internal states using sparse autoencoders to understand whether they appear to rely on time-specific pieces of knowledge versus generalizable patterns. Our analyses identify features associated with both time-aware reasoning and look-ahead-biased reasoning. We then apply the LLMs to an entirely different domain and intervene on these features. We find that amplifying time-awareness features substantially reduces look-ahead bias on forecasting prompts while preserving general reasoning performance. In contrast, steering the candidate look-ahead-bias features does not produce an effect. These results suggest that interpretable temporal features can be used to causally shift LLMs toward more historically grounded reasoning.