Organizations: Institute of Science Tokyo · NII LLMC · Nara Institute of Science and Technology · Center of Juris-Informatics, ROIS-DS · Osaka University
Legal reasoning requires identifying which version of a statute applies to a case, typically based on when the legally relevant events occurred rather than when the case is adjudicated. Yet whether large language models (LLMs) can reliably perform such temporal legal reasoning remains largely unexplored. We formalize this task as Temporal Applicable-Law Determination (TALD), and construct a TALD-Bench, a benchmark of 51,956 Chinese civil judgments spanning the enactment of the Civil Code. Evaluating nine advanced LLMs, we identify three key findings. First, models exhibit a strong bias toward newer law, achieving over 70% accuracy on new-law cases but below 25% on old-law cases. Second, models demonstrate substantial knowledge of statutory versions and temporal-effect rules, yet struggle to apply this knowledge to cases. Stronger general reasoning performance does not consistently translate into better TALD accuracy, whereas temporal-law hints yield substantial improvements, suggesting a bottleneck in how models deploy their existing capabilities. Third, temporal-law hints also increase uncertainty during statutory-version reasoning, suggesting that they may help models reconsider their default preference for newer law. Our findings highlight the importance of adapting how LLMs apply existing capabilities, rather than relying solely on improvements in legal knowledge or general reasoning.
Figures & tables
Figure 1: Temporal Applicable-law Determination (TALD) as a vital step within legal reasoning, and an illustrative instance of how temporal legal reasoning is implemented by LLMs when faced with a legal issue.
Figure 2: TALD accuracy on old-law and new-law cases (Equation 2.2 ) and the distribution of weighted errors pooled across both splits. Each stacked bar sums to 100%.
Figure 3: Accuracy on the two legal knowledge probes.
Figure 4: Spearman correlations among GPQA, HLE, LCB, and old-law TALD accuracy under (a) no hint, (b) weak hint, and (c) strong hint. All panels share a color scale; diagonal entries display self-correlations (1.000). External benchmark scores and matched configurations are fixed across hint conditions. Details see Appendix E.2 .
Model
Mode
Default
Weak
Strong
Claude Opus 4.6
low
0.021 ± 0.001
0.131 ± 0.011
0.185 ± 0.107
medium
0.019 ± 0.002
0.102 ± 0.008
0.143 ± 0.083
high
0.026 ± 0.002
0.099 ± 0.024
0.154 ± 0.089
max
0.014 ± 0.004
0.113 ± 0.033
0.156 ± 0.091
Gemini-3.1-Pro
low
0.146 ± 0.011
0.501 ± 0.019
0.439 ± 0.035
medium
0.131 ± 0.020
0.503 ± 0.021
0.489 ± 0.016
Table 1: Effect of temporal-law hints on old-law cases, measured by TALD accuracy. Each entry reports the mean ± its across-seed standard error. Default denotes the no-hint setting.
Figure 5: TALD accuracy and version-related policy entropy on old-law cases across hint strength, shown separately for each model. Points show means over three random seeds; error bars show standard errors. TALD accuracy uses a common left-axis range; entropy uses model-specific right-axis ranges. Full results are reported in Table 7 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Area of civil law
Predecessor statute(s)
Civil Code counterpart
General provisions
General Principles of the Civil Law; General Provisions of the Civil Law
Book I: General Provisions
Property
Property Law
Book II: Real Rights
Contract
Contract Law
Book III: Contracts
Personality rights
—
Book IV: Personality Rights
Marriage and family
Marriage Law
Book V: Marriage and Family
Inheritance
Succession Law
Book VI: Succession
Appendix
Table 2: Predecessor statutes and their Civil Code counterparts, grouped by area of civil law. A dash indicates that no separate predecessor statute is listed for that area.
(a) API model IDs used in evaluation
Name in paper
Mode(s) in results
API model ID
DeepSeek-V3.2
chat
deepseek-chat
reasoner
deepseek-reasoner
GLM-4.7
instruct; thinking
z-ai/glm-4.7
GPT-5.4
none; low; medium; high
gpt-5.4
Gemini-3.1-Pro
low; medium; high
gemini-3.1-pro-preview
Appendix
Table 3: Identifiers and evaluated configurations of the nine primary models. Panel (a) reports API request identifiers used during evaluation; panel (b) links to the corresponding Hugging Face model repositories.
Model
Mode
Old-law
New-law
ACC ± SE
ACC ± SE
DeepSeek-V3.2
reasoner
0.092±0.0167
0.746±0.0155
GLM-4.7
thinking
0.080±0.0240
0.787±0.0149
GPT-5.4
high
0.237±0.0048
0.701±0.0070
Gemini-3.1-Pro
high
0.135±0.0130
0.843±0.0198
Claude Opus 4.6
max
0.014±0.0040
0.820±0.0198
Appendix
Table 4: Default-instruction TALD accuracy for the nine primary models, corresponding to Figure 2a . Each entry reports the mean ± its standard error across four sampling seeds; each run samples 100 cases from each split.
Mode
Old-law
New-law
GPT-5.6 Sol
none
0.124±0.0153
0.788±0.0147
low
0.406±0.0164
0.688±0.0310
medium
0.410±0.0102
0.682±0.0246
high
0.366±0.0130
0.710±0.0118
xhigh
0.355±0.0098
0.710±0.0292
Appendix
Table 5: Supplementary evaluation of newer frontier models under the default instruction. These configurations assess whether the old-law performance deficit persists beyond the primary model panel and are not included in the knowledge, hint, or entropy analyses. Entries report mean TALD accuracy and its across-seed standard error.
Model
Mode
External benchmark score
Old-law TALD accuracy
GPQA
HLE
LCB
No hint
Weak hint
Strong hint
DeepSeek-V3.2
chat
75.1
11.2
59.3
0.013
0.028
0.080
reasoner
84.0
24.6
86.2
0.092
0.394
0.436
Claude Opus 4.6
max
89.6
39.9
–
0.014
0.113
0.156
GPT-5.4
none
74.8
11.3
–
0.021
0.087
0.167
low
87.1
30.8
–
0.082
0.444
0.456
Appendix
Table 6: Artificial Analysis benchmark scores (0–100) and old-law TALD accuracy (0–1) under no, weak, and strong hints. TALD columns report the across-seed means from Table 1 . The bottom rows give each benchmark’s matched configuration count and Spearman correlation with TALD under each hint condition; matching is fixed across conditions. Dashes indicate unavailable scores.
Model
Mode
No hint
Weak hint
Strong hint
ACC
Hiver
ACC
Hiver
ACC
Hiver
Qwen3-235B
thinking
0.155 ± 0.025
0.238 ± 0.006
0.524 ± 0.025
0.253 ± 0.004
0.523 ± 0.032
0.266 ± 0.005
Qwen3-80B
thinking
0.050 ± 0.002
1.148 ± 0.026
0.459 ± 0.022
1.185 ± 0.029
0.578 ± 0.024
1.211 ± 0.025
Qwen3-30B
thinking
0.020 ± 0.015
0.315 ± 0.005
0.371 ± 0.015
0.425 ± 0.010
0.450 ± 0.020
0.462 ± 0.014
LegalOne-8B
reasoning
0.093 ± 0.008
0.147 ± 0.006
0.280 ± 0.032
0.159 ± 0.002
0.298 ± 0.011
0.173 ± 0.003
Appendix
Table 7: TALD accuracy and version-related policy entropy Hiver on old-law cases across hint strength. Each cell shows the mean above and ± its standard error below, computed over three random seeds.
Large language models are increasingly used for legal research, yet their fixed training cutoffs and reliance on static parametric knowledge are at odds with the evolving nature of statutory law. We study two temporal failure modes: post-cutoff staleness, where models apply superseded rules after legislative amendments, and recency bias, where models prefer newer provisions even when a historical version governs the fact pattern. To this end, we present a benchmark of 312 expert-validated, time-sensitive German statutory QA pairs spanning three categories: Post-Cutoff Amendment Questions, Pre-Amendment Questions, and Multi-Provision Pre-Amendment Questions. We evaluate five LLMs by OpenAI, Anthropic and DeepSeek under four inference settings: Vanilla, Web-search, and two retrieval-augmented variants that enforce temporal validity via a fact date extraction and version filtering. Using an LLM-as-a-judge validated against human expert ratings, we find severe degradation in the Vanilla post-cutoff setting. Both RAG approaches substantially improve performance across all question types, while web search yields unstable gains and exhibits a marked recency bias on historically anchored tasks. Our results indicate that reliable legal QA requires treating temporal validity as a hard constraint.
Max Prior, Andreas Schultz, Matthias Grabmair
Technical University of Munich Garching near Munich, Germany
Large language models (LLMs) have demonstrated strong performance across a wide range of legal tasks. In legal practice, time is a critical concept that governs the validity of statutes, the progression of legal cases, and the enforcement of procedural deadlines. However, legal temporal capabilities remain underexplored in existing legal AI benchmarks. To address this gap, we propose LexKairos, a comprehensive benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal context across three dimensions: statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning. LexKairos comprises nine sub-tasks drawn from real-world Chinese judicial cases and statutes. We conduct systematic evaluations of eight LLMs under multiple inference settings, including vanilla, Chain-of-Thought (CoT), and thinking modes. Our results show that Gemini-3-Flash achieves the strongest overall performance, yet even the best-performing model exhibits notable limitations on tasks demanding precise time-sensitive statutory metadata recall or complex reasoning in time limits, indicating that legal temporal knowledge and reasoning remain open challenges for current LLMs. Data and code are available at https://github.com/thunlp/LexKairos.
Chenyang Li, Zejia Feng, Yuqin Huang +2
University of Oxford · Sun Yat-Sen University · Tsinghua University
While large language models (LLMs) augmented with agentic search capabilities show promise for legal reasoning, they overlook a fundamental constraint that applicable law must match the temporal context of each case, as retroactive application of statutes violates core legal principles and leads to erroneous conclusions. Our observations reveal that current legal LLMs suffer from temporal bias anchored to their training cutoff, while search agents rarely incorporate temporal constraints into queries, and that web search alone cannot provide the precise statute and precedent citations that legal reasoning demands. To address these challenges, we propose LegalSearch-R1, an end-to-end reinforcement learning framework that pairs local statute RAG for precise article matching with online web search for broader legal knowledge, trained on temporally-indexed data spanning multiple amendment periods to enforce temporal consistency. Extensive experiments on our benchmark covering 13 legal tasks demonstrate that our 7B-parameter agent outperforms state-of-the-art deep research frameworks and specialized legal LLMs by 12.9% to 29.8%, surpasses baselines by 57.7% to 80.3% on temporal consistency, and exhibits robust out-of-domain generalization. The code and data are available at https://github.com/AlexFanw/LegalSearch-R1.
Wei Fan, Yining Zhou, Mufan Zhang +8
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada · School of Law, Tsinghua University, Beijing, China