When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

Yiqian Huang; Yang Zhou; Qianying Liu; Shaowen Peng; Yuntao Kong; Kotaro Funakoshi; Chuan Xiao; Manabu Okumura

cs.AIJul 8, 2026

When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

Authors: Yiqian Huang, Yang Zhou, Qianying Liu, Shaowen Peng, Yuntao Kong, Kotaro Funakoshi, Chuan Xiao, Manabu Okumura,

Organizations: Institute of Science Tokyo · NII LLMC · Nara Institute of Science and Technology · Center of Juris-Informatics, ROIS-DS ·

Abstract

Legal reasoning requires identifying which version of a statute applies to a case, typically based on when the legally relevant events occurred rather than when the case is adjudicated. Yet whether large language models (LLMs) can reliably perform such temporal legal reasoning remains largely unexplored. We formalize this task as Temporal Applicable-Law Determination (TALD), and construct a TALD-Bench, a benchmark of 51,956 Chinese civil judgments spanning the enactment of the Civil Code. Evaluating nine advanced LLMs, we identify three key findings. First, models exhibit a strong bias toward newer law, achieving over 70% accuracy on new-law cases but below 25% on old-law cases. Second, models demonstrate substantial knowledge of statutory versions and temporal-effect rules, yet struggle to apply this knowledge to cases. Stronger general reasoning performance does not consistently translate into better TALD accuracy, whereas temporal-law hints yield substantial improvements, suggesting a bottleneck in how models deploy their existing capabilities. Third, temporal-law hints also increase uncertainty during statutory-version reasoning, suggesting that they may help models reconsider their default preference for newer law. Our findings highlight the importance of adapting how LLMs apply existing capabilities, rather than relying solely on improvements in legal knowledge or general reasoning.

Figures & tables

Figure 1: Temporal Applicable-law Determination (TALD) as a vital step within legal reasoning, and an illustrative instance of how temporal legal reasoning is implemented by LLMs when faced with a legal issue.

Panel of Figure 2 — Figure 2: TALD accuracy on old-law and new-law cases (Equation 2.2 ) and the distribution of weighted errors pooled across both splits. Each stacked bar sums to 100%.

Panel of Figure 3 — Figure 3: Accuracy on the two legal knowledge probes.

Figure 4: Spearman correlations among GPQA, HLE, LCB, and old-law TALD accuracy under (a) no hint, (b) weak hint, and (c) strong hint. All panels share a color scale; diagonal entries display self-correlations (1.000). External benchmark scores and matched configurations are fixed across hint conditions. Details see Appendix E.2 .

Model	Mode	Default	Weak	Strong
Claude Opus 4.6	low	0.021 $\pm$ 0.001	0.131 $\pm$ 0.011	0.185 $\pm$ 0.107
	medium	0.019 $\pm$ 0.002	0.102 $\pm$ 0.008	0.143 $\pm$ 0.083
	high	0.026 $\pm$ 0.002	0.099 $\pm$ 0.024	0.154 $\pm$ 0.089
	max	0.014 $\pm$ 0.004	0.113 $\pm$ 0.033	0.156 $\pm$ 0.091
Gemini-3.1-Pro	low	0.146 $\pm$ 0.011	0.501 $\pm$ 0.019	0.439 $\pm$ 0.035
	medium	0.131 $\pm$ 0.020	0.503 $\pm$ 0.021	0.489 $\pm$ 0.016

Table 1: Effect of temporal-law hints on old-law cases, measured by TALD accuracy. Each entry reports the mean

\pm

its across-seed standard error. Default denotes the no-hint setting.

Panel of Figure 5 — Figure 5: TALD accuracy and version-related policy entropy on old-law cases across hint strength, shown separately for each model. Points show means over three random seeds; error bars show standard errors. TALD accuracy uses a common left-axis range; entropy uses model-specific right-axis ranges. Full results are reported in Table 7 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Area of civil law	Predecessor statute(s)	Civil Code counterpart
General provisions	General Principles of the Civil Law; General Provisions of the Civil Law	Book I: General Provisions
Property	Property Law	Book II: Real Rights
Contract	Contract Law	Book III: Contracts
Personality rights	—	Book IV: Personality Rights
Marriage and family	Marriage Law	Book V: Marriage and Family
Inheritance	Succession Law	Book VI: Succession

Appendix

Table 2: Predecessor statutes and their Civil Code counterparts, grouped by area of civil law. A dash indicates that no separate predecessor statute is listed for that area.

(a) API model IDs used in evaluation
Name in paper	Mode(s) in results	API model ID
DeepSeek-V3.2	chat	deepseek-chat
	reasoner	deepseek-reasoner
GLM-4.7	instruct; thinking	z-ai/glm-4.7
GPT-5.4	none; low; medium; high	gpt-5.4
Gemini-3.1-Pro	low; medium; high	gemini-3.1-pro-preview

Appendix

Table 3: Identifiers and evaluated configurations of the nine primary models. Panel (a) reports API request identifiers used during evaluation; panel (b) links to the corresponding Hugging Face model repositories.

Model	Mode	Old-law	New-law
Model	Mode	ACC $\pm$ SE	ACC $\pm$ SE
DeepSeek-V3.2	reasoner	$0.092\,\pm\,0.0167$	$0.746\,\pm\,0.0155$
GLM-4.7	thinking	$0.080\,\pm\,0.0240$	$0.787\,\pm\,0.0149$
GPT-5.4	high	$0.237\,\pm\,0.0048$	$0.701\,\pm\,0.0070$
Gemini-3.1-Pro	high	$0.135\,\pm\,0.0130$	$0.843\,\pm\,0.0198$
Claude Opus 4.6	max	$0.014\,\pm\,0.0040$	$0.820\,\pm\,0.0198$

Appendix

Table 4: Default-instruction TALD accuracy for the nine primary models, corresponding to Figure 2a . Each entry reports the mean

\pm

its standard error across four sampling seeds; each run samples 100 cases from each split.

GPT-5.6 Sol
Mode	Old-law	New-law
none	$0.124$ $\pm\,0.0153$	$0.788$ $\pm\,0.0147$
low	$0.406$ $\pm\,0.0164$	$0.688$ $\pm\,0.0310$
medium	$0.410$ $\pm\,0.0102$	$0.682$ $\pm\,0.0246$
high	$0.366$ $\pm\,0.0130$	$0.710$ $\pm\,0.0118$
xhigh	$0.355$ $\pm\,0.0098$	$0.710$ $\pm\,0.0292$

Appendix

Table 5: Supplementary evaluation of newer frontier models under the default instruction. These configurations assess whether the old-law performance deficit persists beyond the primary model panel and are not included in the knowledge, hint, or entropy analyses. Entries report mean TALD accuracy and its across-seed standard error.

Model	Mode	External benchmark score			Old-law TALD accuracy
Model	Mode	GPQA	HLE	LCB	No hint	Weak hint	Strong hint
DeepSeek-V3.2	chat	75.1	11.2	59.3	0.013	0.028	0.080
	reasoner	84.0	24.6	86.2	0.092	0.394	0.436
Claude Opus 4.6	max	89.6	39.9	–	0.014	0.113	0.156
GPT-5.4	none	74.8	11.3	–	0.021	0.087	0.167
	low	87.1	30.8	–	0.082	0.444	0.456

Appendix

Table 6: Artificial Analysis benchmark scores (0–100) and old-law TALD accuracy (0–1) under no, weak, and strong hints. TALD columns report the across-seed means from Table 1 . The bottom rows give each benchmark’s matched configuration count and Spearman correlation with TALD under each hint condition; matching is fixed across conditions. Dashes indicate unavailable scores.

Model	Mode	No hint		Weak hint		Strong hint
Model	Mode	ACC	$H_{i}^{ver}$	ACC	$H_{i}^{ver}$	ACC	$H_{i}^{ver}$
Qwen3-235B	thinking	0.155 ± 0.025	0.238 ± 0.006	0.524 ± 0.025	0.253 ± 0.004	0.523 ± 0.032	0.266 ± 0.005
Qwen3-80B	thinking	0.050 ± 0.002	1.148 ± 0.026	0.459 ± 0.022	1.185 ± 0.029	0.578 ± 0.024	1.211 ± 0.025
Qwen3-30B	thinking	0.020 ± 0.015	0.315 ± 0.005	0.371 ± 0.015	0.425 ± 0.010	0.450 ± 0.020	0.462 ± 0.014
LegalOne-8B	reasoning	0.093 ± 0.008	0.147 ± 0.006	0.280 ± 0.032	0.159 ± 0.002	0.298 ± 0.011	0.173 ± 0.003

Appendix

Table 7: TALD accuracy and version-related policy entropy

H_{i}^{ver}

on old-law cases across hint strength. Each cell shows the mean above and

\pm

its standard error below, computed over three random seeds.

Legal Reasoning

Legal reasoning studies how people and AI interpret laws, apply rules to facts, weigh evidence, and justify legal conclusions.

See all 111 papers

Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering
May 22, 2026Max Prior, Andreas Schultz, Matthias GrabmairRetrieval-Augmented Generation Temporal IR
Technical University of Munich Garching near Munich, Germany
LexKairos: Benchmarking Legal Temporal Capabilities in LLMs
Aug 10, 2026Chenyang Li, Zejia Feng, Yuqin Huang +2Legal NLP LLM Evaluation
University of Oxford · Sun Yat-Sen University · Tsinghua University
Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
May 25, 2026Wei Fan, Yining Zhou, Mufan Zhang +8Agentic Search Legal QA
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada · School of Law, Tsinghua University, Beijing, China

May 22, 2026J/K move · Enter open · S save

Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

Max Prior, Andreas Schultz, Matthias Grabmair

Technical University of Munich Garching near Munich, Germany

Retrieval-Augmented Generation Temporal IR Legal QA Temporal QA

Open paper AskPDF arXiv

Large language models are increasingly used for legal research, yet their fixed training cutoffs and reliance on static parametric knowledge are at odds with the evolving nature of statutory law. We study two temporal failure modes: post-cutoff staleness, where models apply superseded rules after legislative amendments, and recency bias, where models prefer newer provisions even when a historical version governs the fact pattern. To this end, we present a benchmark of 312 expert-validated, time-sensitive German statutory QA pairs spanning three categories: Post-Cutoff Amendment Questions, Pre-Amendment Questions, and Multi-Provision Pre-Amendment Questions. We evaluate five LLMs by OpenAI, Anthropic and DeepSeek under four inference settings: Vanilla, Web-search, and two retrieval-augmented variants that enforce temporal validity via a fact date extraction and version filtering. Using an LLM-as-a-judge validated against human expert ratings, we find severe degradation in the Vanilla post-cutoff setting. Both RAG approaches substantially improve performance across all question types, while web search yields unstable gains and exhibits a marked recency bias on historically anchored tasks. Our results indicate that reliable legal QA requires treating temporal validity as a hard constraint.

Abstract

Legal Reasoning

Explore similar work

Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

LexKairos: Benchmarking Legal Temporal Capabilities in LLMs

Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning