All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting

Zeyu Zhang; Ryan Chen; Bradly C. Stadie

cs.AIFeb 19, 2026

All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting

Authors: Zeyu Zhang, Ryan Chen, Bradly C. Stadie

Organizations: Department of Statistics and Data Science, Northwestern University · Bridgewater AIA Labs

Abstract

Backtesting LLMs on resolved events assumes models reason only from pre-cutoff knowledge, yet pretrained models inevitably leak post-cutoff knowledge. We introduce a claim-level evaluation framework that decomposes prediction rationales into atomic claims and applies Shapley values to quantify each claim's decision impact, yielding Shapley-DCLR (Shapley-weighted Decision-Critical Leakage Rate) -- an interpretable metric measuring what fraction of decision-driving reasoning is contaminated. We further propose TimeSPEC (Time-Supervised Prediction with Extracted Claims), an inference-time architecture that interleaves temporally-filtered retrieval with claim-level supervision, producing predictions grounded entirely in pre-cutoff evidence. Across three LLMs, the ablation experiments confirm retrieval and supervision are jointly necessary; and a three-task probe further illstrates that the performance cost of temporal enforcement scales with each task's reliance on post-cutoff information.

Figures & tables

Figure 1: The Shapley-DCLR evaluation pipeline: a model-agnostic, claim-level framework for detecting and quantifying temporal leakage. Given any LLM’s rationale $R$ and a reference date $t_{\text{ref}}$ , Phase 1 decomposes $R$ into atomic claims and categorizes each. Phases 2 and 3 run in parallel. Phase 2 estimates each claim’s Shapley contribution $\phi_{i}$ to the prediction via Leverage SHAP at $O(C\!\cdot\!n\log n)$ cost. Phase 3 determines leakage indicators $\{\ell_{i}\}$ via category-based routing, where deterministic categories bypass external search, reducing verification cost. Phase 4 fuses $\{\phi_{i}\}$ and $\{\ell_{i}\}$ into Shapley-DCLR (Eq. 4 ).

Figure 2: The TimeSPEC architecture: an inference-time pipeline that interleaves temporally-filtered retrieval with claim-level supervision under a closed-world guarantee. The Generator issues date-filtered web queries ( $t<t_{\text{ref}}$ ) and produces a draft prediction with rationale. The Supervisor extracts and categorizes claims, partitioning them into validated ( $\mathcal{C}_{\text{valid}}^{(1)}$ ) and violated sets. If violations exist, the Regenerator re-queries with diversified searches and the Resupervisor audits the new rationale, yielding $\mathcal{C}_{\text{valid}}^{(2)}$ ; if none, the validated claims route directly to the Aggregator . The Aggregator synthesizes $\hat{y}_{\text{final}}=\mathcal{A}(\mathcal{I},\,\mathcal{C}_{\text{valid}}^{(1)}\cup\mathcal{C}_{\text{valid}}^{(2)})$ under a closed-world constraint: only the task input and validated claims enter the final synthesis.

Method	Search	Sup.	DCLR (%) $\downarrow$	Acc (%) $\uparrow$	DCLR (%) $\downarrow$	Acc (%) $\uparrow$	DCLR (%) $\downarrow$	Acc (%) $\uparrow$
			Qwen3.5-35B		Kimi-K2.5		Claude Sonnet 4
Temporal Hint	–	–	5.2	80.5	2.7	86.8	1.0	83.9
Self-Correction	–	$\checkmark$	1.6	79.9	1.7	86.8	0.8	84.0
Temporal RAG	$\checkmark$	–	13.1	85.4	8.9	87.8	4.7	84.2
TimeSPEC	$\checkmark$	$\checkmark$	1.8	83.9	1.0	87.4	0.8	83.9

Table 1: Treatment-group results across the (Search, Supervision) ablation on three LLMs. DCLR: Shapley-DCLR (%,

\downarrow

, lower is less leakage); Acc:

1{-}\text{Brier}

(%,

\uparrow

). Temporal RAG achieves the highest accuracy but also the highest leakage on every model; TimeSPEC recovers most of the accuracy gain while suppressing DCLR to the supervised floor. Bold marks the best value per column.

Method	Qwen3.5	Kimi-K2.5	Claude
Temporal Hint	0.6	0.5	0.2
Self-Correction	0.5	0.4	0.2
Temporal RAG	16.7	17.0	18.5
TimeSPEC	3.7	2.7	4.2

Table 2: Control-group Shapley-DCLR (%,

\downarrow

) on three LLMs. Control questions resolve after every model’s knowledge cutoff, so parametric leakage is impossible by construction. Temporal Hint and Self-Correction confirm the near-zero empirical floor.

Figure 3: Task-level performance vs. Shapley-DCLR across the four baselines on Qwen3.5-35B. Each panel plots task-specific performance (

\uparrow

) against Shapley-DCLR (%,

\downarrow

); lower-right is the ideal operating region. Performance metrics are

1{-}(\hat{p}{-}y)^{2}

for Legal,

1{-}|\hat{y}{-}y|/|y|

for NBA Salary, and

(\rho{+}1)/2

for Stock, all normalized to

[0,1]

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Source	$N$	Format	Leakage Status
Treatment	Autocast ( Zou et al., 2022 )	1,942	Binary	Possible (pre-cutoff resolution)
Control	ForecastBench ( Karger et al., 2025 )	827	Binary	Impossible (post-cutoff resolution)
Total		2,769

Appendix

Table 3: Large-scale ablation dataset summary. Treatment questions admit parametric leakage; Control questions do not.

Task	Type	$N$	Period	Cutoff Design	Metric
Stock	Ranking	100	Dec 2019–Jun 2020	Period start	$(\rho{+}1)/2$ ( $\uparrow$ )
Salary	Regression	137	2019–2025	Pre-signing	$1{-}\|\hat{y}{-}y\|/\|y\|$ ( $\uparrow$ )
Legal	Classification	100	Recent terms	Pre-decision	$1{-}(\hat{p}{-}y)^{2}$ ( $\uparrow$ )

Appendix

Table 4: Task-sensitivity probe dataset summary. Tasks are ordered from highest to lowest leakage sensitivity.

ID	Type	Example Claim	Status	$\tau(c)$ Rule
C0	Timeless Definition	“The Brier score is $(p-o)^{2}$ for predicted probability $p$ and binary outcome $o$ .”	Never leaked	$\tau(c)=-\infty$ ; time-invariant by definition.
C1	Stable Entity Attribute	“The U.S. Supreme Court is composed of nine justices.”	Usually safe	Deterministic; attribute must be stable through $t_{\text{ref}}$ .
C2	Time-Indexed State	“As of Q3 2019, U.S. crude-oil demand had been declining for three consecutive quarters.”	Verify date	Earliest source date corroborating the stated condition.
C3	Quantitative Measurement	“Apple’s FY2019 revenue was $260.2B.”	Verify date	Date of public release (e.g., 10-K filing date).
C4	Comparative / Relational	“In 2019, Boeing’s debt-to-equity ratio exceeded that of Airbus by $>$ 3 $\times$ .”	Verify date	Latest source date among the comparands.
C5	Discrete Event	“On Oct. 25, 2019, Microsoft won the $10B JEDI cloud contract.”	Verify date	Event date, or announcement date if unreported.

Appendix

Table 5: Full nine-class claim taxonomy. The Status column indicates how

\ell(c)

is determined: deterministic categories (C0–C1, C7–C8) bypass external search entirely, while C2–C6 require date verification via

\tau(c)

. Horizontal rules separate the three operational tiers.

Method	Prompt	RAG	Supervisor	Search	Regen	Aggregator
Temporal Hint	$\checkmark$	–	–	–	–	–
Self-Correction	$\checkmark$	–	LLM-only	–	$\checkmark$	$\checkmark$
Temporal RAG	$\checkmark$	$\checkmark$	–	–	–	–
TimeSPEC	$\checkmark$	$\checkmark$	LLM+Search	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 6: Architectural comparison of the four generation methods.

\checkmark

indicates the component is present; – indicates absent.

	Legal ( $n{=}100$ )				NBA Salary ( $n{=}137$ )				Stock ( $n{=}100$ )
Method	DCLR $\downarrow$	Top-1 $\downarrow$	Top-3 $\downarrow$	Top-5 $\downarrow$	DCLR $\downarrow$	Top-1 $\downarrow$	Top-3 $\downarrow$	Top-5 $\downarrow$	DCLR $\downarrow$	Top-1 $\downarrow$	Top-3 $\downarrow$	Top-5 $\downarrow$
Temporal Hint	11.7	17.0	11.3	10.1	58.5	70.8	50.4	48.2	26.0	28.0	26.7	26.2
Self-Correction	5.6	7.0	5.0	4.0	11.3	11.7	9.7	9.0	10.2	7.0	8.7	8.6
Temporal RAG	32.5	41.0	35.3	30.7	61.7	75.2	59.1	52.0	73.3	80.0	74.3	72.8
TimeSPEC	5.9	5.0	5.7	5.8	12.8	14.6	12.5	10.6	12.2	7.0	7.0	7.0

Appendix

Table 7: Leakage concentration among the most decision-critical claims. DCLR is the Shapley-weighted aggregate from Section 4.2 ; Top-

K

is the fraction of the

K

highest-Shapley claims that are leaked (Eq. 7 ), averaged across instances. All values in %. Top-1

>

DCLR indicates that leakage concentrates at the most influential position. Bold marks the lowest value per column within each task group.

Prompt	T. Hint	Self-Corr.	T. RAG	TimeSPEC
Generation (System + User)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
RAG Context Block	–	–	$\checkmark$	$\checkmark$
Query Generation	–	–	$\checkmark$	$\checkmark$
Taxonomy Verification	–	$\checkmark$	–	$\checkmark$
Search Re-Verification	–	–	–	$\checkmark$
Regeneration	–	$\checkmark$	–	$\checkmark$

Appendix

Table 8: Prompt usage across methods.

\checkmark

= used; – = not used.

Data Contamination in Language Models

Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.

See all 64 papers

TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting
May 13, 2026Zeyu Zhang, Bradly C. StadieLLM Evaluation RL for Language Model Reasoning
Department of Statistic and Data Science Northwestern University
Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
Aug 4, 2026Zeyu Zhang, Bradly C. StadieLLM Evaluation LLM Auditing
Department of Statistics and Data Science Northwestern University
Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning
May 14, 2026Chenlu Ding, Jiancan Wu, Yanchen Luo +3LLM Prompting LLM Fine-Tuning
University of Science and Technology of China · The Hong Kong Polytechnic University · University of Notre Dame

May 13, 2026J/K move · Enter open · S save

TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting

Zeyu Zhang, Bradly C. Stadie

Department of Statistic and Data Science Northwestern University

LLM Evaluation RL for Language Model Reasoning Temporal Data Leakage Language Model Auditing

Open paper AskPDF arXiv

Backtesting large language models on historical events requires reasoning exclusively from information available before a specified cutoff date. Yet models routinely leak post-cutoff knowledge from pre-training into their reasoning, inflating apparent accuracy and undermining evaluation validity. Prompt-based constraints fail when suppressed content is causally related to the prediction, and knowledge unlearning cannot address this problem because temporal compliance is instance-specific: the same fact may be legitimate evidence for one cutoff date and a violation for another. Rather than erasing knowledge, the model must learn temporal discipline: selecting evidence conditioned on each instance's cutoff date. We propose TEMPO (Temporal Enforcement via Mode-separated Policy Optimization), which trains this discipline via two contributions: (1) a two-mode reward where a leakage mode drives post-cutoff claims to zero as a hard prerequisite before a performance mode optimizes task performance; and (2) a GRPO-based training pipeline that enables the model to discover temporally valid reasoning strategies. We prove that training monotonically decreases leakage, converges to the leak-free optimum, and improves task performance once compliance is achieved. On three prediction tasks and two models, TEMPO reduces leakage from 2~~13% to 0.6~~3.7% across all conditions, with task performance improving 6~13% where strong pre-cutoff signals exist and maintained where the prediction task is inherently difficult from valid information alone.

Abstract

Data Contamination in Language Models

Explore similar work

TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting

Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores

Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning