As automated fact-checking scales on social media, large language model (LLM) verdict scores can look stronger than warranted. One reason is that evaluations mix in information that was not knowable at claim time. Two channels are easy to conflate: outcomes memorized in pre-training and retrieved evidence published after the claim. Yet standard benchmarks rarely separate the two. In this study we measure both channels on AVeriTeC and QuanTemp++ by reconstructing point-in-time evidence conditions and probing for outcome information encoded in model representations. We find substantial evidence of parametric leakage, that can be hidden by the aggregate accuracy, while a simple representation bottleneck reduces this future leakage more efficiently than a mutual-information-based training penalty. We also find that allowing post-claim evidence inflates zero-shot accuracy by 6.3 points in AVeriTeC while the effect is negligible in QuanTemp++, where retrieval provides little post-claim evidence. These results show that misinformation benchmarks can overstate fact-checking performance when they do not account for what information was actually available at claim time.
Figures & tables
Trust
Signal
Answers
Share
slug
date in the URL path ( /2020/10/30/ )
533
9.6%
published
JSON-LD datePublished ,
1,603
28.8%
article:published_time metadata
meta
generic date metadata, <time> tags
96
1.7%
archive
snapshot timestamp (upper bound only)
2,252
40.5%
—
no signal recovered
1,081
19.4%
Table 1: Publication-date signals on AVeriTeC binary claims, from most to least reliable. “Archive” is an upper bound only, not a publication date.
Figure 1: Trained representation pipeline. Two knobs—bottleneck width dR and penalty weight λ —are studied against future-information leakage.
Metric
Original
Masked
Δ
Majority
Accuracy
0.710
0.678
−0.032
0.679
Macro-F1
0.640
0.502
−0.138
0.405
Supported recall
0.421
0.131
−0.291
0.000
Refuted recall
0.846
0.937
+0.091
1.000
Table 2: Entity-masking probe on Qwen3-14B, 2,554 entity-bearing binary AVeriTeC claims, no evidence shown. Majority is always-Refuted (1,735/2,554).
Claim (original / masked)
P(true)
Russia has developed a new vaccine to fight COVID-19.
1.00→0.00
[GPE_1] has developed a new vaccine to fight [PERSON_1].
American rapper Kanye West is running for President of the United States in the 2020 elections.
1.00→0.00
[NORP_1] rapper [PERSON_1] is running for President of [GPE_1] in the [DATE_1] elections.
Table 3: Supported claims where masking flips full confidence into rejection ( P(true) , original → masked).
Model
Training data
Acc.
Supp. recall (o → m)
Recall drop [95% CI]
ΔP(true) [95% CI]
Qwen3-14B
in-window (2020)
0.710
0.421→0.131
+0.291[.258,.322]
+0.279[.250,.309]
phi-4
in-window (2020)
0.717
0.277→0.039
+0.238[.210,.268]
+0.250[.227,.274]
GPT-2-XL
WebText ≤ 2017
0.606
0.394→0.686
−0.292[−.322,−.260]
−0.087[−.092,−.082]
Table 4: The identical probe on a pre-cutoff control and a second in-window model. Recall figures and CIs on the entity-bearing subset; ΔP(true) is the confidence change on gold-Supported claims (positive = masking lowers confidence). Both in-window models collapse toward “False” but the control moves in the opposite direction.
Model
gold-True
gold-False
ΔΔP [95% CI]
Qwen3-14B
+0.279
+0.089
+0.190[.156,.225]
phi-4
+0.250
+0.075
+0.175[.150,.202]
GPT-2-XL
−0.087
−0.096
+0.009[.002,.015]
Table 5: The confidence change under masking, split by gold label, with ΔΔP the gold-True minus the gold-False change (positive = masking lowers confidence). A label-blind drift moves both classes equally and cancels in ΔΔP .
Figure 2: Validation macro-F1 (left) and leakage (right) as the penalty λ grows, on AVeriTeC (top) and QuanTemp++ (bottom) with top-5 point-in-time evidence, one line per width dR . F1 falls as λ grows for at most a marginal leakage return. The width, not the penalty, is what moves leakage. Means over 5 seeds.
top-5 point-in-time
claim-only
dR
F1
leakage
F1
leakage
8
0.607
0.168
0.626
0.268
2
0.577
0.046
0.582
0.086
1
0.512
0.029
0.492
0.057
Table 6: AVeriTeC: the λ=0 grid with and without retrieved point-in-time evidence. Leakage is the held-out R2 gain of predicting K from (R,E) over E alone. Means over 5 seeds. In the claim-only setting the conditioner E is a constant, so its leakage column reads unconditional outcome decodability.
top-5 point-in-time
claim-only
dR
F1
leakage
F1
leakage
8
0.783
0.015
0.730
0.392
2
0.789
0.010
0.731
0.307
1
0.783
0.008
0.730
0.288
Table 7: QuanTemp++: the λ=0 grid with and without retrieved point-in-time evidence, as Table 6 . The claim-only leakage is further inflated by outcome statements that restate the claim.
Figure 3: Zero-shot Qwen3-14B verdicts. (a) macro-F1 by evidence view: on AVeriTeC quality rises from no-evidence to point-in-time to leaked; on QuanTemp++ it is flat, because its retrieval surfaces almost no post-claim evidence (Table 13 ). (b) leaked − point-in-time inflation under the default prompt and an evidence-only prompt; the gap grows when the memory fallback is removed. Bars are on the full binary claim sets ( n=2,513 / 5,537 ).
Figure 4: Leaked − point-in-time Supported-recall of the trained head vs. (a) λ at dR=32 and (b) width dR at λ=0 (Section 5.2 ). On AVeriTeC the gap is near zero while the parametric shortcut remains, then re-emerges; on QuanTemp++ it falls toward zero (Table 13 ). Means over 10 seeds, ±1 SD.
Claims
n
PIT
Leaked
Δ
No pre-claim evidence
1,361
0.746
0.835
+0.090
Has pre-claim evidence
1,152
0.824
0.855
+0.031
All claims
2,513
0.782
0.844
+0.063
Table 8: The zero-shot accuracy gap on AVeriTeC, split by whether any pre-claim evidence exists for the claim. Where none exists, the point-in-time view gives the model an empty evidence set. Both gaps are significant.