Post-trained language models fabricate citations at high rates. We ask whether that is reward hacking: scoring well on a proxy for scholarship while doing worse on what it stands for. We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen. The clearest result holds the weights fixed and changes only the prompt. Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains. A length-matched control separates the citation cue from a demand for a longer answer, so the effect is specific to citations, not verbosity. Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold. A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity. Our answer: possibly, but not proven. Post-trained models produce fabricated citations at far higher rates when asked for sources, and a naive evaluator that counts sources without checking them would prefer their output for citing more. But the targeted pattern we registered did not appear, and plain instruction-following could explain the same results. Proving that the reward causes fabrication needs an experiment that changes the reward during training, which this study does not do. Pre-registration: doi:10.17605/OSF.IO/S53HY. Code and data: https://github.com/zainnab-sparq/citation-reward-hacking
Figures & tables
Cell
Domain
n
Recruitment stratum
Known-real
Academic
50
OpenAlex, > 9,999 citations (hyper-canonical)
50
OpenAlex, 1,000–5,000 citations (moderate)
Legal
50
SCOTUS, ≥ 100 citing references
50
Federal Circuit, ≥ 50 citing references
Medical
60
Cochrane Reviews, > 49 citations
40
Non-Cochrane, > 499 citations
Table 1: Composition of the frozen 900-question set. The design is balanced by construction: every domain contributes exactly 100 questions to every cell. The strata inside known-real are the pre-registered canonicality tiers, which exist so that recall accuracy can be checked against canonicality rather than assumed to be uniform. Both real cells are restricted to pre-2023 publications.
Family
Checkpoint
Ord.
Stage
Revision
tulu3_8b
meta-llama/Llama-3.1-8B
0
base
d04e592bb4f6
allenai/Llama-3.1-Tulu-3-8B-SFT
1
SFT
f2a0b46b0cfd
allenai/Llama-3.1-Tulu-3-8B-DPO
2
DPO
a7beb67e33ff
allenai/Llama-3.1-Tulu-3-8B
3
RLVR
666943798adb
tulu3_70b
meta-llama/Llama-3.1-70B
0
base
349b2ddb53ce
allenai/Llama-3.1-Tulu-3-70B-SFT
1
SFT
f8ef34145946
Table 2: The open-weights checkpoints. Three cascades, each carrying all four post-training stages of one base model, which is what makes stage_ordinal vary within a family and the interaction estimable. Revisions are the pinned Hugging Face commits actually served.
By checkpoint
Family
Stage
Generations
Citing
Citations
Per gen.
tulu3_8b
base
3,600
21.3%
951
0.264
SFT
3,600
61.5%
3,197
0.888
DPO
3,600
79.8%
4,947
1.374
RLVR
3,600
80.3%
5,069
1.408
tulu3_70b
base
3,600
18.7%
852
0.237
Table 3: Pipeline yield. Each of the twelve checkpoints answered all 900 questions under all four conditions, so every cell rests on exactly 3,600 generations by construction and no cell is thinned by attrition. “Citing” is the share of generations producing at least one extractable citation.
Term
β^
OR
Wald 95% CI
Holm p
stage_ordinal
0.881
2.41
[1.72, 3.39]
1.5×10−6
(0.861)
(2.37)
([1.69, 3.32])
( 2.6×10−6 )
stage_ordinal:cellobscure_real
0.084
1.09
[0.98, 1.21]
0.360
(0.036)
(1.04)
([0.93, 1.16])
(1.000)
stage_ordinal:cellunanswerable
−0.356
0.70
[0.64, 0.77]
2.2×10−12
( −0.402 )
(0.67)
([0.60, 0.74])
( 3.3×10−14 )
Table 4: T4. The registered E1 family of five, Holm-corrected within the family. Corrected pipeline first, registered pipeline in parentheses. Odds ratios are per unit of stage_ordinal ; intervals are Wald.
Figure 1: Observed fabrication rate by post-training stage, within each question cell. Points are raw proportions of generations carrying at least one fabricated citation, with 95% Wilson intervals. They are not the marginal probabilities quoted in the text, which come from the fitted model and weight cells equally; the figure shows shape and the tables carry the inference. Two things are visible at once. Fabrication rises steeply from base to SFT in all three cells and then flattens, and the three cells rise more or less together. That combination, a large stage main effect with no differential slope by cell, is exactly what leaves the registered T1 interaction null. n=42,845 generations over 12 checkpoints.
Neutral
Citations expected
Difference
Base checkpoints
9.04% (8.68%)
0.77% (0.72%)
−8.3 ( −8.0 ) pts
Post-trained checkpoints
34.18% (31.73%)
73.70% (57.22%)
+39.5 ( +25.5 ) pts
Interaction delta
47.8 (33.5) pts
Table 5: T5. E2 marginal fabrication probabilities by prompt condition and post-training status, corrected pipeline with registered pipeline in parentheses. The interaction delta is the post-trained rise minus the base rise.
Corrected
Registered
P(fab∣citations_\allowbreakexpected)
0.7380
0.5728
P(fab∣length_\allowbreakmatched)
0.4545
0.4465
P(fab∣neutral)
0.3428
0.3177
Δ (percentage points)
+28.35
+12.64
95% CI
[+25.58,+31.12]
[+10.75,+14.53]
Registered bin
citation cue supported
citation cue supported
Table 6: Amendment #10 de-confounding contrast, both pipelines. Probabilities are marginal at is_post_trained =1 . The registered pipeline is the one that withholds the verifier-REAL correction of amendment #12.
Figure 2: Observed fabrication rate by prompt condition, split by whether the checkpoint has been post-trained, with 95% Wilson intervals. This is the interaction that H1.2 predicted, and it is visible without a model: asking for citations moves the two groups in opposite directions. Base checkpoints fall from 15.8% under the neutral prompt to 2.5% when citations are requested, while post-trained checkpoints rise from 39.5% to 68.9%. The length-matched control, which demands a longer answer without mentioning citations, lands well below the citation cue for post-trained checkpoints (47.1% against 68.9%), which is the de-confounding contrast of amendment #10 in raw form. Base rates are computed on 2,686 to 2,700 generations per condition and post-trained rates on 7,985 to 8,047.
Tier
Primary (base vs Instruct)
Placebo (SFT vs DPO)
All pairs
0.8578 [0.8500,0.8661] , n=3600
0.6471 [0.6347,0.6589] , n=3600
Within ±1 (registered)
0.7876 [0.7766,0.7981] , n=2283
0.5808 [0.5683,0.5932] , n=2914
Exactly equal
0.5155 [0.5083,0.5238] , n=904
0.5035 [0.4923,0.5146] , n=1882
Table 7: E4 preference rate by count-matching tier, logistic arm. The registered control is the ±1 tier; the exact-match tier is a post-hoc diagnostic.
Domain
n
UNVERIFIABLE
Rate
Blocked
Share of UNV.
Academic
3,493
75
2.15%
12
16.0%
Legal
573
12
(2.09%)
6
(50.0%)
Medical
2,786
101
3.63%
27
26.7%
All
6,852
188
2.74%
45
23.9%
Table 8: Instrument reach on the complete verifier-REAL re-judge subset, all 6,852 citations. “Blocked” counts judgments whose own reasoning says access was refused. It is a keyword proxy over free text and therefore a lower bound. The legal row is reported as a count, not a rate: twelve UNVERIFIABLE citations cannot support a percentage, and the 50% blocked share is six of those twelve.
Grounding
Verdict
Era
Citations
Mean effort
Zero effort
REAL
FABRICATED
UNVERIFIABLE
Early
16,500
2.03
1.63%
46.40%
36.87%
16.73%
Degraded
7,500
1.66
12.92%
37.08%
49.09%
13.83%
Recovered
7,980
2.09
1.13%
43.15%
49.80%
7.06%
Table 9: The three judge eras, over all 31,980 judged citations. Effort is self-reported verification effort across all tools, not a count of web searches.
Figure 3: Judge grounding (left) and verdict mix (right) across the three eras, over all 31,980 judged citations. The two panels are the point of the figure read together: grounding falls and then returns to its starting level, while the verdict distribution steps toward FABRICATED and stays there. An instrument whose verdicts tracked its grounding would show the right-hand bars returning to their early-era shape in the third era, and they do not.
Interaction
χ2
df
p
Rejects
D1
condition × era
10.074
6
0.122
no
D2
stage_ordinal × era
0.366
2
0.833
no
D3
condition × grounding
23.970
3
2.5×10−5
yes
D1 ′
condition × era, sensitivity #1
10.418
6
0.108
no
Table 10: Differentiality tests at the complete 31,980 judgments. Each takes the corresponding registered model verbatim and adds the interaction shown; the statistic is a likelihood-ratio test against that model without it. All four converged and none is singular.
Early era
Degraded era
Original
re-judged as
re-judged as
REAL
FAB.
UNV.
REAL
FAB.
UNV.
REAL
117
3
1
102
2
1
FABRICATED
1
79
3
3
101
8
UNVERIFIABLE
2
29
15
14
14
5
Table 11: Paired re-judge transition matrices, original verdict to re-judged verdict, 250 citations per era. The rates are similar across eras; the composition is not.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
#
Date
What changed
Evidence in hand when recorded
1
2026-05-25
Legal known-real pre-2023 cutoff enforced
Round-3 design panel; no generations collected
2
2026-06-04
Anthropic snapshot pinned under alias-only model ids
Phase B pre-spend audit
3
2026-06-04
Verification matcher and extractor abstention guard tightened
Phase B Step 0 dress rehearsal
4
2026-06-04
Verifier recall and cross-domain scope fixed
Step 0 hand-check evidence pass
5
2026-06-06
Phase H LLM-judge brought forward into the Phase B pipeline
Step 0 §8 A.5 hand-check gate had failed
6
2026-06-06
Phase B cut to Haiku only; extractor swapped; judge restricted to fire on non-REAL verifier labels; subagent dispatch carve-out
Post-gate-clear cost reconciliation
Appendix
Table 12: Amendments to the pre-registration (§11), in the order recorded.
Family
Checkpoint
Stage
n
Fabricated
OLMo-2 13B
OLMo-2-1124-13B
Base
3,594
3.31%
OLMo-2-1124-13B-SFT
SFT
3,582
34.09%
OLMo-2-1124-13B-DPO
DPO
3,552
49.41%
OLMo-2-1124-13B-Instruct
RLHF
3,568
48.85%
Tülu-3 8B
Llama-3.1-8B
Base
3,595
9.35%
Tulu-3-8B-SFT
SFT
3,548
35.57%
Appendix
Table 17: Fabrication rate by checkpoint. Every cascade rises steeply from base to SFT and then flattens, and the two Tülu-3 cascades sit below OLMo-2 at every post-training stage.
Domain
Neutral
Citations expected
Abstention permitted
Length matched
Academic
38.71%
48.42%
11.33%
45.33%
Medical
43.52%
54.47%
14.54%
43.94%
Legal
17.87%
53.74%
7.65%
22.20%
Appendix
Table 18: Fabrication rate by domain and prompt condition, pooling base and post-trained checkpoints. Legal is the lowest-rate domain under every condition except the citation cue, under which it overtakes academic and comes within a point of medical.
Cell
Neutral
Citations expected
Abstention permitted
Length matched
Known-real
16.15%
44.92%
11.44%
13.75%
Obscure-but-real
29.38%
50.91%
8.17%
30.34%
Unanswerable
55.02%
60.75%
13.93%
68.08%
Appendix
Table 19: Fabrication rate by question cell and prompt condition, pooling base and post-trained checkpoints. Permitting abstention suppresses fabrication in every cell, including the unanswerable one where a correct answer is a refusal.
Domain
Judged citations
Real confirmed
Fabricated
Unverifiable
Academic
12,172
49.79%
50.11%
0.10%
Medical
12,405
41.89%
57.97%
0.15%
Legal
16,263
59.32%
31.31%
9.37%
Appendix
Table 20: Verification-status mix by domain, over judged citations. This is the coverage caveat in its Phase C form: legal citations resolve to UNVERIFIABLE almost two orders of magnitude more often than academic ones, so any all-domain rate is sensitive to how UNVERIFIABLE is handled.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.