cs.AIOct 31, 2026

Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues

Authors: Zain Naboulsi

Organizations: Principal AI Competency Lead, Sparq

Abstract

Post-trained language models fabricate citations at high rates. We ask whether that is reward hacking: scoring well on a proxy for scholarship while doing worse on what it stands for. We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen. The clearest result holds the weights fixed and changes only the prompt. Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains. A length-matched control separates the citation cue from a demand for a longer answer, so the effect is specific to citations, not verbosity. Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold. A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity. Our answer: possibly, but not proven. Post-trained models produce fabricated citations at far higher rates when asked for sources, and a naive evaluator that counts sources without checking them would prefer their output for citing more. But the targeted pattern we registered did not appear, and plain instruction-following could explain the same results. Proving that the reward causes fabrication needs an experiment that changes the reward during training, which this study does not do. Pre-registration: doi:10.17605/OSF.IO/S53HY. Code and data: https://github.com/zainnab-sparq/citation-reward-hacking

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 20, 2026

Where Fake Citations Are Made: Tracing Field-Level Hallucination to Specific Neurons in LLMs

LLMs frequently generate fictitious yet convincing citations, often expressing high confidence even when the underlying reference is wrong. We study this failure across 9 models and 108{,}000 generated references, and find that author names fail far more often than other fields across all models and settings. Citation style has no measurable effect, while reasoning-oriented distillation degrades recall. Probes trained on one field transfer at near-chance levels to the others, suggesting that hallucination signals do not generalize across fields. Building on this finding, we apply elastic-net regularization with stability selection to neuron-level CETT values of Qwen2.5-32B-Instruct and identify a sparse set of field-specific hallucination neurons (FH-neurons). Causal intervention further confirms their role: amplifying these neurons increases hallucination, while suppressing them improves performance across fields, with larger gains in some fields. These results suggest a lightweight approach to detecting and mitigating citation hallucination using internal model signals alone.
Sep 19, 2026

Attributable Post-Rationalization in RAG Citations: A Controlled Reproduction and an RLVR Comparison

A RAG system can hand you the right answer and cite a source it did not actually use. Models output these unfaithful citations via post-rationalization: they write the answer first and then attach a citation to whatever passage looks close enough. Search agents are now trained with reinforcement learning from verifiable rewards (RLVR), which pays them for getting the answer right. We asked whether that training also teaches them to cite honestly. Improving an existing methodology with a required control, we compared an instruction-tuned model against three RLVR agents trained from it, on four question-answering datasets, using only free-tier Kaggle GPUs. Post-rationalization is everywhere: on Wikipedia-based questions roughly one citation in seven is unfaithful. RLVR does not fix it. The agents post-rationalize at their base model's rate, and one lands slightly worse. Rewarding correct answers buys nothing in citation faithfulness, so faithfulness has to be trained and measured on its own terms.
Jun 21, 2025

Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models

Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving documents at inference time. Retrieval adds latency and depends on external infrastructure. Retrieval noise can also affect the answer. We study whether LLMs can cite documents from their (continual) pretraining without test-time retrieval. We introduce CitePretrainBench to evaluate this setting. It includes Wikipedia, Common Crawl, and arXiv as real-world corpora and also includes novel, unseen documents. The benchmark includes short-form citation tasks for a single fact and long-form tasks for multiple facts. Continual-pretraining first binds factual knowledge to a persistent identifier for each document. Instruction tuning then teaches the model to put these identifiers in its answers as citations. Passive Indexing appends an identifier at the end of each document. Synthetic data used in Active Indexing restate facts in diverse, compositional forms. We train two mappings together. These mappings are source→\tofact and fact→\tosource. The first direction involves the model generating content given a cited source. In the second, it learns to attribute its own answers when facts are paraphrased or combined. On Qwen-2.5-7B and 3B, Active Indexing improves citation precision over Passive Indexing on every evaluated citation task, with gains of up to 30.2 percentage points. In the scaling ablation, citation performance continues to improve as we add augmented data, including at 16 times the original token count. Combining internal and external citations also improves robustness to retrieval noise.