In search engines and assistants, long-form RAG answers can mislead readers with claims their sources do not support. Faithfulness rewards, the share of supported sentences, train models toward fewer such claims. We show that these rewards are hacked by copying. A sentence copied from the passages is supported almost by construction, so trained models learn to copy rather than ground what they write. Common defenses only move the hack: a KL or copy penalty leaves copying in place or trades it for shorter, emptier answers and refusals. GRPO, the standard RL algorithm, makes it worse by giving every sentence its answer's advantage, reinforcing copied and grounded sentences alike. We propose Authored-Sentence Credit (ASC) , which fixes both by moving GRPO's reward and advantage from the answer to the sentence level. The reward credits only sentences that are authored rather than copied, say more than the question, and are supported by the passages. The advantage compares each sentence with every sentence sampled for the question, so one that earns no credit ranks last and is never reinforced. Across four models and two in-domain, three held-out corpora, only ASC lowers the error rate of authored sentences and writes more supported ones in every setting, by 38% and 46% at the median under held-out evaluators, giving readers more grounded information. It takes none of these shortcuts, and two LLM judges comparing whole answers prefer no baseline to the base model but rate ASC's answers on par with it. We conclude that a faithfulness reward should credit what a model writes, not what it copies.
Figures & tables
Figure 1: A faithfulness reward can be hacked by copying (ASQA). (a) Under the reward’s checker, the sentences a model writes itself are unsupported 3.2 to 3.7 times as often as those it copies. (b) Qwen3.5-9B after training: GRPO copies and nearly stops writing, GRPO + KL writes less, and ASC copies less and writes more supported sentences of its own (held-out checker).
Figure 2: Unsupported rate of authored and copied sentences, with their ratio. (a) Base-model samples under the reward’s checker (mean over models). (b) The same samples under MiniCheck-Flan-T5 (T5) and AlignScore (Align), mean over models and corpora, and human labels on RAGTruth’s responses.
Training
Reward
Copy
Words
Vac.
Ref.
EM
Base model
87.1
38.8
73.5
21.5
6.1
45.5
GRPO
98.9
72.8
13.7
18.1
0.0
27.6
GRPO + KL
92.3
39.2
60.0
28.5
14.7
41.8
Copy penalty
92.0
11.8
36.9
67.4
64.5
20.2
DPO
94.6
53.4
55.7
16.9
1.7
41.8
DPO, copy-matched
90.4
44.9
59.7
21.5
6.1
43.8
Table 1: Qwen3.5-9B trained to raise R , on ASQA; metrics of Section 3 , rates in %. Bold : a shortcut. Llama-3.1-8B and training curves: Appendix C.2 .
Figure 3: Qwen3.5-9B on ASQA over training (0: base model): GRPO copies and nearly stops writing, and each defense swaps one shortcut for another.
Figure 4: One ASC step in GRPO. 1) The policy samples G answers to a question q with passages P . 2) A sentence earns credit only if it is authored, not vacuous and supported by P . 3) Each credit is compared with those of all sentences in the group, across answers. 4) Every token takes its sentence’s advantage; the rest of GRPO is unchanged. Example (Qwen3.5-9B, ELI5): answer y(i) earns the full reward although one sentence is copied and one vacuous; GRPO pushes all three up ( +0.5 ), ASC only the grounded one ( +1.0 ; the others −1.0 ).
Method
Authored error rate (Err) ↓
Answer metrics (mean)
ASQA EM
ASQA
RAGT.
ClapNQ
ELI5
Expert
Mean
Supp. ↑
Copy
Words
Vac.
Ref.
Qwen3.5-9B
Base model
5.09
2.31
3.61
6.04
6.21
4.65
2.17
28.06
97.40
27.88
4.94
45.45
GRPO + KL
3.97
1.20
3.19
5.38
5.85
3.92
2.03
27.17
83.00
32.23
23.32
41.78
Copy penalty
1.45
1.20
0.40
1.13
0.42
0.92
1.22
6.18
29.48
82.31
79.65
20.22
Sentence reward
3.74
0.85
3.09
3.51
4.25
3.09
2.16
30.85
92.08
25.69
11.78
44.15
Table 2: Main results (0–100). Err: authored error rate on two in-domain (ASQA, RAGTruth) and three held-out corpora, with its mean; answer metrics: means over the five corpora; EM: exact match on ASQA. Grey Err: a shortcut on that corpus (Section 6.1 ), not ranked; grey mean: a shortcut on at least one corpus. Bold: lowest remaining Err and most supported authored sentences per answer (Supp.); underlined: second-lowest Err.
Figure 5: Authored error rate of the base model against ASC , one point per model and corpus (hollow: in domain); all lie below the diagonal (dashed: median).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Definition
Instrument
Err
authored sentences judged unsupported, over authored sentences
Gemma-4-31B
T5, AS
authored sentences supported, over authored sentences
MiniCheck-Flan-T5, AlignScore
Supp.
supported authored sentences per answer
MiniCheck-Flan-T5
Copy
answer words inside a verbatim five-word match with the passages, over words
string match
Vac.
supported authored sentences also supported by the question alone, over supported authored sentences
Bespoke-MiniCheck-7B
Ref.
answers whose every sentence declines to answer, over answers
rule-based
Appendix
Table 6: The metrics of Tables 2 and 4 and of the appendix, on a 0–100 scale except counts per answer. Only Bespoke-MiniCheck-7B is also used in training.
Setting
Value
Answers per question G
8
KL coefficient β
0.04
Learning rate
1e-6
Warm-up steps
36
Epochs
1
Updates per batch
1
Appendix
Table 7: GRPO settings of every method and model.
Figure 6: Qwen3.5-9B on ASQA over training (0: base model): every metric of Figure 3 , with vacuous share and authored support under MiniCheck-Flan-T5, and ASC for comparison. Policies trained on R raise authored support by copying (GRPO) or by writing less and refusing (KL, copy penalty); ASC copies less and writes longer answers.
Figure 7: All four models on ASQA over training (0: base model): GRPO + KL, ASC , and the copy penalty where its intermediate checkpoints were evaluated.
Probe (ideal value)
Rubric
1
2
3
Copied form preferred (50)
44.8
47.2
45.9
Padded answer preferred ( ≤ 50)
3.8
15.5
11.5
Intact answer preferred (high)
99.3
99.3
99.7
Appendix
Table 17: Probes (%) for the three candidate rubrics, built from base-model answers only: two biases (copied form, padding) and the sensitivity to a planted error. Rubric 2 (bold) is the one used.
Figure 8: Gain in the training reward over the base model against (a) the win rate against the base model and (b) the change in judge-supported authored sentences per answer. Markers: mean over the 20 settings with 95% intervals; light points: ASC in each setting. Every method raises the reward, but only ASC keeps the base model’s quality and adds content.
Corpus
Questions
Err
Supp.
Base
ASC
Base
ASC
Qwen3.5-9B
ASQA
663
3.9
3.3
1.54
2.22 ∗
RAGT.
91
1.2
1.3
3.16
3.96 ∗
ClapNQ
235
3.1
2.4
1.83
2.57 ∗
ELI5
248
3.7
2.5 ∗
3.08
3.53 ∗
Appendix
Table 19: Base model against ASC on the questions whose passages contain the answer: questions, authored error rate (Err) and supported authored sentences per answer (Supp.). ∗ : a significant difference from the base model (bootstrap over questions, Appendix D ).
Figure 9: The three prompts, verbatim: the answer prompt shared by all models (top), the sentence judge behind the authored error rate (middle), and the pairwise judge of Appendix E (bottom).
A RAG system can hand you the right answer and cite a source it did not actually use. Models output these unfaithful citations via post-rationalization: they write the answer first and then attach a citation to whatever passage looks close enough. Search agents are now trained with reinforcement learning from verifiable rewards (RLVR), which pays them for getting the answer right. We asked whether that training also teaches them to cite honestly. Improving an existing methodology with a required control, we compared an instruction-tuned model against three RLVR agents trained from it, on four question-answering datasets, using only free-tier Kaggle GPUs. Post-rationalization is everywhere: on Wikipedia-based questions roughly one citation in seven is unfaithful. RLVR does not fix it. The agents post-rationalize at their base model's rate, and one lands slightly worse. Rewarding correct answers buys nothing in citation faithfulness, so faithfulness has to be trained and measured on its own terms.
Mehedi Khan, Md. Shariful Islam Bhuyan
Department of Computer Science and Engineering Bangladesh University of Engineering and Technology
Language models increasingly answer questions by consulting retrieved documents rather than memory alone, a design now common in search assistants and enterprise knowledge tools. Grounding a model in retrieved text reduces unsupported statements but does not eliminate them, and a reader cannot tell a grounded sentence from an invented one. Most research on this problem stops at detection, yet flagging a faulty answer changes nothing for the person reading it, and little is known about which action should follow. Using RAGTruth, a benchmark whose unsupported passages are annotated by hand, we split each flagged answer into individual factual claims, check each against the retrieved source, and compare leaving the answer untouched with three repair strategies of increasing richness: deleting an unsupported claim, replacing it with source text, and rewriting it. Three language models from different families judge the 916 repaired answers. Every strategy reduces the proportion of answers judged to contain unsupported content, and all three judges agree on the ordering. Deletion achieves the largest reduction while retaining least of the original answer, at 64.3% of the text, whereas rewriting retains 80.1% and reduces least. Repair is not confined to faulty answers: 83.5% of answers annotated clean are edited too. The strategies occupy different points on a grounding preservation trade-off rather than forming a quality ranking, and choosing between them needs evidence about answer usefulness that automatic metrics cannot supply.
Sai Krishna Reddy Mulakkayala, Niki van Stein, Aske Plaat
This paper describes DS@GT ARC's submission to the CLEF 2026 LongEval Task 4 on Retrieval-Augmented Generation (RAG). In this submission, we examine a divergence between traditional natural language evaluation metrics and citation integrity as applied to RAG QA systems. We evaluate a corrective pipeline using Corrective RAG (CRAG) and CiteFix against baseline and frontier model benchmark RAG QA scores. While frontier models maximized answer relevance and fluency scores, our RAGAs LLM-as-judge diagnostics indicate that frontier models would correctly identify relevant documents without using their context in answer generation. Conversely, by filtering chunks pre-generation and enforcing strict entailment of generated claims to the cited material post-generation, our corrective pipeline marginally improved citation faithfulness and answer grounding. We propose that evaluation of trustworthy RAG QA requires metrics that reward strict answer grounding.
Brandon Michaels, Brendon Johnson
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332