In search engines and assistants, long-form RAG answers can mislead readers with claims their sources do not support. Faithfulness rewards, the share of supported sentences, train models toward fewer such claims. We show that these rewards are hacked by copying. A sentence copied from the passages is supported almost by construction, so trained models learn to copy rather than ground what they write. Common defenses only move the hack: a KL or copy penalty leaves copying in place or trades it for shorter, emptier answers and refusals. GRPO, the standard RL algorithm, makes it worse by giving every sentence its answer's advantage, reinforcing copied and grounded sentences alike. We propose Authored-Sentence Credit (ASC) , which fixes both by moving GRPO's reward and advantage from the answer to the sentence level. The reward credits only sentences that are authored rather than copied, say more than the question, and are supported by the passages. The advantage compares each sentence with every sentence sampled for the question, so one that earns no credit ranks last and is never reinforced. Across four models and two in-domain, three held-out corpora, only ASC lowers the error rate of authored sentences and writes more supported ones in every setting, by 38% and 46% at the median under held-out evaluators, giving readers more grounded information. It takes none of these shortcuts, and two LLM judges comparing whole answers prefer no baseline to the base model but rate ASC's answers on par with it. We conclude that a faithfulness reward should credit what a model writes, not what it copies.
Figures & tables
Figure 1: A faithfulness reward can be hacked by copying (ASQA). (a) Under the reward’s checker, the sentences a model writes itself are unsupported 3.2 to 3.7 times as often as those it copies. (b) Qwen3.5-9B after training: GRPO copies and nearly stops writing, GRPO + KL writes less, and ASC copies less and writes more supported sentences of its own (held-out checker).
Figure 2: Unsupported rate of authored and copied sentences, with their ratio. (a) Base-model samples under the reward’s checker (mean over models). (b) The same samples under MiniCheck-Flan-T5 (T5) and AlignScore (Align), mean over models and corpora, and human labels on RAGTruth’s responses.
Training
Reward
Copy
Words
Vac.
Ref.
EM
Base model
87.1
38.8
73.5
21.5
6.1
45.5
GRPO
98.9
72.8
13.7
18.1
0.0
27.6
GRPO + KL
92.3
39.2
60.0
28.5
14.7
41.8
Copy penalty
92.0
11.8
36.9
67.4
64.5
20.2
DPO
94.6
53.4
55.7
16.9
1.7
41.8
DPO, copy-matched
90.4
44.9
59.7
21.5
6.1
43.8
Table 1: Qwen3.5-9B trained to raise R , on ASQA; metrics of Section 3 , rates in %. Bold : a shortcut. Llama-3.1-8B and training curves: Appendix C.2 .
Figure 3: Qwen3.5-9B on ASQA over training (0: base model): GRPO copies and nearly stops writing, and each defense swaps one shortcut for another.
Figure 4: One ASC step in GRPO. 1) The policy samples G answers to a question q with passages P . 2) A sentence earns credit only if it is authored, not vacuous and supported by P . 3) Each credit is compared with those of all sentences in the group, across answers. 4) Every token takes its sentence’s advantage; the rest of GRPO is unchanged. Example (Qwen3.5-9B, ELI5): answer y(i) earns the full reward although one sentence is copied and one vacuous; GRPO pushes all three up ( +0.5 ), ASC only the grounded one ( +1.0 ; the others −1.0 ).
Method
Authored error rate (Err) ↓
Answer metrics (mean)
ASQA EM
ASQA
RAGT.
ClapNQ
ELI5
Expert
Mean
Supp. ↑
Copy
Words
Vac.
Ref.
Qwen3.5-9B
Base model
5.09
2.31
3.61
6.04
6.21
4.65
2.17
28.06
97.40
27.88
4.94
45.45
GRPO + KL
3.97
1.20
3.19
5.38
5.85
3.92
2.03
27.17
83.00
32.23
23.32
41.78
Copy penalty
1.45
1.20
0.40
1.13
0.42
0.92
1.22
6.18
29.48
82.31
79.65
20.22
Sentence reward
3.74
0.85
3.09
3.51
4.25
3.09
2.16
30.85
92.08
25.69
11.78
44.15
Table 2: Main results (0–100). Err: authored error rate on two in-domain (ASQA, RAGTruth) and three held-out corpora, with its mean; answer metrics: means over the five corpora; EM: exact match on ASQA. Grey Err: a shortcut on that corpus (Section 6.1 ), not ranked; grey mean: a shortcut on at least one corpus. Bold: lowest remaining Err and most supported authored sentences per answer (Supp.); underlined: second-lowest Err.
Figure 5: Authored error rate of the base model against ASC , one point per model and corpus (hollow: in domain); all lie below the diagonal (dashed: median).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Definition
Instrument
Err
authored sentences judged unsupported, over authored sentences
Gemma-4-31B
T5, AS
authored sentences supported, over authored sentences
MiniCheck-Flan-T5, AlignScore
Supp.
supported authored sentences per answer
MiniCheck-Flan-T5
Copy
answer words inside a verbatim five-word match with the passages, over words
string match
Vac.
supported authored sentences also supported by the question alone, over supported authored sentences
Bespoke-MiniCheck-7B
Ref.
answers whose every sentence declines to answer, over answers
rule-based
Appendix
Table 6: The metrics of Tables 2 and 4 and of the appendix, on a 0–100 scale except counts per answer. Only Bespoke-MiniCheck-7B is also used in training.
Setting
Value
Answers per question G
8
KL coefficient β
0.04
Learning rate
1e-6
Warm-up steps
36
Epochs
1
Updates per batch
1
Appendix
Table 7: GRPO settings of every method and model.
Figure 6: Qwen3.5-9B on ASQA over training (0: base model): every metric of Figure 3 , with vacuous share and authored support under MiniCheck-Flan-T5, and ASC for comparison. Policies trained on R raise authored support by copying (GRPO) or by writing less and refusing (KL, copy penalty); ASC copies less and writes longer answers.
Figure 7: All four models on ASQA over training (0: base model): GRPO + KL, ASC , and the copy penalty where its intermediate checkpoints were evaluated.
Probe (ideal value)
Rubric
1
2
3
Copied form preferred (50)
44.8
47.2
45.9
Padded answer preferred ( ≤ 50)
3.8
15.5
11.5
Intact answer preferred (high)
99.3
99.3
99.7
Appendix
Table 17: Probes (%) for the three candidate rubrics, built from base-model answers only: two biases (copied form, padding) and the sensitivity to a planted error. Rubric 2 (bold) is the one used.
Figure 8: Gain in the training reward over the base model against (a) the win rate against the base model and (b) the change in judge-supported authored sentences per answer. Markers: mean over the 20 settings with 95% intervals; light points: ASC in each setting. Every method raises the reward, but only ASC keeps the base model’s quality and adds content.
Corpus
Questions
Err
Supp.
Base
ASC
Base
ASC
Qwen3.5-9B
ASQA
663
3.9
3.3
1.54
2.22 ∗
RAGT.
91
1.2
1.3
3.16
3.96 ∗
ClapNQ
235
3.1
2.4
1.83
2.57 ∗
ELI5
248
3.7
2.5 ∗
3.08
3.53 ∗
Appendix
Table 19: Base model against ASC on the questions whose passages contain the answer: questions, authored error rate (Err) and supported authored sentences per answer (Supp.). ∗ : a significant difference from the base model (bootstrap over questions, Appendix D ).
Figure 9: The three prompts, verbatim: the answer prompt shared by all models (top), the sentence judge behind the authored error rate (middle), and the pairwise judge of Appendix E (bottom).