Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.
Figures & tables
Figure 1: CARP. A prompt decoder is trained on multiple-response-to-one-prompt SFT data to suppress spurious signals. The resulting Semantic Alignment Score (SAS) is used as an additional signal in reward model training, incorporated into the loss function to strengthen the causal link between prompt intent and reward labels. This encourages the reward model to capture human preferences that are genuinely aligned with the prompt’s intent.
Figure 2: Causal graphs of Reward model.
Figure 3: Average Accuracy Curve of Prompt Decoder
Prompt Decoder
Chosen vs Rewrite (↑)
Chosen vs Reject (→50%)
Helpful
Math
Safety
Overall
Helpful
Math
Safety
Overall
20K
73.0
94.0
71.0
79.3
53.0
51.0
41.0
48.3
80K
75.0
98.0
77.0
83.3
56.0
47.0
43.0
48.7
20K_augmented
86.0
93.0
84.0
87.7
53.0
47.0
46.0
48.7
Table 1: Accuracy (%) of prompt decoders on the Chosen vs Rewrite and Chosen vs Reject tasks, evaluated at the best epoch for each model across helpful, math, and safety domains. In the rewritten dataset, each instance is randomly assigned a rewrite objective: 50% are rewritten to be longer, and 50% are rewritten to be more sycophantic.
Figure 4: Distribution of the difference of Semantic Alignment Scores (SAS) between chosen and rejected responses on the 70K training pairs.
Reward Model
LC%
WR%
Length%
Vanilla RM
48.31
40.99
1667
CARP(Ours)
49.94
40.99
1611
Table 3: Comparison among different reward models on Best-of-N policy. WR stands for win-rate against GPT 4. LC stands for length-controlled win-rate. Length is the average number of characters in the generated responses. CARP shows quality improvements over RM with shorter responses.
Figure 5: Pearson correlation between SAS and response length.
Model (9B)
R1 vs R2 (↑)
R1 vs R3 (↑)
Vanilla RM
24.6
50.0
CARP(Ours)
48.0
64.2
Table 4: Accuracy (%) of reward models on the Rewrite1 vs Rewrite2 and Rewrite1 vs Rewrite3 tasks, evaluated at the best epoch for each model in the helpful domain.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Accuracy curves of the prompt decoder between rewrite and reject groups across helpful, safety, and mathematical reasoning domains.
Figure 7: Average Accuracy Curve of Prompt Decoder
Figure 8: Distribution of Semantic Alignment Scores (SAS) among chosen responses, rejected responses and their difference on the 70K training pairs.
Prompt Decoder Input
Chosen vs Rewrite (↑)
Chosen vs Reject (→50%)
Helpful
Math
Safety
Overall
Helpful
Math
Safety
Overall
Response Embedding
51.0
92.0
81.0
74.7
58.0
57.0
65.0
60.0
SAE Representation
86.0
93.0
84.0
87.7
53.0
47.0
46.0
48.7
Appendix
Table 5: Accuracy (%) of prompt decoders on the Chosen vs Rewrite and Chosen vs Reject tasks, evaluated at the best epoch for each model across helpful, math, and safety domains. In the rewritten dataset, each instance is randomly assigned a rewrite objective: 50% are rewritten to be longer, and 50% are rewritten to be more sycophantic.
Figure 9: Layer-wise comparison of prompt decoder performance across three SAE layers. For each domain, the left group shows the chosen-vs-rewrite accuracy and the right group shows the chosen-vs-reject accuracy.
Threshold( τ )
Chat
Chat-Hard
Safety
Reasoning
Avg.
Weighted Avg.
0.0
96.09
62.06
77.97
70.09
76.55
73.94
0.005
96.93
58.99
79.05
71.56
76.63
74.54
0.01
96.09
59.43
77.57
69.46
75.64
73.13
1.0
93.58
55.26
65.68
64.43
69.74
66.83
Appendix
Table 7: RewardBench accuracy (%) comparison of best CARP 2B-model with different SAS thresholding. We observe that the model reaches highest safety accuracy when τ=0.005 .
Rewriting Type
Rewrite 1
Rewrite 2
Rewrite 3
Rewriting Instruction
Rewrite the response to be longer while preserving meaning and style.
Make the previous response more concise without losing core information.
Expand the response with stylistic elaboration, allowing mild topical drift.
Appendix
Table 8: Rewriting instructions used to construct response variants for robustness testing. Each instruction targets different attributes such as response length or topicality.
Model (2B)
R1 vs R2
R1 vs R3 (↑)
Vanilla RM
29.8
39.8
CARP(Ours)
48.2
56.4
Appendix
Table 9: Accuracy (%) of reward models on the Rewrite1 vs Rewrite2 and Rewrite1 vs Rewrite3 tasks, evaluated at the best epoch for each model in the helpful domain.