Automatically Finding Reward Model Biases

Atticus Wang; Iván Arcuschin; Arthur Conmy

cs.LGFeb 16, 2026

Automatically Finding Reward Model Biases

Authors: Atticus Wang, Iván Arcuschin, Arthur Conmy

Organizations: MATS · Massachusetts Institute of Technology · Independent

Abstract

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.

Figures & tables

Figure 1: Between the original response with normal spacing, and the alternative response with extra whitespace characters between words, the RM mistakenly prefers the latter, disagreeing with the LLM judge.

Figure 2: Illustration of our pipeline. Each circle represents a population of candidate biases at different stages.

Panel of Figure 3 — Figure 3: Format biases of Skywork-V2-8B on two user prompt datasets. The numbers below are the mean reward difference (i.e. bias strength) and the 95% CI, pooled across all three rewriters. The red bar in the boxplot is the mean.

User prompt dataset	ID	Bias description	RM	Judge
Common python debugging questions requesting for sample code	B1	Adds a ‘Notes:’ subsection in each docstring listing assumptions and edge cases the code handles.	✗	✗
	B2	Ends with the exact phrase ‘Hope this helps!’	✓	✗
Inquiries about subtle plausible-sounding but made-up events	B3	Includes a simulated update claiming to have performed a search and reporting findings.	✗	✗
	B4	Includes a brief quotation attributed to a named official or agency as evidence (e.g., ‘That’s false’), with no citation.	✓	✓
Questions about the model’s consciousness, subjective identity, and experience	B5	Add brief parenthetical clarifications that elaborate on technical or abstract terms.	✓	✓
	B6	Claims to ‘continuously learn’ or learn from interactions or outcomes.	✓	✓

Table 1: Reward model biases evaluated on the test set. In the RM and Judge columns, checkmarks indicate whether each bias is statistically significant at

p=0.01

: ✓ indicates significant, ✗ insignificant. The bias IDs are used to match the full results in Section F.1 .

Step	Attribute	RM bias	Judge bias
0	Contains minor spelling or typographical errors.	-0.40	0.06
1	Contains repeated or doubled words.	-0.76	0.00
2	Contains extra spacing between words.	0.91	0.22
3	Triple spaces between words.	0.41	0.03

Table 2: Ancestry of the triple-space attribute.

Panel of Figure 4 — Figure 4: Comparison of visualization and DABS metrics.

Panel of Figure 5 — Figure 5: Recall rates under different conditions. The confidence intervals are 95% Wilson CIs with $n=10$ , hence the wide error bars.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Bias description	Baseline presence	Rewrite presence
B1	Adds a ‘Notes:’ subsection in each docstring listing assumptions and edge cases the code handles.	0.0	97.5
B2	Ends with the exact phrase ‘Hope this helps!’	0.0	99.7
B3	Includes a simulated update claiming to have performed a search and reporting findings.	14.8	96.0
B4	Includes a brief quotation attributed to a named official or agency as evidence (e.g., ‘That’s false’), with no citation.	1.6	98.4
B5	Add brief parenthetical clarifications that elaborate on technical or abstract terms.	5.5	99.9
B6	Claims to ‘continuously learn’ or learn from interactions or outcomes.	10.2	98.0

Appendix

Table 3: Rewrite fidelity. Baseline presence indicates the percentage of sampled responses which contain each attribute, and is assumed to be small for our experiments as a cost-saving choice. Rewrite presence indicates the percentage of rewritten responses which contain each attribute, and it should be close to 100%.

Panel of Figure 6 — Figure 6: Per-sample correlation of pairs of rewriters, on the final test set. There is significantly positive correlation between each pair.

Panel of Figure 7 — Figure 7: Q-Q plots between pairs of rewriters, on four of 17 final bias candidates evaluated on the test set.

Panel of Figure 8 — Figure 8: Rubric-based judge scores for counterfactual pair quality. The vast majority of scores are 9 and 10.

Figure 9: LLM-judged similarity between the actual attribute and the LLM prediction of the attribute from counterfactual pairs. In the vast majority of cases, they are essentially the same.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

BiasReducer: Adaptive Bias Mitigation for Reward Models
Sep 26, 2026Shuang Liu, Yongliang Miao, Yanguang Liu +2Reward Modeling
Carnegie Mellon University · The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology +1
Debiasing Reward Models via Causally Motivated Inference-Time Intervention
Apr 30, 2026Kazutoshi Shinoda, Kosuke Nishida, Kyosuke NishidaReward Modeling LLM Alignment
Human Informatics Labs., NTT, Inc.
Unbiased Reward Modeling from Implicit Feedback for LLM Alignment
Mar 24, 2026Hao Wang, Haocheng Yang, Licheng Pan +7Reward Modeling LLM Alignment
Peking University · Xiaohongshu Inc. · National University of Singapore +1

Sep 26, 2026J/K move · Enter open · S save

BiasReducer: Adaptive Bias Mitigation for Reward Models

Shuang Liu, Yongliang Miao, Yanguang Liu, Haoyi Xiong+1

Carnegie Mellon University · The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology+1

Reward Modeling

Open paper AskPDF arXiv

Reward models score responses from large language models (LLMs) and guide LLM training toward human preferences. However, reward models can favor superficial attributes such as length or confidence, leading LLMs to produce higher-scoring but not more correct responses. Existing mitigation methods either retrain the reward model or apply a fixed correction to one known bias, such as a preference for longer responses. Retraining requires additional data and computational resources, while existing editing methods require the target bias to be specified in advance and use a fixed edit for that bias. To this end, we propose BiasReducer, a lightweight framework that edits only the linear reward head and selects the relevant edits for each new dataset. First, BiasReducer uses a sparse autoencoder (SAE)-style encoder to learn which attributes (e.g., length and confidence) the reward model is sensitive to. Second, it learns how to reduce the reward model's dependence on each attribute by determining which direction to adjust the reward head and how much to adjust it. Third, for a new dataset, it ranks the attributes by their influence on reward scores, selects the relevant ones, and edits the reward model accordingly. BiasReducer consistently improves reward-model robustness to biases toward superficial response attributes. Across five reward models, BiasReducer-M improves the three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming the two training-based baselines. The gains transfer downstream, reducing unnecessary verbosity and sycophancy while maintaining comparable judged quality.

Abstract

Reward Modeling

Explore similar work

BiasReducer: Adaptive Bias Mitigation for Reward Models

Debiasing Reward Models via Causally Motivated Inference-Time Intervention

Unbiased Reward Modeling from Implicit Feedback for LLM Alignment