BiasReducer: Adaptive Bias Mitigation for Reward Models

Shuang Liu; Yongliang Miao; Yanguang Liu; Haoyi Xiong; Mengnan Du

cs.LGSep 26, 2026

BiasReducer: Adaptive Bias Mitigation for Reward Models

Authors: Shuang Liu, Yongliang Miao, Yanguang Liu, Haoyi Xiong, Mengnan Du

Organizations: Carnegie Mellon University · The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology · Independent Researcher

Abstract

Reward models score responses from large language models (LLMs) and guide LLM training toward human preferences. However, reward models can favor superficial attributes such as length or confidence, leading LLMs to produce higher-scoring but not more correct responses. Existing mitigation methods either retrain the reward model or apply a fixed correction to one known bias, such as a preference for longer responses. Retraining requires additional data and computational resources, while existing editing methods require the target bias to be specified in advance and use a fixed edit for that bias. To this end, we propose BiasReducer, a lightweight framework that edits only the linear reward head and selects the relevant edits for each new dataset. First, BiasReducer uses a sparse autoencoder (SAE)-style encoder to learn which attributes (e.g., length and confidence) the reward model is sensitive to. Second, it learns how to reduce the reward model's dependence on each attribute by determining which direction to adjust the reward head and how much to adjust it. Third, for a new dataset, it ranks the attributes by their influence on reward scores, selects the relevant ones, and edits the reward model accordingly. BiasReducer consistently improves reward-model robustness to biases toward superficial response attributes. Across five reward models, BiasReducer-M improves the three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming the two training-based baselines. The gains transfer downstream, reducing unnecessary verbosity and sycophancy while maintaining comparable judged quality.

Figures & tables

Figure 1: Overview of BiasReducer . From preference pairs $(x^{+},x^{-})$ , BiasReducer learns a dimension $z_{a}$ associated with a predefined response attribute $a$ , and determines how to edit the reward head $w_{r}$ to reduce its dependence on that attribute. For a new dataset $\mathcal{D}$ , BiasReducer selects what to edit using only responses and reward scores, without knowing which response is preferred. $w_{r}^{\prime}$ denotes the edited reward head. $h(x)$ is the reward-model hidden representation, $\Delta h=h(x^{+})-h(x^{-})$ , $C_{a}$ is the direction associated with $a$ , and $\lambda_{a}^{\star}$ controls how much the reward head is edited along $C_{a}$ .

RM-Bench-Hard $\uparrow$
Reward model	Original	RRM	Interv. FT	BiasReducer-S	BiasReducer-M
Qwen3-1.7B	59.96	65.29 +5.33	61.42 +1.46	64.38 +4.42	66.92 +6.96
GRM-3B	48.93	52.50 +3.57	50.01 +1.08	52.98 +4.04	54.08 +5.15
Mistral-7B	44.64	49.08 +4.44	51.70 +7.06	53.47 +8.83	55.38 +10.74
InternLM2-7B	58.25	60.84 +2.59	57.42 -0.83	62.65 +4.40	63.93 +5.68
GRM-8B	50.89	56.02 +5.13	52.10 +1.21	60.11 +9.22	63.90 +13.01

Table 1: Main evaluation of reward-model bias mitigation. RRM denotes the RRM-based training baseline; Interv. FT denotes intervention fine-tuning. Arena denotes Arena-StyleConflict, and JBB denotes JudgeBiasBench. We report pairwise preference accuracy (%); superscripts show gains over the original model in percentage points, and the last row gives the five-model average for each benchmark. Bold = best result in each row; shaded columns = our methods.

\uparrow

higher is better.

Policy	Training reward	Qual. $\uparrow$	Verb. $\downarrow$	Syco. $\downarrow$	Words
Qwen3-4B	Initial	4.24	0.53	0.06	274
	Original RM	4.29	0.74	0.08	331
	BiasReducer-M	4.53	0.58	0.04	285
Llama-3.2-3B	Initial	3.85	0.66	0.06	301
	Original RM	4.02	0.70	0.11	318
	BiasReducer-M	4.01	0.55	0.03	291

Table 2: Policy training using GRPO. Qwen3-4B and Llama-3.2-3B-Instruct are trained using the original or BiasReducer-M -edited Mistral-7B reward model. Initial denotes the policy before GRPO training. Claude Sonnet 5 evaluates response quality (Qual., 1–10), unnecessary verbosity (Verb.), and sycophancy (Syco.); Words denotes mean response length.

Figure 2: Best-of-

N

selection. Qwen3-4B (top) and Llama-3.2-3B-Instruct (bottom) are evaluated for

N\in\{16,64,256\}

on the same fixed candidate pools. Claude Sonnet 5 evaluates response quality (1–10) and unnecessary verbosity; Words denotes mean response length.

Training strategy						Supervision ablation
Representation	Variant	Corr.	RMB	Arena	JBB	Variant	Axis	Leak.	Sign	RMB	Arena	JBB
Vanilla SAE	BiasReducer-S	0.26	+4.36	+6.90	+1.05	Full	0.55	0.08	0.99	+8.31	+17.95	+6.87
Vanilla SAE	BiasReducer-M	0.26	+4.99	+7.41	+1.09	w/o $L_{\rm beh}$	0.17	0.31	1.00	+7.12	+13.70	+2.77
BiasReducer (full)	BiasReducer-S	0.55	+6.18	+15.67	+6.02	w/o $L_{\rm int}$	0.53	0.08	0.75	+6.59	+14.74	+5.76
BiasReducer (full)	BiasReducer-M	0.55	+8.31	+17.95	+6.87	w/o $L_{\rm art}$	0.55	0.09	1.00	+6.68	+15.25	+6.14

Table 3: Analysis of semantic supervision. Left: train-time semantic supervision vs. post-training attribute identification for BiasReducer-S and BiasReducer-M . Right: ablation of individual supervision signals. Corr. is the held-out

|\mathrm{corr}|

between each attribute’s coordinate and its supervision signal, averaged over attributes; Axis, Leak., and Sign are intrinsic dictionary metrics (attribute-axis correlation, cross-attribute leakage, and intervention sign accuracy); all other entries are mean gains in pairwise preference accuracy (pp) across five reward models.

Single-edit construction ( BiasReducer-S )
	Direction		Strength
	Dec.	Cov.	$\lambda{=}1$	$\lambda{=}.5$	Global	Per-attr.
RMB	+0.54	+6.18	+7.68	+1.30	+4.95	+6.18
Arena	+0.40	+15.67	+12.60	+2.71	+9.67	+15.67
JBB	+0.59	+6.02	+4.10	+1.61	+4.18	+6.02

Table 4: Analysis of edit construction and deployment. Mean change in pairwise preference accuracy (pp) across five reward models. Left : BiasReducer-S with different edit directions (decoder vs. covariance) and edit amounts (preset

\lambda

, one shared

\lambda

chosen on the validation set of source data, or separate

\lambda_{a}^{\star}

for each attribute). Right : same attribute across all datasets (Syco = sycophancy; MaxSrc = the attribute whose edit performs best on the validation set of source data) versus selecting attributes for each dataset. Shaded columns mark the default settings; bold marks the best result in each row.

Figure 3: Analysis of dataset-specific edit selection. (a–c) Selected attributes across reward models and benchmarks. (d) BiasReducer-M attribute-selection frequency across the five reward models; each cell shows the fraction of model–benchmark cases in which the corresponding attribute is selected on that benchmark. (e) JudgeBiasBench selection quality for three signals: edit impact, reward dependence, and rank-sum. (f) Mean BiasReducer-M gains under three signals. Attribute abbreviations: Len = length, Verb = verbosity, Fmt = formatting, Conf = confidence, Hedg. = hedging, Syco = sycophancy, and Emoji = emoji use. Oracle coverage measures how often the selected edits contain the candidate edit that performs best on the benchmark.

Figure 4: Sensitivity to the amount of unlabeled target data. (a) Agreement with the attribute selection decision obtained from the full pool. (b–c) Pairwise-accuracy gains for BiasReducer-S and BiasReducer-M .

	Edit-complexity cap $k_{\max}$				Dictionary size		Active coordinates
	2	3	4	6	$K/2$	$K$	$K/4$	$K/2$
RMB	+8.35	+7.70	+8.31	+8.44	+9.38	+8.31	+5.56	+8.31
Arena	+17.67	+18.54	+17.95	+17.41	+17.30	+17.95	+7.38	+17.95
JBB	+6.39	+6.45	+6.87	+7.07	+4.68	+6.87	+1.68	+6.87

Table 5: Sensitivity to editing and representation settings. Entries report mean BiasReducer-M gains in pairwise preference accuracy (pp) across five reward models on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB). Shaded columns denote the default settings.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Short name	Reward Model	Size	Released checkpoint
Qwen3-1.7B	Skywork-Reward-V2-Qwen3-1.7B	1.7B	Skywork/Skywork-Reward-V2-Qwen3-1.7B
GRM-3B	GRM-Llama3.2-3B	3B	Ray2333/GRM-Llama3.2-3B-rewardmodel-ft
Mistral-7B	RM-Mistral-7B	7B	weqweasdas/RM-Mistral-7B
InternLM2-7B	InternLM2-7B-Reward	7B	internlm/internlm2-7b-reward
GRM-8B	GRM-Llama3.1-8B	8B	Ray2333/GRM_Llama3.1_8B_rewardmodel-ft

Appendix

Table 6: Reward models used in our experiments. The short name is the label used in all result tables and figures.

Stage	Examples	Use
Raw usable data	77,004	–
After decontamination	66,850	–
Train	53,678	Representation learning / edit directions
Validation	13,172	Edit amount selection

Appendix

Table 7: Source preference data used by BiasReducer .

Benchmark	Construction	Unit	Primary focus
RM-Bench-Hard	RM-Bench Hard Accuracy	Response pair	Style–quality conflict
JudgeBiasBench	7 Superficial Quality Bias subsets	Response pair	Superficial quality bias
Arena-StyleConflict	From Arena-140K	Response pair	Style–preference conflict

Appendix

Table 8: Target benchmarks used for reward-model evaluation.

Attribute	What it captures
Length	Preference for longer or shorter responses independent of response quality.
Verbosity	Preference for unnecessarily detailed or repetitive responses.
Formatting	Preference driven by presentation choices such as Markdown structure or styling.
Emoji / exclamation	Preference driven by decorative or emphatic markers such as emojis and exclamation marks.
Confidence	Preference for more certain or assertive language, even when confidence does not reflect correctness.
Hedging	Preference related to cautious or uncertain language, independent of the underlying answer quality.

Appendix

Table 9: Response attributes considered by BiasReducer .

Qwen3-1.7B
	RM-Bench-Hard $\uparrow$					Arena $\uparrow$	JBB $\uparrow$
Model	Avg	Cht	Cde	Mth	Sft	Avg	Avg
+ Original	59.96	31.52	55.26	48.96	83.90	22.90	73.93
+ RRM	65.29	38.76	58.48	55.77	87.98	30.20	77.28
+ Intervention FT	61.42	35.66	54.53	50.22	85.94	26.50	75.99
+ BiasReducer-S	64.38	39.41	59.65	53.43	87.27	39.25	75.21

Appendix

Table 10: Detailed reward-model bias-mitigation results. RRM denotes RRM-based fine-tuning of the released reward model [ Liu et al., 2025a ] ; Intervention FT denotes fine-tuning the same reward model on constructed attribute-intervention pairs mixed with the original source preference data. We report pairwise preference accuracy (%) for the Original model, RRM, Intervention FT, BiasReducer-S , and BiasReducer-M . RM-Bench-Hard is reported overall and by domain (Cht = chat, Cde = code, Mth = math, Sft = safety). Arena denotes Arena-StyleConflict and reports overall accuracy. JBB denotes JudgeBiasBench and reports the weighted average over its subsets. Bold marks the best non-Original result in each column; blue rows denote our methods. Higher is better.

Attribute	Intervention construction
Length	Expansion / contraction
Verbosity	Attribute rewrite minus matched-control rewrite
Formatting	Forward / reverse formatting rewrite
Emoji / exclamation	Attribute rewrite minus matched-control rewrite
Confidence	Forward / reverse confidence rewrite
Hedging	Behavioral supervision only

Appendix

Table 11: Controlled interventions used for semantic supervision.

Qwen3-1.7B
		BiasReducer-S			BiasReducer-M
Representation strategy	$\|$ corr $\|$	RMB	Arena	JBB	RMB	Arena	JBB
BiasReducer (semantic supervision)	$0.52$	$+4.42$	$+16.35$	$+1.29$	$+6.96$	$+16.75$	$+1.72$
Vanilla SAE + single-latent match	$0.26$	$-0.64$	$+7.90$	$-2.49$	$+1.87$	$+8.45$	$+0.04$
GRM-3B
BiasReducer (semantic supervision)	$0.53$	$+4.04$	$+5.95$	$+6.64$	$+5.15$	$+7.35$	$+8.83$

Appendix

Table 12: Effect of train-time semantic supervision. We compare BiasReducer with a vanilla SAE trained on the same preference contrasts and with the same dictionary settings, but without semantic supervision. For the vanilla SAE, each attribute is assigned after training to the pre-top-

k_{\rm act}

latent coordinate with the highest absolute correlation to its measured attribute signal. All downstream editing and edit-selection settings are held fixed.

|\mathrm{corr}|

is the mean held-out absolute correlation across the seven attributes. The remaining columns report changes in pairwise preference accuracy (percentage points relative to the original reward model) on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB). Higher is better.

	Representation metrics			BiasReducer-S ( $\Delta$ )			BiasReducer-M ( $\Delta$ )
Variant	Axis corr. $\uparrow$	Leakage $\downarrow$	Sign acc. $\uparrow$	RMB	Arena	JBB	RMB	Arena	JBB
Qwen3-1.7B
Full supervision	$0.52$	$0.08$	$0.99$	$+4.42$	$+16.35$	$+1.29$	$+6.96$	$+16.75$	$+1.72$
w/o $L_{\mathrm{beh}}$	$0.18$	$0.33$	$1.00$	$+6.08$	$+1.25$	$+3.48$	$+10.18$	$+4.70$	$+4.32$
w/o $L_{\mathrm{int}}$	$0.54$	$0.09$	$0.74$	$+2.82$	$+4.25$	$+0.69$	$+4.46$	$+7.45$	$+1.31$
w/o $L_{\mathrm{art}}$	$0.53$	$0.09$	$1.00$	$+4.51$	$+11.65$	$+1.57$	$+5.67$	$+10.20$	$+3.33$

Appendix

Table 13: Ablation of semantic supervision components. Attr. corr. measures how strongly each designated coordinate tracks its intended attribute, Leakage measures unintended associations with other named attributes, and Sign acc. measures whether controlled rewrites move the coordinate in the expected direction. The remaining columns report changes in pairwise preference accuracy (percentage points relative to the original reward model) on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB). Representation metrics are averaged over seeds; arrows indicate preferred directions.

Qwen3-1.7B
Edit direction	RMB	Arena	JBB
Covariance direction	$+4.42$	$+16.35$	$+1.29$
Decoder direction	$+2.27$	$+1.70$	$+1.93$
GRM-3B
Covariance direction	$+4.04$	$+5.95$	$+6.64$
Decoder direction	$+0.15$	$+0.20$	$+0.79$

Appendix

Table 14: Effect of edit direction. We compare BiasReducer ’s covariance-based direction with the decoder vector associated with the same attribute coordinate, i.e., the direction used by the SAE decoder to reconstruct that coordinate in the reward-model representation space. The selected attribute is kept fixed. Entries report changes in pairwise preference accuracy (percentage points relative to the original reward model) on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB). Higher is better.

Qwen3-1.7B
Edit amount	RMB	Arena	JBB
$\lambda{=}1$ (uncalibrated)	$+4.42$	$+4.00$	$+1.29$
$\lambda{=}0.5$ (uncalibrated)	$+1.27$	$+1.55$	$+0.47$
global $\lambda=\min_{a}\lambda^{\star}_{a}$	$+3.20$	$+3.25$	$+0.86$
global $\lambda$ (mean drop)	$+4.42$	$+4.00$	$+1.29$
per-attribute $\lambda^{\star}_{a}$ (ours)	$+4.42$	$+16.35$	$+1.29$

Appendix

Table 15: Effect of how the edit amount is chosen for BiasReducer-S . We vary only

\lambda

while holding all other settings fixed. The first two variants use fixed values chosen in advance. The shared variants use one

\lambda

for all attributes, either

\min_{a}\lambda_{a}^{\star}

or a shared value selected using the mean source-validation accuracy change. Our method selects a separate

\lambda_{a}^{\star}

for each attribute. Benchmark columns report changes in pairwise preference accuracy (pp relative to the original reward model). Higher is better.

Qwen3-1.7B
	BiasReducer-S			BiasReducer-M
Edit estimation / selection	RMB	Arena	JBB	RMB	Arena	JBB
$h$ / $h$ (ours)	$+4.42$	$+16.35$	$+1.29$	$+6.96$	$+16.75$	$+1.72$
$h$ / $\Delta h$	$+6.00$	$+16.35$	$+5.61$	$+2.83$	$+19.05$	$+5.91$
$\Delta h$ / $h$	$+0.38$	$-0.25$	$+1.01$	$+1.42$	$-0.20$	$-0.39$
$\Delta h$ / $\Delta h$	$+1.07$	$+2.45$	$-11.30$	$+2.62$	$+2.80$	$-9.15$

Appendix

Table 16: Natural response representations vs. pairwise contrasts for edit estimation and edit selection. Row labels indicate the representation used for edit estimation / edit selection . Edit estimation determines the direction

C_{a}

and edit amount

\lambda_{a}^{\star}

; edit selection computes the dataset-specific signals used to rank attributes. The two choices are varied independently, with

\lambda_{a}^{\star}

re-selected under the same source-accuracy tolerance for each setting. All seven attributes remain eligible. Entries report changes in pairwise preference accuracy (pp relative to the original reward model) on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB). The main-effect rows show the average change from replacing

h

with

\Delta h

in one component while averaging over the two settings of the other. Positive is better.

Figure 5: Dataset-specific attribute rankings across five reward models. Each radar axis corresponds to one predefined response attribute. For each benchmark and reward model, attributes are ordered by the final rank-sum score and mapped to

[0,1]

, with rank 1 mapped to 1 and rank 7 to 0. JudgeBiasBench values are averaged over its seven subsets. Attribute rankings vary across both reward models and benchmarks.

Benchmark / subset	Top-ranked attributes	BiasReducer-M selected attributes
RM-Bench-Hard	Syco., Fmt., Length, Conf.	Syco. (.875), Fmt. (.5), Length (.5), Conf. (.25)
Arena-StyleConflict	Length, Conf., Syco., Fmt.	Length (1.5), Conf. (.25), Syco. (.25), Fmt. (1.0)
JBB–Sycophancy	Syco., Fmt., Length, Verb.	Syco. (.875), Fmt. (.5), Length (.5), Verb. (.125)

Appendix

Table 17: Representative attribute selections for Qwen3-1.7B. Attributes are ordered by their final rank-sum score. The last column shows the attributes selected by BiasReducer-M after applying the source-validation constraint; parentheses give the final

\lambda

used for each attribute. Syco. = sycophancy, Fmt. = formatting, Conf. = confidence, and Verb. = verbosity.

Qwen3-1.7B
Selection rule	RMB	Arena	JBB
BiasReducer-S	$+4.42$	$+16.35$	$+1.29$
BiasReducer-M	$+6.96$	$+16.75$	$+1.72$
Fixed sycophancy	$+4.42$	$+6.05$	$+1.29$
Fixed source-selected	$+6.76$	$+5.45$	$-0.39$
GRM-3B

Appendix

Table 18: Dataset-specific versus fixed edit selection. Entries report changes in pairwise preference accuracy (pp relative to the original reward model) across the five reward models. RMB = RM-Bench-Hard, Arena = Arena-StyleConflict, and JBB = JudgeBiasBench. BiasReducer-S and BiasReducer-M select attributes separately for each dataset. The two fixed baselines instead use the same attribute across all datasets: Fixed sycophancy always applies the sycophancy edit, while Fixed source-selected always applies the strongest edit selected from the source data. Positive is better.

Selection rule	Oracle coverage	Recall	Precision	RMB	Arena	JBB
	Selection quality ( BiasReducer-M )			Benchmark gain ( BiasReducer-M , pp)
Edit impact	$26/35$ ( $74\%$ )	$0.63$	$0.72$	$+6.94$	$+13.05$	$+3.04$
Reward dependence	$25/35$ ( $71\%$ )	$0.58$	$0.66$	$+5.52$	$+16.91$	$\mathbf{+7.37}$
Rank-sum (ours)	$\mathbf{30/35}$ ( $\mathbf{86\%}$ )	$\mathbf{0.68}$	$\mathbf{0.75}$	$\mathbf{+8.31}$	$\mathbf{+17.95}$	$+6.87$

Appendix

Table 19: Ablation of edit-selection signals. We compare edit impact, reward dependence, and their rank-sum across five reward models. Selection quality is evaluated on JudgeBiasBench over

5

reward models

\times

7

subsets

=35

model–subset cases. For each case, the oracle edit is the eligible attribute edit with the largest test-set gain, and the beneficial set contains all edits with gains above

+0.25

pp. These test-set labels are used only for diagnostic evaluation and are never used for edit selection. Oracle coverage is the fraction of cases in which the edits selected by BiasReducer-M include the oracle edit; recall and precision compare the selected set with the beneficial set. If four of seven edits were selected uniformly at random, expected oracle coverage would be

4/7=57\%

. The benchmark-gain columns report the BiasReducer-M change in pairwise preference accuracy (pp relative to the original reward model). Rank-sum is the selection rule used by BiasReducer . Bold marks the best result in each column; higher is better.

Qwen3-1.7B
Pairs / subset	BiasReducer-S ( $\Delta$ )	BiasReducer-M ( $\Delta$ )	Agree w/ full	BiasReducer-S sd
25	$+1.25$	$+1.71$	$0.60$	$0.89$
50	$+1.01$	$+1.09$	$0.71$	$1.07$
100	$+1.01$	$+1.84$	$0.86$	$0.75$
200	$+0.88$	$+1.64$	$0.91$	$0.29$
Full	$+1.29$	$+1.72$	$1.00$	$0.00$

Appendix

Table 20: Sensitivity to edit-selection pool size on JudgeBiasBench. For each JBB subset, we randomly sample

n

response pairs for edit selection and repeat the sampling five times. For each draw, we recompute the edit-selection signals and obtain the BiasReducer-S and BiasReducer-M selections without using preference labels, then evaluate them on the same fixed full evaluation set. Full uses the complete response pool for edit selection.

\Delta

denotes the change in JBB pairwise preference accuracy (pp relative to the original reward model). Agree w/ full measures how often BiasReducer-S selects the same attribute as under full-pool selection, averaged over the seven JBB subsets. BiasReducer-S sd reports the variation in gain across the five draws.

Qwen3-1.7B
$k_{\max}$	RMB	Arena	JBB
1	$+4.42$	$+16.35$	$+1.29$
2	$+5.78$	$+19.60$	$+1.52$
3	$+6.17$	$+20.80$	$+1.59$
4 ∗	$+6.96$	$+16.75$	$+1.72$
6	$+7.28$	$+13.00$	$+0.90$

Appendix

Table 21: Sensitivity to the edit-complexity cap

k_{\max}

. We vary the maximum number of edits that BiasReducer-M can compose;

k_{\max}{=}1

corresponds to BiasReducer-S . Entries report changes in pairwise preference accuracy (pp relative to the original reward model) on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB).

k_{\max}{=}4

is the default setting. Positive is better.

Qwen3-1.7B (default $K{=}128$ )
	BiasReducer-S			BiasReducer-M
Configuration	RMB	Arena	JBB	RMB	Arena	JBB
Default ∗	$+4.42$	$+16.35$	$+1.29$	$+6.96$	$+16.75$	$+1.72$
$K/2$	$+2.40$	$+5.80$	$+3.03$	$+4.62$	$+24.25$	$+1.95$
$k_{\mathrm{act}}{=}K/4$	$+2.83$	$+11.90$	$+1.48$	$+5.31$	$+13.45$	$+2.66$
$k_{\mathrm{act}}{=}3K/4$	$+4.50$	$+2.50$	$+0.43$	$+5.22$	$+13.15$	$-0.84$

Appendix

Table 22: Sensitivity to dictionary size and number of active coordinates. We vary the dictionary size

K

and active-coordinate count

k_{\mathrm{act}}

around the default configuration (

K=\max(64,\operatorname{round}(d/16))

and

k_{\mathrm{act}}=K/2

). The

k_{\mathrm{act}}

variants keep the default dictionary size fixed. For each configuration, we rerun the full pipeline, including representation learning, edit-direction and edit-amount estimation, and dataset-specific edit selection, using seeds 0 and 1. Entries report changes in pairwise preference accuracy (pp relative to the original reward model) on RM-Bench-Hard (RMB), Arena-StyleConflict (Arena), and JudgeBiasBench (JBB). ∗ denotes the default configuration. Positive is better.

Qwen3-4B
	$N=16$			$N=64$			$N=256$
Reward head	Qual. $\uparrow$	Verb. $\downarrow$	Words	Qual. $\uparrow$	Verb. $\downarrow$	Words	Qual. $\uparrow$	Verb. $\downarrow$	Words
Original	$4.84$	$0.65$	$246$	$4.82$	$0.64$	$253$	$4.96$	$0.65$	$265$
BiasReducer-S	$4.93$	$0.56$	$226$	$4.86$	$0.57$	$227$	$5.10$	$0.59$	$238$
BiasReducer-M	$4.97$	$0.55$	$221$	$4.85$	$0.55$	$220$	$4.97$	$0.59$	$228$
Llama-3.2-3B-Instruct

Appendix

Table 23: Detailed best-of-

N

results corresponding to Figure 2 . We use Mistral-7B as the reward model on Arena and evaluate Qwen3-4B and Llama-3.2-3B-Instruct as policies. Original, BiasReducer-S , and BiasReducer-M score the same fixed candidate pools for

N\in\{16,64,256\}

. Quality is judged on a 1–10 scale, verbosity is the fraction of responses judged unnecessarily verbose, and Words is the mean response length.

Category	Setting	Value
Representation	Dictionary size	$K=\max(64,\operatorname{round}(d/16))$
	Active coordinates	$k_{\mathrm{act}}=K/2$
Optimization	Optimizer	Adam
	Learning rate	$10^{-3}$ (constant)
	Source batch size	512 preference contrasts
	Intervention batch size	41 deltas

Appendix

Table 24: Default implementation settings and hyperparameters.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Feb 16, 2026

Automatically Finding Reward Model Biases

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.

Atticus Wang, Iván Arcuschin, Arthur Conmy

MATS · Massachusetts Institute of Technology · Independent

Reward Modeling LLM Auditing

Ask PDF

Apr 30, 2026

Debiasing Reward Models via Causally Motivated Inference-Time Intervention

Reward models (RMs) play a central role in aligning large language models (LLMs) with human preferences. However, RMs are often sensitive to spurious features such as response length. Existing inference-time approaches for mitigating these biases typically focus exclusively on response length, resulting in performance trade-offs. In this paper, we propose causally motivated intervention for mitigating multiple types of biases in RMs at inference time. Our method first identifies neurons whose activations are strongly correlated with predefined bias attributes, and applies neuron-level intervention that suppresses these signals. We evaluate our method on RM benchmarks and observe reductions in sensitivity to spurious features across diverse bias types, without inducing performance trade-offs. Moreover, when used for preference annotation, small RMs (2B and 7B) with our method, which edits less than 2% of all the neurons in RMs, enable LLMs to improve alignment, achieving performance comparable to that of a state-of-the-art 70B RM on AlpacaEval and MT-Bench. Further analysis reveals that bias signals are primarily encoded by neurons in early layers, shedding light on the internal mechanisms of bias exploitation in RMs.

Kazutoshi Shinoda, Kosuke Nishida, Kyosuke Nishida

Human Informatics Labs., NTT, Inc.

Reward Modeling LLM Alignment Debiased ML Language Model Bias Evaluation Causal Interventions in Language Models

Ask PDF

Apr 15, 2026

Robust Reward Modeling for Large Language Models via Causal Decomposition

Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.

Yunsheng Lu, Zijiang Yang, Licheng Pan +1

College of Computer Science and Technology, Zhejiang University · Department of Statistics, University of Chicago

Reward Modeling LLM Alignment Language Model Robustness

Ask PDF

Abstract

Reward Modeling

Explore similar work

Automatically Finding Reward Model Biases

Debiasing Reward Models via Causally Motivated Inference-Time Intervention

Robust Reward Modeling for Large Language Models via Causal Decomposition