Reward models (RMs) are critical components of alignment pipelines, yet they exhibit biases toward superficial stylistic cues, preferring better-presented responses over semantically superior ones. Existing debiasing methods typically require retraining or architectural modifications, while direct activation suppression degrades performance due to representation entanglement. We propose SteerRM, the first training-free method for debiasing reward models using Sparse Autoencoder (SAE)-based interventions. SteerRM isolates stylistic effects using contrastive paired responses, identifies bias-related SAE features with a strength-stability criterion, and suppresses them at inference time. Across six reward models on RM-Bench, SteerRM improves Hard-split accuracy by 7.3 points on average while preserving overall performance. Results on a Gemma-based reward model and a controlled non-format bias further suggest generalization across RM architectures and bias types. We further find that format-related features are concentrated in shallow layers and transfer across models, revealing shared architecture-level bias encoding patterns. These results show that SAE-based interventions can mitigate reward-model biases without retraining, providing a practical and interpretable solution for alignment pipelines.
Figures & tables
Figure 1: Overview of SteerRM. Our training-free pipeline consists of three stages: (1) synthesizing paired responses with different surface formats or styles, (2) identifying format-related SAE features, and (3) suppressing these features at inference time.
Figure 2: SAE reconstruction quality across layers. (Left) Reconstruction MSE. (Middle) Absolute reward difference between original and reconstructed scores. (Right) L0 sparsity measured as the mean number of active features per sample. Layers 0-9 are selected based on these metrics.
Figure 3: Distribution of top-100 candidate format-related SAE features across layers. Format-related features are concentrated in early layers (1-3), revealing that formatting information is encoded at shallow layers of the Transformer.
Table 1: Main results on RM-Bench. We compare baseline reward models, direct activation suppression (Activation), and SAE-based feature intervention (SteerRM). SteerRM consistently improves performance on the Hard split while preserving Normal and Average accuracy, whereas Activation improves Hard performance at the cost of substantial degradation on other splits.
Method
Easy
Normal
Hard
Average
Baseline
87.6
68.2
39.1
65.0
SteerRM
83.3
68.1
44.8
65.4
Delta
-4.3
-0.1
+5.7
+0.4
Table 2: Cross-architecture evaluation on a Gemma-based reward model. We apply SteerRM to Ray2333/GRM-Gemma2-2B-sftreg using Gemma Scope SAEs. Delta denotes SteerRM minus Baseline.
Method
Test Set
Accuracy
Rel. Change
Baseline
Original
65.5
–
Baseline
Politeness-Injected
61.0
-6.9%
SteerRM
Politeness-Injected
66.0
+0.8%
Table 3: Generalization to politeness bias on Skywork. Politeness-Injected denotes the adversarial variant with minor politeness markers added to rejected responses. Relative change is computed against the baseline accuracy on the original clean set.
Target
Method
Easy
Normal
Hard
Average
QRM 4 4 footnotemark: 4
Baseline
87.8
73.4
47.5
69.5
SteerRM
75.9
70.0
56.1
67.3
URM 5 5 footnotemark: 5
Baseline
83.9
73.7
53.7
70.5
SteerRM
72.7
72.6
67.9
71.1
Base-RM 6 6 footnotemark: 6
Baseline
85.5
73.3
51.0
69.9
SteerRM
72.4
70.1
52.3
64.9
Table 4: Feature transferability across models. Format-related SAE features identified on Skywork 3 are applied to other reward models. We report Baseline and SteerRM accuracies on the Easy, Normal, and Hard splits, as well as the Average score. SteerRM consistently improves performance on the Hard split across target models, indicating effective cross-model transfer of format-related features.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Comparison of SAE reconstruction metrics on the Base Model (Llama-3.1-8B-Base). While both variants achieve comparable MSE, the Residual Stream SAEs (LXR) demonstrate more stable and lower L0 sparsity compared to the MLP Output SAEs (LXM), indicating a more efficient sparse representation.
Figure 5: Reconstruction quality for SAEs trained on the MLP output (LXM) when evaluated on the Reward Model . Compared to the Residual Stream SAEs (Figure 2 ), these models show higher instability in reward preservation (Reward Delta) and sparsity patterns across layers.
Figure 6: Layer-wise distribution of selected format-related features across four additional models.
Synthetic Pairs ( N )
Top-10 Overlap
Easy
Normal
Hard
Average
50
30%
80.8 (-8.1)
70.3 (-4.6)
48.1 (+0.8)
66.4 (-3.9)
100
60%
88.0 (-0.9)
75.1 (+0.2)
48.7 (+1.4)
70.6 (+0.3)
500 (default)
100%
83.9 (-5.0)
74.1 (-0.8)
56.5 (+9.2)
71.5 (+1.2)
1000
90%
84.2 (-4.7)
74.0 (-0.9)
54.1 (+6.8)
70.8 (+0.5)
Appendix
Table 5: Sensitivity to the number of synthetic probing pairs. Overlap is measured against the top-10 features identified with the default N=500 configuration. Numbers in parentheses show changes relative to the baseline model.
Method
Data Source
Easy
Normal
Hard
Average
Baseline
–
88.9
74.9
47.3
70.3
SteerRM (Synth)
Synthesized Pairs
83.9 (-5.0)
74.1 (-0.8)
56.5 (+9.2)
71.5 (+1.2)
SteerRM (Existing)
Existing Samples
82.0 (-6.9)
74.4 (-0.5)
60.5 (+13.2)
72.3 (+2.0)
Appendix
Table 6: Feature identification from synthesized versus existing samples. Numbers in parentheses show changes relative to the baseline model.
Figure 7: Performance sensitivity to the number of suppressed features ( K ). The plot shows accuracy across Easy, Normal, Hard difficulty levels and the overall average on RM-Bench.