Reward models are central to aligning language models with human preferences via reinforcement learning (RL). As RL is increasingly applied to settings such as verifiable rewards and multi-objective alignment, RMs are expected to encode more complex and multifaceted preference distributions. However, classifier RMs remain static once trained, limiting their adaptability at test time. We propose Variational In-Context Reward Modeling (ICRM), a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations. ICRM casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior. We show that ICRM adapts to unseen preference distributions at test time for both single and multi-objective settings. With more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to 70.8, achieves lower calibration error than a generative judge on moral dilemma preferences, and expands the attainable Pareto frontier under conflicting preferences. We further study the practical applicability of ICRM for RL training, showing that it can effectively encode verifiable rewards by outperforming a conventional RM in math reasoning. Finally, we provide theoretical guarantees that the variational objective admits a global interior optimum with finite confidence, followed by an ablation study on its role for the test-time steerability, and we analyze how KL regularization mitigates reward over-optimization.
Figures & tables
Figure 1: Variational in-context reward modeling ( ICRM ) with Beta prior for the Bradley-Terry (BT) model. ICRM directly encodes the mean and sharpness of the Beta posterior given in-context preference demonstrations, allowing multi-objective test-time steerability for any unseen preferences.
Figure 2: Calibrated steerability. Calibration analysis with ECE ( ↓ ) and RMSE ( ↓ ) between the flipping ratio of demonstrations and the preference accuracy on moral dilemma preference pairs.
Figure 3: Directional Steerability. The mean accuracy and standard deviation of ICRM by number of in-context demonstrations from a held-out set on HHH-Alignment and SafeRLHF.
RM-Bench
Chat
Safety
Code
Math
Avg.
Generative
LLM-as-a-Judge
69.2
90.9
54.3
67.6
70.3
GRAM-r 2
61.1
92.9
54.7
61.6
67.6
Classifier
BTRM
69.3
96.0
53.2
62.1
70.2
Table 1: RM-Bench evaluation results with the standard deviation of four random seeds for ICRM.
Figure 4: Multi-objective steerability. Pareto frontiers of ICRM trained on Llama-3.2-3B (Figure 4(a) ) and Qwen3-4B (Figure 4(b) ), and the Hypervolume (HV) of the Pareto frontiers (Figure 4(c) ).
PKU-SafeRLHF
RM-Bench
Variant
N
ECE ↓
RMSE ↓
HV ↑
HV ↑
IC-BT
4
0.25
0.29
0.33
0.89
ICRM, no KL
4
0.28
0.34
0.29
0.84
ICRM, fixed KL
4
0.20
0.22
0.38
0.96
ICRM, weighted KL
4
0.24
0.25
0.38
0.94
IC-BT
16
0.27
0.31
0.34
0.93
Table 2: Objective ablation with Qwen3-4B-Base. Best and second-best rounded values within each N are bold and underlined, respectively.
Figure 5: Accuracy mean (“Accuracy (%)”) and average rewards (“Training Reward”) of three reward models (Figure 5(a) ) and performance of the training policy with four reward models (Figure 5(b) ).
Pearson r
ROLS2
RIso2
MATH500
Qwen3-1.7B (Base)
-
-
37.2%
Verifier
-
-
-
50.4%
Bradley-Terry
0.663
0.439
0.428
51.4%
ICRM ( λ=0.5 )
0.685
0.469
0.461
50.2%
ICRM ( λ=0.1 )
0.691
0.477
0.459
53.8%
Table 3: Correlation analysis for the alignment between RM and exact match accuracy on RLVR.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training dynamics of ICRM with different KL regularization coefficients.
Model
N=0
N=1
N=4
N=8
N=64
BTRM
70.20
69.60
68.00
67.80
67.80
GRM
68.20
66.20
65.80
65.70
64.90
ICRM
–
60.50
65.70
67.60
70.80
Appendix
Table 4: Mean RM-Bench accuracy (%) across four evaluation seeds. Best and second-best values within each N are bold and underlined, respectively.
Figure 7: Extended calibrated steerability analysis across different base models and λ choices in Section 5.1 .
Hyperparameter
Value
Number of Rollouts ( n )
8
Number of Unique Prompts Per Batch ( m )
64
Learning Rate
10−6
Learning Rate Scheduler
Constant
KL penalty ( β )
0.0
Appendix
Table 5: Hyperparameters for GRPO training in Section 6 .