Reward models are central to aligning language models with human preferences via reinforcement learning (RL). As RL is increasingly applied to settings such as verifiable rewards and multi-objective alignment, RMs are expected to encode more complex and multifaceted preference distributions. However, classifier RMs remain static once trained, limiting their adaptability at test time. We propose Variational In-Context Reward Modeling (ICRM), a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations. ICRM casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior. We show that ICRM adapts to unseen preference distributions at test time for both single and multi-objective settings. With more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to 70.8, achieves lower calibration error than a generative judge on moral dilemma preferences, and expands the attainable Pareto frontier under conflicting preferences. We further study the practical applicability of ICRM for RL training, showing that it can effectively encode verifiable rewards by outperforming a conventional RM in math reasoning. Finally, we provide theoretical guarantees that the variational objective admits a global interior optimum with finite confidence, followed by an ablation study on its role for the test-time steerability, and we analyze how KL regularization mitigates reward over-optimization.
Figures & tables
Figure 1: Variational in-context reward modeling ( ICRM ) with Beta prior for the Bradley-Terry (BT) model. ICRM directly encodes the mean and sharpness of the Beta posterior given in-context preference demonstrations, allowing multi-objective test-time steerability for any unseen preferences.
Figure 2: Calibrated steerability. Calibration analysis with ECE ( ↓ ) and RMSE ( ↓ ) between the flipping ratio of demonstrations and the preference accuracy on moral dilemma preference pairs.
Figure 3: Directional Steerability. The mean accuracy and standard deviation of ICRM by number of in-context demonstrations from a held-out set on HHH-Alignment and SafeRLHF.
RM-Bench
Chat
Safety
Code
Math
Avg.
Generative
LLM-as-a-Judge
69.2
90.9
54.3
67.6
70.3
GRAM-r 2
61.1
92.9
54.7
61.6
67.6
Classifier
BTRM
69.3
96.0
53.2
62.1
70.2
Table 1: RM-Bench evaluation results with the standard deviation of four random seeds for ICRM.
Figure 4: Multi-objective steerability. Pareto frontiers of ICRM trained on Llama-3.2-3B (Figure 4(a) ) and Qwen3-4B (Figure 4(b) ), and the Hypervolume (HV) of the Pareto frontiers (Figure 4(c) ).
PKU-SafeRLHF
RM-Bench
Variant
N
ECE ↓
RMSE ↓
HV ↑
HV ↑
IC-BT
4
0.25
0.29
0.33
0.89
ICRM, no KL
4
0.28
0.34
0.29
0.84
ICRM, fixed KL
4
0.20
0.22
0.38
0.96
ICRM, weighted KL
4
0.24
0.25
0.38
0.94
IC-BT
16
0.27
0.31
0.34
0.93
Table 2: Objective ablation with Qwen3-4B-Base. Best and second-best rounded values within each N are bold and underlined, respectively.
Figure 5: Accuracy mean (“Accuracy (%)”) and average rewards (“Training Reward”) of three reward models (Figure 5(a) ) and performance of the training policy with four reward models (Figure 5(b) ).
Pearson r
ROLS2
RIso2
MATH500
Qwen3-1.7B (Base)
-
-
37.2%
Verifier
-
-
-
50.4%
Bradley-Terry
0.663
0.439
0.428
51.4%
ICRM ( λ=0.5 )
0.685
0.469
0.461
50.2%
ICRM ( λ=0.1 )
0.691
0.477
0.459
53.8%
Table 3: Correlation analysis for the alignment between RM and exact match accuracy on RLVR.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training dynamics of ICRM with different KL regularization coefficients.
Model
N=0
N=1
N=4
N=8
N=64
BTRM
70.20
69.60
68.00
67.80
67.80
GRM
68.20
66.20
65.80
65.70
64.90
ICRM
–
60.50
65.70
67.60
70.80
Appendix
Table 4: Mean RM-Bench accuracy (%) across four evaluation seeds. Best and second-best values within each N are bold and underlined, respectively.
Figure 7: Extended calibrated steerability analysis across different base models and λ choices in Section 5.1 .
Hyperparameter
Value
Number of Rollouts ( n )
8
Number of Unique Prompts Per Batch ( m )
64
Learning Rate
10−6
Learning Rate Scheduler
Constant
KL penalty ( β )
0.0
Appendix
Table 5: Hyperparameters for GRPO training in Section 6 .
Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) with human preferences, yet most pipelines learn a single reward model that overlooks individual differences in preferences. Personalized reward models (PRMs) address this by conditioning rewards on user-specific feedback, most commonly through in-context learning (ICL), where a user's historical comparisons are supplied as contextual preference pairs. However, we identify a key limitation of ICL-based PRMs: they fail to capture the preference relations conveyed by contextual pairs. To address this, we propose Preference-Aligned Test-Time Training (P-TTT), which explicitly encodes these relations into user-specific fast weights for personalized reward prediction. P-TTT introduces sequence-level update and apply operations to match the response-level granularity of preference feedback, together with a preference-aligned objective that directly uses pairwise preference relations to guide fast-weight adaptation. Notably, P-TTT is simple to implement and computationally efficient, updating fast weights within a single forward pass without inference-time backpropagation. Extensive experiments show that P-TTT more effectively captures historical preference relations and outperforms state-of-the-art methods by a large margin.
Bohao Wang, Xiaoyan Zhao, Yang Zhang +4
Zhejiang University · National University of Singapore
Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.
Reward models play a fundamental role in aligning large language models with human preferences. Existing methods predominantly follow two paradigms: scalar discriminative preference models, which are efficient but lack interpretability, and generative judging models, which offer richer reasoning at the cost of higher computational overhead. We observe that the log-probability margin between verdict tokens strongly correlates with prediction correctness, providing a reliable proxy for instance difficulty without additional inference cost. Building on this insight, we propose CAMEL, a confidence-gated reflection framework that performs a lightweight single-token preference decision first and selectively invokes reflection only for low-confidence instances. To induce effective self-correction, we train the model via reinforcement learning with counterfactual prefix augmentation, which exposes the model to diverse initial verdicts and encourages genuine revision. Empirically, CAMEL achieves state-of-the-art performance on three widely used reward-model benchmarks with 82.9% average accuracy, surpassing the best prior model by 3.2% and outperforming 70B-parameter models using only 14B parameters, while establishing a strictly better accuracy-efficiency Pareto frontier.