Learning Ordinal Probabilistic Reward from Preferences

Longze Chen; Qi Deng; Lu Wang; Renke Shan; Ze Gong; Run Luo; Jiaming Li; Jing Luo

cs.CLFeb 13, 2026

Learning Ordinal Probabilistic Reward from Preferences

Authors: Longze Chen, Qi Deng, Lu Wang, Renke Shan, Ze Gong, Run Luo, Jiaming Li, Jing Luo,

Organizations: Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Ritzz-AI · Shenzhen University of Advanced Technology

Abstract

Reward models are crucial for aligning large language models (LLMs) with human values and intentions. Existing approaches follow either Generative (GRMs) or Discriminative (DRMs) paradigms, yet both suffer from limitations: GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation. To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response. To make this paradigm practical, we present its closed-form, discrete realization: the Ordinal Probabilistic Reward Model (OPRM), which discretizes the quality score into a finite set of ordinal ratings. Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT). It enables rewards to better reflect absolute text quality by incorporating quality-level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions. Experiments on various reward model benchmarks show that our method improves accuracy by $\textbf{2.9%}\sim\textbf{7.4%}$ compared to prior reward models, demonstrating strong performance and data efficiency. Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.

Figures & tables

Baselines	Input Format	Output Format	Margin Sensitivity	Require Training	Value Head Free	Input Flexible
Bradley-Terry ( Bradley & Terry, 1952 )	Single Response	Continuous Score	✓	✓	✗	✓
PairRM ( Jiang et al., 2023 )	Response Pairs	Continuous Score	✓	✓	✗	✗
CLoud ( Ankner et al., 2024 )	Single Response	Critique + Continuous Score	✓	✓	✗	✓
LLM-as-a-Judge ( Zheng et al., 2023 )	Response Pairs	Discrete Score	✗	✗	✓	✗
Pointwise GRM ( Liu et al., 2025b )	Single Response	Critique + Discrete Score	✗	✓	✓	✓
OPRM (Ours)	Single Response	Continuous Score	✓	✓	✓	✓

Table 1: Comparison of OPRM with baseline reward models across multiple dimensions . Margin Sensitivity (whether distinguish samples with subtle preference differences), Require Training (whether requires training on preference data), Value Head Free (whether eliminates the need for additional value head), and Input Flexibility (whether supports rating single and multiple responses).

Figure 1: The architectures of Ordinal Probabilistic Reward Model . Given a problem and a pair of responses, designated as chosen and rejected , the OPRM utilizes its language model (LM) head to obtain the ordinal rating probabilities for each response. A joint probability matrix is then constructed by computing the Cartesian product of these two sets of probabilities for optimization.

Figure 2: Region Flooding Tuning . To ensure the correctness of the reward modeling, region flooding is applied to each of the three partition combinations, resulting in a lower triangular form.

Model	Reward Bench	PPE-P	PPE-C	RMB	Overall	$\text{Overall}^{*}$
Reported Results of Public Models
Skywork-Reward-Gemma-2-27B	93.8	56.6	56.6	60.2	66.8	57.8
DeepSeek-V2.5-0905	81.5	62.8	58.5	65.7	67.1	62.3
Gemini-1.5-Pro	86.8	66.1	59.8	56.5	67.3	60.8
ArmoRM-8B-v0.1	90.4	60.6	61.2	64.6	69.2	62.1
InternLM2-20B-Reward	90.2	61.0	63.0	62.9	69.3	62.3

Table 2: Overall results of different methods and models on four RM benchmarks. bold numbers indicate the best performance. Underlined numbers indicate the second best. The Overall* score is the average performance excluding Reward Bench due to its known data contamination issues.

Model	MMLU-Pro	MATH	GPQA	MBPP-Plus	IFEval	PPE Correctness
Results of Our Method
OPRM-Qwen2.5-7B	65.2	70.1	56.3	59.0	56.1	61.3
OPRM-Qwen2.5-14B	66.7	70.7	57.1	67.4	59.5	64.3
OPRM-Qwen2.5-32B	71.2	73.2	57.9	66.2	62.2	66.1
OPRM-Qwen2.5-72B	73.4	75.9	58.6	54.1	59.5	64.3
Results of Our Method (w/ Region Flooding Tuning)

Table 3: Detailed results of different methods on the PPE Correctness benchmark. Bold numbers indicate the best performance. Underlined numbers indicate the second best.

Method	Pair-Accuracy	Best-of-N	Best-of-N-plus	Worst-of-N	Overall
Random Baseline	50.0	25.5	31.3	68.7	43.9
Training on Role Play Data Only
BT Model	70.4	48.6	51.3	83.6	63.5
BT Model - w/ Margin	71.0	49.3	52.3	84.2	64.2
OPRM (ours)	71.3	49.4	52.5	84.1	64.3
OPRM-RgFT (ours)	72.1( $\uparrow$ 0.8 )	50.7( $\uparrow$ 1.3 )	53.6( $\uparrow$ 1.1 )	85.1( $\uparrow$ 1.0 )	65.4( $\uparrow$ 1.1 )

Table 4: Detailed results of Qwen2.5-7B with different methods on the Role Play benchmark. Bold numbers indicate the best performance. Underlined numbers indicate the second best.

Model	Acc (%)	ECE-10 (%, $\downarrow$ )
Qwen-2.5-32B	69.56	26.72
OPRM-32B	81.04	10.62
OPRM-RgFT-32B	90.90	5.18

Table 5: Comparison of accuracy and calibration (ECE-10) on the RewardBench.

Figure 3: Comparison of score distributions for responses of high-quality ( Top ) and low-quality ( Bottom ).

Figure 4: Ablation Study : (a) Assessing the superiority of OPRM over the BT Model. (b) Evaluating the efficacy of Weighted Average Decoding. (c) Validating the necessity of Region Flooding.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	RewardBench	PPE-P	PPE-C	RMB	Overall
Irregular Variant (Scale $0\dots 9$ )
bad $\{0,1,2,3\}$ , normal $\{4,5\}$ , good $\{6,7,8,9\}$	89.1	64.1	67.7	74.4	73.8
Default Uniform (Scale $1\dots 9$ )
bad $\{1,2,3\}$ , normal $\{4,5,6\}$ , good $\{7,8,9\}$	88.9	64.6	67.3	74.8	73.9

Appendix

Table 6: Performance comparison between the default uniform configuration and an irregular boundary variant. The results indicate that the method is highly robust to binning strategies.

Method	RewardBench	PPE-P	PPE-C	RMB	Overall
Baseline A (Hard Classification)	71.0	43.9	46.6	55.7	54.3
Baseline B (Scalar-Weighted Classification)	85.4	61.5	64.3	71.3	70.6
OPRM-RgFT-32B (Ours)	88.9	64.6	67.3	74.8	73.9

Appendix

Table 7: Comparative analysis of OPRM-RgFT against standard classification baselines utilizing identical supervision signals. The results demonstrate that the proposed region flooding mechanism yields significant gains over pure classification approaches.

Method	Chat	Chat Hard	Safety	Reasoning	Reward Bench
Reported Results of Public Models
Skywork-Reward-Gemma-2-27B	95.8	91.4	91.9	96.1	93.8
DeepSeek-V2.5-0905	-	-	-	-	81.5
Gemini-1.5-Pro	94.1	77.0	85.8	90.2	86.8
ArmoRM-8B-v0.1	96.9	76.8	90.5	97.3	90.4
InternLM2-20B-Reward	98.9	76.5	89.5	95.8	90.2

Appendix

Table 8: Detailed results of different methods on the Reward Bench benchmark.

Method	MMLU-Pro	MATH	GPQA	MBPP-Plus	IFEval	PPE Correctness
Reported Results of Public Models
Skywork-Reward-Gemma-2-27B	54.0	63.0	53.0	59.0	54.0	56.6
DeepSeek-V2.5-0905	-	-	-	-	-	58.5
Gemini-1.5-Pro	-	-	-	-	-	59.8
ArmoRM-8B-v0.1	66.0	71.0	57.0	54.0	58.0	61.2
InternLM2-20B-Reward	68.0	70.0	57.0	58.0	62.0	63.0

Appendix

Table 9: Detailed results of different methods on the PPE Correctness benchmark.

Method	Helpfulness BoN	Helpfulness Pair	Harmlessness BoN	Harmlessness Pair	RMB
Reported Results of Public Models
Skywork-Reward-Gemma-2-27B	47.2	65.3	56.1	72.1	60.2
DeepSeek-V2.5-0905	-	-	-	-	65.7
Gemini-1.5-Pro	53.6	76.3	29.9	66.1	56.5
ArmoRM-8B-v0.1	63.6	78.7	49.7	66.3	64.6
InternLM2-20B-Reward	58.5	76.3	49.9	67.0	62.9

Appendix

Table 10: Detailed results of different methods on the RMB benchmark.

Figure 5: Annotation Region Flooding Tuning . As annotation ambiguity increases, the target optimization region "floods" to encompass a wider set of plausible outcomes. A more uncertain annotation results in a larger target region than a more certain one.

Method	Training Time	Inference Time	Hardware (Train)	Hardware (Infer)
BTRM (Standard DRM)	7.3h	5.5 min	$32\times\text{H200}$	$4\times\text{H200}$
OPRM (Ours)	6.9h	2.1 min	$32\times\text{H200}$	$4\times\text{H200}$

Appendix

Table 11: Computational efficiency comparison between BTRM and OPRM.

Example

Question : You are a highly skilled mathematician tasked with solving complex math problems. Your goal is to provide clear, step-by-step solutions that can be easily parsed and evaluated. Here is the math problem you need to solve: <problem> The length of a rectangle is twice its width. Given the length of the diagonal is

5\sqrt{5}

, find the area of the rectangle. </problem> Box your final answer using LaTeX, for example:

x = \boxed{[Your final numerical or algebraic answer]}

. Now, please solve the given math problem and provide your solution in the specified format.

Chosen Response : Here’s my solution to the problem: Let the width of the rectangle be w . Since the length is twice the width, the length of the rectangle is 2 w . We are given that the length of the diagonal is

5\ \sqrt{5}

. Using the Pythagorean theorem, we can write:

\left(5\sqrt{5}\right)^{2}=w^{2}+\left(2w\right)^{2}

Simplifying the equation, we get:

25\cdot 5=w^{2}+4w^{2}

Combine like terms:

25\cdot 5=5w^{2}

Divide both sides by

5

25=w^{2}

Take the square root of both sides:

w=\sqrt{25}=5

Now that we have the width, we can find the length:

2w=2\cdot 5=10

The area of the rectangle is the product of the width and length:

A=w\cdot 2w=5\cdot 10=\boxed{50}

Therefore, the area of the rectangle is

50

Rejected Response : To solve this problem, we can start by using the Pythagorean theorem to relate the length and width of the rectangle to the length of the diagonal:

d^{2}=l^{2}+w^{2}

where

d

is the length of the diagonal,

l

is the length of the rectangle, and

w

is the width of the rectangle. Since the length of the rectangle is twice its width, we can write:

l=2w

Substituting this into the Pythagorean theorem, we get:

d^{2}=(2w)^{2}+w^{2}

Simplifying this expression, we get:

d^{2}=5w^{2}

Since we are given that the length of the diagonal is

5\sqrt{5}

, we can set up the equation:

d=5\sqrt{5}

Substituting this into the equation above, we get:

5\sqrt{5}^{2}=5w^{2}

Simplifying this expression, we get:

25\cdot 5=5w^{2}

Simplifying further, we get:

125=w^{2}

Taking the square root of both sides, we get:

w=\sqrt{125}=5\sqrt{5}

Now that we have found the width, we can find the length by substituting this value back into the equation:

l=2w=2(5\sqrt{5})=10\sqrt{5}

The area of the rectangle is given by:

A=lw=(10\sqrt{5})(5\sqrt{5})=50\sqrt{5}^{2}=50\cdot 5=\boxed{250}

Final Answer: The final answer is

250

BTRM-Qwen2.5-32B

Chosen Score: 0.03

Rejected Score: -2.70

Appendix

Table 12: Case 1

Example

Question : is

(x^{y})^{z}=(x^{z})^{y}

Chosen Response : Yes, the equation

(x^{y})^{z}=(x^{z})^{y}

is indeed true, and it is a direct consequence of the properties of exponents in mathematics. To see why this is true, let’s consider the equation on both sides: Left side:

(x^{y})^{z}

Right side:

(x^{z})^{y}

First, we can simplify the left side by applying the power of a power rule, which states that

(a^{m})^{n}=a^{(mn)}

: Left side:

(x^{y})^{z}=x^{(yz)}

Now, let’s simplify the right side using the same power of a power rule: Right side:

(x^{z})^{y}=x^{(zy)}

Now we can see that both sides of the equation are equal:

x^{(yz)}=x^{(zy)}

Since the bases

(x)

are the same and the exponents

(yz)

and

(zy)

are also the same, we can conclude that the equation

(x^{y})^{z}=(x^{z})^{y}

is true. This property holds for any real numbers

x

y

, and

z

, as long as

x\neq 0

and

y

and

z

are non-zero.

Rejected Response : A classic question about exponentiation! Unfortunately, the answer is no,

(x^{y})^{z}

is not necessarily equal to

(x^{z})^{y}

in general. To see why, let’s break it down:

(x^{y})^{z}

means “raise

x

to the power of

y

, and then raise the result to the power of

z

”. This can be written as:

(x^{y})^{z}=x^{(y*z)}

On the other hand,

(x^{z})^{y}

means “raise

x

to the power of

z

, and then raise the result to the power of

y

”. This can be written as:

(x^{z})^{y}=x^{(z*y)}

As you can see, the two expressions are not equal in general, because

y*z

is not necessarily equal to

z*y

. For example, if

x=2

y=3

, and

z=4

, then:

(2^{3})^{4}=2^{(3*4)}=2^{12}=4096

while

(2^{4})^{3}=2^{(4*3)}=2^{12}=4096

But in this case, it just so happens that

y*z=z*y

, so the two expressions are equal. However, this is not always the case. For example, if

x=2

y=2

, and

z=3

, then:

(2^{2})^{3}=2^{(2*3)}=2^{6}=64

while

(2^{3})^{2}=2^{(3*2)}=2^{6}=64

Again, it just so happens that the two expressions are equal in this case, but this is not a general rule. So, to summarize:

(x^{y})^{z}

is not necessarily equal to

(x^{z})^{y}

, unless

y

and

z

commute, i.e.,

y*z=z*y

BTRM-Qwen2.5-32B

Chosen Score: -0.96

Rejected Score: -0.39

Appendix

Table 13: Case 2

Example

Question : Anita ___ some application letters to the company but they never call her. A.has send B.have sent C.have send D.have sending E.has sent

Chosen Response : The correct answer is E. "has sent". Explanation: In this sentence, we need to use the present perfect tense, which is formed by "has/have + past participle". The past participle of the verb "send" is "sent". A. "has send" is incorrect because "send" is the base form of the verb, not the past participle. B. "have sent" is incorrect because the subject "Anita" is singular, so we should use "has" instead of "have". C. "have send" is incorrect for two reasons: "have" is used with a plural subject, and "send" is not the past participle. D. "have sending" is incorrect because "sending" is the present participle, not the past participle. E. "has sent" is the correct form, using "has" for the singular subject "Anita" and the past participle "sent". Therefore, the correct sentence should be: "Anita has sent some application letters to the company, but they never call her."

Rejected Response : The correct option is E. "has sent." The sentence should read: "Anita has sent some application letters to the company but they never call her." The verb "has sent" is used because the subject of the sentence, "Anita," is singular and the action of sending the letters happened in the past. The present perfect tense is used to indicate that the action was completed in the past, but its effects are still being felt in the present. Therefore, "has sent" is the appropriate form of the verb to use in this sentence.

BTRM-Qwen2.5-32B

Chosen Score: 0.24

Rejected Score: -1.12

Appendix

Table 14: Case 3

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Mar 5, 2026

VRM: Teaching Reward Models to Understand Authentic Human Preferences

Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.

Biao Liu, Ning Xu, Junming Yang +2

Southeast University, Nanjing, China

Reward Modeling LLM Alignment Human Preference Modeling

Ask PDF

Jul 31, 2026

Learning Latent Reasoning Traces for Scalar Reward Models End-to-End

Reward models (RMs) are central to aligning large language models with human preferences via reinforcement learning. Although traditional scalar RMs enable efficient and probabilistic reward modeling, they rely on superficial cues that fail to generalize to complex or out-of-distribution (OOD) tasks. Conversely, generative RMs leverage extensive reasoning to improve robustness on challenging tasks, but their natural language-based scores lack the numerical flexibility and probabilistic interpretability that scalar RMs offer. While recent approaches combine both paradigms through off-policy multi-task learning, such parallel optimization does not guarantee that generated reasoning traces actively align with or benefit downstream scalar reward prediction. To address this mismatch, we propose LatentRM, a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize the likelihood of downstream scalar rewards. Through on-policy optimization of the latent reasoning space end-to-end, LatentRM tightly couples deep reasoning-based evaluation with precise scoring. Extensive validations on in-distribution and OOD datasets and RLHF show that LatentRM outperforms scalar, generative, and hybrid RMs on preference modeling and policy alignment across tasks ranging from open-ended conversation to complex reasoning.

Sanwoo Lee, Clive Bai, Hsiu-Yuan Huang +3

National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent

Reward Modeling LLM Alignment Latent Variable Models Process Reward Models Preference-Based RL

Ask PDF

Feb 27, 2026

RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models

Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq.

Daniel Yang, Samuel Stante, Florian Redhardt +5

ETH Zurich · ETH AI Center

Reward Modeling Reward Model Evaluation

Ask PDF

Abstract

Reward Modeling

Explore similar work

VRM: Teaching Reward Models to Understand Authentic Human Preferences

Learning Latent Reasoning Traces for Scalar Reward Models End-to-End

RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models