CAMEL: Confidence-Gated Reflection for Reward Modeling

Zirui Zhu; Hailun Xu; Yang Luo; Yong Liu; Kanchan Sarkar; Kun Xu; Yang You

cs.CLFeb 24, 2026

CAMEL: Confidence-Gated Reflection for Reward Modeling

Authors: Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Kun Xu, Yang You

Organizations: National University of Singapore · TikTok

Abstract

Reward models play a fundamental role in aligning large language models with human preferences. Existing methods predominantly follow two paradigms: scalar discriminative preference models, which are efficient but lack interpretability, and generative judging models, which offer richer reasoning at the cost of higher computational overhead. We observe that the log-probability margin between verdict tokens strongly correlates with prediction correctness, providing a reliable proxy for instance difficulty without additional inference cost. Building on this insight, we propose CAMEL, a confidence-gated reflection framework that performs a lightweight single-token preference decision first and selectively invokes reflection only for low-confidence instances. To induce effective self-correction, we train the model via reinforcement learning with counterfactual prefix augmentation, which exposes the model to diverse initial verdicts and encourages genuine revision. Empirically, CAMEL achieves state-of-the-art performance on three widely used reward-model benchmarks with 82.9% average accuracy, surpassing the best prior model by 3.2% and outperforming 70B-parameter models using only 14B parameters, while establishing a strictly better accuracy-efficiency Pareto frontier.

Figures & tables

Figure 1: Given a query $q$ and two candidate responses $(r_{a},r_{b})$ , a scalar reward model assigns scores to the responses and induces a pairwise preference, whereas a generative reward model produces a textual judgment when outputting the preferred response.

Panel of Figure 2 — Figure 2: Confidence score distribution and its relationship with prediction accuracy on Skywork-Reward-Preference-80K using Qwen3-14B. (a) Distribution of confidence scores for correct and incorrect predictions. Correct predictions exhibit a heavier tail toward higher confidence scores, while incorrect predictions are concentrated in the low-confidence region. (b) Accuracy as a function of confidence score. Each point represents the accuracy within a binned confidence interval, with color intensity indicating sample count. It is clear that predictions with higher confidence scores are substantially more likely to be correct.

Panel of Figure 3 — Figure 3: CAMEL Preference Judgment Prompt.

Models	RewardBench	RM-Bench	JudgeBench	Avg.
ScalarRMs
Llama-3-OffsetBias-RM-8B	89.0	71.3	63.5	74.6
ArmoRM-Llama3-8B-v0.1	90.4	69.3	59.7	73.1
Internlm2-20b-reward	90.2	68.3	64.3	74.3
Skywork-Reward-Llama-3.1-8B-v0.2	93.1	72.1	62.9	76.0
LDL-Reward-Gemma-2-27B-v0.1	95.0	71.1	64.2	76.8

Table 1: Performance comparison of recent reward models across three popular evaluation benchmarks. Bold values denote the best results, and underlined values denote the second-best.

Panel of Figure 4 — Figure 4: Accuracy vs. Average Output Tokens Trade-off on RewardBench/RM-Bench. The Pareto curve illustrates the performance-efficiency trade-off of CAMEL under varying confidence thresholds $\tau$ . CAMEL-Fast uses only token-level confidence without reflection, while CAMEL-Reflection applies full reflection to all samples. By adaptively selecting when to reflect based on confidence, CAMEL achieves superior accuracy with significantly fewer tokens compared to RM-R1 baselines. The color gradient indicates the confidence threshold $\tau$ from low to high.

Models	RewardBench	RM-Bench	JudgeBench	Avg.
Qwen3-14B	81.9	71.1	62.6	71.9
Qwen3-14B + Reflection	83.3	73.2	65.0	73.8
Qwen3-14B-SFT	90.6	72.7	64.8	76.0
Qwen3-14B-SFT + Reflection	90.5	72.7	66.1	76.4
Qwen3-14B-GRPO	91.2	83.5	62.9	79.2
Qwen3-14B-GRPO + Reflection	90.0	84.0	74.2	82.7

Table 2: Ablation of training and inference components on three benchmarks. “+ Reflection” applies reflection to all test inputs. SFT: supervised fine-tuning on preference data. GRPO: one-epoch GRPO on verdicts without counterfactual prefix augmentation (with doubled rollouts for parity). CAMEL combines GRPO with counterfactual prefix augmentation and confidence-gated reflection, yielding competitive overall accuracy and stronger robustness on three benchmarks.

Figure 5: Confidence score distribution before and after CAMEL training.

Panel of Figure 6 — Figure 6: Confusion matrices of fast token-level predictions versus reflection reasoning outcomes on (a) RewardBench and (b) RM-Bench. Rows indicate whether the fast prediction is correct; columns indicate whether the reflection output is correct.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Confidence score vs. accuracy on three training datasets using Qwen3-14B. The three panels correspond to Code-Preference-Pairs, Math-Step-DPO-10K, and Skywork-Reward-Preference-80K from top to bottom. Across all datasets, accuracy increases steadily with the confidence score, indicating that the verdict-token log-probability margin is a reliable proxy for prediction correctness and instance difficulty across diverse training domains.

Panel of Figure 8 — Figure 8: Prompt used for ablation model training.

Figure 9: Confidence score distribution on Skywork-Reward-Preference-80K across six models. Each panel shows, for one base model, the density of confidence scores for correct and incorrect predictions. Correct predictions consistently place more mass in the high-confidence region, whereas errors are concentrated near smaller margins, providing the distributional counterpart to the accuracy trends in Figure 10 .

Figure 10: Accuracy vs. confidence score on Skywork-Reward-Preference-80K across six models. Each panel reports binned prediction accuracy as a function of the confidence score for one base model. The overall increasing trend shows that higher-confidence predictions are consistently more reliable across model families and scales.

Figure 11: Confidence score distribution on Code-Preference-Pairs across six models. Each panel shows the confidence-score densities of correct and incorrect predictions for one base model. Despite the coding-specific nature of the dataset, the same separation appears across all six models, with errors concentrated in the low-confidence region.

Figure 12: Accuracy vs. confidence score on Code-Preference-Pairs across six models. Each panel reports binned prediction accuracy as confidence increases for one base model. The overall increasing trend indicates that the confidence score remains informative even when the judgment depends on implementation correctness and subtle bug patterns.

Figure 13: Confidence score distribution on Math-Step-DPO-10K across six models. Each panel plots the density of confidence scores for correct and incorrect predictions for one base model. Correct predictions are shifted toward larger confidence margins, while incorrect predictions remain concentrated near the low-confidence region.

Figure 14: Accuracy vs. confidence score on Math-Step-DPO-10K across six models. Each panel reports binned prediction accuracy as a function of confidence for one base model. The overall increasing curves indicate that the confidence score remains predictive even when the preference judgment depends on stepwise mathematical correctness.

Figure 15: Confidence score distribution on RewardBench across six models. Each panel shows the confidence-score densities of correct and incorrect predictions for one base model on this external benchmark. The same high-confidence separation between correct and incorrect cases persists beyond the training data, though the exact spread differs across models.

Figure 16: Accuracy vs. confidence score on RewardBench across six models. Each panel reports binned accuracy as a function of the confidence score for one base model. The overall increasing trend shows that the confidence signal generalizes to an out-of-training benchmark with broader domain coverage.

Figure 17: Confidence score distribution on RM-Bench across six models. Each panel shows, for one base model, the density of confidence scores for correct and incorrect predictions. Even on this harder benchmark, incorrect predictions remain concentrated in the low-confidence region, while correct predictions contribute the heavier high-confidence tail.

Figure 18: Accuracy vs. confidence score on RM-Bench across six models. Each panel reports binned prediction accuracy as the confidence score increases for one base model. The overall increasing trend confirms that the confidence score still tracks prediction reliability on a benchmark that stresses fine-grained preference discrimination and robustness to stylistic cues.

Benchmark	High-Conf Fraction	Error (High-Conf)	Error (Low-Conf)	Recovered by Forced Reflection
RewardBench	67.3%	3.0%	19.0%	10 / 61 (16.4%)
RM-Bench	58.7%	12.5%	37.3%	385 / 877 (43.9%)
JudgeBench	33.9%	22.9%	42.9%	8 / 48 (16.7%)

Appendix

Table 3: High-confidence vs. low-confidence error rates at

\tau=5

. Each benchmark is partitioned by whether the fast-pass confidence exceeds the default routing threshold. “High-Conf Fraction” is the proportion of samples that bypass reflection; “Error (High-Conf)” and “Error (Low-Conf)” report the error rates within the two partitions. “Recovered by Forced Reflection” counts high-confidence mistakes that would become correct if reflection were run anyway, quantifying how many bypassed errors are still recoverable.

Benchmark / Threshold	Accuracy	Reflection Rate	Avg. Output Tokens
RewardBench, $\tau=3$	93.5%	10.2%	104.5
RewardBench, $\tau=5$	94.2%	32.7%	298.2
RewardBench, $\tau=7$	94.3%	74.2%	560.0
RM-Bench, $\tau=3$	81.3%	16.7%	219.3
RM-Bench, $\tau=5$	84.9%	41.3%	505.3
RM-Bench, $\tau=7$	87.2%	76.1%	784.3

Appendix

Table 4: Example operating points under different global thresholds. For each benchmark, we sweep the routing threshold

\tau

and report the resulting task accuracy, the fraction of samples sent to reflection, and the average number of output tokens per example. “Reflection rate” is the fraction of samples routed to reflection. The table illustrates CAMEL’s core trade-off: larger

\tau

values generally improve accuracy by reflecting on more borderline cases, but they also increase generation cost.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
Apr 8, 2026Kai Qin, Liangxin Liu, Yu Liang +7Preference Modeling Self-Reflection in Language Model Agents
Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China
VRM: Teaching Reward Models to Understand Authentic Human Preferences
Mar 5, 2026Biao Liu, Ning Xu, Junming Yang +2Reward Modeling LLM Alignment
Southeast University, Nanjing, China
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

Apr 8, 2026J/K move · Enter open · S save

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang+6

Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China

Preference Modeling Self-Reflection in Language Model Agents Reward Modeling LLM Alignment Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM.

Abstract

Reward Modeling

Explore similar work

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

VRM: Teaching Reward Models to Understand Authentic Human Preferences

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment