Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Guiquan Sun; Bing Bai; Martin Renqiang Min; Dongjin Song

cs.CLOct 20, 2026

Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Authors: Guiquan Sun, Bing Bai, Martin Renqiang Min, Dongjin Song

Organizations: School of Computing, University of Connecticut · NEC Labs America

Abstract

Generative reward models (GenRMs) generate natural language rationales to justify preference judgments, yet a correct verdict does not guarantee a sound justification. Existing approaches often rely on reference explanations to supervise rationales, but obtaining such references can be costly and does not necessarily establish whether generated rationales support the resulting verdict. To address these limitations, we propose DeCert, a framework that evaluates the validity and verdict consistency of generated rationales without requiring reference explanations. Crucially, DeCert translates these assessments into preference supervision that accounts for both verdict correctness and justification quality. In particular, when multiple sampled judgments share the same correct verdict, differences in certificate quality enable DeCert to construct rationale preference pairs that distinguish stronger from weaker justifications. Extensive empirical studies across two model families and multiple in- and out-of-distribution benchmarks demonstrate that DeCert consistently outperforms strong generative judge baselines while improving rationale quality. Moreover, DeCert effectively mitigates positional bias, improving position-consistent accuracy by up to 14.8 points. These results highlight the importance of supervising not only which verdict is correct, but also how that verdict is justified.

Figures & tables

Figure 1: Illustrations of risks arising from unreliable judging in GenRMs, where the stated rationale conflicts with the final output. Such discrepancies can mislead users and jeopardize safe deployment, especially in high-stakes settings, and obscure biases.

Figure 2: Overview of DeCert . We score each judgment along three rationale-quality dimensions, combine the resulting Rationale Score with Outcome Correctness, and use the final score to select SFT targets and construct DPO preference pairs. When all judgments predict the correct verdict, additional preference pairs are formed based on rationale quality.

LLM-as-a-Judge
Models	RM-Bench					JudgeBench					Avg.
Models	Chat	Math	Code	Safety	Overall	Knwl.	Reas.	Math	Code	Overall	Avg.
GPT-4o †	67.2	67.5	63.6	91.7	72.5	50.6	54.1	75.0	59.5	59.8	66.2
Claude-3.5-Sonnet †	62.5	62.6	54.4	64.4	61.0	62.3	66.3	66.1	64.3	64.8	62.9
Generative Reward Models
Nemotron-30B-A3B	68.3	69.0	59.1	90.1	71.6	46.8	41.8	62.5	47.6	48.0	59.8

Table 1: Light Blue and underlined highlight the best and second-best results, respectively. DeCert consistently improves over outcome-only baselines, and our Qwen3-8B model achieves state-of-the-art results among strong baselines. JudgeBench reports positional consistent accuracy. “Avg.” averages the two Overall columns.

\dagger

: reported by Wang et al. (2026a) ;

\ddagger

: reported by the original papers ( Whitehouse et al., 2026 , J1;) .

Base Models
Models	#Training	Overall			PCA by domain
Models	Pref. Pairs	Acc.	Swap	PCA	Code	General	Multiling.	STEM
Llama-3.1-8B	–	63.9	61.2	45.2	51.4	43.3	44.3	42.5
Qwen3-8B	–	65.2	67.9	53.6	62.1	47.3	61.6	53.1
Open Generative Reward Models
Con-J-7B	80K	66.2	69.9	51.1	57.4	48.3	51.4	50.0

Table 2: Results on HelpSteer3 comparing judgment bias. Acc.: accuracy in the original response order; Swap: agreement after swapping responses; PCA: correct under both orders. Reported gains are relative to the corresponding base models. Bold and underlined mark the best and second-best scores, respectively.

Figure 3: Effect of rationale preference pairs. Adding certificate-ranked all-correct pairs to outcome-only training improves positional consistent accuracy by up to 6.2 points.

Method	Overall		Multiling.
Method	Score	$\Delta$	Score	$\Delta$
Base	0.2059	–	0.2287	–
w/o Rati. Pairs	0.2212	$\uparrow$ 0.0153	0.2973	$\uparrow$ 0.0686
Ours	0.2624	$\uparrow$ 0.0565	0.3589	$\uparrow$ 0.1302

Table 3: Rationale quality on HelpSteer3-Atomic, measured by RationaleRM-style average precision. w/o Rati. Pairs removes rationale preference pairs.

Figure 4: Counterfactual evidence sensitivity. Evidence Adaptation Rate (EAR;

\uparrow

) measures whether the rationale updates its assessment of the edited issue, while Stale Rationale Rate (SRR;

\downarrow

) measures reuse of an invalidated claim.

Figure 5: Rates of seven non-exclusive rationale flaws. Lower rates indicate better rationale quality.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Corr.	Inst.	Lang.	Comp.	Clar.
Code/STEM	0.40	0.25	–	0.20	0.15
General	0.30	0.30	–	0.25	0.15
Multilingual	0.30	0.25	0.20	0.15	0.10

Appendix

Table 4: Domain-specific evaluation weights. Corr.: correctness; Inst.: instruction following; Lang.: language consistency; Comp.: completeness; Clar.: clarity.

Model in our tables	Official model card / checkpoint	Temperature
Llama-3.1-8B ( AI, 2024 )	meta-llama/Llama-3.1-8B-Instruct	0.0
Qwen3-8B ( Yang et al., 2025 )	Qwen/Qwen3-8B	0.6
Con-J-7B ( Ye et al., 2025 )	ZiyiYe/Con-J-Qwen2-7B	0.6
R-Align-8B ( Lai et al., 2026 )	lyn22333/GenRM-R-Align-8B	0.6
RewardAnything-8B ( Yu et al., 2025b )	zhuohaoyu/RewardAnything-8B-v1	0.6
GRAM-R 2 ( Wang et al., 2026b )	wangclnlp/GRAM-RR-LLaMA-3.1-8B-RewardModel	0.6

Appendix

Table 5: Model checkpoints and decoding temperatures. Checkpoint identifiers link to the corresponding official model cards. The listed temperatures are those used in our evaluation.

Threshold $\delta$	RM-Bench	JudgeBench
0.15	51.9	42.6
0.20	52.4	41.4
0.25 (default)	52.9	42.6

Appendix

Table 6: PCA (%;

\uparrow

) with different rationale score-gap thresholds on Llama-3.1-8B. The default is

\delta=0.25

. Best results, including ties, are in bold .

Aggregation	RM-Bench	JudgeBench
Arithmetic	43.0	40.6
Geometric (Ours)	52.9	42.6

Appendix

Table 7: Effect of score aggregation. We report position-consistent accuracy (PCA, %;

\uparrow

Figure 6: Ablation of rationale scoring under controlled perturbations. (a) Individual component scores. (b) Geometric and arithmetic aggregation. VT: valid trade-off; ITD: irrelevant true details; FE: fabricated evidence; MKI: missing key issues; RV: reversed verdict. Success rates are measured at

\tau=0.05

on a shared 0–100% scale; higher is better. Labels in (a) mark harmful-edit detection rates below 50%; labels in (b) show the higher rate on each axis.

Analyzer	Recall	Precision	F1	Issues	API Price ($)
GPT-4o	0.299	0.416	0.327	2.71	2.50/10.00
GPT-4o-mini	0.293	0.317	0.288	3.52	0.15/0.60
Qwen3-8B	0.241	0.411	0.281	2.39	–

Appendix

Table 8: Material-issue analyzer comparison. API prices are reported per 1M input/output tokens.

Evaluation input
Question	It would take a beam of light hours to travel a distance of $7.15\times 10^{12}$ m.
Response A	Uses $c\approx 3.00\times 10^{8}$ m/s and computes $t=\frac{7.15\times 10^{12}}{3.00\times 10^{8}}=2.3833\times 10^{4}\ {\rm s},\qquad\frac{2.3833\times 10^{4}}{3600}\approx\mathbf{6.62\ {\rm h}}.$ It concludes that the light travel time is approximately 6.62 hours .
Response B	Starts from $c=299{,}792{,}458$ m/s but attempts to convert the speed to meters per hour as $299{,}792{,}458\ {\rm m/s}\ \mathbf{/}\ 3600\approx 83{,}275{,}683{,}000\ {\rm m/h},$ and then reports $t\approx 0.0859\ {\rm h}\approx\mathbf{5.15\ {\rm min}}.$ It concludes that the travel time is “about 5 minutes.” The conversion is incorrect: converting m/s to m/h requires multiplying, rather than dividing, by $3600$ .
Gold verdict	Response A

Appendix

Table 9: Case study of a same-verdict rationale preference pair. Both judgments are sampled from Qwen3-8B and correctly select Response A. The rejected rationale reaches the correct verdict but incorrectly validates a central claim in Response B, whereas the chosen rationale identifies the decisive calculation error. Bold text highlights the key difference.

Panel of Figure 7 — Figure 7: Prompt template used to generate structured judgment certificates. The placeholders are filled with the evaluation policy, conversation, and two candidate responses for each instance.

Panel of Figure 8 — Figure 8: Prompt template used to verify whether each observation in a decision certificate is grounded in the corresponding response and is substantively correct.

Panel of Figure 9 — Figure 9: Prompt template used to match each independently identified material issue to at most one claim in a decision certificate.

Panel of Figure 10 — Figure 10: Prompt template used to extract the material issues that should determine the response preference. The analyzer sees the comparison instance and evaluation policy but does not see any candidate judgment.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Apr 13, 2026Haozhe Wang, Cong Wei, Weiming Ren +3Reward Modeling Multimodal Reward Modeling
HKUST · University of Waterloo · Alibaba
Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models
Mar 2, 2026Qiyuan Zhang, Yufei Wang, Tianhe Wu +5Reward Modeling Reward Model Evaluation
City University of Hong Kong · Tencent Hunyuan · MBZUAI
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward Modeling Process Reward Models
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China

Apr 13, 2026J/K move · Enter open · S save

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu+2

HKUST · University of Waterloo · Alibaba

Reward Modeling Multimodal Reward Modeling RL for Image Generation Prompt Optimization for T2I Generation

Open paper AskPDF arXiv

Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.

Abstract

Reward Modeling

Explore similar work

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans