PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization

Zelin Tan; Zhouliang Yu; Bohan Lin; Zijie Geng; Hejia Geng; Yudong Zhang; Mulei Zhang; Yang Chen

cs.AIMar 27, 2026

PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization

Authors: Zelin Tan, Zhouliang Yu, Bohan Lin, Zijie Geng, Hejia Geng, Yudong Zhang, Mulei Zhang, Yang Chen, +4 more

Organizations: University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong · University of Oxford · Wuhan University

Abstract

We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, treating all correct responses identically regardless of reasoning quality, and gradually lose the advantage signal as groups become uniformly correct. Process reward models (PRM) offer richer supervision, but directly using PRM scores causes reward hacking, where models exploit verbosity to inflate scores while accuracy collapses. PAPO resolves both by composing the advantage from an outcome component Aout, derived from ORM and normalized over all responses, and a process component Aproc, derived from a rubric-based PRM and normalized exclusively among correct responses. This decoupled design ensures that Aout anchors training on correctness while Aproc differentiates reasoning quality without distorting the outcome signal. Experiments across multiple model scales and six benchmarks demonstrate that PAPO consistently outperforms ORM, reaching 51.3% vs.\ 46.3% on OlympiadBench while continuing to improve as ORM plateaus and declines.

Figures & tables

Panel of Figure 1 — Figure 1: (a) ORM (blue) plateaus and declines after step 750 due to signal exhaustion; PRM (red, dashed) collapses via reward hacking; ORM $\times$ PRM (purple, dash-dotted) tracks ORM closely without exceeding it; PAPO (green) continues improving throughout training, reaching 51.3%. (b) Comparison across AIME 2024/2025, OlympiadBench, and their average. Naive multiplicative combination (ORM $\times$ PRM) barely improves over ORM, while PAPO’s decoupled normalization yields substantial gains on all benchmarks.

Figure 2: Reward signal analysis. (a) Training reward: PRM reward climbs to 1.0 (complete reward hacking); ORM

\times

PRM stays moderate. (b) Response length: PRM generates increasingly verbose responses; ORM

\times

PRM shows moderate length increase (up to

\sim

1700 tokens). (c) OlympiadBench accuracy: PRM collapses after step 600; ORM

\times

PRM tracks ORM but fails to exceed it, showing that naive signal combination does not resolve signal exhaustion.

Figure 3: Overview of PAPO. Given a prompt, the policy generates

G

responses. Each response is evaluated by two reward signals: an outcome reward (ORM, binary correctness) and a process reward (PRM, rubric-based quality, only for correct responses ). The advantage is computed through decoupled normalization :

A_{\text{out}}

is normalized over all responses via standard GRPO, while

A_{\text{proc}}

is normalized exclusively among correct responses ( correct subset normalization ). The combined advantage

A_{\text{total}}=A_{\text{out}}+A_{\text{proc}}

provides both correctness direction and quality differentiation.

		Competition Mathematics			Standard	STEM	Code	Average
Model	Method	Olympiad	AIME’24	AIME’25	MATH-500	GPQA	HumanEval	Math	All
Qwen2.5-3B	Base	22.4	2.5	0.8	54.4	27.6	35.3	20.0	23.8
	GRPO	35.3 ±0.4	8.3 ±1.5	4.2 ±1.2	69.0 ±0.6	38.8 ±2.4	39.3 ±2.0	29.2 ±0.5	32.5 ±0.6
	PA-GRPO	38.6 ±0.5	10.0 ±1.4	3.3 ±1.0	73.2 ±0.6	36.3 ±1.8	52.5 ±1.6	31.3 ±0.5	35.7 ±0.5
	$\Delta$	+3.3	+1.7	-0.9	+4.2	-2.5	+13.2	+2.1	+3.2
Qwen2.5-7B	Base	32.8	5.0	7.5	67.3	30.1	54.6	28.2	32.9

Table 1: Accuracy (avg@4, %) on six benchmarks across five model configurations, reported as mean

{}_{\pm\text{std}}

over 3 seeds. Gray rows show the untrained or instruction-tuned starting checkpoint. The lower half of the Qwen2.5-7B block reports PAPO composed on three mainstream GRPO variants (GSPO, SAPO, CISPO). Bold marks the better entry within each pair.

Figure 4: OlympiadBench acc/mean@4 on Qwen2.5-7B-Base. Each panel pairs a base GRPO variant (dashed) with its PA-* counterpart (solid green), where PA-* denotes PAPO composed on top of that base optimizer.

Figure 5: Signal quality comparison on Qwen2.5-7B. (a) Zero-advantage ratio: ORM’s sparsity grows to 69% while PAPO maintains 44%. (b) Advantage standard deviation, reflecting gradient signal strength. (c) Positive-advantage ratio, reflecting reinforcement density.

Figure 6: Length-accuracy trajectories on OlympiadBench (Qwen2.5-7B-Base). Markers are selected training checkpoints colored by step; stars mark the final step.

Figure 7: OlympiadBench acc/mean@4 on Qwen2.5-7B-Base for three additional ablations. (a) Group size

G=4

vs.

G=8

. (b) Process advantage weight

\lambda=0.5

vs.

\lambda=1

. (c) Stronger PRM (5-tier rubric with GPT-OSS-120B judge) vs. default (3-tier with 20B). The GRPO ORM baseline is included in every panel as a dashed reference.

Method	AIME	AIME	Olympiad	MATH
	2024	2025	Bench	-500
ORM	10.8	10.8	46.3	80.2
Mult ( $r^{\text{out}}\!\times\!r^{\text{proc}}$ )	12.1	10.2	46.7	80.2
Fullnorm	12.5	10.4	49.6	81.2
PAPO	15.8	13.1	51.3	82.3

Table 2: Structural ablation results (%, avg@4, Qwen2.5-7B). All methods report accuracy.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Prompts per batch	128
Responses per prompt ( $G$ )	8
Max response length	8192 tokens
Sampling temperature	1.0
PPO clip range ( $\epsilon$ )	0.2
KL penalty coefficient ( $\beta$ )	0.0

Appendix

Table 3: Training hyperparameters.

Hyperparameter	Value
Max response length	8192 tokens
Clip ratio low ( $\epsilon_{\text{low}}$ )	0.2
Clip ratio high ( $\epsilon_{\text{high}}$ )	0.28
Overlong buffer length	4096 tokens
Overlong penalty factor	1.0
Evaluation frequency	every 20 steps

Appendix

Table 4: Additional DAPO-specific hyperparameters for the Qwen3-4B-Base experiments. All other hyperparameters follow Table 3 .

Difficulty	Pass Rate Range	Count
Trivial	$(0.875,\,1.0]$	4,000
Easy	$(0.625,\,0.875]$	4,000
Medium	$(0.375,\,0.625]$	4,000
Hard	$(0.125,\,0.375]$	4,000
Very Hard	$(0,\,0.125]$	4,000
Total		20,000

Appendix

Table 5: Difficulty distribution of the stratified training set. Pass rates are measured using a reference model with 8 attempts at temperature 1.0. Equal sampling across tiers ensures balanced difficulty coverage.

Hyperparameter	Value
length-reward $\alpha$	0.05
difficulty $A$	0.4
difficulty $B$	1.5
difficulty $\rho_{0}$	0.75
difficulty $k$	10.0
explicit_penalty	$-1.0$

Appendix

Table 6: GRPO-LEAD hyperparameters, matching Zhang and Zuo (2025) .

Comparison	$n$	Exact
PRM vs. human, all	500	91.2%
PRM vs. human, answer-correct	332	94.6%

Appendix

Table 7: Agreement between the PRM and majority-vote human labels.

PRM	$n$	Human label (%)			Mean / gap
score		0	0.5	1
0	11	90.9	9.1	0.0	0.045 / 0.045
0.5	76	2.6	92.1	5.3	0.513 / 0.013
1	245	0.4	4.1	95.5	0.976 / 0.024

Appendix

Table 8: PRM calibration on the 332 answer-correct MATH-500 responses.

Group	Tokens	PRM	Human	Gap	Agree.
Shortest 25%	408	0.82	0.81	$+0.01$	95.2%
Middle 50%	712	0.86	0.85	$+0.01$	94.6%
Longest 25%	1508	0.86	0.84	$+0.02$	94.0%

Appendix

Table 9: PRM–human alignment across response-length strata on the answer-correct subset.

Signal	Peak	Step 1090
ORM (GRPO)	45.0	43.0
PAPO	50.0	49.7
CorrectSubset-Rand-3	42.4	37.4
CorrectSubset-Rand-5	42.9	38.5
CorrectSubset-Length	47.0	40.5
Full-Length	43.1	1.7

Appendix

Table 10: OlympiadBench accuracy (%) for counterfactual process signals in the matched control trajectories. Peak is the best reported checkpoint from steps 0–1090.

Figure 8: Training curves (avg@4) across two model scales and three benchmarks. Top row: Qwen2.5-3B; bottom row: Qwen2.5-7B. PAPO consistently outperforms ORM throughout training on both scales, with gains widening in later stages as ORM’s signal exhaustion worsens. The pattern is consistent across OlympiadBench (competition math), MATH-500 (standard math), and HumanEval (code generation). Qwen2.5-14B results are reported in Table 1 .

Figure 9: Ablation: correct subset normalization (PAPO) vs. full normalization (Fullnorm) on OlympiadBench and MATH-500. Both methods improve over ORM, but PAPO’s correct subset design maintains a consistent advantage over Fullnorm throughout training.

Figure 10: Ablation: decoupled normalization (PAPO) vs. multiplicative reward (Mult) on OlympiadBench and MATH-500. Mult improves over ORM but remains substantially below PAPO in these trajectories.

Figure 11: (a) Minimum advantage among correct responses. ORM: identically 0.0 (all correct responses treated equally). PAPO:

\approx-1.49

(the process advantage penalizes correct responses with poor reasoning). (b) Standard deviation of advantages among correct responses. ORM: zero (no differentiation). PAPO: non-zero (quality-based ranking within correct group).

Figure 12: PAPO advantage composition. (a) Signal strength (std) of

A_{\text{out}}

and

A_{\text{proc}}

: the outcome advantage provides the dominant gradient signal while the process advantage contributes a complementary quality signal. (b) Process signal active ratio: the fraction of groups where

A_{\text{proc}}

is non-zero (

\geq 2

correct responses), growing from

\sim

30% to 70% as the model improves. (c) Mean advantage by correctness: correct responses receive positive advantage while wrong responses receive negative, confirming the outcome signal functions correctly despite the added process component.

Problem	Model	Tokens	Correct	Observation
Competition number theory	Base	866	3/3	Focused proofs
	ORM	1583	3/3	Longer but correct
	PRM	3072	0/3	Drift to filler ∗
Algebra (proof)	Base	1059	0/3	Attempts proof
	ORM	1492	0/3	Attempts proof
	PRM	3227	0/3	2 hit cap; 1 drifts

Appendix

Table 11: Case study: response statistics on three math problems (Qwen2.5-7B). PRM reward hacking is severe on harder problems (top two), with 2–3

\times

longer responses and characteristic topic drift. On easier problems, PRM remains comparable to baselines (omitted for brevity).

Explore similar work

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona
The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
Jun 8, 2026Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6Process Reward Models RL for Language Model Reasoning
University of Maryland, College Park · Capital One · University of Central Florida
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Zhengliang Shi +8Reward Hacking Rubric-Based RL
Scale AI · University of Arizona · University of Texas at Dallas

May 19, 2026J/K move · Enter open · S save

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei, Daniel George+4

Scale AI · Persona

Reward Modeling Rubric-Based RL Reinforcement Learning with Verifiable Rewards Rubric-Based Evaluation Rubric-Guided Generation Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins

24

30

base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in

2.5

4\times

fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

Abstract

Explore similar work

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL