Reward-Adaptive Co-Evolution of Policy and Rubric

Yizhu Chen; Jianguang Jiang; Jiameng Huang; Chuyu Fang; Bojun Zhang; Zhenyu Jiao; Xing Yuan; Tong Lin

cs.AIOct 29, 2026

Reward-Adaptive Co-Evolution of Policy and Rubric

Authors: Yizhu Chen, Jianguang Jiang, Jiameng Huang, Chuyu Fang, Bojun Zhang, Zhenyu Jiao, Xing Yuan, Tong Lin

Organizations: Peking University · Baidu Inc. · Institute of Automation, Chinese Academy of Sciences.

Abstract

Rubric-based rewards provide fine-grained, interpretable supervision for open-ended reinforcement learning, where holistic scalar rewards struggle to capture multidimensional response quality. Static rubrics can lose discriminative power as the policy distribution evolves, while existing policy--rubric co-evolution methods rely on globally prescribed update schedules that ignore heterogeneous reward informativeness. We introduce RACE, a Reward-Adaptive Co-Evolution framework. At the group level, per-rollout-group training-signal routing classifies groups as informative, diversity-collapsed, or rubric-failed, updating the policy only with informative groups while caching rubric failures. At the phase level, event-triggered phase switching enters rubric optimization upon persistent reward saturation and returns when rubric-reward improvement plateaus. Rubric phases train the generator on cached pairs using a frozen judge and a scale-normalized pairwise-margin reward. Across both levels, online threshold adaptation adjusts routing and switching thresholds through proportional control. Under a unified evaluation protocol, RACE achieves the highest average performance and outperforms the strongest fixed-alternation baseline on all eight tasks, while improving rubric discrimination.

Figures & tables

Step	Valid.(%)	Label (%)	Redund.(%)	$\sigma=0$ (%)
200	70	9	28	32
400	87	4	16	27
600	99	5	23	27
800	95	10	21	29
1000	82	12	23	44

Table 1: Rubric analysis under fixed-length alternation.

Step	GSM8K	MMLU	IFEval	PopQA	GPQA	Avg.
500	95.8	85.1	38.6	30.7	53.6	60.8
1000	95.5	85.6	38.8	31.2	54.0	61.0

Table 2: Downstream performance.

Figure 1: Framework overview of RACE. A shared model parameterized by $\theta$ alternates between the policy $\pi_{\theta}$ and rubric generator $\mathcal{G}_{\theta}$ . During the policy phase, $\pi_{\theta}$ samples $\{y_{i}\}_{i=1}^{G}$ for prompt $x$ , and the frozen judge $\mathcal{J}$ scores them under the prompt-specific rubric $R$ via Eq. 2 . Per-rollout-group training-signal routing uses informative groups to update the policy, masks diversity-collapsed groups, and caches rubric-failed entries $(x,y^{+},y^{-},m_{\mathrm{old}})$ in $\mathcal{P}$ . During the rubric phase, $\mathcal{G}_{\theta}$ samples candidate rubrics $\{R_{i}\}_{i=1}^{G}$ for each cached entry, each evaluated on the same pair $(y^{+},y^{-})$ using the scale-normalized pairwise-margin reward in Eq. 16 . Event-triggered phase switching moves to rubric optimization upon persistent reward saturation and back upon a rubric-reward plateau.

Mode	Condition	Action
Diversity-collapsed	Low surface/semantic diversity	Mask
Rubric-failed	Degenerate; otherwise	Mask; add to $\mathcal{P}$
Informative	Discriminative	standard GRPO

Table 3: Per-rollout-group training-signal routing.

Prompted Rubrics Source
Method	GSM8K	MBPP+	BBH	MMLU	PopQA	GPQA	Zebra	AGI-E	Avg
GPT-4.1	95.7	65.1	71.7	84.8	30.7	54.9	81.9	82.9	71.0
Qwen3-8B	95.5	58.4	66.8	83.4	31.7	56.9	84.3	83.9	70.1
Scalar Reward Model
Scalar RM (Skywork-RM-V2)	95.6	66.2	54.9	82.0	25.2	46.2	56.4	81.6	63.5
Offline Rubric Generation

Table 4: Downstream policy evaluation. Avg. is the unweighted mean; Bold and underlined values indicate the best and second-best results, respectively.

	JudgeBench						GPT-5.4
Method	Know.	Reason.	Math	Code	Overall	Avg.	Avg.
EvoLM	31.8	36.7	26.8	33.3	32.6	32.2	57.9
RACE	33.1	37.8	28.6	40.5	34.6	35.0	62.2

Table 5: JudgeBench and RewardBench2 evaluations.

Dimension	Variant	RB2	Policy Avg.
Overall	RACE (full)	49.2	73.0
Routing	No Gradient Filtering	48.7	70.8
	Uniform Rubric Sampling	47.9	71.4
	Filter-only	45.1	70.9
Switching	Random Phase-Length Alternation	45.3	71.4
Switching	Fixed 50/50 Alternation + Routing	44.3	71.3

Table 6: Ablation study. RB2 and Policy Avg. denote the average performance on RewardBench2 and the eight main-table downstream benchmarks, respectively. Bold and underlined values indicate the best and second-best results in each column.

Figure 2: Phase-length dynamics. Policy (a) and rubric (b) phase lengths across successive alternations; solid curves denote 5-point moving averages (MA), and dashed horizontal lines mark the target lengths

L^{*}=25

and

M^{*}=12

, respectively.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Description
Training Infrastructure
Batch size (unique prompts)	64	Per policy step
Rollouts per prompt ( $G$ )	8	For GRPO
Max response length	16384	Tokens
GPUs	16	2 nodes $\times$ 8 H800
Learner ranks	8	4 per node, DeepSpeed ZeRO-3

Appendix

Table 7: Complete hyperparameter specification for RACE.

Method	Factuality	Precise IF	Math	Safety	Focus	Ties	Avg.
GPT-4.1 (prompted)	32.1	32.6	34.4	39.1	35.0	46.3	36.6
Qwen3-8B (prompted)	27.8	26.5	38.1	25.0	30.2	9.7	26.2
Scalar RM (Skywork-RM-V2)	88.2	67.8	83.1	97.3	99.2	83.1	86.4
RaR	N/A (frozen GPT-4.1 rubric)
RRD	N/A (frozen GPT-4.1 rubric)
RLCER	33.1	29.8	52.2	48.3	51.9	54.8	45.0

Appendix

Table 8: Full RewardBench2 evaluation. All scores are percentages. Bold and underlined values indicate the best and second-best results in each column, respectively. N/A denotes methods for which a directly comparable trained rubric generator is unavailable.

Generator	#	Criteria emitted for the rectangle prompt
EvoLM, step 1000	4	Relate length and width using the perimeter formula; express area in one variable; identify a maximization method; provide the correct maximum-area value.
RACE, step 700	5	Set up $2x+2y=48$ and $A=xy$ ; apply a valid maximization method; explain why the optimum is a square; give 144 square units; present the solution with a clear logical flow.

Appendix

Table 9: Same-prompt rubric comparison. The descriptions preserve the semantic content of the generated criteria while removing only the repeated three-level scoring anchors for compactness.

Step	#	Representative criteria
50	5	Correct value 144; derive the perimeter relation; use an appropriate maximization method; explain why the optimum is a square; maintain clarity and logical flow.
350	4	Set up perimeter and area equations; substitute $y=24-x$ to obtain $A=24x-x^{2}$ ; identify a valid maximization argument; give 144 with justification.
700	5	Set up perimeter and area; maximize the area; justify the square optimum; give 144 square units; show a clear progression from setup to solution.

Appendix

Table 10: RACE rubric evolution for one fixed math prompt. Criteria are abbreviated, while retaining the emitted equations, expected values, and distinctions between criteria.

Generator	Valid (%)	# Crit.	Len.	Label-only (%)	Specific (%)	Constraint (%)	Grounded (%)
RACE, step 50	73	5.5	56.9	20.9	10.1	10.8	38.1
RACE, step 350	79	4.9	63.0	10.4	14.7	11.9	43.8
RACE, step 700	90	5.0	62.6	9.9	18.0	11.6	40.0
EvoLM, step 1000	82	5.2	59.8	11.8	14.5	10.6	–

Appendix

Table 11: Rubric statistics on 100 fixed prompts. Validity is the percentage of prompts yielding parseable JSON. The remaining metrics are computed over criteria from parse-valid rubrics. “Specific” denotes criteria containing concrete expected values; “Grounded” denotes lexical overlap with prompt-specific entities or constraints.

Diagnostic	Value
near tie ( $\|m_{old}\|<0.025$ )	5.23%
Advantage collapse ( $\sigma(S_{\mathrm{rubric}})=0$ )	4.56%
Base reward at clipping boundary	12.19%

Appendix

Table 12: Rubric-generator reward diagnostics.

Step	Groups	$\sigma(\mathbf{r})=0$ (%)	All $\geq 0.8$ (%)	Mean $\sigma(\mathbf{r})$
50	4,684	24.2	74.4	0.073
100	2,970	28.5	75.3	0.070
200	3,547	28.6	75.8	0.066
250	4,017	23.9	75.1	0.069
350	6,833	22.8	75.2	0.068
500	5,718	24.6	76.2	0.065

Appendix

Table 13: Reward variation within policy rollout groups during RACE training. Each checkpoint row aggregates groups from a disjoint window centered on that checkpoint.

Method	Rubric construction	Answer scoring	Extra checks
RaR	1 rubric proposal	$G$ answer scores	–
RRD	Initial proposal + recursive refinement	$G\times I$ item-level scores	Data-dependent refinement checks
RLCER	$G$ rubric generations (one per rollout)	$G^{2}$ judge calls	$G$ verifier checks
Rubric-ARM	1 rubric generation (frozen model)	$G$ answer scores	–
EvoLM	1 prompt-level rubric generation	$G$ answer scores	–
RACE (policy update)	1 prompt-level rubric generation	$G$ answer scores	–

Appendix

Table 14: Per-prompt online reward computation. Here

G

is the GRPO group size and

I

denotes the number of rubric items scored individually by the corresponding baseline.

Update	Policy generations	Rubric generations	Judge evaluations	Learner samples
Policy	$N_{\mathrm{batch}}G$	$N_{\mathrm{batch}}$	$N_{\mathrm{batch}}G$	$N_{\mathrm{batch}}G$ policy responses
Rubric	0 (cached responses)	$N_{\mathrm{batch}}G$	$2N_{\mathrm{batch}}G$	$N_{\mathrm{batch}}G$ candidate rubrics

Appendix

Table 15: RACE online model calls for one optimizer update. One rubric is shared by all

G

policy responses in a prompt group. During a rubric update, each of the

G

candidate rubrics is evaluated on both responses of the single cached ordered pair

(y^{+},y^{-})

drawn from

\mathcal{P}

Controller Setting	GSM8K	MBPP+	BBH	MMLU	PopQA	GPQA	Zebra	AGI-E	Avg.
Default $(0.20,25,12)$	95.9	68.7	72.3	86.1	33.8	56.1	83.9	87.5	73.0
Short Phase Targets $(0.20,10,5)$	95.9	67.6	66.4	85.2	32.7	54.1	82.0	87.1	71.4
Lower Quarantine Target $(0.10,25,12)$	96.1	65.7	63.8	84.8	32.9	56.9	83.6	87.4	71.4

Appendix

Table 16: Hyperparameter sensitivity. Each tuple denotes

(q^{*},L^{*},M^{*})

, corresponding to the target quarantine rate and the target policy- and rubric-phase lengths.

Rubric-Based RL

Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.

See all 107 papers

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6Automatic Rubric Generation RL for Language Models
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation
Oct 28, 2026Chengsheng Zhang, Wenshen Xu, Yuchuan Wu +6RL for Language Models Rubric-Based RL
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · Nanjing University +1
RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation
Oct 18, 2026Xueyun Tian, Bingbing Xu, Chuyue Li +9Rubric-Based RL Reward Design for RL
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1

Jun 22, 2026J/K move · Enter open · S save

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

Hongxin Ding, Baixiang Huang, Yue Fang, Weibin Liao+5

National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education+2

Automatic Rubric Generation RL for Language Models Rubric-Based RL Rubric-Based Evaluation Rubric-Guided Generation Reward Design for RL

Open paper AskPDF arXiv

Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.

Abstract

Rubric-Based RL

Explore similar work

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation