Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

Chengsheng Zhang; Wenshen Xu; Yuchuan Wu; Zhiyuan Yu; Junfeng Fang; Tianlong Wang; Bingxue Qiu; Yongbin Li

cs.AIOct 28, 2026

Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

Authors: Chengsheng Zhang, Wenshen Xu, Yuchuan Wu, Zhiyuan Yu, Junfeng Fang, Tianlong Wang, Bingxue Qiu, Yongbin Li,

Organizations: University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · Nanjing University · National University of Singapore

Abstract

Rubric-based reinforcement learning (RL) offers interpretable, fine-grained supervision for aligning large language models (LLMs) on open-ended generation tasks. However, static rubric criteria may become saturated or remain beyond the policy's exploration, rendering the reward signals insufficiently discriminative. While recent efforts seek to guide sampling exploration or evolve rubrics, they fail to explicitly factor in how rubric difficulty should track the evolving policy's capabilities, resulting in a persistent mismatch between the optimization objective and the policy's current capability frontier. In this paper, we propose Co-Evolving Policy and Rubric via Capability Awareness (CoCA), a closed-loop framework that jointly adapts the rubric and policy through criterion-level diagnosis. Specifically, CoCA contrasts query-only and rubric-guided rollouts to characterize each criterion relative to the current policy, recalibrates criteria with mismatched difficulty, and identifies guidance-revealed capability gaps for frontier-based learning. Based on this insight, CoCA either expands rubric-conditioned frontiers through guided optimization or transfers qualified gains through on-policy self-distillation. As the policy updates, CoCA improves the capability diagnosis and enables more reliable distillation, thereby completing the co-evolution loop. Extensive experiments across diverse benchmarks and LLMs demonstrate that CoCA consistently improves performance over rubric-based RL methods. Moreover, CoCA promotes the internalization of guidance-revealed capabilities into query-only behavior while mitigating capability degradation.

Figures & tables

Figure 1: Motivation and overview of CoCA. (a) Static criteria may become saturated or remain unmet, limiting reward discriminativeness. (b) Query-only failures can conceal capabilities elicitable under rubric guidance. (c) Distinct from existing rubric-based RL methods, CoCA tracks the policy’s capability boundary to couple rubric adaptation and policy learning in a closed loop.

Figure 2: Pipeline overview of our CoCA. Dual-Group rollouts contrast query-only and rubric-conditioned behavior to diagnose criterion-level capabilities, guiding capability-aware rubric calibration and frontier selection. Normal GRPO remains active for every query, complemented by Frontier-Guided OPSD when a qualified frontier is available. Otherwise, Guided GRPO expands the rubric-conditioned frontier, supplying improved responses for the preceding two stages to close the policy–rubric co-evolution loop. The algorithm pseudocode of CoCA is provided in Appendix B .

Method	Medical Domain			Science Domain		Psychological Domain		Mean
Method	HealthBench	RaR-Medicine	LLMEval-Med	ResearchQA	RaR-Science	CPsyCoun	HeartBench	Mean
Qwen3-4B	$22.38$	$51.78$	$38.38$	$59.93$	$69.76$	$31.39$	$44.17$	$45.40$
LLM-Rank	$30.10$	$52.54$	$42.13$	$73.09$	$\underline{74.91}$	$33.56$	$43.54$	$49.98$
RaR	$37.12$	$56.03$	$42.58$	$78.42$	$73.81$	$77.41$	$\mathbf{50.14}$	$59.36$
RuscaRL	$\underline{40.44}$	$\mathbf{58.37}$	$43.63$	$\underline{79.57}$	$73.12$	$\underline{82.03}$	$45.74$	$\underline{60.41}$
Online Rubrics	$37.77$	$56.28$	$\underline{44.68}$	$78.96$	$74.35$	$79.55$	$44.92$	$59.50$

Table 1: Main results across seven benchmarks spanning the medical, science, and psychological domains. The best scores are highlighted in bold , while the second-best scores are underlined . All scores are normalized to

0\sim 100

. Additional experiments extending CoCA to larger model and other GRPO variants are presented in Appendix D.2 and Appendix D.5 , respectively.

Dataset	Qwen3-4B			Qwen3-8B			Gemma-3-4B-it
Dataset	Base	RuscaRL	CoCA	Base	RuscaRL	CoCA	Base	RuscaRL	CoCA
HealthBench †	$62.96$	$75.54$	$\mathbf{80.34}$	$71.83$	$81.13$	$\mathbf{83.02}$	$65.50$	$74.94$	$\mathbf{77.79}$
HealthBench †	$62.96$	$+12.58$	$+17.38$	$71.83$	$+9.30$	$+11.19$	$65.50$	$+9.44$	$+12.29$
InfoBench	$77.68$	$75.68$	$\mathbf{79.02}$	$78.66$	$78.04$	$\mathbf{80.40}$	$76.66$	$76.44$	$\mathbf{77.11}$
InfoBench	$77.68$	$-2.00$	$+1.34$	$78.66$	$-0.62$	$+1.74$	$76.66$	$-0.22$	$+0.45$
FollowBench	$47.92$	$45.73$	$\mathbf{51.58}$	$52.19$	$48.04$	$\mathbf{52.56}$	$59.51$	$58.65$	$\mathbf{60.73}$

Table 2: Experiment results of rubric-conditioned in-domain capability and out-of-domain instruction-following generalization. All compared models are trained on HealthBench. † : We provide the full ground-truth rubric together with each HealthBench-Hard query.

Eval Judge	Method	HealthBench	RaR-Med.	LLMEval.	Mean
Qwen3.7-Flash ( Qwen Team, 2026 )	Base	$27.31$	$54.56$	$59.22$	$47.03$
	RaR	$43.04$	$60.06$	$62.52$	$55.21$
	CoCA	$\mathbf{47.89}$	$\mathbf{61.35}$	$\mathbf{65.53}$	$\mathbf{58.26}$
Kimi-K2.5 ( Team et al., 2026 )	Base	$17.93$	$55.29$	$25.04$	$32.75$
	RaR	$36.77$	$61.79$	$28.04$	$42.20$
	CoCA	$\mathbf{41.59}$	$\mathbf{63.02}$	$\mathbf{30.13}$	$\mathbf{44.91}$

Table 3: Ablation of evaluation judge models for Qwen3-8B in the medical domain.

CARC	$\mathbf{\mathcal{L}_{\mathrm{Frontier\text{-}OPSD}}}$	$\mathbf{\mathcal{L}_{\mathrm{GRPO}}^{\mathrm{G}}}$	HealthBench	LLMEval.
$\times$	$\times$	$\times$	$41.88$	$49.93$
$\checkmark$	$\times$	$\times$	$46.30$	$48.73$
$\times$	$\checkmark$	$\times$	$42.39$	$50.52$
$\checkmark$	$\checkmark$	$\times$	$45.01$	$51.12$
$\checkmark$	$\times$	$\checkmark$	$44.47$	$52.62$
$\times$	$\checkmark$	$\checkmark$	$43.11$	$50.97$

Table 4: Ablation study of the CoCA’s components on Qwen3-8B.

Figure 3: Overall distribution of rubric capability states on HealthBench-Hard. For each query, we sample two groups of responses: one under the query-only condition and the other with the full rubric additionally provided. Each rubric is then assigned to one of five capability states according to its satisfaction patterns across the two groups.

Type	Transition (Source $\Longrightarrow$ Target)	Qwen3-4B		Qwen3-8B		Gemma-3-4B-it
Type	Transition (Source $\Longrightarrow$ Target)	RaR	Ours	RaR	Ours	RaR	Ours
Internalization $\uparrow$	Learnable ${\color[rgb]{0.0977,0.6484,0.4609}\Longrightarrow}$ Saturated	$8.10$	$\mathbf{9.83}$	$6.96$	$\mathbf{14.02}$	$13.84$	$\mathbf{15.76}$
	Learnable ${\color[rgb]{0.0977,0.6484,0.4609}\Longrightarrow}$ Discriminative	$33.69$	$\mathbf{34.47}$	$32.56$	$\mathbf{36.88}$	$30.97$	$\mathbf{35.70}$
	Too-hard ${\color[rgb]{0.0977,0.6484,0.4609}\Longrightarrow}$ Saturated	$2.63$	$\mathbf{3.65}$	$0.49$	$\mathbf{5.25}$	$6.34$	$\mathbf{8.67}$
	Discriminative ${\color[rgb]{0.0977,0.6484,0.4609}\Longrightarrow}$ Saturated	$41.17$	$\mathbf{43.61}$	$40.68$	$\mathbf{51.21}$	$42.89$	$\mathbf{45.42}$
Degradation $\downarrow$	Saturated ${\color[rgb]{0.9297,0.1992,0.1992}\Longrightarrow}$ Others	$12.86$	$\mathbf{12.21}$	$\mathbf{8.95}$	$12.48$	$13.33$	$\mathbf{12.50}$

Table 5: Rubric capability-state transitions on HealthBench-Hard. For each backbone, we record each rubric’s Source State under the Base model and its Target State under a HealthBench-trained model, and report the types and proportions (%) of the resulting transitions.

Panel of Figure 4 — Figure 4: Training dynamics and inference-time scaling of CoCA on HealthBench. (a) CoCA maintains higher entropy, preserving exploration during training; (b) CoCA matches RaR’s best performance with fewer training steps and time ( Appendix E.2 ); (c) its consistent advantage over the base model and RaR across $K$ indicates a higher capability ceiling accessible through sampling.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Learning Mechanisms			Decision Coupling
Method	Adaptive Rubrics	Token-level Signals	State-based Learning	Decision Coupling
RaR ( Gunjal et al., 2026 )	$\times$	$\times$	$\times$	Fixed Rubric $\to$ Policy Rewards
RuscaRL ( Zhou et al., 2026 )	$\times$	$\times$	$\times$	Scheduled Rubric Guidance
Online Rubrics ( Rezaei et al., 2026b )	$\checkmark$	$\times$	$\times$	Response Contrasts $\to$ Criteria
RGSD ( Rezaei et al., 2026a )	$\times$	$\checkmark$	$\times$	Fixed Rubric $\to$ Teacher Signals
CriPO ( Xia et al., 2026 )	$\times$	$\checkmark$	$\checkmark$	Criterion States $\to$ Injection and Advantage Flipping

Appendix

Table 6: Comparison of representative rubric-based methods. Adaptive rubrics respond to policy feedback. Token-level signals denote rubric-informed supervision beyond scalar advantages. State-based learning uses criterion diagnostics to select policy-update interventions.

Domain	Dataset	Split	# Samples	# Avg. Criteria	Metric
Medical Domain	HealthBench	Train/Test	$4,000/1,000$	$11.35/11.85$	Rubric Score
	RaR-Medicine	Test	$2,242$	$9.59$	Rubric Score
	LLMEval-Med	Test	$667$	–	Overall Performance (OP)
Science Domain	ResearchQA	Train/Valid	$4,000/500$	$7.53/7.46$	Rubric Score
Science Domain	RaR-Science	Test	$500$	$7.46$	Rubric Score
Psychological Domain	CPsyCoun	Train/Test	$2,400/500$	$8.48/8.46$	Rubric Score

Appendix

Table 7: Statistics of the datasets used in our experiments. “# Avg. Criteria” counts rubric items or instruction constraints per sample.

Table 12

Method	Primary	Additional	Total	HealthBench	RaR-Medicine	LLMEval-Med	Mean
Method	Rollouts	Rollouts	Rollouts	HealthBench	RaR-Medicine	LLMEval-Med	Mean
RaR ( Gunjal et al., 2026 )	$4$	–	$4$	$41.88$	$64.29$	$49.93$	$52.03$
	$8$	–	$8$	$45.51$	$64.92$	$50.37$	$53.60$
	$12$	–	$12$	$46.91$	$65.75$	$50.82$	$54.49$
RuscaRL ( Zhou et al., 2026 )	$4$	–	$4$	$46.14$	$64.07$	$45.28$	$51.83$
	$8$	–	$8$	$46.23$	$64.23$	$45.43$	$51.96$

Appendix

Table 10: Ablation of rollout budgets and sampling configurations on Qwen3-8B.

Method	Medical QA (In & Cross Domain)			Instruct Following (Out Of Domain)
Method	HealthBench	RaR-Medicine	LLMEval-Med	InfoBench	FollowBench	AdvancedIF
Qwen3-14B	$28.87$	$65.59$	$50.22$	$79.24$	$55.85$	$40.08$
RaR	$46.86$	$70.61$	$54.57$	$80.00$	$57.68$	$39.97$
RuscaRL	$47.91$	$70.02$	$52.02$	$79.91$	$52.31$	$32.46$
Online Rubrics	$50.50$	$71.12$	$58.77$	$79.68$	$56.95$	$37.73$
CoCA (Ours)	$\mathbf{52.55}$	$\mathbf{72.20}$	$\mathbf{61.02}$	$\mathbf{80.40}$	$\mathbf{62.16}$	$\mathbf{41.80}$

Appendix

Table 11: Scaling CoCA to Qwen3-14B on medical Domain and OOD instruction following.

Distillation Trajectory	Privileged Information	HealthBench	LLMEval.
Off-policy	Learnable Rubric	$42.02$	$48.13$
On-policy	Learnable Rubric	$43.82$	$51.57$
On-policy	+ Off-policy Demo.	$\mathbf{46.86}$	$\mathbf{52.77}$

Appendix

Table 12: Ablation study of the distillation trajectory and privileged information for self-distillation on Qwen3-8B.

Model	Gating	HealthBench	LLMEval.
Qwen3-4B	$\times$	$40.60$	$43.48$
Qwen3-4B	$\checkmark$	$41.01$	$46.48$
Qwen3-8B	$\times$	$45.40$	$50.82$
Qwen3-8B	$\checkmark$	$46.86$	$52.77$
Gemma-3-4B-it	$\times$	$50.92$	$24.14$
Gemma-3-4B-it	$\checkmark$	$52.71$	$26.54$

Appendix

Table 13: Ablation of dynamic gating for Guided GRPO.

\checkmark

denotes dynamic activation, while

\times

denotes always-active Guided GRPO.

GRPO Variants	HealthBench	LLMEval.
GSPO ( Zheng et al., 2025 )	$44.26$	$50.22$
w/ CoCA (Ours)	$48.47$	$51.57$
DAPO ( Yu et al., 2026 )	$45.20$	$51.27$
w/ CoCA (Ours)	$46.70$	$53.37$
CISPO ( Chen et al., 2025 )	$42.35$	$49.33$
w/ CoCA (Ours)	$46.63$	$51.87$

Appendix

Table 14: Extension to other GRPO variants on Qwen3-8B.

Figure 5: Query-level OPSD activation ratio during training on Qwen3-8B.

Figure 6: Training-time breakdown of Qwen3-8B on HealthBench over

500

steps, excluding evaluation and checkpoint saving. We decompose the total training time into distinct phases: rollout generation, reward and rubric evaluation, log-probability computation, and actor parameter updates. “*”: the training time measured through step

250

Dataset	Intent Preservation Rate	Difficulty Direction Accuracy
HealthBench	$98.37$	$98.89$
ResearchQA	$94.13$	$98.91$
CPsyCoun	$95.12$	$93.10$

Appendix

Table 15: Intent preservation rate and difficulty direction accuracy across datasets.

Figure 7: Within-step effects of rubric calibration on the same query-only responses during HealthBench training. We measure the fraction of discriminative criteria (top) and the mean within-group reward standard deviation (bottom). Faint lines denote per-step values; solid lines denote 25-step trailing weighted means.

Backbone	Post-Rewrite	Nonzero-group Coverage	Zero-variance
Backbone	Activation	Before $\rightarrow$ After	Group Recovery
Qwen3-4B	$15.76$	$78.14\rightarrow 87.78$	$44.54$
Qwen3-8B	$13.68$	$74.78\rightarrow 85.27$	$42.06$
Gemma3-4B	$14.48$	$78.27\rightarrow 87.68$	$43.77$

Appendix

Table 16: Effect of rubric calibration on query-only reward signals during Healthbench training (all values in %). “Post-Rewrite Activation”: the fraction of rewritten criteria that become discriminative. “Nonzero-group Coverage”: the fraction of groups with non-identical rewards; “Zero-variance Group Recovery”: the fraction of initially zero-variance groups that acquire reward variation.

Figure 8: Case study of selective rubric calibration on HealthBench.

Figure 9: Prompt template for polarity-aware rewriting of positive-weight rubric criteria during capability-aware calibration.

Figure 10: Prompt template for polarity-aware rewriting of negative-weight rubric criteria during capability-aware calibration.

Figure 11: Prompt template used by LLM-Rank for rubric-free listwise judging in the medical domain.

Figure 12: Prompt template used by the rubric judge to evaluate each response against every rubric criterion independently.

Figure 13: Prompt template for using gpt-5.6-sol (xhigh) to augment CPsyCoun samples with HealthBench-style rubrics to construct psychology-domain training data.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 438 papers

EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
May 28, 2026Xin Guan, Xiaomeng Hu, Shen Huang +6Open-Ended Generation RL for Language Models
Tongyi Lab, Alibaba Group
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6Automatic Rubric Generation RL for Language Models
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Oct 13, 2026Guangcheng Zhu, Yingfan Ma, Xing Zheng +7RL for Language Models Rubric-Based RL
Zhejiang University · Ant Group

May 28, 2026J/K move · Enter open · S save

EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation

Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang+5

Tongyi Lab, Alibaba Group

Open-Ended Generation RL for Language Models Reinforcement Learning Rubric-Based RL Multi-Objective Language Model Alignment Rubric-Based Evaluation+2

Open paper AskPDF arXiv

Reinforcement Learning (RL) has advanced Large Language Models (LLMs) in verifiable domains, while open-ended generation remains challenging due to the absence of definitive rewards. Rubric-based RL provides explicit evaluation criteria, but learning to construct these criteria remains challenging when final-answer correctness is not verifiable. We propose EvoRubric, a co-evolutionary RL framework that combines criterion-validity feedback, response discrimination, and peer agreement to learn rubrics for open-ended generation. A shared policy acts as both a Reasoner and a Rubric Generator, using its current responses and historical rubrics to discover new evaluation dimensions. To combine adaptive rubric discovery with a stable validity check, a frozen copy of the initial policy serves as the Meta-Verifier, while a frozen Grader scores responses against the retained criteria. Discriminative feedback, Leave-One-Out peer consensus, and a persistent memory pool transform this feedback into complementary rewards that jointly optimize both policy roles, closing the loop between response improvement and rubric discovery. EvoRubric improves over matched static and external evolving-rubric baselines across five benchmarks in Medical, Writing, and Science. Across three training seeds, it achieves five-benchmark averages of 56.28 at 8B and 61.13 at 14B, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively. Human audits assess criterion validity and response quality, and experiments with expert-initialized rubrics demonstrate compatibility with human priors.

Abstract

RL for Language Models

Explore similar work

EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning