Organizations: University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · Nanjing University · National University of Singapore
Rubric-based reinforcement learning (RL) offers interpretable, fine-grained supervision for aligning large language models (LLMs) on open-ended generation tasks. However, static rubric criteria may become saturated or remain beyond the policy's exploration, rendering the reward signals insufficiently discriminative. While recent efforts seek to guide sampling exploration or evolve rubrics, they fail to explicitly factor in how rubric difficulty should track the evolving policy's capabilities, resulting in a persistent mismatch between the optimization objective and the policy's current capability frontier. In this paper, we propose Co-Evolving Policy and Rubric via Capability Awareness (CoCA), a closed-loop framework that jointly adapts the rubric and policy through criterion-level diagnosis. Specifically, CoCA contrasts query-only and rubric-guided rollouts to characterize each criterion relative to the current policy, recalibrates criteria with mismatched difficulty, and identifies guidance-revealed capability gaps for frontier-based learning. Based on this insight, CoCA either expands rubric-conditioned frontiers through guided optimization or transfers qualified gains through on-policy self-distillation. As the policy updates, CoCA improves the capability diagnosis and enables more reliable distillation, thereby completing the co-evolution loop. Extensive experiments across diverse benchmarks and LLMs demonstrate that CoCA consistently improves performance over rubric-based RL methods. Moreover, CoCA promotes the internalization of guidance-revealed capabilities into query-only behavior while mitigating capability degradation.
Figures & tables
Figure 1: Motivation and overview of CoCA. (a) Static criteria may become saturated or remain unmet, limiting reward discriminativeness. (b) Query-only failures can conceal capabilities elicitable under rubric guidance. (c) Distinct from existing rubric-based RL methods, CoCA tracks the policy’s capability boundary to couple rubric adaptation and policy learning in a closed loop.
Figure 2: Pipeline overview of our CoCA. Dual-Group rollouts contrast query-only and rubric-conditioned behavior to diagnose criterion-level capabilities, guiding capability-aware rubric calibration and frontier selection. Normal GRPO remains active for every query, complemented by Frontier-Guided OPSD when a qualified frontier is available. Otherwise, Guided GRPO expands the rubric-conditioned frontier, supplying improved responses for the preceding two stages to close the policy–rubric co-evolution loop. The algorithm pseudocode of CoCA is provided in Appendix B .
Method
Medical Domain
Science Domain
Psychological Domain
Mean
HealthBench
RaR-Medicine
LLMEval-Med
ResearchQA
RaR-Science
CPsyCoun
HeartBench
Qwen3-4B
22.38
51.78
38.38
59.93
69.76
31.39
44.17
45.40
LLM-Rank
30.10
52.54
42.13
73.09
74.91
33.56
43.54
49.98
RaR
37.12
56.03
42.58
78.42
73.81
77.41
50.14
59.36
RuscaRL
40.44
58.37
43.63
79.57
73.12
82.03
45.74
60.41
Online Rubrics
37.77
56.28
44.68
78.96
74.35
79.55
44.92
59.50
Table 1: Main results across seven benchmarks spanning the medical, science, and psychological domains. The best scores are highlighted in bold , while the second-best scores are underlined . All scores are normalized to 0∼100 . Additional experiments extending CoCA to larger model and other GRPO variants are presented in Appendix D.2 and Appendix D.5 , respectively.
Dataset
Qwen3-4B
Qwen3-8B
Gemma-3-4B-it
Base
RuscaRL
CoCA
Base
RuscaRL
CoCA
Base
RuscaRL
CoCA
HealthBench †
62.96
75.54
80.34
71.83
81.13
83.02
65.50
74.94
77.79
+12.58
+17.38
+9.30
+11.19
+9.44
+12.29
InfoBench
77.68
75.68
79.02
78.66
78.04
80.40
76.66
76.44
77.11
−2.00
+1.34
−0.62
+1.74
−0.22
+0.45
FollowBench
47.92
45.73
51.58
52.19
48.04
52.56
59.51
58.65
60.73
Table 2: Experiment results of rubric-conditioned in-domain capability and out-of-domain instruction-following generalization. All compared models are trained on HealthBench. † : We provide the full ground-truth rubric together with each HealthBench-Hard query.
Eval Judge
Method
HealthBench
RaR-Med.
LLMEval.
Mean
Qwen3.7-Flash ( Qwen Team, 2026 )
Base
27.31
54.56
59.22
47.03
RaR
43.04
60.06
62.52
55.21
CoCA
47.89
61.35
65.53
58.26
Kimi-K2.5 ( Team et al., 2026 )
Base
17.93
55.29
25.04
32.75
RaR
36.77
61.79
28.04
42.20
CoCA
41.59
63.02
30.13
44.91
Table 3: Ablation of evaluation judge models for Qwen3-8B in the medical domain.
CARC
LFrontier-OPSD
LGRPOG
HealthBench
LLMEval.
×
×
×
41.88
49.93
✓
×
×
46.30
48.73
×
✓
×
42.39
50.52
✓
✓
×
45.01
51.12
✓
×
✓
44.47
52.62
×
✓
✓
43.11
50.97
Table 4: Ablation study of the CoCA’s components on Qwen3-8B.
Figure 3: Overall distribution of rubric capability states on HealthBench-Hard. For each query, we sample two groups of responses: one under the query-only condition and the other with the full rubric additionally provided. Each rubric is then assigned to one of five capability states according to its satisfaction patterns across the two groups.
Table 5: Rubric capability-state transitions on HealthBench-Hard. For each backbone, we record each rubric’s Source State under the Base model and its Target State under a HealthBench-trained model, and report the types and proportions (%) of the resulting transitions.
Figure 4: Training dynamics and inference-time scaling of CoCA on HealthBench. (a) CoCA maintains higher entropy, preserving exploration during training; (b) CoCA matches RaR’s best performance with fewer training steps and time ( Appendix E.2 ); (c) its consistent advantage over the base model and RaR across K indicates a higher capability ceiling accessible through sampling.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Learning Mechanisms
Decision Coupling
Adaptive Rubrics
Token-level Signals
State-based Learning
RaR ( Gunjal et al., 2026 )
×
×
×
Fixed Rubric → Policy Rewards
RuscaRL ( Zhou et al., 2026 )
×
×
×
Scheduled Rubric Guidance
Online Rubrics ( Rezaei et al., 2026b )
✓
×
×
Response Contrasts → Criteria
RGSD ( Rezaei et al., 2026a )
×
✓
×
Fixed Rubric → Teacher Signals
CriPO ( Xia et al., 2026 )
×
✓
✓
Criterion States → Injection and Advantage Flipping
Appendix
Table 6: Comparison of representative rubric-based methods. Adaptive rubrics respond to policy feedback. Token-level signals denote rubric-informed supervision beyond scalar advantages. State-based learning uses criterion diagnostics to select policy-update interventions.
Domain
Dataset
Split
# Samples
# Avg. Criteria
Metric
Medical Domain
HealthBench
Train/Test
4,000/1,000
11.35/11.85
Rubric Score
RaR-Medicine
Test
2,242
9.59
Rubric Score
LLMEval-Med
Test
667
–
Overall Performance (OP)
Science Domain
ResearchQA
Train/Valid
4,000/500
7.53/7.46
Rubric Score
RaR-Science
Test
500
7.46
Rubric Score
Psychological Domain
CPsyCoun
Train/Test
2,400/500
8.48/8.46
Rubric Score
Appendix
Table 7: Statistics of the datasets used in our experiments. “# Avg. Criteria” counts rubric items or instruction constraints per sample.
Table 12
Method
Primary
Additional
Total
HealthBench
RaR-Medicine
LLMEval-Med
Mean
Rollouts
Rollouts
Rollouts
RaR ( Gunjal et al., 2026 )
4
–
4
41.88
64.29
49.93
52.03
8
–
8
45.51
64.92
50.37
53.60
12
–
12
46.91
65.75
50.82
54.49
RuscaRL ( Zhou et al., 2026 )
4
–
4
46.14
64.07
45.28
51.83
8
–
8
46.23
64.23
45.43
51.96
Appendix
Table 10: Ablation of rollout budgets and sampling configurations on Qwen3-8B.
Method
Medical QA (In & Cross Domain)
Instruct Following (Out Of Domain)
HealthBench
RaR-Medicine
LLMEval-Med
InfoBench
FollowBench
AdvancedIF
Qwen3-14B
28.87
65.59
50.22
79.24
55.85
40.08
RaR
46.86
70.61
54.57
80.00
57.68
39.97
RuscaRL
47.91
70.02
52.02
79.91
52.31
32.46
Online Rubrics
50.50
71.12
58.77
79.68
56.95
37.73
CoCA (Ours)
52.55
72.20
61.02
80.40
62.16
41.80
Appendix
Table 11: Scaling CoCA to Qwen3-14B on medical Domain and OOD instruction following.
Distillation Trajectory
Privileged Information
HealthBench
LLMEval.
Off-policy
Learnable Rubric
42.02
48.13
On-policy
Learnable Rubric
43.82
51.57
On-policy
+ Off-policy Demo.
46.86
52.77
Appendix
Table 12: Ablation study of the distillation trajectory and privileged information for self-distillation on Qwen3-8B.
Model
Gating
HealthBench
LLMEval.
Qwen3-4B
×
40.60
43.48
✓
41.01
46.48
Qwen3-8B
×
45.40
50.82
✓
46.86
52.77
Gemma-3-4B-it
×
50.92
24.14
✓
52.71
26.54
Appendix
Table 13: Ablation of dynamic gating for Guided GRPO. ✓ denotes dynamic activation, while × denotes always-active Guided GRPO.
GRPO Variants
HealthBench
LLMEval.
GSPO ( Zheng et al., 2025 )
44.26
50.22
w/ CoCA (Ours)
48.47
51.57
DAPO ( Yu et al., 2026 )
45.20
51.27
w/ CoCA (Ours)
46.70
53.37
CISPO ( Chen et al., 2025 )
42.35
49.33
w/ CoCA (Ours)
46.63
51.87
Appendix
Table 14: Extension to other GRPO variants on Qwen3-8B.
Figure 5: Query-level OPSD activation ratio during training on Qwen3-8B.
Figure 6: Training-time breakdown of Qwen3-8B on HealthBench over 500 steps, excluding evaluation and checkpoint saving. We decompose the total training time into distinct phases: rollout generation, reward and rubric evaluation, log-probability computation, and actor parameter updates. “*”: the training time measured through step 250 .
Dataset
Intent Preservation Rate
Difficulty Direction Accuracy
HealthBench
98.37
98.89
ResearchQA
94.13
98.91
CPsyCoun
95.12
93.10
Appendix
Table 15: Intent preservation rate and difficulty direction accuracy across datasets.
Figure 7: Within-step effects of rubric calibration on the same query-only responses during HealthBench training. We measure the fraction of discriminative criteria (top) and the mean within-group reward standard deviation (bottom). Faint lines denote per-step values; solid lines denote 25-step trailing weighted means.
Backbone
Post-Rewrite
Nonzero-group Coverage
Zero-variance
Activation
Before → After
Group Recovery
Qwen3-4B
15.76
78.14→87.78
44.54
Qwen3-8B
13.68
74.78→85.27
42.06
Gemma3-4B
14.48
78.27→87.68
43.77
Appendix
Table 16: Effect of rubric calibration on query-only reward signals during Healthbench training (all values in %). “Post-Rewrite Activation”: the fraction of rewritten criteria that become discriminative. “Nonzero-group Coverage”: the fraction of groups with non-identical rewards; “Zero-variance Group Recovery”: the fraction of initially zero-variance groups that acquire reward variation.
Figure 8: Case study of selective rubric calibration on HealthBench.
Figure 9: Prompt template for polarity-aware rewriting of positive-weight rubric criteria during capability-aware calibration.
Figure 10: Prompt template for polarity-aware rewriting of negative-weight rubric criteria during capability-aware calibration.
Figure 11: Prompt template used by LLM-Rank for rubric-free listwise judging in the medical domain.
Figure 12: Prompt template used by the rubric judge to evaluate each response against every rubric criterion independently.
Figure 13: Prompt template for using gpt-5.6-sol (xhigh) to augment CPsyCoun samples with HealthBench-style rubrics to construct psychology-domain training data.
Reinforcement Learning (RL) has advanced Large Language Models (LLMs) in verifiable domains, while open-ended generation remains challenging due to the absence of definitive rewards. Rubric-based RL provides explicit evaluation criteria, but learning to construct these criteria remains challenging when final-answer correctness is not verifiable. We propose EvoRubric, a co-evolutionary RL framework that combines criterion-validity feedback, response discrimination, and peer agreement to learn rubrics for open-ended generation. A shared policy acts as both a Reasoner and a Rubric Generator, using its current responses and historical rubrics to discover new evaluation dimensions. To combine adaptive rubric discovery with a stable validity check, a frozen copy of the initial policy serves as the Meta-Verifier, while a frozen Grader scores responses against the retained criteria. Discriminative feedback, Leave-One-Out peer consensus, and a persistent memory pool transform this feedback into complementary rewards that jointly optimize both policy roles, closing the loop between response improvement and rubric discovery. EvoRubric improves over matched static and external evolving-rubric baselines across five benchmarks in Medical, Writing, and Science. Across three training seeds, it achieves five-benchmark averages of 56.28 at 8B and 61.13 at 14B, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively. Human audits assess criterion validity and response quality, and experiments with expert-initialized rubrics demonstrate compatibility with human priors.
Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.
Hongxin Ding, Baixiang Huang, Yue Fang +6
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on open-ended tasks.