cs.LGOct 13, 2026
SaveCoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Organizations: Zhejiang University · Ant Group
Abstract
Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on open-ended tasks.
Figures & tables
Figure 1: (Left) The mismatch between criterion-level evaluation and response-level learning signals in standard rubric-based RL. (Right) Training dynamics across methods on RaR-Medicine over the first 200 steps.
Figure 2: Overview of CoCA. Sampled responses are first evaluated against a shared rubric to derive signed criterion credits through within-group comparisons. Guided by relevance cues from criterion-conditioned distribution shifts, these credits are then allocated across tokens while preserving each criterion’s signed budget. Finally, the resolved credits are aggregated into token-level advantages for GRPO optimization.
| Method | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| Initial | 54.2 (0.0) | 41.9 (0.0) | 51.1 (0.0) | 72.0 (0.0) | 67.6 (0.0) | 50.7 (0.0) | 49.7 (0.0) | 55.3 (0.0) |
| RaR | 65.0 ( 10.8) | 47.5 ( 5.6) | 55.4 ( 4.3) | 73.8 ( 1.8) | 72.4 ( 4.8) | 56.6 ( 5.9) | 51.6 ( 1.9) | 60.3 ( 5.0) |
| OPSD | 44.7 ( 9.5) | 30.6 ( 11.3) | 48.8 ( 2.3) | 69.0 ( 3.0) | 65.5 ( 2.1) | 47.5 ( 3.2) | 46.6 ( 3.1) | 50.4 ( 4.9) |
| RaR+OPSD | 61.8 ( 7.6) | 42.1 ( 0.2) | 53.4 ( 2.3) | 72.1 ( 0.1) | 67.9 ( 0.3) | 50.6 ( 0.1) | 48.8 ( 0.9) | 56.7 ( 1.4) |
| RuscaRL | 58.4 ( 4.2) | 45.3 ( 3.4) | 55.6 ( 4.5) | 73.8 ( 1.8) | 67.9 ( 0.3) | 53.1 ( 2.4) | 44.4 ( 5.3) | 56.9 ( 1.6) |
Table 1: Experimental results (%) on medical and scientific benchmarks, using Qwen3-8B as the base model and gpt-oss-120b as the judge. Avg. denotes the mean score across all seven benchmarks, and the best results are highlighted in bold . Arrows indicate improvements ( ) or declines ( ) relative to the initial model.
| Method | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| CoCA | 69.8 (0.0) | 54.3 (0.0) | 61.8 (0.0) | 77.6 (0.0) | 74.2 (0.0) | 59.6 (0.0) | 54.4 (0.0) | 64.5 (0.0) |
| Uniform | 65.0 ( 4.8) | 47.5 ( 6.8) | 55.4 ( 6.4) | 73.8 ( 3.8) | 72.4 ( 1.8) | 56.6 ( 3.0) | 51.6 ( 2.8) | 60.3 ( 4.2) |
| Shuffled | 50.7 ( 19.1) | 39.4 ( 14.9) | 53.1 ( 8.7) | 69.2 ( 8.4) | 65.2 ( 9.0) | 50.5 ( 9.1) | 43.8 ( 10.6) | 53.1 ( 11.4) |
| Shared | 67.7 ( 2.1) | 50.1 ( 4.2) | 57.6 ( 4.2) | 73.9 ( 3.7) | 72.7 ( 1.5) | 57.7 ( 1.9) | 52.0 ( 2.4) | 61.7 ( 2.8) |
Table 2: Ablation results (%) for credit allocation mechanisms. Arrows indicate changes relative to CoCA.
Figure 3: Criterion-level correction ( ) and regression ( ) rates on RaR-Medicine and RaR-Science datasets. Correction measures the fraction of initially unmet criteria that become satisfied after training, while regression measures the fraction of initially satisfied criteria that become unmet.
| Measure | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| JSD (CoCA) | 69.8 (0.0) | 54.3 (0.0) | 61.8 (0.0) | 77.6 (0.0) | 74.2 (0.0) | 59.6 (0.0) | 54.4 (0.0) | 64.5 (0.0) |
| Forward KL | 60.8 ( 9.0) | 48.8 ( 5.5) | 57.3 ( 4.5) | 73.5 ( 4.1) | 70.3 ( 3.9) | 54.5 ( 5.1) | 49.1 ( 5.3) | 59.2 ( 5.3) |
| Reverse KL | 66.7 ( 3.1) | 49.7 ( 4.6) | 57.1 ( 4.7) | 75.9 ( 1.7) | 72.9 ( 1.3) | 57.2 ( 2.4) | 52.4 ( 2.0) | 61.7 ( 2.8) |
| Log-ratio | 53.4 ( 16.4) | 41.4 ( 12.9) | 50.8 ( 11.0) | 70.0 ( 7.6) | 66.9 ( 7.3) | 50.7 ( 8.9) | 46.3 ( 8.1) | 54.2 ( 10.3) |
Table 3: Ablation results (%) for relevance measures. Arrows indicate changes relative to CoCA.
Figure 4: (a) Rates of two signal conflicts: response-level advantages versus criterion-level credits, and distillation gradients versus outcome-grounded gradients. (b) Effects of and on average scores in each domain.
Figure 5: Visualization of criterion–token relevance in a medical response. Darker colors indicate higher JSD between next-token distributions with and without criterion conditioning.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Settings | Hyperparameters |
|---|---|
| CoCA | , (medicine), (science) |
| Sampling | rollout_n , top_k , top_p , temperature max_prompt_length , max_response_length enable_thinking |
| Training | ppo_mini_batch_size , ppo_micro_batch_size_per_gpu log_prob_micro_batch_size_per_gpu , train_batch_size AdamW, learning_rate , weight_decay schedule , warmup ratio , grad_clip ppo_epochs , clip_ratio , training_steps entropy_coeff , use_kl_loss , use_kl_in_reward |
| Evaluation | top_p , temperature , max_response_length |
| Optimizations | FSDP, vLLM (tensor parallel size ), FlashAttention-2 bf16 mixed precision, gradient checkpointing, padding removal Actor parameter and optimizer offload: disabled |
Table 4: Training and evaluation configuration.
Figure 6: Prompt template for the LLM judge. Blue placeholders denote the task prompt, model response, and numbered rubric criteria.
Figure 7: Prompt templates for criterion-conditioned forward passes in CoCA. Blue placeholders denote the original final user message and the selected criterion.
| Method | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| Qwen3-4B | ||||||||
| Initial | 45.6 (0.0) | 37.8 (0.0) | 48.3 (0.0) | 66.0 (0.0) | 64.3 (0.0) | 43.8 (0.0) | 43.2 (0.0) | 49.9 (0.0) |
| RaR | 55.2 ( 9.6) | 43.6 ( 5.8) | 50.3 ( 2.0) | 66.2 ( 0.2) | 69.2 ( 4.9) | 48.3 ( 4.5) | 44.6 ( 1.4) | 53.9 ( 4.0) |
| RaR+OPSD | 52.4 ( 6.8) | 42.2 ( 4.4) | 49.1 ( 0.8) | 65.8 ( 0.2) | 68.0 ( 3.7) | 47.6 ( 3.8) | 43.7 ( 0.5) | 52.7 ( 2.8) |
| HeRL | 58.0 ( 12.4) | 45.7 ( 7.9) | 51.9 ( 3.6) | 68.7 ( 2.7) | 70.3 ( 6.0) | 49.0 ( 5.2) | 45.4 ( 2.2) | 55.6 ( 5.7) |
Table 5: Experimental results (%) for Qwen3-4B and Olmo3-7B-Instruct-SFT with gpt-oss-120b as the judge. Avg. denotes the mean across all seven benchmarks. The best results within each model block are highlighted in bold . Arrows indicate improvements ( ) or declines ( ) relative to the corresponding initial model.
| Method | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| Initial | 57.9 (0.0) | 52.1 (0.0) | 44.3 (0.0) | 72.0 (0.0) | 67.7 (0.0) | 51.1 (0.0) | 49.7 (0.0) | 56.4 (0.0) |
| RaR | 68.2 ( 10.3) | 57.5 ( 5.4) | 47.4 ( 3.1) | 73.8 ( 1.8) | 73.1 ( 5.4) | 58.0 ( 6.9) | 51.6 ( 1.9) | 61.4 ( 5.0) |
| OPSD | 48.9 ( 9.0) | 38.2 ( 13.9) | 42.9 ( 1.4) | 69.0 ( 3.0) | 66.1 ( 1.6) | 48.7 ( 2.4) | 46.6 ( 3.1) | 51.5 ( 4.9) |
| RaR+OPSD | 64.6 ( 6.7) | 52.3 ( 0.2) | 44.6 ( 0.3) | 72.1 ( 0.1) | 68.3 ( 0.6) | 51.2 ( 0.1) | 48.8 ( 0.9) | 57.4 ( 1.0) |
| RuscaRL | 60.1 ( 2.2) | 54.5 ( 2.4) | 44.9 ( 0.6) | 73.8 ( 1.8) | 66.4 ( 1.3) | 54.1 ( 3.0) | 44.4 ( 5.3) | 56.9 ( 0.5) |
Table 6: Experimental results (%) on medical and scientific benchmarks, using Qwen3-8B as the base model and DeepSeek-V4.1-Flash as the judge. Avg. denotes the mean score across all seven benchmarks, and the best results are highlighted in bold . Arrows indicate improvements ( ) or declines ( ) relative to the initial model.
Figure 8: Complete word-level annotation examples from Medicine (top) and Science (bottom), with the corresponding rubric criterion shown above each response. Dark and pale blue denote annotated and remaining words, respectively. Subword pieces are merged into words for display.
Figure 9: Criterion–token relevance over 1,000 annotated pairs. The scatterplots compare mean JSD at annotated and remaining positions; the dashed diagonal denotes equality. The right panel reports the percentage of pairs meeting each enrichment-ratio threshold.
Figure 10: Training efficiency of Qwen3-8B on Medicine using A100 GPUs. Left: validation performance versus training time. Middle: average training step time for all methods. Right: CoCA’s runtime breakdown, with the total step time shown at the center and component proportions listed below.
| Strategy | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| CoCA | 69.8 (0.0) | 54.3 (0.0) | 61.8 (0.0) | 77.6 (0.0) | 74.2 (0.0) | 59.6 (0.0) | 54.4 (0.0) | 64.5 (0.0) |
| Random | 67.3 ( 2.5) | 49.6 ( 4.7) | 56.1 ( 5.7) | 74.1 ( 3.5) | 73.4 ( 0.8) | 57.9 ( 1.7) | 51.3 ( 3.1) | 61.4 ( 3.1) |
| Top | 69.4 ( 0.4) | 52.3 ( 2.0) | 59.0 ( 2.8) | 76.8 ( 0.8) | 74.0 ( 0.2) | 58.6 ( 1.0) | 52.2 ( 2.2) | 63.2 ( 1.3) |
Table 7: Ablation results (%) for criterion selection strategies. Arrows indicate changes relative to CoCA.
Figure 11: Training dynamics of CoCA and RaR in Medicine (top) and Science (bottom). Columns show reward, response length, and policy entropy, with faint stepwise curves and bold -step moving averages.
| Method | Medicine | Science | Avg. | |||||
|---|---|---|---|---|---|---|---|---|
| RaR-Medicine | HealthBench | LLMEval-Med | MedQA | RaR-Science | ResearchQA | GPQA-D | ||
| RaR | 0.26 | 0.25 | 0.44 | 0.51 | 0.18 | 0.39 | 0.53 | 0.24 |
| CoCA | 0.31 | 0.22 | 0.37 | 0.45 | 0.29 | 0.30 | 0.48 | 0.17 |
Table 8: Standard deviations ( ) of benchmark scores across three random seeds. Avg. reports the standard deviation of the per-seed average score over all seven benchmarks.
More in this topic
RL for Language Models
Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.
See all 433 papers