RuACT: Recovering Inactive Rubrics as Adaptive Critics and Teachers
Organizations: Zhejiang University · Huawei
Abstract
Rubric-based reinforcement learning (RL) extends RL with verifiable rewards to open-ended tasks by scoring each response against query-specific rubrics. Under group-relative optimization, however, a rubric that every sampled response fails (unreachable) or passes (saturated) gives all responses the same outcome and no learning signal; in Qwen3-4B medical training, 60.88% of query-rubric observations are inactive. We find that the two states lack different supervision: stating an unreachable rubric in the prompt raises its satisfaction from 1.13% to 91.25%, whereas responses that all pass a saturated rubric still differ in quality. We therefore propose RuACT (Rubrics as Adaptive Critics and Teachers), which assigns each inactive rubric the role that supplies what it lacks. Rubric as Teacher uses unreachable rubrics as privileged information for reward-gated on-policy self-distillation, and Rubric as Critic decomposes saturated rubrics into child rubrics that separate current responses while sharing the parent's weight. RuACT outperforms GRPO, rubric reweighting, and rubric-guided exploration on all four medical and scientific benchmarks with Qwen3-4B and achieves the best average with Qwen3-1.7B. On initially unreachable rubrics, unguided recovery rises from 35.02% to 42.86%, while child refinement improves reward-quality alignment on independent rollouts from 57.74% to 60.77%. Code is available at https://github.com/ZJU-REAL/RuACT.
Figures & tables
| Medicine | Science | ||||||
| Model | Group | Method | RaR-Med. | HealthBench | RaR-Sci. | ResearchQA | Avg. |
| Qwen3-1.7B | Vanilla | Base | 38.57 0.00 | 23.18 0.00 | 52.56 0.00 | 50.51 0.00 | 41.21 0.00 |
| Baselines | GRPO | 43.26 4.69 | 26.45 3.27 | 55.67 3.11 | 54.79 4.28 | 45.04 3.84 | |
| Reweight | 42.60 4.03 | 25.61 2.43 | 56.77 4.21 | 56.11 5.60 | 45.27 4.07 | ||
| RuscaRL | 43.39 4.82 | 23.95 0.77 | 56.41 3.85 | 59.37 8.86 | 45.78 4.58 | ||
| OPSD ∗ | 26.30 | 38.50 | 39.60 | 38.40 | 35.70 | ||
| Scoring rule | Quality alignment (%) | Reward coverage (%) |
|---|---|---|
| parent rubric-only | 57.74 | 65.92 |
| w/ Decomposed | 60.77 | 88.85 |
| w/ Shuffled Decomposed | 53.62 | 90.44 |
| Method | RaR-Med. | HealthBench | Avg. |
|---|---|---|---|
| Base | 50.38 0.00 | 35.45 0.00 | 42.92 0.00 |
| GRPO | 53.55 3.17 | 37.62 2.17 | 45.59 2.67 |
| DAPO-style | 56.29 5.91 | 39.27 3.82 | 47.78 4.87 |
| GRPO+OPSD | 55.59 5.21 | 38.63 3.18 | 47.11 4.20 |
| OPSD ∗ | 42.00 | 42.10 | 42.05 |
| RuACT | 56.34 5.96 | 40.12 4.67 | 48.23 5.32 |
| MT-Bench | FollowBench | ||||
|---|---|---|---|---|---|
| Model | Method | R1 | R2 | HSR | SSR |
| Qwen3-1.7B | Base | 7.13 | 5.90 | 43.52 | 56.44 |
| RuACT | 7.66 | 6.03 | 45.37 | 57.69 | |
| Qwen3-4B | Base | 8.31 | 7.60 | 58.50 | 67.28 |
| RuACT | 8.64 | 7.88 | 59.89 | 67.75 | |
| Variant | RaR-Med. | HealthBench | Avg. |
|---|---|---|---|
| RuACT | 56.34 | 40.12 | 48.23 |
| Rubric as Teacher | |||
| RuACT-Teacher | 55.15 | 38.57 | 46.86 |
| w/o Gate | 55.18 | 37.45 | 46.32 |
| Rubric as Critic | |||
| RuACT-Critic | 55.50 | 38.77 | 47.14 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Train | Evaluation |
|---|---|---|
| RaR-Medicine | 17,926 | 2,242 |
| RaR-Science | 18,302 | 2,292 |
| HealthBench | – | 500 |
| ResearchQA | – | 500 |
| Hyperparameter | Value |
|---|---|
| Training steps | 200 |
| Queries per batch | 64 |
| Responses per query, | 8 |
| PPO mini-batch size | 32 |
| PPO epochs per batch | 1 |
| Learning rate |
| Hyperparameter | Setting |
| Teacher | |
| Target rubrics per query | 3 |
| Guided responses per selected query | 1 |
| Effective (1.7B / 4B) | 0.10 / 0.05 |
| Coefficient warmup steps | 50 |
| Critic | |
| Medicine | Science | ||||||
| Model | Group | Approach | RaR-Med. | HealthBench | RaR-Sci. | ResearchQA | Avg. |
| Qwen3-1.7B | Vanilla | Base | 41.34 | 20.88 | 49.92 | 48.13 | 40.07 |
| Baselines | GRPO | 46.08 | 24.80 | 53.06 | 52.71 | 44.16 | |
| Reweight | 45.60 | 24.50 | 53.44 | 53.47 | 44.25 | ||
| RuscaRL | 46.51 | 21.29 | 53.13 | 58.44 | 44.84 | ||
| Ours | RuACT-Teacher | 46.44 | 23.69 | 54.78 | 55.71 | 45.15 | |
| ID | Rubric-relevant response excerpts | P | |||
|---|---|---|---|---|---|
| R1 | “ External Oblique : Contracts to pull the ribcage downward and inward (indirectly aiding expiration). … Internal Oblique : Also helps in pulling the ribcage inward. … Transverse Abdominis : Contracts to increase abdominal pressure, which helps push the diaphragm upward.” | 1 | 0 | 1 | 1 |
| R2 | “ Transverse Abdominis : Contracts to compress the abdominal cavity, helping push the diaphragm upward. … Oblique Muscles (Internal and External Obliques): Contract to stabilize the abdominal wall and assist in forced expiration.” | 1 | 0 | 0 | 0 |
| R3 | “ External and Internal Obliques : Contract to compress the abdominal cavity, further helping to expel air from the lungs. … Transverse Abdominis : Contracts to increase intra-abdominal pressure, assisting in expiration.” | 1 | 1 | 1 | 1 |
| R4 | “ External Oblique : Contracts to pull the ribcage inward and compress the abdominal cavity. … Internal Oblique : Similarly contributes to abdominal compression. … Transversus Abdominis : Contracts to increase abdominal pressure, helping push the diaphragm up.” | 1 | 0 | 1 | 1 |
| R5 | “ External Oblique and Internal Oblique : Contract to compress the abdominal cavity, helping push the diaphragm upward. … Transversus Abdominis : Contracts to increase abdominal pressure, assisting in expelling air.” | 1 | 0 | 0 | 0 |
| R6 | “ External and Internal Oblique Muscles : Contract to compress the abdominal cavity, further increasing intra-abdominal pressure and aiding in expiration. … Transversus Abdominis : Contracts to tighten the abdominal wall, assisting in expelling air from the lungs.” | 1 | 1 | 1 | 1 |
Rubric-Based RL
Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.
See all 112 papers