Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability. Under spurious rewards, higher scores do not correspond to better responses. Both reflect a training configuration whose task sampling, exploration, and evaluation stay fixed as the policy evolves. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces for task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory from which a harness evolver retrieves within-task history and cross-task evidence. Using this evidence, the evolver attributes each reward failure to the interface whose update can address it. It then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy, returning rejected candidates for revision. We also curate Rubric-Mix-10k, a five-domain training and evaluation suite for open-ended RL across medical, science, writing, role-play, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint. Code and data are available at https://github.com/Tongyi-ConvAI/Qwen-Character/tree/main/HarnessRL.
Figures & tables
Figure 1: Rubric-reward failures under GRPO on Rubric-Mix-10k at Qwen3.5-4B , and harness-controlled RL training in HarnessRL : failure classes for one criterion (top left), class proportions and GRPO’s training reward against oracle quality on held-out tasks (bottom left), and harness (pt,gt,ct) -controlled RL training (right). S denotes the judge’s binary criterion scores and Q the oracle ranking of the rollouts.
Figure 2: HarnessRL runs an RL loop conditioned on a training-time harness Ht=(pt,gt,ct) , which controls task sampling, rollout guidance and the rubric, respectively. The harness co-evolves with the policy: it retrieves evidence from the training-time memory Mt , attributes failures, and is iteratively refined by a proposer and JIT critic loop.
Prompt
Seed rubric
Reference response
Domain
Train tasks
OOD benchmark
Tokens
Criteria
Tokens/crit.
Negative (%)
Tokens
Tasks (%)
Medical HealthBench, RaR, RubricHub
3,000
MedQA 250 items
121
12.4
43.4
21.0
322
67.7
Science ResearchQA, RaR, RubricHub
2,000
GPQA-Diamond 198 items
50
12.3
31.8
2.0
362
31.1
Writing LongWriter-Zero
1,200
WritingBench 250 items
161
10.5
21.6
28.1
–
0.0
Role-play RubricHub (chat)
1,200
RoleBench 250 items
85
26.5
31.5
0.0
1,205
24.2
Instruction following RubricHub (IF)
1,200
IFBench 250 items
211
3.6
52.0
0.0
528
5.6
Table 1: Domain composition of Rubric-Mix-10k . Each domain has 250 held-out tasks. Token counts are means under the Qwen tokenizer. The Tasks (%) column under Reference response reports the percentage of training tasks with a released reference response.
In-domain held-out
Out of distribution
Method
Med.
Sci.
Writ.
Role
IF
Avg.
Med.
Sci.
Writ.
Role
IF
Avg.
Qwen3.5-4B
GRPO
52.4
57.2
60.7
50.8
68.1
57.8
40.5
29.3
66.5
50.7
32.7
43.9
RuscaRL
56.3
61.4
63.1
53.6
69.5
60.8
41.3
29.8
67.1
51.1
32.2
44.3
SkillRL
55.6
60.9
62.4
53.1
69.8
60.4
41.0
30.1
67.6
51.4
32.9
44.6
EvoRubric
55.0
60.5
64.8
54.2
69.1
60.7
43.4
31.2
69.1
53.1
34.3
46.2
Table 2: Quantitative results of HarnessRL on the five-domain Rubric-Mix-10k with Qwen3.5-4B and Qwen3.5-9B. Bold , underline , and green mark the best result, the second-best result, and HarnessRL ’s margin over the best baseline, respectively.
In-domain held-out
Out of distribution
Method
Med.
Sci.
Writ.
Role
IF
Avg.
Med.
Sci.
Writ.
Role
IF
Avg.
HarnessRL
59.3
64.6
69.2
58.9
70.7
64.5
46.5
35.4
73.0
56.6
37.5
49.8
Component ablation
w/o critic
54.9
60.0
62.1
52.0
68.3
59.5
42.9
31.6
68.5
52.7
33.8
45.9
w/o critic & memory
53.6
58.6
60.8
50.7
67.0
58.1
41.3
29.2
66.9
51.0
32.2
44.1
Interface ablation
Table 3: Ablations of HarnessRL on its components and control interfaces at Qwen3.5-4B. Bold and underline mark the best and second-best results, respectively.
Figure 3: Proportion of pairs by failure class at Qwen3.5-4B during training.
Figure 4: Training dynamics of HarnessRL : proportion of tasks by guidance fields, accepted rubric updates per 100 steps, and reversed rubric updates, the last also for its ablations and AMARIS++.
Figure 5: Training efficiency of HarnessRL measured by cluster hours per 100 steps, decomposed into RL loop and harness evolver overhead (left), and cluster hours to the best OOD checkpoint (right).
Figure 6: Case analysis of a randomly sampled training task “Which properties of cellulose filter paper suit it to analytical and clinical use?” : task sampling and mounted guidance (top), rollout groups scored with the dynamically evolving rubric and oracle quality (middle), and steps at which the harness acted, with labels, rollout excerpts, rubric operations, and criteria met or missed (bottom). Dashed lines mark them.
Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. We propose RISE-RL (Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and 3.3 points at the 14B scale, including a 6.0-point gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.
Jinkun Hou, Zhuo Liu, Huimin Ren +3
1Peking University · 2Beijing Institute of Technology · 3Li Auto Inc.
Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.
Hongxin Ding, Baixiang Huang, Yue Fang +6
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Training open-ended agents via reinforcement learning (RL) is hindered by the lack of verifiable gold answers and scalable rubrics. Moreover, even near the model's capability boundary, long-horizon open-ended agentic tasks often yield brittle and unstable rewards, resulting in weak or noisy rollout contrast that obscures fine-grained optimization signals for group-based policy learning. To address these challenges, we propose ARISE-RL, a novel full-cycle self-evolution framework that couples a task/rubric Generator and a reasoning Solver through rubric-mediated co-evolution. The Generator grounds tool-related rubric criteria in real tool observations and is rewarded for producing valid, intermediate-difficulty tasks aligned with the Solver's evolving capability boundary. The Solver, in turn, learns from fine-grained rubric satisfaction signals through multi-step reasoning and tool use. We further introduce Reward-Gated Self-Evolution Distillation (RG-SED), which selectively distills a memory-augmented variant of the same policy back into itself only when the memory yields empirical reward improvement, thereby reducing distribution mismatch and avoiding blind imitation of noisy guidance. Finally, to support rigorous evaluation, we present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning. Extensive experiments demonstrate that ARISE-RL consistently achieves robust and stable overall state-of-the-art performance across all evaluated benchmarks.