HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

Xiaoqiang Wang; Ting-En Lin; Yuchuan Wu; Pinyi Zhang; Caixin Kang; Ling Li; Bingxue Qiu; Jieping Ye

cs.CLOct 19, 2026

HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

Authors: Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu, Pinyi Zhang, Caixin Kang, Ling Li, Bingxue Qiu, Jieping Ye,

Organizations: Alibaba Token Hub, Alibaba Group · Université de Montréal · Mila - Quebec AI Institute

Abstract

Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability. Under spurious rewards, higher scores do not correspond to better responses. Both reflect a training configuration whose task sampling, exploration, and evaluation stay fixed as the policy evolves. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces for task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory from which a harness evolver retrieves within-task history and cross-task evidence. Using this evidence, the evolver attributes each reward failure to the interface whose update can address it. It then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy, returning rejected candidates for revision. We also curate Rubric-Mix-10k, a five-domain training and evaluation suite for open-ended RL across medical, science, writing, role-play, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint. Code and data are available at https://github.com/Tongyi-ConvAI/Qwen-Character/tree/main/HarnessRL.

Figures & tables

Panel of Figure 1 — Figure 1: Rubric-reward failures under GRPO on Rubric-Mix-10k at Qwen3.5-4B , and harness-controlled RL training in HarnessRL : failure classes for one criterion (top left), class proportions and GRPO’s training reward against oracle quality on held-out tasks (bottom left), and harness $(p_{t},g_{t},c_{t})$ -controlled RL training (right). $S$ denotes the judge’s binary criterion scores and $Q$ the oracle ranking of the rollouts.

Figure 2: HarnessRL runs an RL loop conditioned on a training-time harness $\mathcal{H}{}_{t}=(p_{t},g_{t},c_{t})$ , which controls task sampling, rollout guidance and the rubric, respectively. The harness co-evolves with the policy: it retrieves evidence from the training-time memory $\mathcal{M}_{t}$ , attributes failures, and is iteratively refined by a proposer and JIT critic loop.

			Prompt	Seed rubric			Reference response
Domain	Train tasks	OOD benchmark	Tokens	Criteria	Tokens/crit.	Negative (%)	Tokens	Tasks (%)
Medical HealthBench, RaR, RubricHub	3,000	MedQA 250 items	121	12.4	43.4	21.0	322	67.7
Science ResearchQA, RaR, RubricHub	2,000	GPQA-Diamond 198 items	50	12.3	31.8	2.0	362	31.1
Writing LongWriter-Zero	1,200	WritingBench 250 items	161	10.5	21.6	28.1	–	0.0
Role-play RubricHub (chat)	1,200	RoleBench 250 items	85	26.5	31.5	0.0	1,205	24.2
Instruction following RubricHub (IF)	1,200	IFBench 250 items	211	3.6	52.0	0.0	528	5.6

Table 1: Domain composition of Rubric-Mix-10k . Each domain has

250

held-out tasks. Token counts are means under the Qwen tokenizer. The Tasks (%) column under Reference response reports the percentage of training tasks with a released reference response.

	In-domain held-out						Out of distribution
Method	Med.	Sci.	Writ.	Role	IF	Avg.	Med.	Sci.	Writ.	Role	IF	Avg.
Qwen3.5-4B
GRPO	52.4	57.2	60.7	50.8	68.1	57.8	40.5	29.3	66.5	50.7	32.7	43.9
RuscaRL	56.3	61.4	63.1	53.6	69.5	60.8	41.3	29.8	67.1	51.1	32.2	44.3
SkillRL	55.6	60.9	62.4	53.1	69.8	60.4	41.0	30.1	67.6	51.4	32.9	44.6
EvoRubric	55.0	60.5	64.8	54.2	69.1	60.7	43.4	31.2	69.1	53.1	34.3	46.2

Table 2: Quantitative results of HarnessRL on the five-domain Rubric-Mix-10k with Qwen3.5-4B and Qwen3.5-9B. Bold , underline , and green mark the best result, the second-best result, and HarnessRL ’s margin over the best baseline, respectively.

	In-domain held-out						Out of distribution
Method	Med.	Sci.	Writ.	Role	IF	Avg.	Med.	Sci.	Writ.	Role	IF	Avg.
HarnessRL	59.3	64.6	69.2	58.9	70.7	64.5	46.5	35.4	73.0	56.6	37.5	49.8
Component ablation
w/o critic	54.9	60.0	62.1	52.0	68.3	59.5	42.9	31.6	68.5	52.7	33.8	45.9
w/o critic & memory	53.6	58.6	60.8	50.7	67.0	58.1	41.3	29.2	66.9	51.0	32.2	44.1
Interface ablation

Table 3: Ablations of HarnessRL on its components and control interfaces at Qwen3.5-4B. Bold and underline mark the best and second-best results, respectively.

Figure 3: Proportion of pairs by failure class at Qwen3.5-4B during training.

Figure 4: Training dynamics of HarnessRL : proportion of tasks by guidance fields, accepted rubric updates per 100 steps, and reversed rubric updates, the last also for its ablations and AMARIS++.

Figure 5: Training efficiency of HarnessRL measured by cluster hours per 100 steps, decomposed into RL loop and harness evolver overhead (left), and cluster hours to the best OOD checkpoint (right).

Figure 6: Case analysis of a randomly sampled training task “Which properties of cellulose filter paper suit it to analytical and clinical use?” : task sampling and mounted guidance (top), rollout groups scored with the dynamically evolving rubric and oracle quality (middle), and steps at which the harness acted, with labels, rollout excerpts, rubric operations, and criteria met or missed (bottom). Dashed lines mark them.

Rubric-Based RL

Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.

See all 104 papers

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning
Aug 10, 2026Jinkun Hou, Zhuo Liu, Huimin Ren +3Open-Ended Generation RL for Language Models
1Peking University · 2Beijing Institute of Technology · 3Li Auto Inc.
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6Automatic Rubric Generation RL for Language Models
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning
Sep 1, 2026Fanrui Zhang, Ruixue Ding, Qiang Zhang +13AI Agent Benchmarks Rubric-Based RL
Alibaba ATH Token Foundry · Hema, Alibaba Group

Aug 10, 2026J/K move · Enter open · S save

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

Jinkun Hou, Zhuo Liu, Huimin Ren, Hongsheng Xin+2

1Peking University · 2Beijing Institute of Technology · 3Li Auto Inc.

Open-Ended Generation RL for Language Models RL Exploration Rubric-Based RL Rubric-Guided Generation Reward Design for RL+1

Open paper AskPDF arXiv

Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. We propose

\textbf{RISE-RL}

(Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and

\textbf{3.3 points at the 14B scale}

, including a

\textbf{6.0-point}

gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.

Abstract

Rubric-Based RL

Explore similar work

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning