Harness Scaling for Generalizable Agentic Reinforcement Learning

Fei Bai; Huatong Song; Chuan Hao; Ran Tao; Jian Yang; Bryan Dai; Wayne Xin Zhao; Ji-Rong Wen

cs.LGOct 21, 2026

Harness Scaling for Generalizable Agentic Reinforcement Learning

Authors: Fei Bai, Huatong Song, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Ji-Rong Wen

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University

Abstract

Agent harnesses have become a central component of modern agentic systems, shaping how models use tools, manage context, and coordinate long-horizon interaction. Realizing the benefits of these harnesses depends on how effectively the underlying model can make use of them, motivating reinforcement learning directly through the harness. While recent efforts have explored harness diversification to reduce specialization to particular execution settings, how to systematically construct and organize harness diversity for cross-harness generalization remains insufficiently understood. This work studies Harness Scaling as a distinct scaling dimension for agentic RL, focusing on structured variation in how policies execute and interact with the same underlying task and environment. Concretely, we construct a factorized harness space, select a compact training subset with balanced factorial coverage, and organize cross-harness training through locality-aware task assignment. Across multiple benchmarks, model scales, and RL algorithms, Harness Scaling consistently improves generalization to unseen harness combinations. Moreover, policies trained through structured white-box harness variation transfer effectively to unseen external black-box harnesses, outperforming policies trained through a single complex black-box harness. These results establish harness diversity as a meaningful scaling dimension for agentic RL and highlight the importance of how this diversity is organized for cross-harness generalization.

Figures & tables

Figure 1: Overview of Harness Scaling, from structured harness-space construction to cross-harness training and generalization to unseen configurations.

Method	Harness	ClawGym-Bench		JobBench-Easy		PinchBench		Avg.
Method	Harness	In-Space	Claude Code	In-Space	Claude Code	In-Space	Claude Code	Avg.
Qwen3-4B-Instruct
Base	–	18.08	33.90	7.16	6.78	32.69	32.74	21.89
GRPO	Single	37.33	38.96	10.24	5.79	45.60	40.95	29.81
GRPO	Scaling	52.78	45.37	13.55	14.83	56.24	51.47	39.04
PPO	Single	43.78	40.75	12.63	9.68	48.86	47.74	33.91

Table 1: Pass@1 performance across three benchmarks under different harness evaluation settings. In-Space reports the average over 18 held-out harness configurations, while Claude Code denotes out-of-space evaluation with the external black-box harness. Bold indicates the better result between single-harness training and harness scaling under the same model and optimization method.

Method	ClawGym-Bench	JobBench-Easy	PinchBench	Avg.
Qwen3-4B-Instruct
Black-Box SFT	44.48	11.10	49.66	35.08
Black-Box RL	44.10	9.42	46.61	33.38
Harness Scaling	45.37	14.83	51.47	37.22
Qwen3-30B-A3B-Instruct
Black-Box SFT	49.55	13.16	51.52	38.08

Table 2: Pass@1 performance on Claude Code as an unseen external black-box harness. Black-Box SFT and Black-Box RL are trained through OpenClaw, while Harness Scaling is trained over the structured white-box harness space. For the RL-based methods, both Black-Box RL and Harness Scaling use GRPO.

Panel of Figure 2 — Figure 2: Evaluation curves under in-space harness configurations on ClawGym-Bench.

Setting	ClawGym-Bench	PinchBench
Random-18	47.90 / 42.78	51.28 / 48.74
Random Assign.	50.46 / 43.17	53.20 / 50.23
Ours	52.78 / 45.37	56.24 / 51.47

Table 3: Ablation of structured selection and assignment on Qwen3-4B-Instruct. Each entry reports In-Space / Claude Code.

Panel of Figure 3 — Figure 3: Ablation studies of key design choices in Harness Scaling on Qwen3-4B-Instruct. (a–b) Performance across training harness sets of different sizes under white-box and external black-box (Claude Code) evaluation. The full 243-harness setting includes the white-box evaluation configurations and serves as a fixed-budget stress test rather than a held-out generalization setting. (c–d) Effect of the number of harnesses assigned per task during training under the same two evaluation settings.

Panel of Figure 4 — Figure 4: Behavioral analysis comparing Single-Harness RL and Harness Scaling. (a–b) Capability profiles under in-space evaluation. (c–d) Cross-harness reward consistency on the same tasks under two fixed held-out harnesses, Harness A ( S0T2O1C1W2 ) and Harness B ( S2T1O1C2W0 ); ellipses show 1.5 $\sigma$ covariance regions and large markers denote group means.

	Base Harness	In-Space	Claude Code
Single	59.82	48.12	46.02
Scaling	59.63	58.60	49.62
$\Delta$	$-0.19$	$+10.48$	$+3.60$

Table 4: Base-harness vs. cross-harness performance for Qwen3-30B-A3B-Instruct with GRPO on ClawGym-Bench.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dim.	Aspect	Base	Variant I	Variant II
$S$	System prompt	Direct system instruction specifying the agent role and execution rules.	The same requirements are embedded in a simulated product context, with additional platform and UI descriptions.	The requirements are reframed through a narrative and role-based setting.
$T$	Tool interface	Original tool names and schemas, such as execute_bash and file_editor , without distractors.	Functional tools are exposed through semantically related aliases, together with additional non-functional distractor tools.	Functional tools are exposed through opaque identifiers and augmented with distractor tools, largely removing lexical cues to tool functionality.
$O$	Observation	Raw tool outputs without additional prefixes or suffixes.	Outputs are augmented with simulated infrastructure metadata, including pod names, namespaces, and scheduling information.	Outputs are augmented with structured XML-style metadata, including tool name, exit code, process ID, duration, and execution node.
$C$	Context management	The complete interaction history remains available in the active context.	LLM-based compaction periodically summarizes earlier history while retaining the five most recent turns in context.	A more aggressive compaction policy retains the three most recent turns, with summarized history externalized to an auxiliary file.
$W$	Workflow	Standard free-form execution without additional procedural constraints.	A plan-first workflow requires explicit planning before task execution.	Planning is combined with periodic progress verification every three steps.

Appendix

Table 5: Harness component realizations instantiated by the model-driven transformation procedure for the

3^{5}=243

factorized harness space.

S0T0O0C0W0	S0T1O1C1W1	S0T2O2C2W2
S1T0O0C1W1	S1T1O1C2W2	S1T2O2C0W0
S2T0O1C0W2	S2T1O2C1W0	S2T2O0C2W1
S0T0O2C2W1	S0T1O0C0W2	S0T2O1C1W0
S1T0O1C2W0	S1T1O2C0W1	S1T2O0C1W2
S2T0O2C1W2	S2T1O0C2W0	S2T2O1C0W1

Appendix

Table 6: The 18 configurations in the structured training harness set

\mathcal{H}_{\mathrm{tr}}

Parameter	4B GRPO	4B PPO	30B-A3B GRPO	30B-A3B PPO
Prompt batch size	16	128	16	128
Rollouts per prompt	8	1	8	1
Actor mini-batch size	16	128	16	128
Actor learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$	$1\times 10^{-6}$	$1\times 10^{-6}$
Max. prompt length	8K	8K	4K	4K
Max. response length	32K	32K	60K	60K

Appendix

Table 7: Main training hyperparameters for GRPO and PPO experiments.

S0T2O1C1W2	S2T1O1C2W0	S0T1O0C2W2
S2T2O0C0W1	S0T1O2C0W1	S1T0O0C2W1
S0T0O0C1W2	S0T0O1C0W1	S0T2O2C2W0
S1T0O1C1W0	S1T0O2C2W2	S1T1O0C0W0
S1T2O1C0W2	S1T2O2C1W1	S2T0O2C0W0
S2T1O1C1W2	S2T1O2C2W1	S2T2O0C1W0

Appendix

Table 8: The 18 held-out configurations used for in-space evaluation.

Method	ClawGym-Bench	JobBench-Easy	PinchBench	Avg.
Qwen3-4B-Instruct
Single-Harness RL	43.18	7.67	45.20	32.02
Harness Scaling	46.84	11.25	50.11	36.07
Qwen3-30B-A3B-Instruct
Single-Harness RL	45.31	11.90	52.78	36.66
Harness Scaling	50.16	14.81	58.62	41.20

Appendix

Table 9: Out-of-space performance on OpenClaw as an additional external black-box harness. OpenClaw is not used during the training of Single-Harness RL or Harness Scaling. All results use GRPO-trained checkpoints.

Setting	ClawGym-Bench	PinchBench
Single-Harness RL	37.33 / 38.96	45.60 / 40.95
Harness Scaling, $K=1$	48.21 / 43.17	54.23 / 49.26
Repeat-2, Same Anchor	49.76 / 43.60	50.34 / 48.64
Harness Scaling, $K=2$	52.78 / 45.37	56.24 / 51.47
Harness Scaling, $K=4$	49.56 / 41.99	53.90 / 50.04

Appendix

Table 10: Effect of per-task harness diversity on Qwen3-4B-Instruct with GRPO. Repeat-2 matches the two-step task schedule of

K=2

while using the same sampled anchor harness for both steps. Each entry reports In-Space / Claude Code performance.

Setting	Harness	In-Space	Claude Code
Single-Harness RL	$h_{\mathrm{base}}$	31.06	28.57
Single-Harness RL	$h_{1}$	35.79	36.11
Single-Harness RL	$h_{2}$	33.52	33.11
Single-Harness RL	$h_{3}$	35.15	32.72
Harness Scaling	$\mathcal{H}_{\mathrm{tr}}$	40.86	37.22

Appendix

Table 11: Sensitivity of Single-Harness RL to the fixed training harness on Qwen3-4B-Instruct with GRPO. Results are averaged across the three benchmarks. Harness Scaling is included for reference.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 231 papers

ClawGym II: Exploring Black-Box RL on Agent Harness
Aug 17, 2026Huatong Song, Fei Bai, Ming Yang +17Reinforcement Learning RL for LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research
Harness-RL: Black-Box Reinforcement Learning with Action-Args Decoupling for Central-Agent Multi-Agent Harnesses
Aug 30, 2026Xinke Jiang, Zhixin Zhang, Zhibang Yang +6Multi-Agent LLM Systems Agent Harness Optimization
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +3
HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning
Sep 12, 2026Hongliang Wei, Xiaobing Tu, Yinggui Wang +6RL for LLM Agents Agent Harness Optimization
Harbin Institute of Technology · Alibaba Cloud

Aug 17, 2026J/K move · Enter open · S save

ClawGym II: Exploring Black-Box RL on Agent Harness

Huatong Song, Fei Bai, Ming Yang, Renyuan Li+16

Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research

Reinforcement Learning RL for LLM Agents Agent Harness Optimization LLM Agent Harnesses

Open paper AskPDF arXiv

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.

Abstract

Agentic RL

Explore similar work

ClawGym II: Exploring Black-Box RL on Agent Harness

Harness-RL: Black-Box Reinforcement Learning with Action-Args Decoupling for Central-Agent Multi-Agent Harnesses

HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning