LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

Yiming Du; Yuxin Jiang; Tao Yuan; Jianbo Dai; Shaowei Wang; Jierun Chen; Chaofan Tao; Xianzhi Yu

cs.AIAug 18, 2026

LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

Authors: Yiming Du, Yuxin Jiang, Tao Yuan, Jianbo Dai, Shaowei Wang, Jierun Chen, Chaofan Tao, Xianzhi Yu,

Organizations: Huawei Technologies Co., Ltd · The Chinese University of Hong Kong

Abstract

Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.

Figures & tables

Figure 1: Overview of the Lego-RL training infrastructure.

	Harness-native fidelity				Execution & reward			Observability
Framework	Black-box harness	Token-in/ Token-out	History alignment	R3	Fully async	Sandbox execution	Reward-hack defense	Training observability
verl ( 2025 )	–	$\checkmark$	–	$\checkmark$	$\checkmark$	$\triangle$	–	$\triangle$
slime ( 2025 )	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\triangle$	$\triangle$	–
MOLT ( 2026 )	$\triangle$	$\checkmark$	–	$\checkmark$	$\checkmark$	–	–	–
SkyRL-Agent ( 2025 )	$\triangle$	$\triangle$	$\triangle$	$\checkmark$	$\checkmark$	$\checkmark$	–	–
AReaL ( 2025 )	$\checkmark$	$\triangle$	–	–	$\checkmark$	–	–	–

Table 1: Comparison of representative agentic RL frameworks.

\checkmark

: supported;

\triangle

: partial/conditional support; –: not reported. R3: rollout routing replay. Observability denotes monitoring training runs and diagnosing execution- or trajectory-level failures.

Figure 2: Closed-loop operational workflow of Lego-RL . The five stages cover data preparation, run validation, training, live observability, and human review. Stage (3) corresponds to the training infrastructure shown in Figure 1 , while the agent plugin acts as the control plane.

Panel of Figure 3 — Figure 3: Training behavior of OpenHands SDK, Claude Code, and OpenCode over three epochs ( $126$ training steps), showing training reward, validation reward, policy entropy, and mean response length.

Coding agent	Model	SWE-bench Verified (%)
OpenHands SDK	Qwen3.5-35B-A3B ( Qwen Team, 2026a )	$64.0$
	Qwen3.6-35B-A3B ( Qwen Team, 2026b )	$67.4$
	KAT-Coder-V2.5-Dev ( KwaiKAT Team, 2026 )	$67.0$
	Lego-RL -Qwen3.5-35B-A3B	$\mathbf{70.4\ (+6.4)}$
Claude Code	Qwen3.5-35B-A3B ( Qwen Team, 2026a )	$62.4$
	Qwen3.6-35B-A3B ( Qwen Team, 2026b )	$63.4$

Table 2: SWE-bench Verified performance across the three coding agents. All numbers are measured by us under the same harness version and evaluation protocol (temperature 0.7, 200 turns, 200k context budget).

Figure 4: Trajectory termination profiles across agent scaffolds. Bars show the proportion of trajectories by termination reason; timeout and environment-setup failures are excluded from optimization.

Figure 5: In-batch reward distributions across training. (a–c) Distribution of tasks by the number of successful rollouts out of eight in the first and last epoch for each scaffold. The

0/8

and

8/8

groups provide no group-relative advantage. (d) Combined proportion of these zero-variation groups across epochs.

Figure 6: Task-selection ablation across four

951

-task pools. (a) Held-out validation solve rate, measured over the validation tasks that executed. (b) Training verifier reward; levels are pool-specific, so only the slopes are comparable.

			$\|\Delta\overline{\log p}\|$ per trajectory ( $\times 10^{-3}$ )
Coding agent	Pearson $r$	KL ( $\times 10^{-3}$ )	p50	p90	p99
OpenHands SDK	$0.9993$	$0.75$	$0.7$	$1.2$	$2.1$
Claude Code	$0.9980$	$1.35$	$0.7$	$1.3$	$2.7$
OpenCode	$0.9993$	$0.60$	$0.6$	$1.1$	$2.0$

Table 3: Rollout-to-training alignment over the three matched production runs: probabilities captured at the serving boundary against trainer-side recomputation over the corresponding assistant tokens. All statistics are medians over training steps.

Figure 7: Failure diagnosis with the Live UI. (a) Per-step termination reasons for an environment-failure run. (b) Assisted analysis of a collapsed run and the corresponding early-stop condition. (a) and (b) are two different diagnostic runs, neither is the Claude Code production run reported elsewhere in this section.

Figure 8: Behavioral analysis with the Live UI, Claude Code run. (a) Tool-use trajectories for eight rollouts of one task. (b) Task-level solve rates in the first and last sampled epochs.

Stage	Mean (s)	p50	p90	p99	Mean-time fraction
Sandbox setup	$21.6$	$7.7$	$41.8$	$275.2$	$2.3\%$
Agent setup	$4.2$	$4.0$	$4.8$	$8.4$	$0.5\%$
Agent execution	$840.5$	$708.4$	$1604.3$	$2801.1$	$91.3\%$
Verification	$35.9$	$4.1$	$29.6$	$928.5$	$3.9\%$
Other	$20.3$	$8.1$	$12.3$	$965.5$	$2.2\%$
Trial total	$920.4$	$770.0$	$1742.7$	$3189.1$	$100\%$

Table 4: Stage-wise wall-clock statistics across

3{,}699

OpenHands SDK training trials. Mean-time fraction is computed relative to the mean total trial duration; Other includes scheduling, trajectory handling, and sandbox cleanup.

Figure 9: Trainer schedule under synchronous and asynchronous execution.

Figure 10: Lazy versus full image delivery over the same

100

task images. Nydus streams image chunks on demand, whereas OCI denotes the conventional pull, which materializes the entire image before the container starts. The panels report startup latency, cumulative network and disk traffic, and in-container read throughput.

Optimization	Stage	With	Without	Median	p10–p90	$n$
		Median latency		Paired ratio
Lazy image pull	sandbox setup	$1.57$ s	$2.66$ s	$1.7\times$	—	$100$
Prebuilt task image	sandbox setup	$1.04$ s	$36.2$ s	$33.2\times$	$17.9$ – $67.5\times$	$50$
Mounted agent runtime	agent setup	$0.51$ s	$7.82$ s	$15.4\times$	$14.5$ – $16.6\times$	$50$
Packaged grading toolchain	verification	$3.81$ s	$2.72$ s	$0.71\times$	$0.67$ – $0.73\times$	$50$

Table 5: Ablation of sandbox optimizations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Failure Mode	Incidence	Defense
Agent-side: shortcut exploitation
Reads git history	$4.6$ – $20.5\%$	Rebase history to a single commit during agent phase; restore before grading
Downloads reference fix	$1.9\%$	Per-phase egress firewall in privilege-separated sidecar
Edits test files	$2.4$ – $19.4\%$	Withhold tests until grading; revert test-path edits
Environment-side: reward detached from agent
Grader applies reference patch	$2.5\%$	Audit affected instances out of pool; flag live if degenerate reward propagates

Appendix

Table 6: Reward-integrity failure modes and their mitigations. Incidence rates are measured prior to deploying the defenses; “—” indicates cases not separately quantified.

Routing replay configuration	Pearson $r$	Mean $\|\Delta p\|$	Expert overlap	Top-1 agreement
Disabled	$0.9946$	$0.0062$	—	—
Enabled, misaligned	$0.7503$	$0.0954$	$0.083$	$0.026$
Enabled, aligned	$0.9993$	$0.0025$	$0.996$	$0.985$

Appendix

Table 7: Routing-replay configurations compared. Expert overlap and top-1 agreement are undefined when replay is disabled.

Series	First five steps	Last five steps	$t$
training reward	$0.351$	$0.050$	$-7.5$
turns per trajectory	$18.9$	$0.96$	$-14.8$
response length (tokens)	$12{,}170$	$2{,}276$	$-7.7$
tokens per turn	$650$	$2{,}397$	—
policy entropy	$0.134$	$0.230$	$+7.2$
KL term of the loss	$0.0046$	$0.130$	$+3.4$

Appendix

Table 8: Evidence supplied for the collapsed run.

t

is the least-squares slope divided by its standard error over all logged steps; the gradient norm does not clear

|t|=2

and is therefore read as noise. Tokens per turn is a ratio of two rows above it and carries no separate trend statistic.

Figure 11: Reasoning share of the response over training on a fixed

120

-task validation cohort. (a) Mean over tasks (solid) and character-weighted mean (dashed); (b) Median share at each task’s first (open) and last (filled) sampled epoch, grouped by number of rollouts solved.

Figure 12: Agent behaviors before and after training, computed over

420

trajectories at each end of the production OpenHands SDK run. The pass@

k

and pass k rows are computed over prompt groups in the first and last third of the run.

Behavior	First	Last	$\Delta$
Reads back a file it edited	$73.6\%$	$98.1\%$	$+24.5$
Runs the test suite	$85.0\%$	$93.6\%$	$+8.6$
Files explored before 1st edit	$3.45$	$6.92$	$+3.47$
Ends with an explicit finish	$88.3\%$	$91.9\%$	$+3.6$
Reproduces failure before editing	$6.7\%$	$11.2\%$	$+4.5$
Solves despite a failed command	$63.9\%$	$66.8\%$	$+2.9$

Appendix

Table 9: Agent behaviors before and after training, over

420

trajectories at each end of the production OpenHands SDK run; the pass@

k

rows are over prompt groups in the first and last third of the run.

Panel of Figure 14 — Figure 14: Diagnostic panels from the same run. Top: termination-reason breakdown per step. Bottom: the train–inference consistency panel.

Policy loss	GSPO
Clip range (sequence-level)	$(3\times 10^{-4},\,4\times 10^{-4})$
Advantage estimator	GRPO
Loss aggregation	seq-mean-token-mean
KL reward penalty	none
KL loss coefficient	$10^{-3}$
Learning rate	$1\times 10^{-6}$

Appendix

Table 10: Resolved hyperparameters of the three production runs (Qwen3.5-35B-A3B through the OpenHands SDK, Claude Code, and OpenCode).

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

Self-Evolving Coding Agents via Harness-Policy Optimization
Oct 16, 2026Zhepei Wei, Wenlin Yao, Baolin Peng +5AI Coding Agents Agent Harness Optimization
Microsoft Research · University of Virginia · Carnegie Mellon University
Agent Lightning v1.0: Towards Harnessed Agentic RL
Aug 18, 2026Zhiyuan He, Siwei Zhang, Zhiwen Zhou +7Agentic RL LLM Agent Training
Microsoft · Fudan University · Zhejiang University +1
Multi-Harness Reinforcement Learning for Coding Agents: Training Gains and Harness Dependence
Sep 3, 2026Chenqian Le, Jiayi Cheng, Qijia He +3RL for Code Generation Group Relative Policy Optimization
New York University · Columbia University · Dimension Gate

Oct 16, 2026J/K move · Enter open · S save

Self-Evolving Coding Agents via Harness-Policy Optimization

Zhepei Wei, Wenlin Yao, Baolin Peng, Danqing Wang+4

Microsoft Research · University of Virginia · Carnegie Mellon University

AI Coding Agents Agent Harness Optimization Self-Evolving Agents LLM Agent Reliability RL for Code Generation

Open paper AskPDF arXiv

Large language model (LLM) agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a harness tuned for one checkpoint may not target the failure modes of an updated policy, while policy learning remains limited to the trajectories elicited by a fixed harness. In light of this, we propose Self-HPO, a recursive self-improving framework for coding agents that alternates harness and policy optimization, using the evolving policy as its own harness proposer. Specifically, in each round, the current policy first analyzes its own failures and proposes new harness candidates (H-step). The policy is then updated through reinforcement learning (RL) under the selected harness (P-step), and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy's evolving failure modes. Across three coding benchmarks, Self-HPO consistently outperforms harness-only and policy-only baselines, improving Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains generalizing to out-of-distribution settings. Further analysis shows that both optimized components retain gains even when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains. Self-HPO also improves reliability (Pass@1) and expands the capability frontier (Pass@k) over the base agent. Additionally, we show that with a stronger external model (e.g., Opus 4.8) as the harness proposer, the same framework yields a further performance boost.

Abstract

LLM Agent Training

Explore similar work

Self-Evolving Coding Agents via Harness-Policy Optimization

Agent Lightning v1.0: Towards Harnessed Agentic RL

Multi-Harness Reinforcement Learning for Coding Agents: Training Gains and Harness Dependence