LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

Large language model (LLM) agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a harness tuned for one checkpoint may not target the failure modes of an updated policy, while policy learning remains limited to the trajectories elicited by a fixed harness. In light of this, we propose Self-HPO, a recursive self-improving framework for coding agents that alternates harness and policy optimization, using the evolving policy as its own harness proposer. Specifically, in each round, the current policy first analyzes its own failures and proposes new harness candidates (H-step). The policy is then updated through reinforcement learning (RL) under the selected harness (P-step), and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy's evolving failure modes. Across three coding benchmarks, Self-HPO consistently outperforms harness-only and policy-only baselines, improving Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains generalizing to out-of-distribution settings. Further analysis shows that both optimized components retain gains even when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains. Self-HPO also improves reliability (Pass@1) and expands the capability frontier (Pass@k) over the base agent. Additionally, we show that with a stronger external model (e.g., Opus 4.8) as the harness proposer, the same framework yields a further performance boost.

Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.

Recent systems train coding agents with reinforcement learning (RL) on trajectories collected through multiple execution harnesses. However, gains under the training harnesses leave open whether the improvements transfer to unseen interfaces and how they depend on reward normalization. We present a controlled study that measures RL gains under fixed harnesses and compares the same models across harnesses. We train two Qwen3-8B RL variants from the same supervised fine-tuning (SFT) checkpoint using identical replay data. They differ only in whether rewards for each task are normalized within each harness (Within) or across harnesses (Cross). On SWE-bench Verified, we identify three main findings: (i) Multi-harness RL yields modest gains over the SFT baseline under both normalization rules. Across the four training harnesses, the estimated mean improvements are +0.48 and +0.77 percentage points for Within and Cross, respectively. (ii) Cross increases the fraction of trajectories with nonzero advantage but provides no consistent advantage over Within on an unseen harness across training seeds and evaluation budgets; neither rule significantly improves over SFT on this harness. (iii) Performance varies substantially more across harnesses than with RL training. After averaging over model variants, the highest and lowest source-harness solve rates differ by 7.13 percentage points, far exceeding the estimated training gains. Even after multi-harness RL, the observed performance differences across execution interfaces remain substantially larger than either the gains from training or the differences between normalization rules.