Long-Horizon RL Needs Muon

Kai Ruan; Jinghao Lin; Tao Chen; Ziqi Zhou; Qianshan Wei; Zihe Huang

cs.LGOct 14, 2026

Long-Horizon RL Needs Muon

Authors: Kai Ruan, Jinghao Lin, Tao Chen, Ziqi Zhou, Qianshan Wei, Zihe Huang

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Duke University · Institute of Automation, Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences

Abstract

Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.

Figures & tables

Figure 1: Overview of the training recipe. (a) A software engineering agent inspects, edits, and tests a repository; final tests provide the binary outcome reward. (b) Length and tool-error penalties shape advantages, TIS reweights token-level policy losses, and token-level averaging aggregates the policy loss before the Muon update.

Figure 2: Recipe comparisons for the selected training runs in Figures 4 and 5 . Bars summarize unsmoothed training success rates. Pale green highlights the full recipe.

Figure 3: Selected training runs. Thin translucent lines show raw values; bold lines show exponential moving averages (span 5) from the first observation.

Figure 4: Training success across MiniCPM5-2B recipes: thin lines show unsmoothed success rates, bold lines show span-5 EMA, red marks collapse, green marks the full recipe.

Figure 5: Training success across Qwen3.5-4B recipes: thin lines show unsmoothed success rates, bold lines show span-5 EMA, red marks collapse, green marks the full recipe.

Long-Horizon RL

See all 32 papers

Selecting Long-Horizon Trajectories for Reliable and Efficient Terminal-Agent Training
Oct 5, 2026Cuong Dang, Hoang Anh Just, Ruoxi JiaSupervised Fine-Tuning Trajectory Generation
Department of Electrical and Computer Engineering Virginia Tech Blacksburg, VA, USA
Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents
Oct 4, 2026On Tai Tang, Huanyao Zhang, Jiepeng Zhou +1Reinforcement Learning Deep Research Agents
The Hong Kong University of Science and Technology (HKUST)
QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents
Sep 27, 2026Weiqi Wang, Yuxin Zhou, Mouxiang Chen +9Reinforcement Learning Long-Horizon Agent Tasks
Alibaba Token Hub, Alibaba Group · University of Science and Technology of China · Tsinghua University

Oct 5, 2026J/K move · Enter open · S save

Selecting Long-Horizon Trajectories for Reliable and Efficient Terminal-Agent Training

Cuong Dang, Hoang Anh Just, Ruoxi Jia

Department of Electrical and Computer Engineering Virginia Tech Blacksburg, VA, USA

Supervised Fine-Tuning Trajectory Generation LLM Agent Training Long-Horizon Agent Tasks Long-Horizon LLM Agents

Open paper AskPDF arXiv

Terminal agents are commonly trained by imitating long teacher trajectories, yet how much of each trajectory to supervise remains unexplored. We study the supervision horizon, the number of trajectory tokens retained for training, and show that it is a key design axis for reliability and cost. Reliability improves with longer horizons but saturates: on Terminal-Bench, a 12K-token horizon solves more tasks than 16K (

29\pm0.7

vs.\

26\pm0.8

) while requiring 30% less training time. The horizon also shapes agent behavior: short horizons cause premature termination, intermediate horizons yield productive error recovery, and long horizons induce over-persistence. We analyze this saturation through a bias--complexity bound, in which longer supervision reduces temporal supervision bias but increases finite-sample estimation error from more heterogeneous late-stage histories. Guided by this analysis, we propose selective long-horizon refinement, which first trains on short prefixes and then refines only on continuations that are most likely under the warm-start model. It consistently outperforms full long-horizon training. At 16K, it raises successful attempts from

110\pm2.7

126\pm2.1

and tasks solved in at least six of eight attempts from

9\pm0.7

14\pm0.6

; with half of the long-horizon data, it still reaches

122\pm2.4

while cutting training time by 23%. The gains transfer across benchmarks, from

64\pm2.6

73\pm2.1

on Terminal-Bench v2.0 and from

137\pm2.7

155\pm2.2

on OpenThoughts-TBLite. For long-horizon supervision, selecting the right trajectories matters more than training on all of them.

Abstract

Long-Horizon RL

Explore similar work

Selecting Long-Horizon Trajectories for Reliable and Efficient Terminal-Agent Training

Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents

QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents