QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents

Weiqi Wang; Yuxin Zhou; Mouxiang Chen; Siyuan Zhang; Yi Zhang; Yuyan Luo; Zhiyu Yin; Chencan Wu

cs.LGSep 27, 2026

QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents

Authors: Weiqi Wang, Yuxin Zhou, Mouxiang Chen, Siyuan Zhang, Yi Zhang, Yuyan Luo, Zhiyu Yin, Chencan Wu, +4 more

Organizations: Alibaba Token Hub, Alibaba Group · University of Science and Technology of China · Tsinghua University

Abstract

Large language model (LLM) agents increasingly undertake extreme-long (xlong) horizon tasks, where a single execution can span hours, hundreds of model--environment interactions, and nearly 1M tokens per rollout. Applying online reinforcement learning (RL) to such executions poses two fundamental challenges: (1) severe execution variance and prolonged rollout delays cause massive GPU idling; and (2) complex non-linear branching generates massive trajectory redundancy, crippling training efficiency. To address these, we presents QwenGyre, an end-to-end framework for xlong-horizon online RL. QwenGyre elastically reallocates GPUs between rollout and training without interrupting live executions, while its trajectory processor reconstructs branching histories, scores partial progress, and deduplicates redundant paths to bound training costs. Scaled to our flagship model, Qwen~3.8 2.4T, with 700K tokens per rollout, QwenGyre yields a 6.0% absolute gain on NL2RepoBench (52.5% $\to$ 58.5%) in 48 steps. Across our evaluations on diverse domains of training datasets, QwenGyre delivers up to $1.85\times$ and $1.78\times$ speedups over Colocate and Async, respectively.

Figures & tables

ID	Evolution	Stream	Switch	$({\varphi},\mu)$	Allocation	Total (h)	Time / E0
A0	Original Async	Disabled	–	$(1.5,1)$	16 T / 16 R	25.08	1.556
A1	A0 with adjusted partition	Disabled	–	$(1.5,1)$	8 T / 24 R	27.01	1.676
A2	A0 + multi-step burst	Disabled	–	$(0.5,3)$	16 T / 16 R	32.58	2.021
A3	A0 + streaming	Enabled	–	$(1.5,1)$	16 T / 16 R	22.18	1.376
C0	Original Colocate	Disabled	Global	$(0,4)$	32 C / 0 S	23.23	1.442
C1	C0 + standalone rollout nodes	Disabled	Coarse	$(0.5,3)$	28 C / 4 S	22.56	1.400

Table 1: Scheduling ablations over the first 12 NL2RepoBench training steps with Qwen 3.6 122B at

\mathbb{E}[d]=1.5

. A0, C0, and E0 reuse the Async, Colocate, and QwenGyre base configurations from Section 6.2 . Arrows show the evolution paths; E1 and E2 branch independently from E0. Switch is – for fixed partitions, Global for switching all nodes together, Coarse for switching the Colocate pool together, and Fine for switching cells independently. Standalone nodes remain in rollout. Allocation lists training/rollout (T/R) or Colocate/standalone (C/S) node counts; cell entries give cells

\times

nodes per cell. Total is the time for all 12 steps in hours; Time / E0 normalizes it to E0. Lower is better.

Dataset	$\mathbb{E}[d]$	$({\varphi},\mu)$	vs. Async	vs. Colocate
DeepSWE	1	$(0.5,2)$	1.569	1.610
DeepSWE	1.5	$(0.5,3)$	1.572	1.824
TerminalBench	1	$(0.5,2)$	1.383	1.809
TerminalBench	1.5	$(0.5,3)$	1.430	1.849

Table 2: 24-step speedup of QwenGyre over each baseline with Qwen 3.6 122B. The

(\varphi,\mu)

column gives QwenGyre ’s extra dispatch in batch units and training steps per burst. Async uses

(\varphi,\mu)=(\mathbb{E}[d],1)

; Colocate uses

(\varphi,\mu)=(0,2\mathbb{E}[d]+1)

, following Equation 2 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$B$ , $\mu$	Groups per update; updates per burst.
$\varphi$	Extra boundary dispatch in units of $B$ groups.
$Q_{\mathrm{b}}$	Boundary depth after replenishment: all $(\varphi+\mu)B$ unconsumed groups.
$Q_{\mathrm{c}}$	Continuous depth: unfinished groups only.
$v_{d}$ , $v_{t}$	Published version at admission; trainer version immediately before consumption.
$d$	Staleness $v_{t}-v_{d}$ ; denotes its boundary mean in Section A.5 .

Appendix

Table 3: Notation for dispatch and step-time analysis.

Explore similar work

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
Jul 16, 2026Changhai Zhou, Kieran Liu, Yuhua Zhou +17Efficient Long-Context Inference LLM Inference Optimization
MindLab · Fudan University
PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning
Sep 28, 2026Jiaan Zhu, Wei Gao, Youhui Bai +6Parallel Rollouts Agentic Reinforcement Learning
USTC China
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Sep 12, 2026Junyao Yang, Yucheng Shi, Zhongzhi Li +4Critic-Free Reinforcement Learning Long-Horizon Task Planning
Tencent Hy Foundation Model · National University of Singapore · University of Georgia +2

Jul 16, 2026, cs.LGJ/K move · Enter open · S save

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao+16

MindLab · Fudan University

Efficient Long-Context Inference LLM Inference Optimization Offline Reinforcement Learning Long-Term Recurrent Model

Open paper AskPDF arXiv

A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment. The gap is especially important for AI agents, whose observations, tool outputs, documents, and prior decisions accumulate over long trajectories. LongStraw is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, instantiated with Group Relative Policy Optimization (GRPO). It evaluates the shared prompt without autograd, retains only model-specific state needed by later tokens, and replays short response branches one at a time, reducing the live training graph at the cost of additional replay time. We implement it for the hybrid recurrent and full-attention Qwen3.6-27B and the compressed-attention mixture-of-experts GLM-5.2. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8; increasing the group size adds only 0.21 GB of peak allocated memory, while a separate stress test reaches 4.46M positions. On 32 H20 GPUs, we validate the end-to-end LongStraw execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. These experiments establish execution capacity rather than complete training correctness because the captured prompt state is detached and some distributed forward and gradient composition paths remain incomplete.

Abstract

Explore similar work

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks