LoongSage: A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes

LoongSage Team; Bowei Song; Caiwei Zhao; Chuwen Liu; Dechuang Gu; Dong Fang; Feiyu Xie; Huaiyu Li

cs.LGOct 12, 2026

LoongSage: A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes

Authors: LoongSage Team, Bowei Song, Caiwei Zhao, Chuwen Liu, Dechuang Gu, Dong Fang, Feiyu Xie, Huaiyu Li, Hui Zhang, Jian Tian, Jian Wang, Jie Zhao, Liguang Li, Maoyun Wang, Meiyan Yang, Minghan Zhu, Pai Liu, Pandeng Yao, Renbao Tian, Ru Yang, Shuai Jia, Shuai Tan, Shuailong Li, Shuliang Li, Teng Jin, Tianze Yang, Wenbo Cui, Wenjing Bai, Wenkang Wu, Xiaowei Jiang, Xiaoyi Dong, Yanpeng Wang, Yanwen Wang, Yaohui Ma, Yawen Wang, Yihang Zhang, Yingjie Yan, Zhanbiao Lian, Zhenyu Lu, Zhida Hu, Zhihui Hao, Zhihui Hu, Zhiwen Xiao, Zhiyi Gao, Zhongwei Qi, Zhuangzhuang Wu, Ziqian Liu, Ziyue Zheng

Abstract

Reinforcement learning (RL) has emerged as a critical phase in the post-training pipeline of large language models (LLMs). As the frontier of LLM capabilities shifts from static reasoning to agentic tasks--characterized by external tool invocation, multi-turn interactions, and long-horizon decision-making--existing mainstream RL frameworks have grown increasingly unwieldy, raising the overhead of adoption, usage, and debugging. To address these challenges, we present LoongSage, an agent-native, fully asynchronous RL framework with a lightweight core of roughly 14K lines of code. Built upon Megatron for distributed training and SGLang for high-throughput inference, LoongSage centralizes orchestration within a single controller process and encapsulates all GPU-resident components as Ray actors. It introduces AgentFlow, a unified execution paradigm that orchestrates both white-box and black-box agents. For asynchronous execution, it integrates sliding-window strategies that bound trajectory staleness, and employs TransferMesh, a topology-aware synchronization mechanism that routes weights after each step over the fastest transport available to each receiver. Beyond agentic RL, LoongSage incorporates a memory-efficient Teacher Manager that decouples the scaling of teacher models from GPU memory constraints, natively supporting Multi-Teacher On-Policy Distillation. Extensive evaluations across four complete training runs, spanning complex reinforcement learning and multi-teacher distillation, demonstrate that task rewards improve steadily while the train-inference gap is kept under continuous control, and the fully asynchronous pipeline removes synchronous idle bubbles, sustaining hardware utilization.

Figures & tables

Figure 1: Positioning of LoongSage relative to closely related systems. Each entry is compiled from the publicly available documentation of the corresponding project at the time of writing. A dash indicates a capability that is not documented, rather than one that cannot be realized within the system.

Figure 2: Overall system architecture of LoongSage.

Figure 3: The two orthogonal configuration axes behind LoongSage’s operational modes, their trade-offs, and the combination that the framework does not support.

Figure 4: The AgentFlow request path. The Router intercepts each Chat Completions request, tokenizes the turn, dispatches it to SGLang’s /generate , and records the returned tokens per turn in the TrajectoryStore .

Figure 5: Data representation under TITO: a single flattened token array with its position-aligned response-space arrays, referenced by Segment and Triplet indices rather than copied.

Figure 6: AgentFlow stores a black-box trajectory as a tree of segments: trainable mainline segments ( origin=root or compact ) own the flat token arrays, while a collab_spawn request forks a non-trainable subagent placeholder ( origin=subagent ).

Figure 7: The protocol adapter normalizes each agent’s wire format into one internal request: OpenCode over Chat Completions, Codex over OpenAI Responses, and Claude Code over Anthropic Messages. A single parser, tokenizer, and SGLang path then serve all of them.

Figure 8: The fully asynchronous pipeline’s control points reduce to two gates—a dispatch gate and a collection gate. Table 1 lists the mathematical bounds each registered policy places on them.

	Bound at the control point		Resulting guarantee
Strategy	Dispatch	Collection	Staleness	Training order
no-window	$F+Q\leq W$	unconstrained	unbounded	unconstrained
window-gated	$s\leq s_{\min}+W$	unconstrained	$\leq S$ steps	unconstrained
windowed-fifo	$F+Q\leq W$	$s-s_{\min}<B$	indirect	FIFO within $B$

Table 1: The bounds imposed by each sliding-window strategy at the two control points of the asynchronous runtime, and the guarantees they provide. window-gated inherits the capacity bound of no-window in addition to the sequence-number bound listed here.

Figure 9: The three registered sliding-window policies (a) no-window , (b) window-gated , and (c) windowed-fifo on a shared timeline of dispatched groups (bounds in Table 1 , consequences discussed above).

Figure 10: TransferMesh packs the named tensor stream into one contiguous memory bucket described by a single metadata frame, then dynamically selects a transport protocol per receiver based on device topology: utilizing a zero-copy CUDA IPC handle for colocated engines and an NCCL broadcast for remote receivers.

Figure 11: The Teacher Manager schedules teacher groups that share a GPU-memory slot, offloading inactive teacher weights to host memory and swapping them back on demand.

Figure 12: Full-vocabulary distillation via micro-batch logit reconstruction: within each micro-batch the teacher logits are rebuilt from hidden states, consumed by the loss, and released before the next micro-batch begins.

Figure 13: Train–inference consistency is handled in three groups: source alignment guarantees the trained sequence mathematically matches the sampled sequence, long-context and precision settings anchor numerical stability, and residual corrections continuously bound the remaining gap.

Registry	Configuration field	Scope
agent	data_source.agent.name	Agent loop
reward	data_source.reward.name	Reward computation
sandbox	data_source.sandbox.type	Execution environment
advantage	algorithm.advantage_estimator	Advantage estimation
policy_loss	algorithm.policy_loss	Policy loss
kl_policy	opd.{pg,gkd}_kl_method	KL constraint and estimator

Table 2: The core registries exposed by LoongSage. Each is invoked via a @register_<name> decorator at the definition site and bound to the corresponding configuration field. Fields lacking a top-level section ( agent , reward , dataset.* ) are resolved relative to the specific data_source . The kl_policy row abbreviates opd.pg_kl_method and opd.gkd_kl_method .

Setting	Model	Hardware	Focus
Training runs, one complete run each
SWE	DeepSeek-V4-Flash	8 H20 nodes	Long-horizon software tasks
DAPO	DeepSeek-V4-Flash	6 H20 nodes	Verifiable mathematical reasoning
OpenCode	Qwen3-Coder-30B-A3B	4 H20 nodes	Black-box coding agent
MOPD	Qwen3-30B-A3B	1 H20 node	Multi-teacher distillation
Controlled comparisons between configurations

Table 3: The runs reported in this paper.

Figure 14: Training curves for DeepSeek-V4-Flash on SWE tasks.

Figure 15: Training curves for DeepSeek-V4-Flash on DAPO mathematical reasoning.

Figure 16: Training curves for full-vocabulary multi-teacher MOPD on Qwen3-30B-A3B.

Figure 17: Training curves for Qwen3-Coder-30B-A3B with the OpenCode black-box agent.

		all $100$ steps		excluding step $0$
Configuration	Split	avg (s)	speedup	avg (s)	speedup
Colocated	16	580.79	1.00 $\times$	578.63	1.00 $\times$
Colocated $+$ partial rollout	16	489.63	1.19 $\times$	486.34	1.19 $\times$
Async, no-window , $S{=}1.0$	8:8	454.50	1.28 $\times$	451.74	1.28 $\times$
Async, windowed-fifo , $S{=}1.0$	8:8	434.76	1.34 $\times$	424.30	1.36 $\times$
Async, windowed-fifo , $S{=}2.0$	8:8	428.76	1.35 $\times$	414.58	1.40 $\times$

Table 4: Average step time under colocated and fully asynchronous execution. Speedups are relative to the colocated baseline in the same column.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 232 papers

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic RL Asynchronous RL
Tsinghua University · Work done while ZH and YL interned at Z.AI.
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
Jul 2, 2025Zhenyu Han, Ansheng You, Haibo Wang +16RL for Language Models Asynchronous RL
Huawei · Individual Researcher
ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
Oct 15, 2026Yukun Tian, Junwei Su, Haofeng Xu +6RL Post-Training RL for Language Model Reasoning
The University of Hong Kong · Alibaba Group · University of Science and Technology of China +1

Jul 8, 2026J/K move · Enter open · S save

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

Tsinghua University · Work done while ZH and YL interned at Z.AI.

Agentic RL Asynchronous RL Reinforcement Learning RL Post-Training Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Abstract

Agentic RL

Explore similar work

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training