Heddle: A Distributed Orchestration System for Agentic RL Rollout

Zili Zhang; Yinmin Zhong; Chengxu Yang; Chao Jin; Bingyang Wu; Xinming Wei; Yuliang Liu; Xin Jin

cs.LGMar 30, 2026

Heddle: A Distributed Orchestration System for Agentic RL Rollout

Authors: Zili Zhang, Yinmin Zhong, Chengxu Yang, Chao Jin, Bingyang Wu, Xinming Wei, Yuliang Liu, Xin Jin

Organizations: School of Computer Science, Peking University · Independent Researcher

Abstract

Agentic Reinforcement Learning (RL) enables LLMs to solve complex tasks by alternating between a data-collection rollout phase and a policy training phase. During rollout, the agent generates trajectories, i.e., multi-step interactions between LLMs and external tools. Yet, frequent tool calls induce long-tailed trajectory generation that bottlenecks rollouts. This stems from step-centric designs that ignore trajectory context, triggering three system problems for long-tail trajectory generation: queueing delays, interference overhead, and inflated per-token time. We propose Heddle, a trajectory-centric system to optimize the when, where, and how of agentic rollout execution. Heddle integrates three core mechanisms: trajectory-level scheduling using runtime prediction and progressive priority to minimize cumulative queueing; trajectory-aware placement via presorted dynamic programming and opportunistic migration during idle tool call intervals to minimize interference; and trajectory-adaptive resource manager that dynamically tunes model parallelism to accelerate the per-token time of long-tail trajectories while maintaining high throughput for short trajectories. Evaluations across diverse agentic RL workloads demonstrate that Heddle effectively neutralizes the long-tail bottleneck, achieving up to 2.5 $\times$ higher end-to-end rollout throughput compared to state-of-the-art baselines.

Figures & tables

Figure 1. Illustration of the agentic trajectory and the straggler effect.

Figure 2. Long-tailed distribution of coding agents.

Figure 3. Existing agentic RL framework.

Figure 4. CDF of normalized agentic trajectory completion time.

Figure 5. Trajectory length distribution across different prompts.

Figure 6. Interference of long-tailed trajectories.

Figure 7. Performance under different resource allocation strategies.

Figure 9. Progressive trajectory prediction.

Figure 10. Illustration of the placement algorithm.

Figure 11. Insight of heterogeneous resource allocation.

Figure 12. Rollout throughput of agentic RL systems under different workloads and models.

Figure 13. Precision of progressive trajectory prediction under different workloads and models.

Figure 14. Performance of trajectory-level scheduler.

Figure 15. Performance of trajectory-aware placement.

Figure 16. Performance of trajectory-adaptive resource management.

Models		Coding	Search	Math
Qwen3-8B	Tool Exec. (s)	0.46	1.42	0.051
	Pred. (s)	0.103	0.15	0.26
	Migration (s)	0.26	0.12	0.24
Qwen3-14B	Tool Exec. (s)	0.41	1.41	0.046
	Pred. (s)	0.099	0.15	0.27
	Migration (s)	0.27	0.15	0.25

Table 1. Prediction and migration overhead in Heddle .

Models		Coding	Search	Math
Qwen3-8B	Placement (s)	0.036	0.037	0.036
Qwen3-8B	Resource manager (s)	5.69	5.27	5.06
Qwen3-14B	Placement (s)	0.038	0.037	0.036
Qwen3-14B	Resource manager (s)	5.05	4.99	4.97
Qwen3-32B	Placement (s)	0.037	0.038	0.036
Qwen3-32B	Resource manager (s)	5.06	5.01	4.98

Table 2. Algorithm overheads in Heddle .

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 458 papers

Libra: Efficient Resource Management for Agentic RL Post-Training
Jun 2, 2026Kaiwen Chen, Xin Tan, Jingzong Li +6Agentic RL Reinforcement Learning
The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher
Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning
Oct 12, 2026Chiheng Lou, Xiangwei Deng, Bangjun Xiao +4Prefill-Decode Disaggregation Agentic RL
†School of Computer Science, Peking University · ‡LLM-Core Xiaomi
WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
Jul 19, 2026Ryan Xu, Atlas Zhao, David Bao +1Agentic RL Reinforcement Learning

Jun 2, 2026J/K move · Enter open · S save

Libra: Efficient Resource Management for Agentic RL Post-Training

Kaiwen Chen, Xin Tan, Jingzong Li, Zhi Zhou+5

The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher

Agentic RL Reinforcement Learning Distributed RL LLM Agent Training RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has emerged as a standard post-training paradigm for shaping large language models (LLMs) into capable agents. In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads that expose two fundamental challenges. First, due to the long-tailed response distribution, a small fraction of trajectories dominates rollout makespan.Second, rollout and training differ in their compute patterns, memory demands, and sensitivity to sequence length. As the policy evolves, shifts in the workload distribution further change their relative resource demands, making it difficult to maintain balanced execution across the two stages. We present Libra, an adaptive runtime for agentic RL post-training with two complementary components: (1) intra-stage scheduling via a Causality-Guided Bucket Scheduler that routes requests across execution buckets with different parallelism configurations, reducing delays from rollout stragglers; and (2) cross-stage coordination that dynamically reallocates workers between rollout and training as the workload changes. It moves workers between the two stages through a non-blocking protocol without interrupting ongoing training. Evaluated on a 48x NVIDIA A800 GPU cluster and a 160x Ascend 910B3 NPU cluster across three agentic benchmarks, Libra achieves up to 4.2x higher throughput and up to 2.7x faster reward convergence

Abstract

LLM Agent Orchestration

Explore similar work

Libra: Efficient Resource Management for Agentic RL Post-Training

Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning