cs.DCNov 6, 2026

Adaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning

Authors: Salah Chikhi, Luis Zuin, Entong Li, Zeng Li

Organizations: École polytechnique · Huawei Technologies · Shenzhen Research Institute of Big Data

Abstract

We present the Adaptive Scheduling Pipeline (ASP), an offline rollout scheduler for multi-instance asynchronous reinforcement learning (RL) of large language models (LLMs). ASP jointly reduces three sources of rollout underutilization - padding bubbles from intra-minibatch length heterogeneity, skewness bubbles from long-tail outputs, and sorting bubbles from fluctuations in training-queue occupancy induced by output-length-sorted dispatch - through a single offline preprocessing pass. This enables efficient multi-instance rollout that overcomes the early scaling plateau of single-instance inference while reducing fluctuations in the supply of generated samples to the trainer. We integrate the scheduler into an asynchronous architecture that decouples rollout from training while keeping policy lag observable and bounded. Within our asynchronous architecture, ASP's scheduling reduces mean iteration time by 10.9%. The complete system improves end-to-end throughput by up to 3.22 times and reduces iteration time by up to 76.8% relative to MindSpeed-RL, while reducing iteration time by up to 62.5% relative to verl. It also scales more robustly with additional generation resources than these baselines and shows no visible convergence degradation in the measured bounded-lag regime. ASP has since been used in an internal telecom-domain RL post-training workflow, reducing the reported training period from more than three months to two months and producing a domain-enhanced model for knowledge answering and tool calling.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling

    Mar 24, 2026Yiqi Zhang, Huiqiang Jiang, Xufang Luo +7RL for Language ModelsRL for Language Model Reasoning

  2. Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training

    Sep 29, 2026Chenliang Li, Neiwen Ling, Zijun Wei +1Asynchronous RLRL for Language Model Reasoning