Adaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning
Organizations: École polytechnique · Huawei Technologies · Shenzhen Research Institute of Big Data
Abstract
We present the Adaptive Scheduling Pipeline (ASP), an offline rollout scheduler for multi-instance asynchronous reinforcement learning (RL) of large language models (LLMs). ASP jointly reduces three sources of rollout underutilization - padding bubbles from intra-minibatch length heterogeneity, skewness bubbles from long-tail outputs, and sorting bubbles from fluctuations in training-queue occupancy induced by output-length-sorted dispatch - through a single offline preprocessing pass. This enables efficient multi-instance rollout that overcomes the early scaling plateau of single-instance inference while reducing fluctuations in the supply of generated samples to the trainer. We integrate the scheduler into an asynchronous architecture that decouples rollout from training while keeping policy lag observable and bounded. Within our asynchronous architecture, ASP's scheduling reduces mean iteration time by 10.9%. The complete system improves end-to-end throughput by up to 3.22 times and reduces iteration time by up to 76.8% relative to MindSpeed-RL, while reducing iteration time by up to 62.5% relative to verl. It also scales more robustly with additional generation resources than these baselines and shows no visible convergence degradation in the measured bounded-lag regime. ASP has since been used in an internal telecom-domain RL post-training workflow, reducing the reported training period from more than three months to two months and producing a domain-enhanced model for knowledge answering and tool calling.
Figures & tables
| Method | Async | Multi-inst. rollout | Padding-aware rollout scheduling | Sorting-bubble mitigation |
|---|---|---|---|---|
| Hybrid RL | No | No | No | No |
| StreamRL [ 5 ] | Yes | Yes | Partial | No |
| AsyncFlow [ 6 ] | Yes | Config.-dep. | No | No |
| AReaL [ 8 ] | Yes | Yes | No | No |
| Laminar [ 9 ] | Yes | Yes | No | No |
| verl [ 13 ] | Yes | Config.-dep. | No | No |
| Step | Mechanism | Main effect |
|---|---|---|
| 1 | Output-length prediction | Orders prompts by expected rollout length |
| 2 | Dynamic programming minibatch construction | Reduces intra-minibatch length heterogeneity |
| 3 | Balanced batch-group ordering | Smooths queue arrivals over time |
| Model | ASP | verl | MindSpeed-RL | ASP reduction | |
|---|---|---|---|---|---|
| Mean iteration time (s) | vs. verl | vs. MindSpeed-RL | |||
| Qwen3-30B-A3B | 215.2 | 536.7 | 927 | 59.9% | 76.8% |
| Qwen3-32B | 230.7 | 615.1 | 928 | 62.5% | 75.1% |
| Metric | C0 | C1 | C2 | C3 | C4 |
|---|---|---|---|---|---|
| (a) Qwen3-32B responses; controlled ; three-seed means | |||||
| Intra-minibatch length STD (tokens) | 2138 | 1163 | 1163 | 1163 | 1163 |
| Mean padding rate | 0.322 | 0.226 | 0.224 | 0.224 | 0.224 |
| Batch-group load CV | 0.103 | 0.403 | 0.401 | 0.162 | 0.070 |
| Group-load spread (max/min) | 1.68 | 9.88 | 9.81 | 2.39 | 1.39 |
| (b) Qwen3-8B generation with simulated training; measured predictor | |||||
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Feature | Hybrid RL | StreamRL | AsyncFlow | AReaL | Laminar | ASP |
|---|---|---|---|---|---|---|
| Rollout/training separation | No | Yes | Yes | Yes | Yes | Yes |
| Multi-instance rollout | No | Yes | Config.-dep. | Yes | Yes | Yes |
| Independent training/generation configurations | No | Yes | Yes | Yes | Yes | Yes |
| Predicted-length minibatch construction | No | No | No | No | No | Yes |
| Long-tail rollout mitigation | No | Yes | No | Yes | Yes | Yes |
| Explicit minibatch optimization for rollout padding | No | No | No | No | No | Yes |
| Variance reduction (%) | |||||
|---|---|---|---|---|---|
| Wall (s) | Logn. | Exp. | Pareto | Bim. | |
| 30.6 | 29.8 | 30.5 | 29.6 | ||
| 93.9 | 95.6 | 92.2 | 95.6 | ||
| 98.9 | 99.9 | 97.2 | 99.8 | ||
| 99.4 | 100.0 | 97.6 | 100.0 | ||
| 99.4 | 100.0 | 97.8 | 100.0 | ||
| Qwen3-30B-A3B | Qwen3-32B | |||
| Setting | ASP | MindSpeed-RL | ASP | MindSpeed-RL |
| Workload per iteration | ||||
| Prompts | 16 | 16 | 32 | 32 |
| Responses per prompt | 8 | 8 | 8 | 8 |
| Trajectories | 128 | 128 | 256 | 256 |
| Training and generation | ||||
| Total NPUs | ASP training NPUs | ASP generation NPUs |
| 16 | 8 | 8 |
| 32 | 8 | 24 |
| 64 | 32 | 32 |
| 128 | 48 | 80 |
| Dispersion (tokens) | Mean padding rate | ||||||
| MAE | C0 | C1 | C2–C4 | C0 | C1 | C2–C4 | |
| Qwen3-32B | |||||||
| 1.00 | 0 | 2138 | 5 | 5 | 0.322 | 0.003 | 0.003 |
| 0.75 | 839 | 2138 | 820 | 828 | 0.322 | 0.176 | 0.175 |
| 0.64 | 1203 | 2138 | 1163 | 1163 | 0.322 | 0.226 | 0.224 |
| 0.50 | 1692 | 2138 | 1558 | 1563 | 0.322 | 0.273 | 0.271 |
| C0 | C1 | C2 | C3 | C4 | |
| Qwen3-32B | |||||
| 1.00 | 0.103 | 0.495 | 0.492 | 0.188 | 0.000 |
| 0.75 | 0.103 | 0.447 | 0.445 | 0.175 | 0.052 |
| 0.64 | 0.103 | 0.403 | 0.401 | 0.162 | 0.070 |
| 0.50 | 0.103 | 0.332 | 0.330 | 0.140 | 0.080 |
| 0.25 | 0.103 | 0.183 | 0.183 | 0.115 | 0.092 |
LLM Inference Scheduling
Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.
See all 195 papers