TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts

Tianqi Xu; Lu Lv; Haoyang Huang; Wenjie Huang; Zhanming Shen; Yuhao Shen; Baolin Zhang; Xinyi Hu

cs.AIAug 24, 2026

TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts

Authors: Tianqi Xu, Lu Lv, Haoyang Huang, Wenjie Huang, Zhanming Shen, Yuhao Shen, Baolin Zhang, Xinyi Hu,

Organizations: Qwen Business Unit of Alibaba · Carnegie Mellon University · Zhejiang University · National University of Singapore ·

Abstract

Large-scale rollouts have become a core component of modern LLM systems, spanning reinforcement learning (RL) post-training, on-policy distillation (OPD), and sampling-heavy evaluation pipelines. Unlike online serving, which is typically optimized for request-level latency and throughput, a small number of long-tail generations can dominate the end-to-end makespan of an entire rollout step. In practice, rollout requests are often routed uniformly across replicas, which can place extremely long generations inside high-concurrency decoding batches. To address this, we present TailSieve, a partial-rollout-guided framework that jointly controls tail routing and replica allocation for LLM rollouts. In an idealized setting with known completion lengths, we show that makespan-optimal routing in the long-tail regime combines tail isolation with load balancing, and that a simple top-k policy closely approximates this offline optimum. Leveraging the observation that long-tail prompts tend to remain long-tailed across policy updates, TailSieve uses partial rollouts as a training-free signal for identifying candidate tail groups. A hierarchical controller then jointly adapts the number of isolated groups and the replica split between the tail and bulk pools using collected response-work history and a measured concurrency-throughput model. TailSieve achieves up to 1.67x routing-only speedup over uniform group routing. The resulting low-concurrency tail pool further enables route-specialized speculative decoding with MTP or DFlash, achieving up to 2.59x speedup over uniform routing. Selected prompts are regenerated under the current policy, preserving on-policy generation and avoiding additional routing-induced length bias in steady state.

Figures & tables

Figure 1: Measured decode throughput versus active batch size (a) and the resulting replica-time decomposition (b) .

Figure 2: Oracle routing and top- $k$ tail isolation for a representative 100-request rollout group.

Figure 3: Partial-rollout candidate selection, final-length ranking, and long-tail capture at three cutoff percentages.

Figure 4: Joint-control landscapes showing that the optimal isolation size and replica split vary with tail strength.

Figure 5: TailSieve workflow: warm-up tail identification, joint two-pool execution, and hierarchical control.

DeepScaleR
Routing policy	Qwen3.5-35B-A3B	Qwen3.5-4B	Qwen3.5-2B	Qwen3-30B-A3B	Qwen3-4B
Uniform	1.000 $\times$	1.000 $\times$	1.000 $\times$	1.000 $\times$	1.000 $\times$
Uniform Oracle *	1.123 $\times$	1.151 $\times$	1.154 $\times$	1.085 $\times$	1.225 $\times$
StreamRL-style Oracle *	1.030 $\times$	1.113 $\times$	1.136 $\times$	0.928 $\times$	1.174 $\times$
Seer-style Oracle *	1.189 $\times$	1.162 $\times$	1.169 $\times$	1.082 $\times$	1.358 $\times$
TailSieve (Fixed Replica Allocation)	1.219 $\times$	1.199 $\times$	1.144 $\times$	1.112 $\times$	1.186 $\times$

Table 1: Routing-only speedup over uniform group routing.

Model	DeepScaleR			KodCode-10K
	Routing	+MTP	+DFlash	Routing	+MTP	+DFlash
Qwen3.5-35B-A3B	1.219 $\times$	2.390 $\times$	2.181 $\times$	1.560 $\times$	2.586 $\times$	2.378 $\times$
Qwen3.5-4B	1.199 $\times$	2.414 $\times$	2.091 $\times$	1.310 $\times$	1.921 $\times$	1.774 $\times$
Qwen3.5-2B	1.144 $\times$	2.161 $\times$	–	1.143 $\times$	2.349 $\times$	–

Table 2: Step-wise speedup over uniform routing with fixed replica allocation using routing alone, MTP, or DFlash.

Figure 6: Route-specialized speculative decoding (a) and hierarchical-controller convergence (b) on Qwen3.5-35B-A3B.

Figure 7: Thirty-step GRPO results for Qwen3.5-2B (top) and Qwen3.5-4B (bottom): generation time, reward, and mean response length.

Model	Method	GSM8K pass@1	MATH-500 pass@1	AIME24 pass@10	AIME25 pass@10
Qwen3.5-2B	Base model	$9.40\%$	$13.40\%$	$30.00\%$	$30.00\%$
	Baseline	$75.97\%$ $(+66.57)$	$76.20\%$ $(+62.80)$	$50.00\%$ $(+20.00)$	$43.33\%$ $(+13.33)$
	TailSieve (Routing-only)	$76.57\%$ $(+67.17)$	$74.60\%$ $(+61.20)$	$53.33\%$ $(+23.33)$	$36.67\%$ $(+6.67)$
	TailSieve (MTP)	$76.42\%$ $(+67.02)$	$74.40\%$ $(+61.00)$	$53.33\%$ $(+23.33)$	$53.33\%$ $(+23.33)$
Qwen3.5-4B	Base model	$89.61\%$	$84.40\%$	$90.00\%$	$80.00\%$
	Baseline	$89.16\%$ $(-0.45)$	$88.80\%$ $(+4.40)$	$90.00\%$ $(+0.00)$	$86.67\%$ $(+6.67)$

Table 3: Downstream accuracy of base and RL-trained checkpoints; parentheses report percentage-point changes from the corresponding base model.

Model	DeepScaleR			KodCode-10K
	N=64	N=128	N=256	N=64	N=128	N=256
Qwen3.5-35B-A3B	1.346 $\times$	1.199 $\times$	1.122 $\times$	1.670 $\times$	1.214 $\times$	1.142 $\times$
Qwen3.5-4B	1.233 $\times$	1.225 $\times$	1.191 $\times$	1.403 $\times$	1.453 $\times$	1.240 $\times$
Qwen3.5-2B	1.180 $\times$	1.129 $\times$	1.177 $\times$	1.214 $\times$	1.108 $\times$	1.101 $\times$
Qwen3-30B-A3B-Instruct-2507	1.112 $\times$	1.023 $\times$	1.121 $\times$	1.342 $\times$	1.163 $\times$	1.091 $\times$
Qwen3-4B-Instruct-2507	1.254 $\times$	1.215 $\times$	1.306 $\times$	1.442 $\times$	1.367 $\times$	1.248 $\times$

Table 4: Routing-only speedup over uniform routing as rollout concurrency increases.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Setting
Maximum prompt length	$512$ tokens
Maximum response length	$16{,}384$ tokens
Temperature	$0.7$
Top- $p$	$0.8$
Top- $k$	$20$
Presence penalty	$1.5$

Appendix

Table A.1 : Rollout sampling configuration shared by the step-wise and end-to-end RL experiments.

Parameter	Setting
Batch size	$64$ groups $\times\,8$ responses $=512$ trajectories
Training TP	$1$
Training micro-batch size	$8$
PPO clip ratio	$0.2$
Dual-clip coefficient	$3.0$
KL loss/reward	Disabled

Appendix

Table A.2 : Batch and optimizer configuration for the end-to-end mixed-routing RL experiment.

Figure A.1 : Optimality-gap distributions across 100 rollout groups.

Scene	Length construction	$P_{50}$	$P_{90}$	$P_{90}/P_{50}$	Maximum
Weak Tail	Compress the upper half	$1{,}160$	$1{,}740$	$1.5$	$3{,}206$
Real Workload	Original lengths	$1{,}160$	$4{,}115$	$3.547$	$11{,}800$
Strong Tail	Stretch the upper half	$1{,}160$	$6{,}960$	$6.0$	$22{,}094$

Appendix

Table A.3 : Construction and length statistics of the weak-, real-, and strong-tail workloads.

Figure B.1 : Prompt-tail ranking stability across checkpoints and response-length variance decomposition.

Figure D.1 : Gain regimes for fixed top- $10\%$ isolation under a fixed replica split.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 355 papers

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Jun 17, 2026Minseo Kim, Minjae Lee, Seunghyuk Oh +7RL for Language Models Reinforcement Learning
FuriosaAI · University of California, Berkeley
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
May 9, 2026Yuhang Xu, Kaibin Tian, Yang Tian +6Reinforcement Learning Speculative Decoding
1Shanghai Jiao Tong University · 2Bytedance

Jun 17, 2026J/K move · Enter open · S save

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim+6

FuriosaAI · University of California, Berkeley

RL for Language Models Reinforcement Learning Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Reinforcement learning (RL) has become a representative post-training paradigm for large language models (LLMs), enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive (AR) sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 24.2% and 15.6%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.

Abstract

LLM Routing

Explore similar work

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning