Harvesting Expert-Parallel Bubbles for Fine-Tuning
Organizations: McGill University · ETH Zurich · EPFL
Abstract
GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2 faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3 faster than LLMStation and 14 faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.
Figures & tables
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| System | Capacity targeted | Work unit | Admission scope | Signal |
|---|---|---|---|---|
| FlexLLM Oliaro et al. (2026) | serving headroom | token slot | serving iteration | batch composition |
| LLMStation He et al. (2025) | decode / idle headroom | tasklet | serving iteration | iteration timing |
| Harli Xu et al. (2025) | decode underutilization | SM partition | serving iteration | predicted latency |
| DeltaServe Chen et al. (2026) | serving / idle headroom | LoRA batch | serving iteration | predicted latency |
| BubbleTea | EP bubble | fine-tuning chunk | GPU MoE layer | router top- assignment |
| Planner | Bubble utilization | Overrun rate |
|---|---|---|
| Router-blind | 1.28–1.31% | 1.04% |
| Oracle GPU-agnostic | 1.22–1.30% | 1.04% |
| Router-aware | 42.74–45.14% | 0.83–1.04% |
| FT | LLMS | BT | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Len. | Load | req/s | LLMS | BT | Ratio | TTFT | TPOT | TTFT | TPOT |
| 4k | med. | 2 | 1.58 | 5.60 | 3.5 | 1.05 | 1.08 | 1.04 | 1.05 |
| 4k | high | 4 | 0.95 | 3.46 | 3.6 | 1.02 | 1.01 | 1.02 | 1.00 |
| 8k | med. | 1 | 1.52 | 5.22 | 3.4 | 1.04 | 1.05 | 1.02 | 1.01 |
| 8k | high | 2 | 0.91 | 3.51 | 3.9 | 1.03 | 1.11 | 1.02 | 1.10 |
| 16k | med. | 0.5 | 1.46 | 4.86 | 3.3 | 1.02 | 1.02 | 1.02 | 1.04 |
| 4k, 8 req/s | 8k, 4 req/s | 16k, 2 req/s | ||||
|---|---|---|---|---|---|---|
| ( s) | FT | TPOT | FT | TPOT | FT | TPOT |
| 0 | 0.96 | 1.05 | 1.59 | 1.10 | 0.53 | 1.09 |
| 30 | 0.98 | 1.05 | 1.55 | 1.11 | 0.51 | 1.13 |
| 60 | 0.97 | 1.05 | 1.63 | 1.08 | 0.51 | 1.08 |
| 90 | 0.86 | 1.06 | 1.39 | 1.13 | 0.43 | 1.22 |
| 120 | 0.83 | 1.05 | 1.47 | 1.09 | 0.42 | 1.25 |
LLM Inference Scheduling
Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.
See all 197 papers