AdaFloor: Adaptive Floor Planning and Rank-Time Reuse for Long-Tail MoE RL Rollouts
Organizations: Department of Computer Science and Technology, Tsinghua University
Abstract
Synchronous MoE reinforcement-learning post-training suffers from response-length imbalance. Existing methods reduce request tails, yet completed ranks remain bound to Expert Parallelism (EP) collectives and expert ownership. In a representative 16-accelerator Qwen3-30B-A3B rollout, this dummy run traps 32.83% of rank-time. AdaFloor converts this EP-coupled capacity into bounded leases for auxiliary work to improve system resource efficiency through rank-time reuse. Historical prompt-tail prediction and a calibrated guard construct the KV-safe layout. AdaFloor then selects the floor-specific safe quorum schedule with the largest predicted release area. Its runtime shrinks the EP group and leases detached ranks until full-world restoration. Across three updated-policy seeds on DeepScaleR mathematical reasoning rollouts, its capacity-oriented Natural floor2 configuration with TailGuard reduces Vanilla's 29393.9 rank-seconds of dummy run by 96.15%. Rollout shortening eliminates 18693.6 rank-seconds, while coordinated exits convert another 9568.6 rank-seconds into reusable capacity, leaving 1131.7 rank-seconds. For sidecar reuse, its Planned residency floor4 configuration without TailGuard serves 3134 Qwen3-8B GSM8K queries per epoch at 99.46% accuracy and incurs only a 2 to 3% primary throughput reduction against the same configuration with sidecar execution disabled. Its latency-oriented Planned residency floor4 configuration with TailGuard reduces rollout time by 24.09% relative to Vanilla and by 2.31% relative to LengthSort with TailGuard. On public Eurus-2-RL code rollouts with the same model and 16-rank topology, AdaFloor reduces rollout time by 20.56% and dummy run by 89.13% while generated work and mean reward remain stable.
Figures & tables
| Floor | Natural | Planned residency |
|---|---|---|
| 2 | 131,072 | Not supported |
| 4 | 280,576 | 133,120 |
| 8 | 315,648 | 262,656 |
| 16 | 380,800 | 380,800 |
| Forecast | Time | Release | Reward | TG/P |
|---|---|---|---|---|
| (ks) | (k rank-s) | |||
| 4.91 | 8.28 | 0.8352 | 2/0 | |
| 4.45 | 2.21 | 0.8324 | 265/55 | |
| 4.94 | 10.35 | 0.8375 | 1/0 | |
| Mixed | 5.02 | 6.97 | 0.8336 | 146/16 |
| Planner | Rank 1 | Rank 2 | Rank 3 |
|---|---|---|---|
| Quota-aware ILP | 150 | 0 | 0 |
| MinSkew | 142 | 8 | 0 |
| Stepwise-F4 | 88 | 2 | 60 |
| floor4 policy | Q2.5-1.5B | Q3-8B | Pangu-72B | AICore Rel. |
|---|---|---|---|---|
| Natural | 66,184 | 3,083 | 1,441 | 139.49% |
| Planned residency | 67,272 | 3,134 | 1,465 | 128.94% |
| Natural+TailGuard | 55,164 | 2,570 | 1,201 | 140.09% |
| Planned residency +TailGuard | 21,975 | 1,024 | 478 | 127.44% |
| Mode | Expert availability | floor8 | floor4 | floor2 |
|---|---|---|---|---|
| 1 | Resident replica | 93.65 | 98.12 | 100.58 |
| 2 | CPU swap | 484.57 | 496.78 | 514.71 |
| 3 | CPU double buffer | 197.99 | 199.72 | 265.88 |
| 4 | Remote NPU | 144.91 | 164.35 | 193.43 |
| 5 | Dual source | 151.44 | 185.70 | 201.58 |
RL Post-Training
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
See all 458 papers