EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL
Organizations: University of Massachusetts Amherst, Amherst, MA, USA · University of Massachusetts Lowell, Lowell, MA, USA
Abstract
Low-rank adapters (LoRA) let one base model serve many fine-tuned variants: multi-tenant engines keep a single copy of the base and batch the requests of many tenants, each with its own adapter. As base models move to 8-bit (FP8) and 4-bit formats, their general matrix multiplications (GEMMs) become cheaper, but the adapters do not: engines run them as separate small GPU kernels beside the GEMM, whose latency takes a growing share of every decoding step. An adapter projects its input down to a low rank and back up, and our key observation is that the GEMM needs the down-projection only in its epilogue, the final stage in which it writes its output. EpiLoRA, a plugin for the vLLM engine, therefore launches the down-projection first and the low-precision GEMM as a dependent kernel that streams its weights meanwhile and adds every tenant's up-projection in its epilogue. On NVIDIA Hopper GPUs, EpiLoRA decodes for a single adapter with up to 24% lower time per token than vLLM's fastest configuration, and faster than SGLang and TensorRT-LLM; with many tenants, its adapter kernels deliver more throughput than vLLM's on most GPU-model pairs we tested. The same low-precision base also trains: a LoRA reinforcement-learning trainer that computes on the engine's FP8 weights needs no 16-bit copy, so a 27B model trains on one 48 GB GPU, and an engine that then keeps activations in 16 bits removes most of the remaining mismatch between the model generating training data and the one trained.
Figures & tables
| µs | L40S | H100 | H200 | Blackwell |
|---|---|---|---|---|
| Base GEMM, vLLM | 413 | 126 | 104 | 286 |
| Base GEMM, EpiLoRA | 416 | 121 | 102 | 276 |
| Adapter, vLLM | 97 | 53 | 57 | 54 |
| Adapter, EpiLoRA | 46 | 12 | 10 | 17 |
| EpiLoRA’s path | fused | overlap | overlap | overlap |
| Layer, EpiLoRA / vLLM | 0.91 | 0.75 | 0.70 | 0.86 |
| GPU | Model | vLLM | EpiLoRA | Ratio |
|---|---|---|---|---|
| One adapter at batch 1, ms per token | ||||
| H100 | Qwen3.5-9B | 6.26 | 5.14 | 0.82 |
| H100 | Qwen3-8B | 6.02 | 4.81 | 0.80 |
| H100 | Llama-3.1-8B | 5.59 | 4.76 | 0.85 |
| H200 | Qwen3.5-9B | 6.11 | 4.81 | 0.79 |
| H200 | Qwen3-8B | 5.94 | 4.51 | 0.76 |
| EpiLoRA | vLLM | SGLang b | TRT-LLM | |
| Batch 1: time per token relative to EpiLoRA’s | ||||
| H100 | 1.00 | 1.25 | 1.65 | 1.43 |
| H200 | 1.00 | 1.32 | 1.81 | 1.52 |
| Batch 1: GPU time the adapter adds per token (ms) c | ||||
| H100 | 0.40 | 1.87 | 3.01 | 2.87 |
| 32 tenants: throughput relative to EpiLoRA’s | ||||
| µs | No | vLLM | EpiLoRA | Cost (%) | |||
|---|---|---|---|---|---|---|---|
| Tokens | adapter | tuned | dual | serial | overlap | vLLM | EpiLoRA |
| H100, FP8 base | |||||||
| 1 | 126 | 179 | 180 | 163 | 134 | 42 | 6 |
| 4 | 128 | 179 | 172 | 162 | 133 | 35 | 4 |
| 16 | 128 | 180 | 174 | 167 | 136 | 36 | 7 |
| H200, FP8 base | |||||||
| Median TPOT (ms) | p99 TTFT (s) | Goodput (req/s) | ||||
| Req/s | vLLM | EpiLoRA | vLLM | EpiLoRA | vLLM | EpiLoRA |
| H200 | ||||||
| 2 | 9.1 | 7.0 | 0.09 | 0.09 | 1.98 | 1.99 |
| 4 | 12.8 | 8.9 | 0.10 | 0.09 | 3.88 | 3.92 |
| 8 | 18.0 | 14.6 | 0.12 | 0.11 | 7.26 | 7.43 |
| 16 | 30.6 | 26.4 | 0.16 | 0.16 | 12.08 | 12.70 |
| 9B, one L40S | 27B, one H100 | |||
|---|---|---|---|---|
| Placement of the trainer’s base | s/step | GiB | s/step | GiB |
| One bf16 copy, KV kept a | 2,216 | 8 | OOM | |
| Two FP8 copies | OOM | OOM | ||
| bf16, trainer offloaded | 1,314 | 48 | 659 | 160 |
| FlashRL, trainer offloaded | 1,076 | 41 | 421 | 126 |
| EpiLoRA, FP8 trainer offloaded | 1,025 | 32 | 418 | 94 |
| GPU-h | |||||
|---|---|---|---|---|---|
| Placement | run | per M tok. | Gen. s | Host | Tokens |
| bf16, offloaded | 14.67 | 0.91 | 434 | 160 | 162k |
| FlashRL, offloaded | 9.84 | 0.60 | 247 | 127 | 164k |
| EpiLoRA, FP8 offloaded | 9.56 | 0.62 | 245 | 95 | 155k |
| EpiLoRA, one FP8 copy | 10.05 | 0.61 | 257 | 10 | 165k |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Part | Benefit | Cost |
|---|---|---|
| FP8 trainer weights | 6.5 GiB less GPU memory for the 9B and 22 GiB for the 27B; the 27B trains on one 48 GB L40S | None measurable: 7.2 GPU-hours, the same as FlashRL’s bf16 trainer |
| One shared copy | Pinned host memory falls from 48–159 GiB to 7–11 GiB | None measurable: 7.1 against 7.2 GPU-hours with offloading |
| KV-cache release | A 21 GiB instead of a 10 GiB KV cache during rollouts | None measurable at 2,048 tokens; at 8,192 tokens the 27B’s KV-releasing copy took 1.05 FP8 offloading’s GPU-hours |
| Unrounded (W8A16) rollouts, ours or Marlin’s | No correction needed: uncorrected runs end where W8A8 with TIS does | On the H100, 4.2 GPU-hours (Marlin 3.9) against 3.1 for W8A8 with TIS; on the L40S, 8.8 against 9.3 resident and 7.2 offloaded; with one copy at engine share 0.55, points against bf16 |
| Serving system | Many tenants | Fused | Low-precision adapters |
|---|---|---|---|
| Punica, S-LoRA | ✓ | ||
| vLLM, SGLang | ✓ | ||
| TensorRT-LLM | ✓ | ✓ | |
| SVDQuant | ✓ | ||
| LoRAFusion a | ✓ | ✓ | |
| FlexLLM b | ✓ |
| µs | vLLM’s LoRA path | Ours | PDL over | |||||
| Tokens | CUTLASS alone | default | tuned | dual-stream | previous | no PDL | PDL | CUTLASS (%) |
| H100, FP8 base | ||||||||
| 1 | 126 | 205 | 179 | 180 | 188 | 163 | 134 | +6 |
| 4 | 128 | 195 | 179 | 172 | 192 | 162 | 133 | +4 |
| 8 | 128 | 195 | 179 | 172 | 191 | 157 | 133 | +4 |
| 16 | 128 | 196 | 180 | 174 | 181 | 167 | 136 | +7 |
| µs | Serialized | GEMM, | Shrink alone | Empty | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| GPU | Rank | Decode path | empty kernel | cuBLAS GEMM | cuBLAS GEMV | ours | given | cuBLAS | ours | kernel |
| H200 | 16 | 108.1 | 110.2 | 135.0 | 134.6 | 138.9 | 106.0 | 37.6 | 48.6 | 12.8 |
| 64 | 112.4 | 114.0 | 140.1 | 139.6 | 144.6 | 110.4 | 38.6 | 50.3 | 13.3 | |
| 128 | 117.9 | 120.7 | 146.1 | 146.3 | 150.8 | 117.3 | 40.5 | 51.7 | 12.9 | |
| H100 | 16 | 129.3 | 132.0 | 154.9 | 154.5 | 156.8 | 128.7 | 42.0 | 49.4 | 20.3 |
| 64 | 133.4 | 134.8 | 158.0 | 158.5 | 163.0 | 130.9 | 42.1 | 53.4 | 19.6 | |
| GPU | rank | decode path | serialized | ratio | per projection | shrink alone | |
|---|---|---|---|---|---|---|---|
| µs | µs | its split | fastest | ||||
| H200 | 16 | 109.6 | 118.3 | 1.08 | 1.05–1.12 | 11.1–14.6 | 6.1–8.6 |
| H200 | 64 | 112.1 | 130.8 | 1.17 | 1.11–1.24 | 9.0–17.8 | 6.6–9.0 |
| H200 | 128 | 117.9 | 146.1 | 1.24 | 1.17–1.33 | 9.7–18.2 | 6.9–9.7 |
| H100 | 16 | 128.8 | 140.3 | 1.09 | 1.06–1.15 | 9.3–15.2 | 7.4–9.7 |
| H100 | 64 | 133.1 | 149.9 | 1.13 | 1.09–1.19 | 10.1–18.1 | 7.5–9.9 |
| Microseconds per layer | |||||
| GPU | Tokens | GEMM vs. CUTLASS | Epilogue | Exposed + launch | Adapter total |
| H100 | 1 | 10.2 | 2.2 | 12.5 | |
| 4 | 9.6 | 1.8 | 11.4 | ||
| 8 | 9.0 | 1.8 | 10.8 | ||
| 16 | 7.7 | 3.1 | 10.8 | ||
| H200 | 1 | 8.8 | 1.4 | 10.2 | |
| Path picked at | vLLM’s kernels’ time over EpiLoRA’s at ( ) | |||||
|---|---|---|---|---|---|---|
| Projection | 1–32 tokens | 64–2,048 tokens | 64 | 128 | 256 | 512 |
| linear attention, input (4 outputs) | decode path | EpiLoRA’s kernels | 1.31 | 1.17 | 1.62 | 1.57 |
| attention, query/key/value (3 outputs) | decode path | EpiLoRA’s kernels | 1.45 | 1.25 | 1.78 | 1.89 |
| MLP, gate/up (2 outputs) | decode path | vLLM’s kernels; EpiLoRA’s kernels at 256 | 0.87 | 0.89 | 1.00 | 0.99 |
| attention output | decode path | vLLM’s kernels | 0.81 | 0.72 | 0.84 | 0.70 |
| MLP, down | decode path | vLLM’s kernels | 0.84 | 0.74 | 0.89 | 0.78 |
| ms per step | ns per column | ||||
|---|---|---|---|---|---|
| Outputs (projections) | Columns | vLLM | EpiLoRA | vLLM | EpiLoRA |
| 3–4 unequal (32) | 376,832 | 8.14 | 4.71 | 21.6 | 12.5 |
| gate/up, 2 equal (32) | 786,432 | 10.54 | 9.52 | 13.4 | 12.1 |
| one output (64) | 262,144 | 2.08 | 3.42 | 7.9 | 13.1 |
| Shapes | K2 / CUTLASS | ||||
|---|---|---|---|---|---|
| min | median | max | |||
| Qwen3-8B | 1–16 | 16 | 0.93 | 0.96 | 1.00 |
| Qwen3-1.7B | 1 | 8 | 0.92 | 1.08 | 1.12 |
| Qwen3-0.6B | 1 | 8 | 1.15 | 1.16 | 1.24 |
| Qwen3-8B, TP2 | 1 | 4 | 0.98 | 1.15 | 1.28 |
| Qwen3-8B, TP4 | 1 | 4 | 0.97 | 1.06 | 1.23 |
| Exposed ( s) | error , median ( s) | |||||||
|---|---|---|---|---|---|---|---|---|
| GPU | Shapes | Cases | ( s) | ( s) | median | max | whole loop | first wave |
| H100 | rank 64 | 24 | 10.4–18.3 | 16.7–50.5 | 0.6 | 1.3 | 0.6 | 0.6 |
| H100 | rank 16 | 8 | 11.8–15.5 | 18.7–50.8 | 2.6 | 6.1 | 2.6 | 0.7 |
| H100 | rank 128 | 8 | 14.0–18.5 | 18.7–50.8 | 5.0 | 9.3 | 5.0 | 2.3 |
| H100 | TP 2 shards | 4 | 10.0–14.5 | 15.0–33.3 | 3.7 | 5.1 | 3.7 | 2.1 |
| H100 | TP 4 shards | 4 | 10.7–14.2 | 14.4–20.6 | 2.5 | 4.3 | 2.5 | 1.1 |
| Without LoRA | 1 adapter (%) | 32 adapters (%) | ||||
|---|---|---|---|---|---|---|
| Engine, LoRA path | W1 (ms) | W2 (tok/s) | W1 | W2 | W1 | W2 |
| SGLang 0.5.9, csgmv | 4.91 | 10,125 | +64.9 | 43.8 | +64.8 | 43.7 |
| SGLang 0.5.9, triton | 4.91 | 10,125 | +67.4 | 77.2 | +66.6 | 77.2 |
| vLLM 0.28, default kernels | 4.52 | 10,663 | +52.1 | 27.5 | +83.5 | 54.3 |
| vLLM 0.28, dual-stream | 4.52 | 10,663 | +39.3 | 30.6 | +67.8 | 54.4 |
| ours, decode path | 4.52 | 10,663 | +8.6 | 28.1 | +8.1 | 43.4 |
| Without LoRA | 1 adapter (%) | 32 adapters (%) | ||||
|---|---|---|---|---|---|---|
| Engine, LoRA path | W1 (ms) | W2 (tok/s) | W1 | W2 | W1 | W2 |
| SGLang 0.5.9, csgmv | 4.69 | 10,192 | +74.9 | 45.3 | +74.9 | 45.3 |
| SGLang 0.5.9, triton | 4.69 | 10,192 | +78.4 | 79.1 | +78.1 | 79.0 |
| vLLM 0.28, default kernels | 4.11 | 10,742 | +67.9 | 28.8 | +106.9 | 56.6 |
| vLLM 0.28, dual-stream | 4.11 | 10,742 | +52.1 | 30.8 | +90.7 | 55.9 |
| ours, decode path | 4.11 | 10,742 | +10.4 | 28.6 | +12.5 | 44.4 |
| H100 | H200 | |||
|---|---|---|---|---|
| TRT-LLM | EpiLoRA | TRT-LLM | EpiLoRA | |
| Time per token, no LoRA (ms) | 4.59 b | 4.52 | 4.08 b | 4.11 |
| Time per token, one adapter (ms) | 6.93 a | 4.85 | 6.89 a | 4.53 |
| Throughput, one adapter (tok/s) | 6,858 b | 7,703 | 7,256 b | 7,654 |
| Throughput, 32 adapters (tok/s) | 5,901 c | 6,072 c | 6,269 b | 5,985 |
| Time per token (ms) | Added GPU | ||
|---|---|---|---|
| System | no adapter | one adapter | time (ms) |
| vLLM, two streams | 4.39 | 6.52 | 1.87 |
| SGLang | 5.10 | 8.23 | 3.01 |
| TensorRT-LLM | 4.60 | 7.50 | 2.87 |
| EpiLoRA | 4.39 | 4.92 | 0.40 |
| Accuracy (%) | One | Difference | |||
| Benchmark | vLLM | EpiLoRA | only | (95%) | |
| bf16-trained adapter | |||||
| MATH-500 | 91.4 | 90.4 | 14/9 | 0.40 | |
| Olympiad | 61.8 | 63.4 | 36/47 | 0.27 | |
| Minerva | 54.4 | 54.4 | 7/7 | 1.00 | |
| W8A8-trained adapter | |||||
| Median TPOT / vLLM’s | p99 TTFT / vLLM’s | |||||||
| H100 , requests/s | 2 | 4 | 8 | 16 | 2 | 4 | 8 | 16 |
| rank 64 | 0.77 | 0.77 | 0.86 | 0.95 | 0.98 | 1.37 | 1.08 | 1.03 |
| rank 16 | 0.90 | 0.95 | 1.01 | 0.93 | 1.07 | 1.06 | 1.13 | 1.13 |
| L40S , requests/s | 1 | 2 | 4 | 6 | 1 | 2 | 4 | 6 |
| rank 64 | 0.92 | 0.95 | 0.99 | 0.97 | 1.00 | 0.98 | 0.99 | 1.21 |
| rank 64, NVFP4-stored | 0.91 | 0.88 | 0.86 | 0.86 | 1.00 | 0.98 | 0.92 | 0.10 |
| Median TPOT (ms) | p99 TTFT (ms) | ||||||
|---|---|---|---|---|---|---|---|
| GPU | Rate | vLLM | EpiLoRA | Ratio | vLLM | EpiLoRA | Ratio |
| H100 | 2 | 9.9 | 7.9 | 0.79 | 213 | 254 | 1.20 |
| 4 | 15.5 | 13.0 | 0.84 | 238 | 274 | 1.15 | |
| 8 | 28.9 | 29.0 | 1.00 | 268 | 305 | 1.14 | |
| 16 | 54.8 | 64.7 | 1.18 | 360 | 400 | 1.11 | |
| H100, new tiles | 2 | 9.9 | 7.7 | 0.78 | 214 | 242 | 1.13 |
| Median TPOT (ms) | p99 TTFT (ms) | ||||||
| GPU | Requests/s | vLLM | ours | ratio | vLLM | ours | ratio |
| Qwen3.5-9B, 32 tenants, 2,048-token prompts | |||||||
| H200 | 1 | 8.00 | 5.88 | 0.73 | 175 | 173 | 0.99 |
| 2 | 9.79 | 7.08 | 0.72 | 235 | 210 | 0.90 | |
| 4 | 16.70 | 11.09 | 0.66 | 338 | 277 | 0.82 | |
| H100 | 1 | 8.08 | 6.41 | 0.79 | 242 | 250 | 1.03 |
| H100 | H200 | L40S | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 512 tokens | 2,048 | 512 tokens | 2,048 | 512 tokens | 2,048 | |||||||
| Path, µs | host | GPU | host | GPU | host | GPU | host | GPU | host | GPU | host | GPU |
| vLLM’s quantize, GEMM and LoRA kernels | 648 | 916 | 639 | 2211 | 299 | 982 | 287 | 2332 | 460 | 1579 | 454 | 4320 |
| the vendor GEMM alone | 136 | 208 | 138 | 732 | 68 | 215 | 69 | 730 | 101 | 713 | 100 | 2952 |
| ours, hybrid | 541 | 870 | 532 | 1769 | 260 | 875 | 258 | 1756 | 405 | 1689 | 402 | 4211 |
| ours, hybrid, direct launches | 245 | 873 | 243 | 1768 | 125 | 876 | 125 | 1756 | 180 | 1686 | 181 | 4354 |
| ms | H100 | H200 | ||||||
|---|---|---|---|---|---|---|---|---|
| vLLM | EpiLoRA | own kernels | own, new tiles | vLLM | EpiLoRA | own kernels | own, new tiles | |
| Step time (ms) | 66.7 | 71.0 | 71.7 | 67.4 | 65.5 | 61.4 | 68.6 | 64.5 |
| Host time, profiled (ms) | 75.3 | 88.2 | 74.3 | 71.9 | 65.7 | 59.2 | 66.2 | 62.8 |
| GPU time (ms) | 65.6 | 60.6 | 67.3 | 62.9 | 68.0 | 59.0 | 66.5 | 61.1 |
| CUTLASS FP8 GEMM | 22.6 | 22.7 | 22.7 | 22.7 | 22.4 | 22.3 | 22.4 | 22.4 |
| down-projection (shrink) | 5.0 | 5.2 | 11.5 | 6.8 | 5.6 | 6.8 | 13.0 | 7.3 |
| Path | W1 (ms) | over no LoRA | W2 (tok/s) |
|---|---|---|---|
| vLLM, tuned | 20.40 [20.31, 20.54] | +24% | 2,141 |
| vLLM, dual, tuned | 20.31 [20.25, 20.34] | +24% | 2,017 |
| EpiLoRA, searched | 17.75 [17.70, 17.77] | +8% | 2,124 |
| Path | W1 (ms) | over no LoRA | W2, one (tok/s) | W2, 8 (tok/s) |
|---|---|---|---|---|
| vLLM, tuned for the 72B | 28.20 [28.06, 28.27] | +21% | 1,042 | 980 |
| vLLM, tuned, two streams | 29.18 [29.17, 29.19] | +25% | 994 | 926 |
| EpiLoRA, timed dispatch (searched tiles) | 26.50 [26.48, 26.51] | +13% | 1,038 | 974 |
| EpiLoRA, decode path forced | 27.32 [27.31, 27.32] | +17% | 1,041 | 975 |
| Code | Engine | Trainer, correction | GPU | Runs | GPU-h | Accuracy (%) | (points) | ||
| pre-RL | final | [95% CI] | pre-RL | ||||||
| Qwen3.5-9B math; against S0G | |||||||||
| S0G | bf16 | bf16 | H100 a | 5 | 6.9 | 49.8–50.5 | 57.4–61.8 | reference | 0/5 |
| S0T | bf16, asleep | bf16 | H100 | 3 | 5.6 | 50.5 | 61.0–61.7 | [ , ] | 0/3 |
| S8G | W8A8 | FP8 (deq.) | H100 | 5 | 5.6 | 50.6–51.5 | 2.3–60.7 | [ , ] | 1/5 |
| S4F | W8A8 | bf16 | H100 | 3 | 5.3 | 49.7–51.0 | 4.3–52.4 | [ , ] | 2/3 |
| Code | Engine | Trainer, correction | GPU | Runs | GPU-h | Accuracy (%) | (points) | |
| final | [95% CI] | pre-RL | ||||||
| Qwen3.8-27B math, MATH-500, OlympiadBench and Minerva (before RL 61.6–62.8%); against Q27 | ||||||||
| Q27 | bf16 | bf16 | 2 A100 | 3 | 15.7 | 70.8–74.6 | reference | 0/3 |
| S8T1g | W8A16, Marlin | FP8 (deq.), TIS | A100 | 3 | 15.3 | 71.9–73.3 | [ , ] | 0/3 |
| S8T27H | W8A8 | FP8 (ours), TIS | H100 | 2 + 1 b | 4.4 | 72.4–73.5 | [ , ] | 0/2 |
| S0T27A | bf16, offloaded | bf16 + FLA | A100 | 3 | 7.3 | 71.2–72.0 | [ , ] c | 0/3 |
| Each policy’s own engine | Paired by seed | One bf16 engine, avg@8 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Comparison | GPU | Runs | Welch 95% interval | Seeds | interval | Welch 95% interval | ||||
| W8A8 + TIS against tuned bf16 | H100 | 9/9 | [ , ] | 1.000 | 9 | [ , ] | [ , ] | 1.000 | ||
| FlashRL’s recipe against tuned bf16 | H100 | 9/9 | [ , ] | 1.000 | 9 | [ , ] | [ , ] | 1.000 | ||
| matched engine against tuned bf16 | H100 | 9/9 | [ , ] | 1.000 | 9 | [ , ] | [ , ] | 1.000 | ||
| W8A8 + TIS, offloaded, against bf16 offloaded | L40S | 7/7 | [ , ] | 1.000 | 0.005 | 7 | [ , ] | [ , ] | 1.000 | |
| FlashRL’s recipe, offloaded, against bf16 offloaded | L40S | 3/7 | [ , ] | 1.000 | 0.106 | 3 | [ , ] | [ , ] | 1.000 | 0.046 |
| KL over the bf16 pair’s | Fake-quant | Rounded alike | |||
|---|---|---|---|---|---|
| Adapter | W8A8 | matched W8A16 | Marlin W8A16 | share | over W16 trainer |
| bf16-trained, seed 0 | 6.47 | 1.35 | 1.34 | 0.76 | 1.35 |
| bf16-trained, seed 1 | 6.80 | 1.20 | 1.49 | 0.74 | 1.36 |
| bf16-trained, seed 2 | 7.41 | 1.51 | 1.60 | 0.86 | 1.50 |
| W8A8-trained, seed 0 | 6.31 | 1.14 | 1.30 | 0.79 | 1.49 |
| W8A8-trained, seed 1 | 7.16 | 1.38 | 1.30 | 0.76 | 1.37 |
| Mismatch, mean , at steps ( ) | |||||
|---|---|---|---|---|---|
| Configuration | Runs | 1–10 | 91–110 | 181–200 | Late/early |
| bf16 engine and trainer | 9 | 0.0055 | 0.0061 | 0.0064 | 1.18 |
| FlashRL’s recipe (W8A8, bf16 trainer, TIS) | 9 | 0.0272 | 0.0308 | 0.0348 | 1.28 |
| W8A8, FP8 trainer, TIS | 9 | 0.0173 | 0.0192 | 0.0220 | 1.27 |
| matched engine, no correction | 9 | 0.0070 | 0.0068 | 0.0075 | 1.07 |
| W8A8, FP8 trainer, no correction | 6 | 0.0174 | 0.0244 | 0.0394 | 2.26 |
| Configuration | Runs | Generation | Training | Total | Generation share |
|---|---|---|---|---|---|
| bf16 engine and trainer | 9 | 2.00 | 1.22 | 3.22 | 62% |
| FlashRL’s recipe | 9 | 1.77 | 1.19 | 2.96 | 60% |
| W8A8 with TIS, FP8 trainer | 9 | 1.74 | 1.36 | 3.10 | 56% |
| matched engine, FP8 trainer | 9 | 2.81 | 1.39 | 4.20 | 67% |
| Trainer’s base | Engine | Seconds per step | acc. | ||||||||
| Engine | Trainer | Corr. | Where | GiB | Share | KV (GiB) | Rollout | Train | GPU-h | Runs | (points) |
| Qwen3.5-9B on one L40S (48 GB), 200 steps | |||||||||||
| bf16 | bf16 | none | GPU | 16.7 | out of memory at every engine share tried | ||||||
| bf16 | bf16 | none | host | 16.7 | 0.80 | 15.3 | 105 | 46 | 8.4 | 7 | 61.8 % |
| W8A8 | bf16 | TIS | host | 16.7 | 0.80 | 22.7 | 84 | 47 | 7.2 | 3 | |
| W8A8 | FP8 (ours) | TIS | host | 10.2 | 0.80 | 21.3 | 84 | 47 | 7.2 | 7 | |
| Seconds per step | ||||||
| Model, GPU | Copy | Share | KV (GiB) | Rollout | All | Host peak (GiB) |
| Qwen3.5-9B, L40S | one bf16 | 0.55 | 5.5 | 187 | 245 | 8.6 |
| Qwen3.5-9B, L40S | one bf16 | 0.70 | 12.2 | 120 | 174 | 10.1 |
| Qwen3.8-27B, H100 | one bf16 | 0.80 | 9.1 | 136 | 165 | 14.2 |
| Qwen3.5-9B, L40S | one FP8 | 0.55 | 10.3 | 122 | 173 | 7.4 |
| Qwen3.5-9B, L40S | bf16, offloaded | 0.80 | 15.3 | 127 | 182 | 48.2 |
| Host, training | 48 GB L40S | 80 GB H100, the 27B | |||
| Placement of the trainer’s base | 27B (GiB) | 27B, 2,048 | 9B, 8,192 | 8,192 | 16,384 |
| FlashRL: FP8 engine, bf16 trainer, two copies | – | ||||
| FP8 engine and trainer, two copies | 9 | OOM | OOM | ||
| bf16, one shared copy, KV cache kept | 6 b | OOM | OOM | OOM | |
| bf16, trainer’s base offloaded (veRL-style) | 159 | 1314 | 659 | OOM | |
| FlashRL, trainer’s base offloaded | 125 b | 1076 | 421 | 606 d | |
| Placement | s/step | generation (s) | response (tokens) | trained tokens | KV (GiB) |
| 9B, one L40S | |||||
| One bf16 copy, KV kept a | 2,216 | 1,886 | 7,992 | 527k | 4.2 |
| bf16, trainer offloaded | 1,314 | 963 | 7,902 | 558k | 15.3 |
| FlashRL, trainer offloaded | 1,076 | 732 | 7,906 | 549k | 21.4 |
| EpiLoRA, FP8 trainer offloaded | 1,025 | 725 | 7,888 | 544k | 21.4 |
| EpiLoRA, one FP8 copy | 1,012 | 722 | 7,795 | 531k | 21.4 |