Existing serving systems for large language models batch requests for many different LoRA adapters on one GPU. However, these systems reject adapters from other parameter-efficient fine-tuning (PEFT) methods such as DoRA, (IA), VeRA and OFT. One solution to this is to convert these adapters to LoRA, but this induces a significant memory and accuracy cost. Instead, we show that, under some modifications, a batched LoRA kernel can serve all of these methods exactly and efficiently. We achieve this by writing each method in one normal form, a LoRA computation with six slots for an adapter's own vectors and matrices. Some slots can merge into an adapter's weights at load time; the others remain live and induce a cost. We give ways to efficiently pay this cost and predict it from a zero-parameter byte model. We package this system, Scute, and build it on top of vLLM. Scute serves all five methods, coming as checkpoints from the Hugging Face PEFT library, in one batch, and it matches that library with a mean error of at most 1.5% of each adapter's effect. Under a burst of requests on an RTX A6000, a mix of these methods takes 5-17% more decode time than the same number of LoRA adapters. Furthermore, using Scute, the same GPU can serve 64 to 1024 (IA) and VeRA adapters at 1.5x to at least 2.5x the request rate of their LoRA conversions.
Figures & tables
Figure 1: The LoRA schedule and its six slots. The shaded GEMM is shared by all adapters. Thick circles mark slots that never fold. Thin circles mark slots that fold when the adapter owns the factor next to them. The expand already reads h and writes h′ , so it can apply s , b and ek without an extra pass.
method
owned matrices
folds
live slots
LoRA
A,B
c→A
none
DoRA
A,B
e→B , c→A
s
(IA) 3
none
none
s
SSF
none
none
s , b
VeRA
none
c→d
d , e
OFT
R
none
R
Table 1: Members of the normal form ( 1 ); we identify which matrices each adapter owns (VeRA’s A,B are shared), the folds that Proposition 1 allows, and the slots that stay live.
method
route
exact
GB
ms
LoRA, rank 16
expand
✓
1.34
1.97
VeRA, rank 256
route 2 (shared factors)
✓
1.19
1.75
as LoRA, rank 256
✓
21.47
31.58
as LoRA, rank 64
✗
5.37
7.90
OFT, blocks of 32
route 3 (pre-transform)
✓
1.28
1.88
as LoRA, rank 64
✗
5.37
7.90
Table 2: The bytes that each route reads per decode step with 16 tenants in the batch, for Llama-3.1-8B (fp16, every adapted projection of 32 layers), and that time at copy bandwidth. ✓: the route serves the adapter exactly. Bold: the fewest bytes for each method.
Figure 2: Predicted against measured time on the RTX A6000, for OFT’s input transform and for four routes that apply DoRA’s scale outside the expand (App. F ). The line marks equality.
Figure 3: Time per output token under a decode burst against the number of tenants of one method, RTX A6000. Scute serves each adapter exactly. Stock serves each adapter converted to LoRA, at rank 64 and, for OFT, also at rank 256, with every adapter resident. The base model, without adapters, is shown for reference. Stock and the base model run as shipped. With OFT, Scute runs the split base GEMM of § 6.1 , which is faster than the fused base GEMM. Median of six repetitions.
LoRA of rank 64
method
task
PEFT
Scute
SVD
ASVD
ASVD g
dist.
(IA) 3
GSM8K
59.8
+0.4
− 18.6
− 0.8
− 19.6
− 0.4
MNLI
90.2
0.0
− 23.2
+0.5
− 16.9
− 0.3
BoolQ
87.9
− 0.1
− 4.8
+0.8
− 1.6
+0.3
OFT
GSM8K
63.0
+0.2
− 5.6
− 1.2
− 4.6
− 2.0
MNLI
90.9
0.0
− 2.9
+0.3
− 0.9
− 0.1
Table 3: Task accuracy of the trained (IA) 3 , OFT and VeRA adapters. PEFT is the accuracy in percent when the PEFT library runs the adapter. The other columns give the difference from PEFT in points, in Scute (all fifteen adapters in one batch) and converted to a LoRA of rank 64. Bold marks the smallest absolute difference in each row. SVD is a truncated SVD of the update. ASVD is activation-aware SVD fitted on 128 examples of the tenant’s own task, and ASVD g is the same fitted on generic text. Dist. is distillation on the tenant’s task. VeRA was not distilled. App. C gives other ranks and fits.
Figure 4: Share of requests that meet the latency target (first token within 1 s, then at most 100 ms per token) against arrival rate, under decode-heavy load on one RTX A6000. The pools hold 64, 256 and 1024 tenants, half (IA) 3 and half VeRA. Scute serves them exactly, and stock serves their rank-64 conversions. Median of three repetitions.
Scute /
Conversion /
Conversion /
pool
load
LoRA-only
Scute
Scute (stock)
five methods
decode
1.17
1.65
1.73
prefill
1.18
1.14
1.12
VeRA at 64
decode
1.11
1.12
—
prefill
1.12
0.94
—
no VeRA
decode
1.05
1.14
1.25
Table 4: Here we measure serving times for pools of several methods in one engine, under a burst, RTX A6000. The five methods pool holds 20 tenants, four of each method, and stock keeps 16 of them resident. VeRA at 64 is a separate run of the same pool, with stock serving VeRA converted at rank 64 instead of 256. The no VeRA pool holds 16 tenants, four each of LoRA, DoRA, (IA) 3 and OFT. We measure the ratios of serving times for Scute against serving the same number of plain rank-16 LoRA adapters, for the conversions against Scute on the same GEMM path, and for the conversions on vLLM as shipped against Scute. Ratios are the median over six repetitions. App. G gives every rate of vLLM as shipped.
Figure 5: Latency against arrival rate in the two pools, RTX A6000. Scute decodes faster than stock at every rate, and under prefill-heavy load stock’s TTFT grows fastest. Stock, LoRA-only and the base model run as shipped, so the gap between Scute and LoRA-only differs from the same-GEMM-path ratio of Table 4 . Top, median TPOT under decode-heavy load. Bottom, mean TTFT under prefill-heavy load. Median of six repetitions.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
degree
sharding
slices
max ∣Δg∣
2
column-parallel
512
0
2
row-parallel
256
2.4×10−7
4
column-parallel
512
0
4
row-parallel
512
2.4×10−7
8
column-parallel
512
0
8
row-parallel
1024
2.4×10−7
Appendix
Table A1: DoRA’s g computed on real tensor-parallel shards against g computed on one GPU, four DoRA adapters.
proj.
shape
rank
left, r=16
rms g−1
q
4096×4096
4096
61.6%
0.0022
k
4096×1024
1024
59.7%
0.0019
v
4096×1024
1024
83.2%
0.0031
up
4096×14336
4096
92.8%
0.0023
down
14336×4096
4096
95.9%
0.0018
Appendix
Table A2: Wdiag(g−1) for each projection, of shape din×dout : the minimum numerical rank, the minimum share of its squared Frobenius norm outside the best rank-16 approximation, and the median RMS deviation of g , over four adapters and three layers.
rank 16
rank 64
method
task
PEFT
SVD
ASVD
ASVD x
ASVD g
dist.
SVD
ASVD
ASVD x
ASVD g
dist.
(IA) 3
GSM8K
59.8
− 29.8
− 24.6
− 34.4
− 32.8
− 3.2
− 18.6
− 0.8
− 37.8
− 19.6
− 0.4
MNLI
90.2
− 35.9
− 0.9
− 23.2
− 25.6
+0.3
− 23.2
+0.5
− 15.4
− 16.9
− 0.3
BoolQ
87.9
− 10.4
− 0.1
− 2.1
− 4.4
+0.5
− 4.8
+0.8
− 0.3
− 1.6
+0.3
OFT
GSM8K
63.0
− 23.4
− 3.0
− 12.6
− 16.2
− 2.4
− 5.6
− 1.2
− 2.0
− 4.6
− 2.0
MNLI
90.9
− 10.7
+0.3
− 6.5
− 9.6
− 0.8
− 2.9
+0.3
− 0.6
− 0.9
− 0.1
Appendix
Table A3: Every conversion of the trained (IA) 3 , OFT and VeRA adapters (VeRA not distilled): SVD, ASVD fitted on the task’s own data, on another task’s data (x) and on generic text (g), and distillation, evaluated with the training harness. PEFT is the accuracy in percent, and the other columns give the difference from PEFT in points.
converted to LoRA, rank
method
task
PEFT
Scute
16
64
256
(IA) 3
GSM8K
59.8
+0.4
− 29.8
− 18.2
− 6.2
MNLI
90.2
0.0
− 35.8
− 23.2
− 7.1
BoolQ
87.9
− 0.1
− 10.5
− 4.7
+0.5
OFT
GSM8K
63.0
+0.2
− 23.4
− 5.6
+0.8
MNLI
90.9
0.0
− 10.7
− 2.9
+0.1
Appendix
Table A4: Every trained adapter. PEFT is the accuracy in percent when the PEFT library runs the adapter. The other columns give the difference from PEFT in points, in Scute with all fifteen in one batch, and converted to LoRA by SVD and served by stock vLLM. Bold marks the smallest absolute difference in each row. For LoRA, the rank-16 column is the adapter as trained. Stock converts VeRA exactly, at its own rank of 256; Table A3 gives its truncated fits. The base model scores 26.2%, 36.8% and 74.0%.
method
task
lr
steps
metric
LoRA
GSM8K
0.0002
467
0.664
LoRA
MNLI
0.0002
1000
0.908
LoRA
BoolQ
0.0002
588
0.900
DoRA
GSM8K
0.0002
467
0.660
DoRA
MNLI
0.0002
1000
0.919
DoRA
BoolQ
0.0002
588
0.887
Appendix
Table A5: The trained adapters: learning rate, optimizer steps, and the task metric in the PEFT library.
method
effect (nats/token)
error / effect, TP 1
TP 2
LoRA
0.20
1.27%
1.51%
DoRA
1.98
0.55%
0.47%
(IA) 3
1.09
0.54%
0.55%
VeRA
0.22
1.41%
1.49%
OFT
0.86
0.34%
0.37%
Appendix
Table A6: One batch of 9 amplified adapters against the PEFT library in float32. The effect is the mean change in log-probability per token that the adapter, run by the PEFT library, makes to the base model. The error is the engine’s largest mean error per method, as a share of that effect, on one GPU and on two GPUs with tensor parallelism.
route
cells
increment
total
launch
side kernel
27
3.2%
1.4%
2%
gathered scale
27
7.4%
3.4%
23%
index scale
27
20.0%
14.0%
24%
vLLM outside
28
1.2%
0.7%
21%
OFT transform
120
12.3%
1.2%
—
Appendix
Table A7: Median absolute error of the byte model on the RTX A6000. Increment : error on the time that the extra passes add. Total : error on the route’s total time. Both are over the cells where ΔQ/β>λ . Launch : error on total time over the other, launch-bound cells.
stock /
Scute /
pool
load
req/s
Scute
LoRA-only
20 tenants
decode
4
1.92 [1.89, 1.94]
1.13 [1.13, 1.14]
decode
8
1.82 [1.78, 1.83]
1.09 [1.06, 1.12]
decode
16
1.66 [1.65, 1.67]
1.09 [1.08, 1.09]
decode
burst
1.73 [1.73, 1.75]
1.05 [1.05, 1.06]
prefill
2
1.36 [1.33, 1.43]
1.23 [1.15, 1.24]
Appendix
Table A8: Every rate of the mixed pools on the RTX A6000, vLLM as shipped: median per-repetition ratio and 95% interval. TPOT for decode-heavy load, mean TTFT for prefill-heavy load.
method
arm
predicted
measured
VeRA
Scute
0.00
0.05
rank 256
1.97
1.94
OFT
Scute
0.12
0.16
rank 64
0.49
0.53
rank 256
1.97
1.94
(IA) 3
Scute
0.00
0.12
Appendix
Table A9: Decode time added per tenant (ms) on the RTX A6000: the byte model’s prediction against the least-squares slope of TPOT over 1–16 tenants. Stock is counted on every LoRA layer at the padded rank, because vLLM runs its kernels there for every adapter.
20 tenants
16, no VeRA
conversions at own rank
13.0
5.1
+ zero factors elsewhere
14.3
6.4
+ padding (vLLM)
31.6
7.9
stock, measured
34.7
10.5
Scute, predicted
3.1
1.5
Scute, measured
7.1
2.4
Appendix
Table A10: Decode time over the base model under a decode burst (ms), vLLM as shipped. The byte model’s account of stock, built up step by step, against stock’s measured time and Scute’s.
burst
method
tenants
ratio
TPOT
TTFT
VeRA
1
stock / Scute
1.06
1.04
4
stock / Scute
1.15
1.05
16
stock / Scute
1.83
1.10
OFT
1
rank 64 / Scute
1.07
1.00
1
rank 256 / Scute
1.23
1.13
Appendix
Table A11: The A100: median per-repetition ratio under a decode burst (TPOT) and a prefill burst (mean TTFT), stock with every adapter resident.
stock /
Scute /
pool
load
req/s
Scute
LoRA-only
20 tenants
decode
8
1.69 [1.68, 1.71]
1.34 [1.28, 1.36]
decode
16
1.69 [1.64, 1.70]
1.26 [1.25, 1.30]
decode
burst
1.63 [1.62, 1.63]
1.23 [1.23, 1.23]
prefill
4
0.77 [0.70, 0.81]
1.38 [1.32, 1.65]
prefill
16
0.95 [0.84, 1.43]
1.89 [1.81, 2.02]
Appendix
Table A12: The mixed pools on the A100: median per-repetition ratio and 95% interval. TPOT for decode-heavy load, mean TTFT for prefill-heavy load.
tenants
8
12
16
20
24
64 Scute
100
100
100
100
47
stock
100
83
21
14
12
256 Scute
100
70
12
12
12
stock
100
24
13
12
12
1024 Scute
100
33
12
12
9
stock
32
10
4
2
1
Appendix
Table A13: The A100: requests that meet the latency target (%) against decode-heavy arrival rate (requests per second), median of three repetitions.