Multi-LoRA serving allows many task-specific adapters to share a common base model, but scaling to large adapter fleets introduces substantial storage, loading, and execution overheads. Post-hoc compression can reduce the memory footprint of trained LoRA adapters, yet smaller ranks do not automatically translate into faster serving. For instance, padding, factor access, and generic kernels can obscure the retained low-rank structure, while independently recompressing thousands of modules also incurs significant preparation overhead. We present LoRAForge, a system connecting batched factor construction to rank-specialized execution through an explicit retained-rank representation. It supports two allocation policies: uniform retention ratios (LoRAForge-U) and spectral-energy budget allocation (LoRAForge-S). Both share FlashTSQR factor construction and the same serving backend. FlashTSQR exploits parallelism across modules and row tiles, accelerating factor construction by up to 2.28× over matched library QR. In a fixed-work Qwen control, the shared runtime adds 1.17--1.22× throughput across both compressed policies, with negligible gains on original adapters. Across 85 trained open-weighted Qwen and Llama adapters, the spectral policy's floored quality configuration retains about 7.7% of adapter parameters with fleet-average score drops of at most 1.2 points on a 0--100 scale. The same rank-floored 1/12 spectral exports achieve substantial token-throughput gains on both AMD and NVIDIA GPUs.
Figures & tables
Figure 1: Rank redundancy in our two task-adapter fleets. Llama-3.1-8B-Instruct: 35 adapters and 7,840 LoRA matrices; Qwen3-4B-Instruct-2507: 50 adapters and 12,492 matrices. Energy panels (a) show the median normalized squared singular values; rank panels (b) show the CDF of removable-rank fractions at three energy thresholds, with dotted vertical lines marking the medians. Matrices are weighted equally within each fleet, and corresponding panels use shared axis ranges. At τ=.90 , median retained-rank fraction is 28.1% for Llama and 21.9% for Qwen.
Figure 2: LoRAForge Overview. Two rank-allocation policies share factor construction and a retained-rank interface to the same serving runtime.
Figure 3: FlashTSQR’s parallel QR schedule. Our batched schedule launches M×P leaf blocks across M matrices and P row tiles, then performs level-wise reductions for all M merge trees.
Figure 4: RowExpand hardware mapping. Each thread loads one B row directly into registers and reuses it across T tokens. The shared Z tile serves C output threads, each completing its rank reduction locally and adding sat to the corresponding output. Here M and N denote input tokens and output channels, km is the retained rank, and ρm≥km is the padded rank bucket width.
Retained parameter budget
Configuration
1/2
1/4
1/8
1/12
Qwen50 Base: 43.12 Original: 70.21
CTS
70.02
69.44
67.90
62.54
LoRAForge-U
69.99
69.67
69.09
68.33
LoRAForge-S
70.22
70.23
70.11
69.41
Llama35 Base: 52.20 Original: 71.13
Table 1: Task quality under matched compact-parameter ceilings. Adapter-macro native scores (0–100, ↑ ) over 21,592 Qwen and 26,336 Llama requests per arm. Shaded rows give Base and Original references; bold marks the best compressed score per budget. LoRAForge-U uses a uniform retention ratio; LoRAForge-S uses spectral allocation with rank floors. Both are LoRAForge configurations. CTS uses JD-Full.
Figure 5: Full-fleet recompression time on one GH200. FLoRIST and SpecTraL process one matrix pair per call. Both FraQ arms use the same precision, batching, and GPU FP64 eigensolver; only QR changes. Bars show median factor-construction times with min–max whiskers over five runs; brackets compare FraQ medians.
Figure 6: Single-device serving of two models and 85 adapters on AMD and NVIDIA. Each row co-locates Qwen50 and Llama35. Markers show five-seed means and bands sample SD; smooth lines guide the eye between measured loads. LoRA arms use natural stopping; Base Only replays LoRAForge-S’s per-request output lengths. Both platforms use the same rank-floored 1/12 spectral exports; compare configurations within each row. GH200 latency axes are logarithmic.
Figure 7: The same runtime benefits both compressed policies. Qwen50 at (a) 16 and (b) 32 requests/s, with matched inputs, output lengths, stopping rules, and capacity limits. Bars show mean token throughput and dots three input seeds.
Figure 8: Capacity, compression, and execution controls. GH200 Mixed85 at 128 requests/s, two paired seeds with matched output work. Bars are mean token throughput; dots are seeds and ratios divide means. (a) Original/native limited and full, then Spectral/native and Spectral/LoRAForge full. (b) Both weight sets fully resident; brackets compare execution paths. Full capacity allows extra memory and has zero timed paging.
Figure 9: Shrink optimizations serve different execution regimes. Native/tested shrink-only latency on MI250X, summarized over ranks 1–16. Tight pools match the retained rank; Pool 128 keeps capacity fixed at 128. Each bar uses its own matched native baseline. Values are geometric means of per-case median ratios, with cache flushing outside the timed interval.
Figure 10: RowExpand’s two reuse mechanisms. (a) Staged expand versus its Tmax=1 variant; (b) direct-Z versus staged-Z at the same tile limit. Ratios are geometric means over 128 decode and 128 prefill cases.
Figure 11: Rank-aware composition extends the useful kernel regimes. Qwen, tight rank capacity: geometric-mean native/tested shrink-plus-expand latency over four projection shapes. HIP pairs our custom shrink with RowExpand.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Cohort
Spectral exports
Table 1 , 1/12
Qwen50 / Llama35
Shared per model
Figure 7
Qwen50
Same as quality
AMD deployment
Qwen50 / Llama35
Same as quality
GH200 deployment
Qwen50 / Llama35
Same as AMD
Appendix
Table 2: Shared Spectral checkpoint configuration. All rows use budget 1/12 and rank floor ℓm=max(1,⌈um/2⌉) . Each model’s compressed weights are shared across its quality and serving runs.
Figure 12: Four-way mixed-fleet serving comparison in vLLM on MI250X. All configurations share a four-decode-step scheduling policy. Markers show five-seed means and bands sample SD over 1,700 requests per seed; smooth lines guide the eye. LoRA configurations use natural stopping; Base Only matches LoRAForge-S’s per-request output lengths.
Figure 13: Rank and batch crossovers on Llama35. Tight rank capacity, with forced HIP (top) and the frozen automatic policy (bottom), complementing the Qwen results in Figure 11 . Values are geometric-mean native/tested shrink-plus-expand latency ratios across four projection shapes with cache flushing. Rank 33 is omitted; forced HIP is unsupported above rank 64. Black boundaries mark 1× crossovers between measured cells.
Figure 14: Forced HIP with a native pool capacity of 128. The mathematical retained rank is unchanged; this control exposes the combined effects of native padding, kernel execution, and compact B storage.
Figure 15: Frozen automatic policy with a native pool capacity of 128. Both fleets follow the same previously frozen policy. These operator ratios do not include model execution, adapter loading, or CPU–GPU paging.
Allocation
Runtime
TTFT P50 (s)
TPOT P99 (ms)
Peak (GiB)
16
32
16
32
allocated
Original
Native
1.74
14.77
245.1
286.4
45.88
Original
LoRAForge
1.46
13.85
224.0
274.0
51.62
Uniform
Native
0.73
8.95
170.7
227.0
23.88
Uniform
LoRAForge
0.41
6.67
147.4
203.7
24.60
Spectral
Native
0.96
10.42
192.5
242.4
33.30
Appendix
Table 3: Latency and allocated-memory footprint in the six-cell control. Latency columns give the arithmetic mean of each run’s quantile across three input seeds, not a pooled quantile. Column labels 16/32 are offered requests/s. Peak is the mean PyTorch allocated-memory high-water mark over each engine’s entire lifetime, including loading, validation, and graph capture; each engine serves both rates. It is neither timed-phase-only memory nor device-wide usage.
Figure 16: Mixed-fleet gains under matched output work. MI250X at 64 requests/s, three paired seeds with identical inputs, arrivals, output lengths, and stopping rules. Bars show means and dots seeds. (a) Token throughput; the bracket gives the geometric mean of paired LoRAForge-S/Original-native ratios. Here opt. denotes the LoRAForge runtime. (b) Adapter loads during the same runs. KV and adapter-storage budgets are unchanged.
Path
tok/s
TTFT
TPOT
Loads
H2D
P50 (s)
P99 (ms)
GiB
Llama35: 2 paired seeds, provisional
Orig./native
421.5
7.96
466.8
–
–
Orig./LoRAForge
439.9
6.78
429.6
–
–
Spectral/LoRAForge
559.6
5.05
369.7
–
–
Mixed85: 3 paired seeds, complete
Appendix
Table 4: Fixed-work throughput, latency, and paging. Identical complete seed sets are used for all arms within each scenario. Latency entries average per-run quantiles, not pooled request quantiles. Loads and H2D source bytes count only the timed request window; source bytes are not a measurement of physical PCIe traffic or transfer latency. Llama rows are provisional; Mixed85 rows include all three planned seeds.
Figure 17: RowExpand versus native CSGMV. Geometric means of native/staged-RowExpand latency ratios over 128 decode and 128 prefill cases; higher is better. Tmax caps same-adapter token rows per work item. Bands span five timing-block aggregates of cache-flushed measurements. Token-reuse and staging ablations appear in Figure 10 .
Variant
Registers/thread
Static LDS (B)
Local (B)
Shared-Z
101
12,352
0
Direct-Z
128
64
52
Appendix
Table 5: Compiled RowExpand resource attributes. HIP function and occupancy APIs, 256 threads per block. Theoretical occupancy is 50% for both variants (four active blocks per compute unit).
System
Stop
Trace
Output
TTFT
TPOT
Batch
tok/s
P50 (s)
P99 (ms)
peak
Base
E-I
On
417.5
8.64
212.1
82
Base
E-N
On
467.9
7.96
221.4
81
Base
E-I
Off
417.3
8.65
212.2
–
LoRAForge-S
N
On
343.7
26.97
103.6
43
LoRAForge-S
E-I
On
360.5
10.85
245.9
81
Appendix
Table 6: Stopping-contract controls on the same engines. Seed 101, 64 offered requests/s, 1,700 successful requests per row. These are individual runs, not five-seed means. Spectral uses the optimized path.
Figure 18: Full-fleet recompression time on one MI250X GCD. Medians and min–max whiskers over five runs; boxed ratios compare FlashTSQR with stabilized FraQ. CPU decomposition and transfers are timed; input I/O, upload, and JIT are excluded. Baseline definitions and numerical checks are described in this appendix.
Configuration
req/s
tok/s
TTFT
TPOT
Original/native
12.47
511
44.39
222.4
Original/LoRAForge
6.99
285
101.12
367.1
Spectral/LoRAForge
23.58
981
6.43
40.8
Base (Spectral lengths)
37.67
1,567
1.08
31.6
Appendix
Table 7: Mixed85 at 128 offered req/s. Five-seed means on one GH200; TTFT P50 is in seconds and TPOT P99 in milliseconds; Original and Spectral retain natural stopping. Only Base is forced to the corresponding Spectral per-request output lengths.
Configuration
1/2
1/4
1/8
1/12
Qwen50
CTS
−9.80/1
−14.40/4
−35.80/11
−52.00/29
LoRAForge-U
−2.80/0
−8.00/2
−14.05/4
−31.17/5
LoRAForge-S
−2.34/0
−4.00/0
−4.00/0
−19.00/3
LoRAForge-S (w/o floor)
−1.20/0
−5.40/1
−11.60/2
−10.80/5
Llama35
Appendix
Table 8: Per-adapter regressions across retained budgets. Each entry is Δmin/N>5 relative to Original; negative Δmin indicates a score loss.
Figure 19: Four-arm serving comparisons on two frozen adapter fleets. All six offered rates use five input seeds and the same prefill/decode interval of four. Markers are measured means; bands are sample SD. Smooth connections are shape-preserving guides, not additional measurements. LoRA outputs use natural EOS or the task cap. Base replays frozen per-request Spectral lengths: the earlier Qwen backend at 8/16/32, optimized Qwen at 4/48/64, and optimized Llama at all rates. Natural-EOS throughput includes output-length and execution effects; the two fleets also differ in model size and task mix.
Output
Request
TTFT
TPOT
System
tok/s
req/s
P50 (s)
P99 (ms)
Base, length replay
661.8
12.37
11.10
217.6
Original / native
427.3
7.78
23.30
278.6
Original / optimized
401.5
7.34
27.63
225.8
Spectral / LoRAForge
562.2
10.51
14.18
243.6
Appendix
Table 9: At 64 offered requests/s: five-seed means. All 1,000-request runs complete without failed requests. Base exactly matches the optimized Spectral’s per-request output lengths at this offered rate.