Multi-LoRA serving allows many task-specific adapters to share a common base model, but scaling to large adapter fleets introduces substantial storage, loading, and execution overheads. Post-hoc compression can reduce the memory footprint of trained LoRA adapters, yet smaller ranks do not automatically translate into faster serving. For instance, padding, factor access, and generic kernels can obscure the retained low-rank structure, while independently recompressing thousands of modules also incurs significant preparation overhead. We present LoRAForge, a system connecting batched factor construction to rank-specialized execution through an explicit retained-rank representation. It supports two allocation policies: uniform retention ratios (LoRAForge-U) and spectral-energy budget allocation (LoRAForge-S). Both share FlashTSQR factor construction and the same serving backend. FlashTSQR exploits parallelism across modules and row tiles, accelerating factor construction by up to 2.28× over matched library QR. In a fixed-work Qwen control, the shared runtime adds 1.17--1.22× throughput across both compressed policies, with negligible gains on original adapters. Across 85 trained open-weighted Qwen and Llama adapters, the spectral policy's floored quality configuration retains about 7.7% of adapter parameters with fleet-average score drops of at most 1.2 points on a 0--100 scale. The same rank-floored 1/12 spectral exports achieve substantial token-throughput gains on both AMD and NVIDIA GPUs.
Figures & tables
Figure 1: Rank redundancy in our two task-adapter fleets. Llama-3.1-8B-Instruct: 35 adapters and 7,840 LoRA matrices; Qwen3-4B-Instruct-2507: 50 adapters and 12,492 matrices. Energy panels (a) show the median normalized squared singular values; rank panels (b) show the CDF of removable-rank fractions at three energy thresholds, with dotted vertical lines marking the medians. Matrices are weighted equally within each fleet, and corresponding panels use shared axis ranges. At τ=.90 , median retained-rank fraction is 28.1% for Llama and 21.9% for Qwen.
Figure 2: LoRAForge Overview. Two rank-allocation policies share factor construction and a retained-rank interface to the same serving runtime.
Figure 3: FlashTSQR’s parallel QR schedule. Our batched schedule launches M×P leaf blocks across M matrices and P row tiles, then performs level-wise reductions for all M merge trees.
Figure 4: RowExpand hardware mapping. Each thread loads one B row directly into registers and reuses it across T tokens. The shared Z tile serves C output threads, each completing its rank reduction locally and adding sat to the corresponding output. Here M and N denote input tokens and output channels, km is the retained rank, and ρm≥km is the padded rank bucket width.
Retained parameter budget
Configuration
1/2
1/4
1/8
1/12
Qwen50 Base: 43.12 Original: 70.21
CTS
70.02
69.44
67.90
62.54
LoRAForge-U
69.99
69.67
69.09
68.33
LoRAForge-S
70.22
70.23
70.11
69.41
Llama35 Base: 52.20 Original: 71.13
Table 1: Task quality under matched compact-parameter ceilings. Adapter-macro native scores (0–100, ↑ ) over 21,592 Qwen and 26,336 Llama requests per arm. Shaded rows give Base and Original references; bold marks the best compressed score per budget. LoRAForge-U uses a uniform retention ratio; LoRAForge-S uses spectral allocation with rank floors. Both are LoRAForge configurations. CTS uses JD-Full.
Figure 5: Full-fleet recompression time on one GH200. FLoRIST and SpecTraL process one matrix pair per call. Both FraQ arms use the same precision, batching, and GPU FP64 eigensolver; only QR changes. Bars show median factor-construction times with min–max whiskers over five runs; brackets compare FraQ medians.
Figure 6: Single-device serving of two models and 85 adapters on AMD and NVIDIA. Each row co-locates Qwen50 and Llama35. Markers show five-seed means and bands sample SD; smooth lines guide the eye between measured loads. LoRA arms use natural stopping; Base Only replays LoRAForge-S’s per-request output lengths. Both platforms use the same rank-floored 1/12 spectral exports; compare configurations within each row. GH200 latency axes are logarithmic.
Figure 7: The same runtime benefits both compressed policies. Qwen50 at (a) 16 and (b) 32 requests/s, with matched inputs, output lengths, stopping rules, and capacity limits. Bars show mean token throughput and dots three input seeds.
Figure 8: Capacity, compression, and execution controls. GH200 Mixed85 at 128 requests/s, two paired seeds with matched output work. Bars are mean token throughput; dots are seeds and ratios divide means. (a) Original/native limited and full, then Spectral/native and Spectral/LoRAForge full. (b) Both weight sets fully resident; brackets compare execution paths. Full capacity allows extra memory and has zero timed paging.
Figure 9: Shrink optimizations serve different execution regimes. Native/tested shrink-only latency on MI250X, summarized over ranks 1–16. Tight pools match the retained rank; Pool 128 keeps capacity fixed at 128. Each bar uses its own matched native baseline. Values are geometric means of per-case median ratios, with cache flushing outside the timed interval.
Figure 10: RowExpand’s two reuse mechanisms. (a) Staged expand versus its Tmax=1 variant; (b) direct-Z versus staged-Z at the same tile limit. Ratios are geometric means over 128 decode and 128 prefill cases.
Figure 11: Rank-aware composition extends the useful kernel regimes. Qwen, tight rank capacity: geometric-mean native/tested shrink-plus-expand latency over four projection shapes. HIP pairs our custom shrink with RowExpand.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Cohort
Spectral exports
Table 1 , 1/12
Qwen50 / Llama35
Shared per model
Figure 7
Qwen50
Same as quality
AMD deployment
Qwen50 / Llama35
Same as quality
GH200 deployment
Qwen50 / Llama35
Same as AMD
Appendix
Table 2: Shared Spectral checkpoint configuration. All rows use budget 1/12 and rank floor ℓm=max(1,⌈um/2⌉) . Each model’s compressed weights are shared across its quality and serving runs.
Figure 12: Four-way mixed-fleet serving comparison in vLLM on MI250X. All configurations share a four-decode-step scheduling policy. Markers show five-seed means and bands sample SD over 1,700 requests per seed; smooth lines guide the eye. LoRA configurations use natural stopping; Base Only matches LoRAForge-S’s per-request output lengths.
Figure 13: Rank and batch crossovers on Llama35. Tight rank capacity, with forced HIP (top) and the frozen automatic policy (bottom), complementing the Qwen results in Figure 11 . Values are geometric-mean native/tested shrink-plus-expand latency ratios across four projection shapes with cache flushing. Rank 33 is omitted; forced HIP is unsupported above rank 64. Black boundaries mark 1× crossovers between measured cells.
Figure 14: Forced HIP with a native pool capacity of 128. The mathematical retained rank is unchanged; this control exposes the combined effects of native padding, kernel execution, and compact B storage.
Figure 15: Frozen automatic policy with a native pool capacity of 128. Both fleets follow the same previously frozen policy. These operator ratios do not include model execution, adapter loading, or CPU–GPU paging.
Allocation
Runtime
TTFT P50 (s)
TPOT P99 (ms)
Peak (GiB)
16
32
16
32
allocated
Original
Native
1.74
14.77
245.1
286.4
45.88
Original
LoRAForge
1.46
13.85
224.0
274.0
51.62
Uniform
Native
0.73
8.95
170.7
227.0
23.88
Uniform
LoRAForge
0.41
6.67
147.4
203.7
24.60
Spectral
Native
0.96
10.42
192.5
242.4
33.30
Appendix
Table 3: Latency and allocated-memory footprint in the six-cell control. Latency columns give the arithmetic mean of each run’s quantile across three input seeds, not a pooled quantile. Column labels 16/32 are offered requests/s. Peak is the mean PyTorch allocated-memory high-water mark over each engine’s entire lifetime, including loading, validation, and graph capture; each engine serves both rates. It is neither timed-phase-only memory nor device-wide usage.
Figure 16: Mixed-fleet gains under matched output work. MI250X at 64 requests/s, three paired seeds with identical inputs, arrivals, output lengths, and stopping rules. Bars show means and dots seeds. (a) Token throughput; the bracket gives the geometric mean of paired LoRAForge-S/Original-native ratios. Here opt. denotes the LoRAForge runtime. (b) Adapter loads during the same runs. KV and adapter-storage budgets are unchanged.
Path
tok/s
TTFT
TPOT
Loads
H2D
P50 (s)
P99 (ms)
GiB
Llama35: 2 paired seeds, provisional
Orig./native
421.5
7.96
466.8
–
–
Orig./LoRAForge
439.9
6.78
429.6
–
–
Spectral/LoRAForge
559.6
5.05
369.7
–
–
Mixed85: 3 paired seeds, complete
Appendix
Table 4: Fixed-work throughput, latency, and paging. Identical complete seed sets are used for all arms within each scenario. Latency entries average per-run quantiles, not pooled request quantiles. Loads and H2D source bytes count only the timed request window; source bytes are not a measurement of physical PCIe traffic or transfer latency. Llama rows are provisional; Mixed85 rows include all three planned seeds.
Figure 17: RowExpand versus native CSGMV. Geometric means of native/staged-RowExpand latency ratios over 128 decode and 128 prefill cases; higher is better. Tmax caps same-adapter token rows per work item. Bands span five timing-block aggregates of cache-flushed measurements. Token-reuse and staging ablations appear in Figure 10 .
Variant
Registers/thread
Static LDS (B)
Local (B)
Shared-Z
101
12,352
0
Direct-Z
128
64
52
Appendix
Table 5: Compiled RowExpand resource attributes. HIP function and occupancy APIs, 256 threads per block. Theoretical occupancy is 50% for both variants (four active blocks per compute unit).
System
Stop
Trace
Output
TTFT
TPOT
Batch
tok/s
P50 (s)
P99 (ms)
peak
Base
E-I
On
417.5
8.64
212.1
82
Base
E-N
On
467.9
7.96
221.4
81
Base
E-I
Off
417.3
8.65
212.2
–
LoRAForge-S
N
On
343.7
26.97
103.6
43
LoRAForge-S
E-I
On
360.5
10.85
245.9
81
Appendix
Table 6: Stopping-contract controls on the same engines. Seed 101, 64 offered requests/s, 1,700 successful requests per row. These are individual runs, not five-seed means. Spectral uses the optimized path.
Figure 18: Full-fleet recompression time on one MI250X GCD. Medians and min–max whiskers over five runs; boxed ratios compare FlashTSQR with stabilized FraQ. CPU decomposition and transfers are timed; input I/O, upload, and JIT are excluded. Baseline definitions and numerical checks are described in this appendix.
Configuration
req/s
tok/s
TTFT
TPOT
Original/native
12.47
511
44.39
222.4
Original/LoRAForge
6.99
285
101.12
367.1
Spectral/LoRAForge
23.58
981
6.43
40.8
Base (Spectral lengths)
37.67
1,567
1.08
31.6
Appendix
Table 7: Mixed85 at 128 offered req/s. Five-seed means on one GH200; TTFT P50 is in seconds and TPOT P99 in milliseconds; Original and Spectral retain natural stopping. Only Base is forced to the corresponding Spectral per-request output lengths.
Configuration
1/2
1/4
1/8
1/12
Qwen50
CTS
−9.80/1
−14.40/4
−35.80/11
−52.00/29
LoRAForge-U
−2.80/0
−8.00/2
−14.05/4
−31.17/5
LoRAForge-S
−2.34/0
−4.00/0
−4.00/0
−19.00/3
LoRAForge-S (w/o floor)
−1.20/0
−5.40/1
−11.60/2
−10.80/5
Llama35
Appendix
Table 8: Per-adapter regressions across retained budgets. Each entry is Δmin/N>5 relative to Original; negative Δmin indicates a score loss.
Figure 19: Four-arm serving comparisons on two frozen adapter fleets. All six offered rates use five input seeds and the same prefill/decode interval of four. Markers are measured means; bands are sample SD. Smooth connections are shape-preserving guides, not additional measurements. LoRA outputs use natural EOS or the task cap. Base replays frozen per-request Spectral lengths: the earlier Qwen backend at 8/16/32, optimized Qwen at 4/48/64, and optimized Llama at all rates. Natural-EOS throughput includes output-length and execution effects; the two fleets also differ in model size and task mix.
Output
Request
TTFT
TPOT
System
tok/s
req/s
P50 (s)
P99 (ms)
Base, length replay
661.8
12.37
11.10
217.6
Original / native
427.3
7.78
23.30
278.6
Original / optimized
401.5
7.34
27.63
225.8
Spectral / LoRAForge
562.2
10.51
14.18
243.6
Appendix
Table 9: At 64 offered requests/s: five-seed means. All 1,000-request runs complete without failed requests. Base exactly matches the optimized Spectral’s per-request output lengths at this offered rate.
Low-Rank Adaptation (LoRA) has become a key mechanism for efficiently specializing foundation models into many task- and user-specific variants. Serving many LoRA adapters under a shared resource budget turns post-hoc compression into a two-level allocation problem: rank must be distributed across layers within each adapter and across adapters within the fleet. Although trained LoRAs contain substantial spectral redundancy, spectral reconstruction alone does not determine how much compression an adapter can functionally tolerate. We formulate fleet compression through a functional rank-distortion frontier for each adapter, followed by fleet-level minimax allocation under a shared rank budget. Given exact frontiers, this decomposition is equivalent to joint optimization over all layer ranks. We introduce Functional Rank Allocation (FRA), an unlabeled method for constructing practical frontiers. FRA measures isolated module truncations end to end, uses these measurements as an additive search surrogate solved by dynamic programming, remeasures every complete candidate, and takes the lower-risk envelope with a complementary spectral proposal before exact minimax allocation over the constructed frontiers. Across 78 adapters from three public fleets, FRA improves worst-case and tail utility over uniform spectral compression in constrained-budget regimes. Fixed-budget probes show that isolated costs rank candidates well, but their calibration shifts across proposal shapes; end-to-end verification and the spectral proposal therefore limit functional-search error.
Shenghui Li, Thiemo voigt
Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden
Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance: higher-rank adapters generally achieve better downstream task performance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Conversely, ultra-low-rank adapters (r≤2) minimize both VRAM footprint and PCIe transfer overhead, but suffer from downstream task performance degradation. To solve this problem, we propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. Our solution operates in three phases. First, a provider jointly trains high-capacity domain centroids on public data within the domain using a novel alignment regularizer that coheres in-domain task subspaces into a unified basis. Next, users fine-tune ultra-low-rank task residual adapters on private data atop those frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user's task residual on demand. Across LLMs of varying scales, SALT recovers high-rank accuracy using r≤2 residuals, achieving up to 18.5% absolute accuracy gains over state-of-the-art compression baselines and reducing per-adapter memory by up to 16x. When integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints for Llama-3.2-3B.
LoRA offers a parameter-efficient adaptation strategy, supporting multi-user workloads via customized adapters on a shared backbone. However, serving concurrent requests with different adapters introduces substantial computation and memory-management overhead. We ask the research question: Do all requests need their adapters throughout the whole generation process? Our motivational study reveals that employing LoRA adapters at every decoding step is unnecessary, releasing them early has limited impact on accuracy while substantially accelerating generation. Although skipping adapters early offers theoretical workload reduction, translating these computational savings into concrete throughput gains requires tailored GPU kernels. We present LoRazor, a multi-LoRA serving framework that combines early adapter release with active-row aware GPU kernels. LoRazor determines when to permanently release a request's adapter and continue decoding with the backbone alone. By dynamically resizing LoRA kernel launches as adapters are released, LoRazor trims idle GPU blocks. Across two backbones and four tasks, LoRazor achieves up to 3.02x request throughput and 2.53x output-token throughput over vLLM multi-LoRA, with a mean relative task-accuracy decrease of 2.89%.