LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

Shenghui Li; Thiemo voigt

cs.AIOct 26, 2026

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

Authors: Shenghui Li, Thiemo voigt

Organizations: Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden

Abstract

Low-Rank Adaptation (LoRA) has become a key mechanism for efficiently specializing foundation models into many task- and user-specific variants. Serving many LoRA adapters under a shared resource budget turns post-hoc compression into a two-level allocation problem: rank must be distributed across layers within each adapter and across adapters within the fleet. Although trained LoRAs contain substantial spectral redundancy, spectral reconstruction alone does not determine how much compression an adapter can functionally tolerate. We formulate fleet compression through a functional rank-distortion frontier for each adapter, followed by fleet-level minimax allocation under a shared rank budget. Given exact frontiers, this decomposition is equivalent to joint optimization over all layer ranks. We introduce Functional Rank Allocation (FRA), an unlabeled method for constructing practical frontiers. FRA measures isolated module truncations end to end, uses these measurements as an additive search surrogate solved by dynamic programming, remeasures every complete candidate, and takes the lower-risk envelope with a complementary spectral proposal before exact minimax allocation over the constructed frontiers. Across 78 adapters from three public fleets, FRA improves worst-case and tail utility over uniform spectral compression in constrained-budget regimes. Fixed-budget probes show that isolated costs rank candidates well, but their calibration shifts across proposal shapes; end-to-end verification and the spectral proposal therefore limit functional-search error.

Figures & tables

Figure 1: LoRA fleets contain substantial compressible rank. Among

96{,}370

matrices, the median layer retains only

31.2\%

of rank at

\tau=.90

Pool				Metric	Full rank	Moderate	Binding	Severe
Fleet	$r$	$N$	Modules	Metric	Full rank	( $\tau_{\mathrm{uni}}{=}.90$ )	( $\tau_{\mathrm{uni}}{=}.70$ )	( $\tau_{\mathrm{uni}}{=}.50$ )
LoRA Land	$8$	$12$	$64\times$ (q,v)	Directions	$6{,}144$	$2{,}288$ ( $37.2\%$ )	$1{,}320$ ( $21.5\%$ )	$936$ ( $15.2\%$ )
LoRA Land	$8$	$12$	$64\times$ (q,v)	Params (M)	$40.89$	$15.28$ ( $37.4\%$ )	$8.70$ ( $21.3\%$ )	$6.09$ ( $14.9\%$ )
Lots-of-LoRAs	$16$	$25$	$96\times$ (q,k,v)	Directions	$38{,}400$	$19{,}200$ ( $50.0\%$ )	$8{,}800$ ( $22.9\%$ )	$4{,}912$ ( $12.8\%$ )
Lots-of-LoRAs	$16$	$25$	$96\times$ (q,k,v)	Params (M)	$235.93$	$119.59$ ( $50.7\%$ )	$55.57$ ( $23.6\%$ )	$30.50$ ( $12.9\%$ )
LoRARetriever	$8$	$41$	$64\times$ (q,v)	Directions	$20{,}992$	$5{,}880$ ( $28.0\%$ )	$3{,}184$ ( $15.2\%$ )	$2{,}680$ ( $12.8\%$ )

Table 1: Fleet scale and what each budget costs.

N

is the number of adapters in each pool and

r

is the nominal rank. Grouped-query attention makes a

q

direction cost

8{,}192

parameters against

5{,}120

for

k/v

, so on the Mistral fleets the parameter share departs from the rank share.

Pool	Method	Probes	Moderate ( $\tau_{\mathrm{uni}}{=}.90$ )			Binding ( $\tau_{\mathrm{uni}}{=}.70$ )			Severe ( $\tau_{\mathrm{uni}}{=}.50$ )
Pool	Method	Probes	Mean	$p_{10}$	Worst	Mean	$p_{10}$	Worst	Mean	$p_{10}$	Worst
LoRA Land	Cand. Oracle	labeled	1.009	0.990	0.966	0.983	0.949	0.879	0.921	0.878	0.672
	Uniform	✗	- $0.913$	- $0.873$	- $0.081$	- $0.725$	- $0.010$	$-0.293$	- $0.748$	- $0.047$	$-0.015$
	Intra-Spec	✗	- $0.895$	- $0.898$	- $0.036$	- $0.764$	- $0.185$	- $0.000$	- $0.554$	$-0.011$	$-0.414$
	E2E-Spec	curve	- $0.974$	- $0.920$	- $0.897$	- $0.829$	- $0.640$	- $0.203$	- $0.530$	$-0.030$	$-1.309$
	FuncDP	curve + layer	- $0.985$	- $0.949$	- $0.931$	- $0.963$	- $0.915$	- $0.879$	- $0.907$	- $0.692$	- $0.620$

Table 2: Nested ablation at matched fleet rank-direction budgets. Uniform

\rightarrow

Intra-Spec

\rightarrow

E2E-Spec

\rightarrow

FuncDP

\rightarrow

FRA progressively adds within-adapter reallocation, cross-adapter reallocation, functional inner search, and the spectral safeguard. The Candidate Oracle gives, per statistic, the best value any allocation in the union of the five families reaches at that budget, so a block’s entries need not share one allocation. FRA is bold; green marks the best deployable entry.

Rule	Probes	What they measure
Uniform	$0$	—
Intra-Spec	$0$	—
E2E-Spec	$\|\mathcal{G}\|$	each spectral proposal
FuncDP	$Lr+\|\mathcal{G}\|$	ranks $0{:}r{-}1$ , then proposals
FRA	$Lr+2\|\mathcal{G}\|$	both proposal families

Table 3: Calibration cost per adapter. Each probe evaluates 16 unlabeled prompts. Counts exclude a reusable uncompressed reference; all probes are offline.

Figure 2: Ranking fidelity does not explain the cross-proposal reversal. (a) fixed-budget Spearman fidelity, fleet means above. (b) per-adapter median surrogate versus measured risk ratio on a shared

10^{-1}

–

10^{1}

scale, colors as in (a); the legend counts spectral-favoring adapters and triangles mark the two adapters beyond the axes.

Low-Rank Compression

Low-rank compression reduces model size, memory use, and computation by representing weights, activations, or caches with compact factors.

See all 107 papers

Apr 30, 2026

Post-Optimization Adaptive Rank Allocation for LoRA

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.

Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith

1Indian Institute of Technology Hyderabad · 2Deakin University

Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Neural Network Pruning

Ask PDF

Jun 2, 2026

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment. We study the problem of merging

T

LoRAs into a single rank-

r

LoRA, thereby preserving the benefits of low-rank structure. Existing Merge-then-Compress pipelines treat the rank constraint as an afterthought: they merge adapters in the full parameter space, then compress the merged result to rank

r

via truncated SVD. However, full-parameter merging may destroy the low-rank structure, making it difficult for subsequent compression to recover an effective rank-

r

LoRA. We propose Compress-then-Merge (CtM), a reversed pipeline that enforces the rank-

r

bottleneck before merging: CtM computes shared

r

-dimensional subspaces using only the LoRA weights to capture cross-adapter common structure, projects each adapter into the shared subspaces to obtain

r\times r

coordinates, and then applies standard merging rules in this reduced space. CtM guarantees a rank-

r

LoRA by construction, avoiding post-hoc truncation, and enables efficient computation in the core space spanned by concatenated LoRA factors. Experiments across multiple models and tasks show that CtM consistently outperforms existing single-LoRA-output baselines while narrowing the performance gap to full-parameter merging methods.

Zhengbao He, Ruiqi Ding, Zhehao Huang +3

Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

Adapter Tuning Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Model Merging Parameter-Efficient Fine-Tuning

Ask PDF

Aug 4, 2026

FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation

Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data. However, LoRA's two-factor parameterization creates an aggregation mismatch across clients: naively averaging the factors does not recover the average of their induced updates. This mismatch can be avoided by forming the exact aggregate in the full weight space and then recompressing it, but decomposing the resulting dense matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that exactly represent the aggregate, FraQ factorizes it into an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying a prescribed energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show that FraQ achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.

Shenghui Li, Thiemo Voigt

Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden

Federated Learning Aggregation Communication-Efficient Distributed Training Low-Rank Matrix Decomposition Low-Rank Adaptation Low-Rank Compression Federated Fine-Tuning+1

Ask PDF

Abstract

Low-Rank Compression

Explore similar work

Post-Optimization Adaptive Rank Allocation for LoRA

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation