LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

Thamirawaran Sathiyalogeswaran; Navindu De Silva; Chamikara Siriwardane; W. K. Alwis; Jason Mars; Udaya Sampath K. Perera Miriya Thanthrige; Ranga Rodrigo

cs.CVOct 20, 2026

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

Authors: Thamirawaran Sathiyalogeswaran, Navindu De Silva, Chamikara Siriwardane, W. K. Alwis, Jason Mars, Udaya Sampath K. Perera Miriya Thanthrige, Ranga Rodrigo

Organizations: University of Moratuwa, Sri Lanka · University of Michigan, USA

Abstract

Low-rank adaptation (LoRA) attaches small low-rank matrices to a frozen backbone, and public repositories host many LoRAs per backbone, each for a different concept or task. Combining several LoRAs in one generation forces a trade-off. Merging their weights into one update needs a single forward pass, but LoRAs trained for different concepts interfere and quality drops. Running each LoRA in its own forward pass (composite) preserves quality, but inference time grows linearly with the number of active LoRAs. We study a property of LoRA pools overlooked by prior multi-LoRA work. In real LoRA pools, the stacked low-rank factors need far fewer directions than their summed ranks. Using singular value decomposition, we find the minimal subspace that keeps 99% of the factor energy. On three diffusion pools, only 46% to 57% of the summed LoRA ranks is needed, and the subspace grows sub-linearly with pool size. A 48-LoRA sub-pool of LoRA-Hub on FLAN-T5 tasks shows comparable overlap (62%), while the diversity-curated LoRA-Retriever benchmark sits near the no-overlap baseline (96%), so pool composition rather than modality predicts where compression is large. We build on this overlap with LoRASpace, a training-free method that computes the subspace once per pool and reuses it in two inference modes, a static mode at merge cost and a dynamic mode for input-dependent routing. On the diffusion pools, static LoRASpace matches composite quality at merge cost, and dynamic LoRASpace is more than 6x faster than composite with six active LoRAs on a 17-LoRA pool, mainly because it runs one backbone pass instead of six. In an ablation, static variants at the same rank match LoRASpace, so the quality gain does not depend on the pool-wide basis. On the near-orthogonal LoRA-Retriever benchmark, compression is small and gains come from coupling with external retrieval.

Figures & tables

Method family (representative work)	Pool-wide basis	Active per step	Mode	Per-image cost
Static merge (Task Arith. ( Ilharco et al., 2023 ) , TIES ( Yadav et al., 2023 ) , DARE ( Yu et al., 2024 ) )	X	$N$	static	$1\times$
Shared-basis merge (Knots ( Stoica et al., 2025 ) , Core-Space ( Panariello et al., 2025 ) )	per merge	$N$	static	$1\times$
Composite & step/mask routing	X	$N$	dynamic	$\sim N\times$
(ComposLoRA ( Zhong et al., 2024 ) , CMLoRA ( Zou et al., 2025 ) , FreeFuse ( Liu et al., 2025 ) , MultLFG ( Roy et al., 2025 ) )
Per-input router-bound composition	X	$1$ to $N$	dynamic	$\sim N\times$
(SpectR ( Fleshman and Van Durme, 2025 ) , K-LoRA ( Ouyang et al., 2025 ) , FreeLoRA ( Zheng et al., 2025 ) , LoRAHub ( Huang et al., 2024 ) , LoRA-Retriever ( Zhao et al., 2024 ) )

Table 1: Where LoRASpace sits among training-free multi-LoRA methods. Columns: pool-wide basis

(U,V)

(defined in § 3.4 ), active LoRAs per step, deployment mode (static vs. dynamic), and per-image cost normalised to ambient merge (

1\times

). To our knowledge, no other reviewed family combines a pool-wide basis with simultaneous

N{\geq}2

composition and both deployment modes from one precompute (Eq. ( 5 )).

Panel of Figure 1 — Figure 1: Three strategies for composing $N$ active LoRAs from a Combo (the active subset for one generation). Merge folds them into one delta with one forward pass. Composite runs $N$ forward passes. LoRASpace projects via $V^{\top}$ , mixes $\sum_{i}w_{i}\hat{B}_{i}\hat{A}_{i}$ , and back-projects via $U$ in one pass. Static-mode pre-folds the inner sum when $\{w_{i}\}$ are time-invariant; dynamic-mode evaluates it per UNet pass (precompute illustration in App. A ).

Pool	$K$	$\sum_{i}r_{i}$	$D$ (mean)	$\rho$	$\beta$
Pool A (SD 1.5)	11	784	446	0.57	0.43
Pool B (SD 1.5)	17	1112	521	0.47	0.49
Pool C (SDXL)	17	1792	819	0.46	0.55
LoRA-Hub (FLAN-T5-Large)	48	768	474	0.62	0.88
LoRA-Retriever (LLaMA-2-7B)	48	384	368	0.96	0.94

Table 2: Rank overlap across pools at

\tau=0.99

\rho=D/\sum_{i}r_{i}

(lower means more overlap).

\beta

is the power-law exponent of

D

vs.

K

. The LoRA-Hub row is a

K{=}48

sub-sample matched in size to LoRA-Retriever (full 196-LoRA pool in App. T ).

Method	CLIP-T $\uparrow$	ImageReward $\uparrow$	GPT-4o Total $\uparrow$	GPT-4o Win% $\uparrow$
merge	$33.60\pm 0.19$	$0.760\pm 0.008$	14.32	21.9
composite	$35.27\pm 0.19$	$0.953\pm 0.011$	15.24	29.5
LoRASpace (uniform)	$\mathbf{36.01\pm 0.16}$	$\mathbf{1.041\pm 0.035}$	15.61	48.6

Table 3: Pool A quality (mean

\pm

std over 3 seeds,

240

combos/seed, averaged over

N\in\{2,\dots,5\}

, with GPT-4o from one seed). Higher is better for all metrics.

Figure 2: Pool C (SDXL) speed and memory. Left: per-image wall-clock vs.

N

. Composite scales linearly. Dynamic LoRASpace stays near-flat (

6.1\times

speedup at

N{=}6

). Static LoRASpace inherits ambient-merge cost. Right: GPU memory. LoRASpace has the lowest load and peak memory (

6.5

16.2

GB). PEFT baselines (merge, switch, composite, cmlora) exceed

22

GB peak. RTX 3090,

1024^{2}

, 50 DPM-Solver++ steps.

Panel of Figure 3 — Figure 3: Pool C $N{=}5$ example. VioletXL + catwalk + god rays + old copper + anger, SDXL. L $\to$ R : merge , composite , LoRASpace . Time $36/187/42$ s.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Pool-wide precompute (paid once at pool-creation time). Per-LoRA factors are stacked and decomposed via paired marginal SVDs into shared bases $(U,V)$ (red) plus projected factors $(\hat{B}_{i},\hat{A}_{i})$ . The cache is reused across all subsequent compositions.

Method	Per-step compute	Peak GPU memory	Scaling in $N$
merge	$O(Ld^{2})$	$O(Ld^{2})$	$\Theta(1)$
switch	$O(L(d^{2}+\bar{r}d))$	$O(L(d^{2}+K_{\mathrm{pool}}\bar{r}d))$	compute $\Theta(1)$ , mem $\Theta(K_{\mathrm{pool}})$
composite	$O(NLd^{2}S)$	$O(L(d^{2}+K_{\mathrm{pool}}\bar{r}d))$	both $\Theta(N)$
cmlora	$O(NLd^{2}S)$	$O(L(d^{2}+K_{\mathrm{pool}}\bar{r}d))$	both $\Theta(N)$
knots	$O(Ld^{2})$	$O(Ld^{2})$	$\Theta(1)$
LoRASpace	$O(L(dD+N\bar{r}D)S)$ , $D\leq\min(\textstyle\sum_{i}r_{i},d)$	$O(L(d^{2}+dD))$ GPU $+$ $O(LK_{\mathrm{pool}}D\bar{r})$ CPU	linear in $N$ inside subspace; $D$ caps with $K$

Appendix

Table 4: Per-step compute and peak GPU memory of training-free multi-LoRA composition methods.

N

is active LoRAs per call,

\bar{r}

is representative rank,

d

is layer width,

L

is LoRA-bearing layers,

S

is tokens per layer, and

K_{\mathrm{pool}}

is pool size. LoRASpace ’s

D\leq\min(\sum_{i}r_{i},d)

saturates at the layer width.

$\tau$	$D_{\min}$	$D$ (mean)	$D_{\max}$	CLIP-T	ImageReward
$0.95$	224	356	568	35.76	1.071
$0.97$	248	392	611	35.81	1.072
$0.99$	283	446	663	35.77	1.064
$0.999$	314	508	741	35.80	1.067

Appendix

Table 5: Pool A energy-threshold ablation. Subspace dimension

D

and uniform- LoRASpace quality at four

\tau

values (mean over

N\in\{2,\dots,5\}

240

combos, seed

111

). Quality varies non-monotonically within seed-level noise while

D

grows by

1.43\times

$K$	$\sum_{i}r_{i}$ (mean)	$D$ (mean $\pm$ std)	$\rho$
2	224	$213\pm 50$	0.95
4	304	$275\pm 22$	0.91
6	456	$353\pm 35$	0.77
8	584	$397\pm 38$	0.68
11	784	$446\pm 0$	0.57

Appendix

Table 6: Pool A rank-fill ratio

\rho

vs. pool size

K

(mean

\pm

std over

3

random subsets per

K

;

K{=}11

has zero std as the full pool admits one subset). Pairs (

K{=}2

) are nearly rank-orthogonal (

\rho\approx 0.95

); at full pool,

43\%

of rank is redundant.

Pool	$\rho$	$\rho_{\mathrm{intra}}$	$\rho_{\mathrm{cross}}$	$D/D_{\mathrm{null}}$
Pool C (SDXL, $K{=}17$ )	0.46	0.84	0.55	0.98
Six stylized SDXL LoRAs ( $K{=}6$ )	0.97	1.00	0.97	1.00
Community SDXL, uncurated ( $K{=}20$ )	0.87	0.99	0.88	1.00
Pool C + community union ( $K{=}37$ )	0.43	0.89	0.48	0.99
LoRA-Hub (FLAN-T5, $K{=}48$ )	0.60	1.00	0.60	0.70
Wild FLAN-T5 LoRAs, uncurated ( $K{=}12$ )	0.84	1.00	0.85	0.92

Appendix

Table 7: Rank-fill decomposition at

\tau=0.99

D/D_{\mathrm{null}}<1

means the pool needs fewer dimensions than a spectrum-matched random pool. The LoRA-Hub row uses a different 48 of the 196 adapters than Tab. 2 , and LoRA-Retriever is an independent re-download. The stylized, community and wild pools are described in App. H .

		Per-LoRA error		Composed error ( $N{=}5$ )
Setting	$D$	marginal	joint	marginal	joint
Full pool, $K{=}17$ , 24 layers	822	0.88%	0.35%	0.84%	0.43%
Small pool, $K{=}6$ , 24 layers	374	0.52%	0.22%	0.56%	0.32%
Full pool, $K{=}17$ , all 722 layers	per layer	1.01%	0.43%	0.95%	0.53%

Appendix

Table 8: Reconstruction error of marginal and joint bases at equal

D

(Pool C).

$\tau$	mean $D$	CLIP-T	ImageReward	DINO
0.99	785	25.33	1.137	1.000
0.70	107	25.67	1.158	0.981
0.50	49	25.61	1.148	0.967
0.30	22	25.27	1.117	0.949

Appendix

Table 9: Quality at compressed energy thresholds. DINO is the similarity to the

\tau=0.99

images.

Pool	$K$	$\sum_{i}r_{i}$	$D$	$\rho$
Pool C (control)	17	1792	819	0.46
Six stylized LoRAs alone	6	192	186	0.97
Pool C + six stylized	23	1984	848	0.43
Community LoRAs, uncurated	20	673	580	0.87
Pool C + community	37	2337	921	0.43

Appendix

Table 10: Rank fill of heterogeneous and uncurated SDXL pools at

\tau=0.99

Update path	Captured energy	Extra directions per layer
Projection, unchanged bases	0.30	0
Extension, $q=4$	0.92	4
Extension, $q=8$	0.96	8
Extension, $q=16$	0.98	16
Extension, $q=32$	1.00	32

Appendix

Table 11: Captured energy of a held-out Pool C adapter under each update path (mean over all

17

adapters).

Requirement	CtS	KnOTS	Core-Space	Composite	Per-combo truncated merge	LoRASpace
Serve any active subset, static or routed, from one precompute	no	no	no	n/a	no	yes
Context-dependent weights without $N$ backbone passes	no	no	no	no	no	yes
Predict before deployment whether a pool compresses	partial ∗	no	no	no	no	yes ( $\rho$ )
Admit a newly published adapter without recomputing the pool	no	no	no	n/a	no	yes (extension)

Appendix

Table 12: Requirements of multi-LoRA composition under a memory budget. Composite covers ComposLoRA [ Zhong et al., 2024 ] and CMLoRA [ Zou et al., 2025 ] .

Method	Composition (native)	Composition ( compact )	$\Delta$ composition	$\Delta$ image
spatial- LoRASpace	4.15	4.27	$+0.12$	$-0.19$
temporal- LoRASpace	3.82	4.38	$\mathbf{+0.56}$	$-0.01$
semantic- LoRASpace	3.78	4.20	$+0.42$	$-0.13$
Mean	3.92	4.28	$+0.36$	$-0.11$

Appendix

Table 13: GPT-4o composition and image quality under native (truncated) triggers versus the compact deduplication rule,

15

Pool C combos at

N{=}7

	$N{=}7$			$N{=}8$
Method	CLIP-T	ImgReward	DINO (vs. comp.)	CLIP-T	ImgReward	DINO (vs. comp.)
merge	32.16	0.06	0.508	32.49	0.01	0.434
switch	36.43	1.23	0.689	36.10	1.10	0.650
composite	35.65	0.61	n/a	34.47	0.39	n/a
cmlora	34.59	0.60	0.833	34.60	0.38	0.809
spatial- LoRASpace	34.79	0.70	0.512	34.03	0.59	0.399

Appendix

Table 14: Pool B (SD 1.5,

17

LoRAs,

48

combos per cell) at

N\in\{7,8\}

with compact triggers. Bold = best; underline = best non- LoRASpace .

Router	CLIP-T $\uparrow$	ImageReward $\uparrow$
uniform	$35.82\pm 0.10$	$1.050\pm 0.029$
spatial	$35.43\pm 0.34$	$0.857\pm 0.062$
temporal	$\mathbf{35.92\pm 0.10}$	$\mathbf{1.086\pm 0.030}$
semantic	$35.83\pm 0.09$	$1.050\pm 0.026$

Appendix

Table 15: Routing ablation on Pool A with the LoRASpace substrate fixed (mean

\pm

across-seed std over 3 seeds,

240

combos/seed,

N\in\{2,\dots,5\}

). All four routers exceed the merge baseline (

33.60

0.760

, Tab. 3 ); temporal gives the largest ImageReward gain over uniform.

	truncated-merge		subspace-merge
$N$	CLIP-T $\uparrow$	ImgRew $\uparrow$	CLIP-T $\uparrow$	ImgRew $\uparrow$
2	34.57	1.053	34.45	1.022
3	35.31	1.037	35.39	1.028
4	36.13	1.078	36.11	1.081
5	37.00	1.126	37.10	1.115
Mean	35.75	1.073	35.76	1.062

Appendix

Table 16: Mechanism ablation per

N

. truncated-merge (combo-specific basis) vs. subspace-merge (pool-wide basis), Pool A, seed

111

. Both run at ambient-merge inference cost; they differ only in configure-time precompute.

	merge		composite		LoRASpace (uniform)
$N$	CLIP-T $\uparrow$	ImgRew $\uparrow$	CLIP-T $\uparrow$	ImgRew $\uparrow$	CLIP-T $\uparrow$	ImgRew $\uparrow$
2	29.33	1.300	31.50	1.292	31.67	1.289
3	31.32	0.894	32.95	1.108	33.60	1.089
4	28.82	0.799	31.79	1.058	32.95	1.106
5	28.53	0.426	30.48	0.829	32.01	1.028
6	29.06	0.184	32.85	0.577	34.22	0.691

Appendix

Table 17: Pool C (SDXL,

17

LoRAs) quality per

N

. Seed

111

200

combos per method (

40

per

N

, identical across methods).

95\%

bootstrap CIs (mean over

N

): merge CLIP-T

[28.97,29.86]

, ImgRew

[0.60,0.84]

; composite

[31.5,32.4]

[0.88,1.06]

; uniform LoRASpace

[32.5,33.3]

[0.95,1.13]

. Merge ImgRew drops from

1.30

N{=}2

0.18

N{=}6

; LoRASpace retains

0.69

Method	Time @ $N{=}2$ (s)	Time @ $N{=}7$ (s)	Peak VRAM	Load VRAM
merge	23.0	43.5 †	22.34 GB	18.19 GB
switch	23.3	23.4	22.83 GB	19.68 GB
composite	76.3	262.2	22.83 GB	19.69 GB
cmlora	77.1	261.5	22.83 GB	19.69 GB
spatial- LoRASpace	34.6	48.0	15.60 to 16.80 GB	6.46 GB
temporal- LoRASpace	32.9	42.7	15.60 to 16.80 GB	6.46 GB

Appendix

Table 18: Per-image speed and GPU memory on Pool C (SDXL,

1024^{2}

50

steps, RTX 3090). Peak VRAM measured at

N{\in}\{2,4,6\}

N{=}7

entries linearly extrapolated. Static-mode LoRASpace ( subspace-merge ) inherits ambient merge ’s per-image cost by construction.

Method	Time @ $N{=}2$ (s)	Time @ $N{=}5$ (s)	Peak VRAM	Load VRAM
merge	18.9	32.7	3.49 GB	2.88 GB
switch	17.5	22.0	3.49 GB	2.88 GB
composite	68.2	142.6	3.49 GB	2.89 GB
cmlora	59.5	138.5	3.49 GB	2.89 GB
spatial- LoRASpace	37.6	30.7	4.09 GB	2.88 GB
temporal- LoRASpace	28.7	44.3	4.09 GB	2.88 GB

Appendix

Table 19: Per-image speed and GPU memory on Pool A (SD 1.5,

512^{2}

200

steps, RTX 3090,

10

combos,

2

warmup

+3

timed repeats per cell, mean across repeats). Body uses

100

steps so wall-clock halves at deployment; relative ratios are preserved. spatial-LoRASpace cells contain measurement noise from the

\leq 3

repeats; the dominant signal is the

N

-scaling separation between LoRASpace and composite .

Condition	Backbone passes	Adapter path	Median s/image
composite	$5$	none (per-pass guidance)	$181.9$
Ambient single-pass	$1$	$N$ thin factor pairs	$37.3$
LoRASpace	$1$	shared subspace (Eq. ( 5 ))	$34.5$

Appendix

Table 20: Pool C per-image time at

N{=}5

, with a single-pass ambient mixture between composite and LoRASpace .

Panel of Figure 5 — Figure 5: Pool C $N{=}4$ : VioletXL + catwalk + god rays + old gold. Left to right : ambient merge , composite , uniform LoRASpace .

Panel of Figure 6 — Figure 6: Pool C $N{=}6$ (high-cardinality): VioletXL + catwalk + chiaroscuro + corrosion + fear + sunglasses.

Pool	Setting	CLIP-T $\uparrow$	ImageReward $\uparrow$	DINO (vs. comp.)
Pool A SD 1.5	mean $N=2$ to $5$ , 240 combos	35.81	1.14	0.73
Pool C SDXL	mean $N=2$ to $6$ , 200 combos	32.10	1.15	0.79
Pool B SD 1.5	$N=7$ , 48 combos	36.43	1.23	0.69
Pool B SD 1.5	$N=8$ , 48 combos	36.10	1.10	0.65
LoRA-Retriever	Macro Task / samp.s -1	$51.54$ Macro Task at $1.82$ samp.s -1

Appendix

Table 21: Switch baseline (one active LoRA at a time, same protocol as main results). Reported for completeness; not directly comparable to composition methods.

Method	Macro Task	Macro Domain	samp/s
base (no LoRA)	4.24	4.17	0.60
oracle (task-matched)	57.54	59.55	1.91
merge (Task Arithmetic)	45.72	46.96	2.18
ties	48.71	51.39	1.93
dare	40.86	41.22	2.04
knots	45.44	47.01	0.31

Appendix

Table 22: Full LoRA-Retriever benchmark [ Zhao et al., 2024 ] on LLaMA-2-7B [ Touvron et al., 2023 ] (

48

LoRAs,

2{,}395

samples,

14

methods). Bold = best non-Oracle; underline = best non- LoRASpace non-Oracle.

ID	Name	Trigger words
character
character_1	Real-person likeness LoRA	identity trigger and appearance descriptors (omitted)
character_2	Real-person likeness LoRA	identity trigger and appearance descriptors (omitted)
character_3	Real-person likeness LoRA	identity trigger and appearance descriptors (omitted)
clothing
clothing_1	Thai University Uniform	mahalaiuniform, white shirt short sleeves, black pencil skirt

Appendix

Table 23: Pool A LoRAs with trigger words. Source: CivitAI (URLs and SHA-256 hashes in the pool manifests).

ID	Name	Trigger words
detail (quality enhancers, no explicit trigger)
detail_1	Detail Tweaker	(none)
detail_2	Add More Details	(none)
pose
pose_1	Shirt Tug Pose	shirt tug
pose_2	Covering Eyes Pose	covering eyes

Appendix

Table 24: Pool B additional LoRAs (Pool A’s 11 plus 6 new).

ID	Name	Trigger words
character
character_1	SL Nobodies CamelliaXL	CamelliaXL, a young woman, detailed face
character_2	SL Nobodies HelenaXL	HelenaXL, a young woman, detailed face
character_3	SL Nobodies VioletXL	VioletXL, a young woman, detailed face
background
background_1	Polyhedron Catwalk	catwalk, runway, fashion show

Appendix

Table 25: Pool C LoRAs (SDXL).

Domain (count)	Tasks
closed-book QA (4)	arc_challenge , arc_easy , natural_questions , trivia_qa
commonsense (4)	copa , hellaswag , piqa , story_cloze
coreference (2)	definite_pronoun_resolution , wsc
NLI (10)	anli_r1 , anli_r2 , anli_r3 , cb , mnli_matched , mnli_mismatched , qnli , rte , snli , wnli
paraphrase (4)	glue_mrpc , glue_qqp , paws_wiki , stsb
RC w/ commonsense (2)	cosmos_qa , record

Appendix

Table 26: LoRA-Retriever 48 task LoRAs by domain.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Jun 2, 2026

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment. We study the problem of merging

T

LoRAs into a single rank-

r

LoRA, thereby preserving the benefits of low-rank structure. Existing Merge-then-Compress pipelines treat the rank constraint as an afterthought: they merge adapters in the full parameter space, then compress the merged result to rank

r

via truncated SVD. However, full-parameter merging may destroy the low-rank structure, making it difficult for subsequent compression to recover an effective rank-

r

LoRA. We propose Compress-then-Merge (CtM), a reversed pipeline that enforces the rank-

r

bottleneck before merging: CtM computes shared

r

-dimensional subspaces using only the LoRA weights to capture cross-adapter common structure, projects each adapter into the shared subspaces to obtain

r\times r

coordinates, and then applies standard merging rules in this reduced space. CtM guarantees a rank-

r

LoRA by construction, avoiding post-hoc truncation, and enables efficient computation in the core space spanned by concatenated LoRA factors. Experiments across multiple models and tasks show that CtM consistently outperforms existing single-LoRA-output baselines while narrowing the performance gap to full-parameter merging methods.

Zhengbao He, Ruiqi Ding, Zhehao Huang +3

Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

Adapter Tuning Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Model Merging Parameter-Efficient Fine-Tuning

Ask PDF

Jun 27, 2026

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.

Yupeng Chang, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Low-Rank Adaptation LLM Fine-Tuning Parameter-Efficient Fine-Tuning

Ask PDF

Oct 26, 2026

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

Low-Rank Adaptation (LoRA) has become a key mechanism for efficiently specializing foundation models into many task- and user-specific variants. Serving many LoRA adapters under a shared resource budget turns post-hoc compression into a two-level allocation problem: rank must be distributed across layers within each adapter and across adapters within the fleet. Although trained LoRAs contain substantial spectral redundancy, spectral reconstruction alone does not determine how much compression an adapter can functionally tolerate. We formulate fleet compression through a functional rank-distortion frontier for each adapter, followed by fleet-level minimax allocation under a shared rank budget. Given exact frontiers, this decomposition is equivalent to joint optimization over all layer ranks. We introduce Functional Rank Allocation (FRA), an unlabeled method for constructing practical frontiers. FRA measures isolated module truncations end to end, uses these measurements as an additive search surrogate solved by dynamic programming, remeasures every complete candidate, and takes the lower-risk envelope with a complementary spectral proposal before exact minimax allocation over the constructed frontiers. Across 78 adapters from three public fleets, FRA improves worst-case and tail utility over uniform spectral compression in constrained-budget regimes. Fixed-budget probes show that isolated costs rank candidates well, but their calibration shifts across proposal shapes; end-to-end verification and the spectral proposal therefore limit functional-search error.

Shenghui Li, Thiemo voigt

Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden

Low-Rank Compression Adaptive Rank Allocation Low-Rank Adaptation

Ask PDF

Abstract

Low-Rank Adaptation

Explore similar work

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets