LoRA Superposition across Pretrained Models

Keigo Watanabe; Hiroshi Kera; Kazuhiko Kawamoto

cs.CVOct 22, 2026

LoRA Superposition across Pretrained Models

Authors: Keigo Watanabe, Hiroshi Kera, Kazuhiko Kawamoto

Organizations: Chiba University, Chiba, Japan · National Institute of Informatics, Tokyo, Japan

Abstract

Low-rank adaptation (LoRA) is usually tied to the pretrained backbone on which it is learned. We ask whether LoRAs trained independently for the same task on different pretrained backbones can be summed and applied unchanged to all contributing backbones. We call this construction LoRA superposition. Across seven ViT-B/16 CLIP backbones and eight image-classification tasks, the exact sum improves all 56 backbone-task pairs and achieves a mean accuracy gain of 30.21 percentage points over the unadapted backbone. The exact sum achieves a mean accuracy only 0.28 percentage points below that of individually trained solo LoRAs. In contrast, a leave-one-model-out (LOMO) sum produces a mean gain of only 0.06 percentage points over the unadapted backbones. Backbone-specific solo LoRAs are nearly orthogonal in parameter space, while pairwise cross-backbone responses are strongly concentrated on the diagonal. These results support a component-wise non-interference hypothesis, under which the frozen backbone acts as an implicit selector of its own component without explicit routing or cross-model alignment.

Figures & tables

Panel of Figure 1 — Figure 1: Cross-backbone LoRA superposition. Backbone-specific solo LoRAs for the same task are summed into one shared adapter, which is stored and applied unchanged to all contributing frozen backbones. Under the non-interference hypothesis, each backbone responds mainly to its own component in the superposed update. The exact sum is a composite update whose rank can grow with the number of contributing backbones.

Panel of Figure 2 — Figure 2: Schematic geometry of LoRA superposition. For backbone $A$ , the superposed update is projected onto $\bm{\tau}_{A,t}$ , giving $\bm{\Pi}_{A}\bm{\tau}_{S,t}\approx\bm{\tau}_{A,t}$ .

Figure 3: Representative samples from the eight downstream datasets.

Backbone	Data scale	Source	Main curation strategy
OpenAI	400M	Public Web data	Proprietary collection and filtering
LAION-400M	400M	Common Crawl	CLIP-score filtering
LAION-2B	2.3B	Common Crawl	CLIP-score and language filtering
DataComp-1B	1.4B	CommonPool	Image-based and CLIP-score filtering
DFN-2B	2B	CommonPool	Learned data-filtering network
MetaCLIP-400M	400M	Common Crawl	Metadata-based concept balancing

Table 1: Pretraining data of the seven CLIP backbones. All datasets contain web-scale image–text pairs but differ in source pools, filtering criteria, and dataset sizes.

Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC	SUN397	SVHN	Average
Backbone	81.82	55.13	54.17	45.12	61.21	67.63	69.10	44.61	59.85
Solo	$85.14$ [-1pt] $\pm$ 0.06	$78.54$ [-1pt] $\pm$ 0.11	$98.55$ [-1pt] $\pm$ 0.04	$96.22$ [-1pt] $\pm$ 0.03	$99.09$ [-1pt] $\pm$ 0.01	$91.23$ [-1pt] $\pm$ 0.04	$79.30$ [-1pt] $\pm$ 0.02	$94.67$ [-1pt] $\pm$ 0.02	$90.34$ [-1pt] $\pm$ 0.02
Joint shared	$84.63$ [-1pt] $\pm$ 0.08	$76.68$ [-1pt] $\pm$ 0.02	$98.09$ [-1pt] $\pm$ 0.03	$94.63$ [-1pt] $\pm$ 0.04	$98.74$ [-1pt] $\pm$ 0.04	$89.79$ [-1pt] $\pm$ 0.07	$78.25$ [-1pt] $\pm$ 0.01	$93.96$ [-1pt] $\pm$ 0.01	$89.34$ [-1pt] $\pm$ 0.02
Exact sum	$85.03$ [-1pt] $\pm$ 0.13	$77.89$ [-1pt] $\pm$ 0.14	$98.49$ [-1pt] $\pm$ 0.03	$95.91$ [-1pt] $\pm$ 0.03	$99.03$ [-1pt] $\pm$ 0.01	$91.03$ [-1pt] $\pm$ 0.03	$78.54$ [-1pt] $\pm$ 0.02	$94.55$ [-1pt] $\pm$ 0.05	$90.06$ [-1pt] $\pm$ 0.03
LOMO sum	$81.72$ [-1pt] $\pm$ 0.03	$55.07$ [-1pt] $\pm$ 0.09	$54.60$ [-1pt] $\pm$ 0.26	$45.27$ [-1pt] $\pm$ 0.21	$61.33$ [-1pt] $\pm$ 0.33	$67.63$ [-1pt] $\pm$ 0.06	$68.53$ [-1pt] $\pm$ 0.05	$45.10$ [-1pt] $\pm$ 0.13	$59.91$ [-1pt] $\pm$ 0.13

Table 2: Top-1 accuracy (%) averaged over seven backbones. Average denotes the unweighted mean over the eight datasets. The top and bottom values denote the mean and standard deviation, respectively, over three random seeds.

Figure 4: Absolute cosine similarity between backbone-specific solo LoRAs, averaged over eight tasks. The top and bottom values denote the mean and standard deviation, respectively, over three random seeds.

Figure 5: Cross-backbone response of individual solo LoRAs. Each cell shows the top-1 accuracy gain in percentage points, averaged over eight tasks, when the solo LoRA trained for the column backbone is applied to the row backbone. The top and bottom values denote the mean and standard deviation, respectively, over three random seeds.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 538 papers

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition
Oct 20, 2026Thamirawaran Sathiyalogeswaran, Navindu De Silva, Chamikara Siriwardane +4Low-Rank Adaptation LLM Inference Acceleration
University of Moratuwa, Sri Lanka · University of Michigan, USA
A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need
Apr 9, 2026Hananel Hazan, Yanbo Zhang, Benedikt Hartl +1Random Feature Methods Parameter-Efficient Fine-Tuning
Allen Discovery Center at Tufts University, Medford, Massachusetts, USA · Institute for Theoretical Physics, TU Wien, Wien, Austria · Wyss Institute for Biologically Inspired Engineering, Harvard University, Boston, Massachusetts, USA
Convergent Performance, Divergent Mechanisms in LoRA-Adapted Vision Foundation Models for Brain MRI
Oct 20, 2026Cecilia Diana-Albelda, Fiona Kekwick, Raghav Mehta +6Vision Foundation Model Adaptation Low-Rank Adaptation
Department of Electronic Technology and Communications, Universidad Autónoma de Madrid, Madrid, Spain · Department of Computing, Imperial College London, London, United Kingdom · Department of Brain Sciences, Imperial College London, London, United Kingdom +3

Oct 20, 2026J/K move · Enter open · S save

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

Thamirawaran Sathiyalogeswaran, Navindu De Silva, Chamikara Siriwardane, W. K. Alwis+3

University of Moratuwa, Sri Lanka · University of Michigan, USA

Low-Rank Adaptation LLM Inference Acceleration Low-Rank Compression

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) attaches small low-rank matrices to a frozen backbone, and public repositories host many LoRAs per backbone, each for a different concept or task. Combining several LoRAs in one generation forces a trade-off. Merging their weights into one update needs a single forward pass, but LoRAs trained for different concepts interfere and quality drops. Running each LoRA in its own forward pass (composite) preserves quality, but inference time grows linearly with the number of active LoRAs. We study a property of LoRA pools overlooked by prior multi-LoRA work. In real LoRA pools, the stacked low-rank factors need far fewer directions than their summed ranks. Using singular value decomposition, we find the minimal subspace that keeps 99% of the factor energy. On three diffusion pools, only 46% to 57% of the summed LoRA ranks is needed, and the subspace grows sub-linearly with pool size. A 48-LoRA sub-pool of LoRA-Hub on FLAN-T5 tasks shows comparable overlap (62%), while the diversity-curated LoRA-Retriever benchmark sits near the no-overlap baseline (96%), so pool composition rather than modality predicts where compression is large. We build on this overlap with LoRASpace, a training-free method that computes the subspace once per pool and reuses it in two inference modes, a static mode at merge cost and a dynamic mode for input-dependent routing. On the diffusion pools, static LoRASpace matches composite quality at merge cost, and dynamic LoRASpace is more than 6x faster than composite with six active LoRAs on a 17-LoRA pool, mainly because it runs one backbone pass instead of six. In an ablation, static variants at the same rank match LoRASpace, so the quality gain does not depend on the pool-wide basis. On the near-orthogonal LoRA-Retriever benchmark, compression is small and gains come from coupling with external retrieval.

Abstract

Low-Rank Adaptation

Explore similar work

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need

Convergent Performance, Divergent Mechanisms in LoRA-Adapted Vision Foundation Models for Brain MRI