SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization

Seyed Mahmoud Sajjadi Mohammadabadi; Xiaolong Ma; Lei Yang; Feng Yan; Junshan Zhang

cs.LGApr 9, 2026

SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization

Authors: Seyed Mahmoud Sajjadi Mohammadabadi, Xiaolong Ma, Lei Yang, Feng Yan, Junshan Zhang

Organizations: University of Nevada, Reno, Reno, NV, USA · Argonne National Laboratory, Lemont, IL, USA · University of Houston, Houston, TX, USA · University of California, Davis, Davis, CA, USA

Abstract

Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices.

Figures & tables

Figure 1: Overview of SOLAR. Given fine-tuned adapters $(A,B)$ , SOLAR projects them onto structured subspaces derived from the pretrained model’s SVD. A seeded pseudo-random generator (seeded with a known value) deterministically creates the basis matrices. Top- $k$ coefficients $\alpha$ and $\beta$ are selected under a budget to reconstruct $\tilde{A}$ and $\tilde{B}$ , while the bases are never stored or transmitted. Only the coefficients $\alpha$ , $\beta$ , and the seed need to be communicated or stored.

Model	Method	#	CIFAR-10		CIFAR-100		Food-101		T-ImageNet
		Param	10	Full	10	Full	10	Full	10	Full
ViT-B	Full-FT	86M	91.1 $\pm$ .8	94.6 $\pm$ .5	78.2 $\pm$ .7	87.7 $\pm$ .3	65.8 $\pm$ .9	85.2 $\pm$ .4	78.1 $\pm$ 1.0	85.4 $\pm$ .6
	LoRA ( $r$ =4)	74K	92.3 $\pm$ .6	98.3 $\pm$ .2	81.8 $\pm$ .8	90.3 $\pm$ .4	72.4 $\pm$ .7	87.6 $\pm$ .3	77.9 $\pm$ .9	88.8 $\pm$ .4
	NOLA	48K	92.2 $\pm$ .6	94.7 $\pm$ .5	81.3 $\pm$ .8	86.6 $\pm$ .4	72.6 $\pm$ .5	85.9 $\pm$ .2	78.4 $\pm$ .7	82.8 $\pm$ .5
	SOLAR ${}_{r=4(4\!\rightarrow\!1.6)}$	41K	92.3 $\pm$ .7	98.3 $\pm$ .4	81.5 $\pm$ .7	89.8 $\pm$ .2	71.8 $\pm$ .6	87.0 $\pm$ .5	77.9 $\pm$ .8	87.9 $\pm$ .4
	SOLAR ${}_{\text{NOLA}(4\!\rightarrow\!1.2)}$	32K	92.1 $\pm$ .7	94.5 $\pm$ .3	81.1 $\pm$ .6	85.4 $\pm$ .3	72.5 $\pm$ .6	85.4 $\pm$ .3	78.3 $\pm$ .8	82.3 $\pm$ .5

Table 1: Top-1 classification accuracy (%) of ViT-B and ViT-L on benchmark datasets under two settings: (1) few-shot (10 samples/class, 25 epochs) and (2) full-data (5 epochs). Results report mean

\pm

std over 5 runs. SOLAR is applied with configuration

{}_{\text{method}(N\rightarrow k)}

, where

N

and

k

are in thousands.

Method	Byte Footprint	Oxford Pets	SUN397	CUB-200	ImageNet-1K
LoRA ( $r$ =1)	74KB	93.0 $\pm$ .3	74.3 $\pm$ .2	84.7 $\pm$ .2	81.5 $\pm$ .4
NOLA	48KB	90.4 $\pm$ .5	61.7 $\pm$ .4	79.4 $\pm$ .4	77.4 $\pm$ .3
SOLAR r=1(2→0.2)	8KB (89% $\downarrow$ )	92.6 $\pm$ .4	73.9 $\pm$ .2	84.2 $\pm$ .3	81.3 $\pm$ .2

Table 2: Additional evaluation on vision datasets using ViT-B. The table shows bit-level representation footprint (32-bit baseline) and Top-1 accuracy. All models are trained for 10 epochs.

Table 4

Model	LLaMA-3.2 1B			LLaMA-2 13B (4-bit)
Method	LoRA	NOLA	SOLAR	LoRA	NOLA	SOLAR
Method	$r$ =8	1000	${r=8(4\!\rightarrow\!1.2)}$	$r$ =1	1000	${r=1(1\!\rightarrow\!0.3)}$
# Params	852K	64K	81K (90% $\downarrow$ )	819K	140K	51K (94% $\downarrow$ )
Val Loss	1.51	1.87	1.52	1.05	1.29	1.05
MMLU Acc	30.1	25.9	28.3	54.5	51.8	54.5

Table 5: Model representation efficiency for LLaMA models. SOLAR compresses LoRA adapter updates across various model sizes.

Method	GPT-2 Small		GPT-2 Medium
Method	MET	# Params	MET	# Params
Full-FT	28.4	124M	46.2	355M
LoRA ( $r$ =4)	29.7	147K	47.2	393K
NOLA	29.1	48K	46.8	350K
SOLAR ( $r$ =4, $1\!\rightarrow\!0.3$ )	29.7	15K (90% $\downarrow$ )	46.4	30K (92% $\downarrow$ )
SOLAR ( $r$ =1, $0.1\!\rightarrow\!0.1$ )	26.1	4K (97% $\downarrow$ )	44.8	9K (98% $\downarrow$ )

Table 6: Performance and parameter efficiency on E2E NLG using GPT-2 Small and Medium. All methods use rank-4 adapters applied to the Query and Value projections.

Figure 2: Subspace similarity between the $W$ and $\Delta W$ matrices (Q, K, V) from the first layer of the ViT-B model using LoRA with rank $r=4$ .

Dataset	LoRA	SOLAR	Overhead (%)
CIFAR-10	1176	14	1.19
CIFAR-100	1165	14	1.20
Food-101	3480	67	1.92
Tiny-ImageNet	2081	15	0.72
ImageNet-1K	56634	155	0.27

Table 8: Runtime Overhead: LoRA (10 epochs) vs. SOLAR post-training on ViT-B across vision datasets. Times in seconds.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Training Samples	Number of Classes
CIFAR-10	50,000	10
CIFAR-100	50,000	100
Food-101	75,750	101
Tiny-ImageNet	100,000	200
ImageNet-1K	1,281,167	1,000

Appendix

Table 9: Dataset statistics used in experiments. Each dataset includes the number of training samples and classes.

Dataset	Samples	Avg. Seq. Length	Context
Stanford Alpaca	52,000	$\sim$ 256 tokens	LLaMA-3 instruction tuning
MMLU	15,858	$\sim$ 200 tokens	LLaMA-3 Generalization evaluation
E2E NLG	42,000	$\sim$ 35 tokens	GPT-2 generation fine-tuning

Appendix

Table 10: Dataset statistics in LLM experiments.

Method	Dataset	Query (Q)	Classifier Head
SOLAR	CIFAR-10	$12\times\left((1600+1600)+\frac{4000+4000}{32}\right)+1=$ 41,401$$	$10\times 768+10=$ 7,690$$
	CIFAR-100	$41,401$	$100\times 768+100=$ 76,900$$
	Food-101	$41,401$	$101\times 768+101=$ 77,669$$
	Tiny-ImageNet	$41,401$	$200\times 768+200=$ 154,000$$
LoRA	CIFAR-10	$12\times[(768\times 4)+(4\times 768)]=$ 73,728$$	$10\times 768+10=$ 7,690$$
	CIFAR-100	$73,728$	$100\times 768+100=$ 76,900$$

Appendix

Table 11: Number of representation parameters for ViT-B (Rank = 4). Each row reports the parameter count for query projections and the classifier head using SOLAR and LoRA across different datasets. The classifier head parameter count is shared across methods and is computed as (num_classes × 768 + num_classes) . For SOLAR, the query projection count corresponds to: number of layers

\times

(top k coefficients for

A

+ top k coefficients for

B

+ encoded basis for

A

+ encoded basis for

B

)

+1

(seed value). All SOLAR rows follow the form

N\!\rightarrow\!\text{top}_{k}

where

N

is the original subspace size. For LoRA, the query projection count corresponds to: number of layers

\times

(input dimension

\times

rank for

A

+ rank

\times

output dimension for

B

), where rank is 4.

Method	Dataset	Query (Q)	Classifier Head
SOLAR	CIFAR-10	$24\times\left((500+500)+\frac{1000+1000}{32}\right)+1=$ 25,501$$	$10\times 1024+10=$ 10,250$$
	CIFAR-100	$25,501$	$100\times 1024+100=$ 102,500$$
	Food-101	$25,501$	$101\times 1024+101=$ 103,625$$
	Tiny-ImageNet	$25,501$	$200\times 1024+200=$ 204,800$$
LoRA	CIFAR-10	$24\times[(1024\times 4)+(4\times 1024)]=$ 196,608$$	$10\times 1024+10=$ 10,250$$
	CIFAR-100	$196,608$	$100\times 1024+100=$ 102,500$$

Appendix

Table 12: Number of representation parameters for ViT-L (Rank = 4). Each row shows the parameter counts for Query projections and the classifier head using SOLAR and LoRA across different datasets. The classifier head parameter count is shared across methods and is calculated as (num_classes × 1024 + num_classes) .

Method	Dataset	MLP	Classifier Head
NOLA	CIFAR-10	$12\times 2\times 2\times 1000+1=$ 48,001$$	$10\times 768+10=$ 7,690$$
	CIFAR-100	$48,001$	$100\times 768+100=$ 76,900$$
	Food-101	$48,001$	$101\times 768+101=$ 77,669$$
	Tiny-ImageNet	$48,001$	$200\times 768+200=$ 154,000$$

Appendix

Table 13: Number of representation parameters for ViT-B (Rank = 4). Each row shows the parameter counts for MLP projections (for NOLA) and classifier head across datasets. The classifier head parameter count is shared across methods and is calculated as (num_classes × 768 + num_classes) .

Method	Representation Footprint (Bytes)
LoRA ( $r{=}1$ )	$12\times\left[(768\times 1)+(1\times 768)\right]\times 4=$ 73,728$$
SOLAR for ViT-B 8Bit ( $r{=}1$ , $500\!\rightarrow\!50$ )	$12\times\left[(50+50)+\frac{500}{8}\right]\times 1+1=$ 1,951$$
SOLAR for ViT-B 8Bit ( $r{=}1$ , $100\!\rightarrow\!10$ )	$12\times\left[(10+10)+\frac{100}{8}\right]\times 1+1=$ 391$$
LoRA ( $r{=}4$ )	$24\times\left[(1024\times 4)+(4\times 1024)\right]\times 4=$ 786,432$$
SOLAR for ViT-L 32Bit ( $r{=}4$ , $4000\!\rightarrow\!1600$ )	$24\times\left[(1600+1600)+\frac{4000}{32}\right]\times 4+1=$ 319,201$$
SOLAR for ViT-L 16Bit ( $r{=}4$ , $4000\!\rightarrow\!1600$ )	$24\times\left[(1600+1600)+\frac{4000}{16}\right]\times 2+1=$ 165,601$$

Appendix

Table 14: Byte-level footprint of representation parameters for ViT-B and ViT-L using LoRA and SOLAR. Each value reflects the total number of bytes required to store adapter updates (excluding classifier heads). For LoRA, storage is computed as: number of layers

\times

( rank

\times

output dimension for

B

+ input dimension

\times

rank for

A

)

\times

precision in bytes (e.g., 4 bytes for 32-bit float). For SOLAR, storage is computed as: number of layers

\times

( top k coefficients for

A

+ top k coefficients for

B

+ encoded basis vectors for

A

+ encoded basis for

B

)

\times

precision in bytes, plus 1 byte to store a random seed. For example, the row "

500\!\rightarrow\!50

" denotes that

500

-dimensional subspaces are sparsified to top-

k=50

coefficients, with encoded bases represented at 1 bit per element (8 elements per byte).

Model (Rank)	Configuration	Total Parameters
LLaMA-3.2 1B ( $r$ =8)	16 layers (Q, V)	$16\times[(2048\times 8+8\times 2048)+(2048\times 8+8\times 512)]=$ 851,968$$
NOLA	16 layers (Q, V)	$16\times 2\times(1000+1000)=$ 64,000$$
SOLAR ( $r$ =8, $4\text{K}\!\rightarrow\!1.2\text{K}$ )	16 layers (Q, V)	$16\times 2\times\left(1200+1200+\frac{4000}{32}\right)+1=$ 80,801$$
LLaMA-3.2 3B ( $r$ =1)	28 layers (Q, V)	$28\times[(3072\times 1+1\times 3072)+(3072\times 1+1\times 1024)]=$ 286,720$$
NOLA	28 layers (Q, V)	$28\times 2\times(1000+1000)=$ 112,000$$
SOLAR ( $r$ =1, $1000\!\rightarrow\!150$ )	28 layers (Q, V)	$28\times 2\times\left(150+150+\frac{1000}{32}\right)+1=$ 18,551$$

Appendix

Table 15: Number of representation parameters for LLaMA-3 models using LoRA, NOLA, and SOLAR. Each row reports total adapter parameters for attention projections (Q and V for LoRA and NOLA; Q and K for SOLAR). Output heads and MLP layers are frozen. For LoRA, the parameter count is computed as: number of layers

\times

( input dimension

\times

rank for

B

+ rank

\times

output dimension for

A

+ ) . Due to differing dimensions between

A

and

B

in LoRA, the table computes the contributions for Q and V projections separately. For NOLA, it is computed as: number of layers

\times

\times

(number of random basis vectors), assuming separate basis sets for

A

and

B

. For SOLAR, the count is: number of layers

\times

\times

( top k coefficients for

B

+ top k for

A

+ encoded bases for

B

+ encoded bases for

A

) , plus 1 byte to communicate or store the shared seed.

Model (Rank)	Configuration	Total Parameters
GPT-2 Small ( $r$ =4)	12 layers (Q, V)	$12\times 2\times(768\times 4+4\times 768)=$ 147,456$$
NOLA	12 layers (Q, V)	$12\times 2\times(1000+1000)=$ 48,000$$
SOLAR ( $r$ =1, $1000\!\rightarrow\!300$ )	12 layers (Q, V)	$12\times 2\times\left(300+300+\frac{1000}{32}\right)+1=$ 15,150$$
SOLAR ( $r$ =1, $100\!\rightarrow\!90$ )	12 layers (Q, V)	$12\times 2\times\left(90+90+\frac{100}{32}\right)+1=$ 4,396$$
GPT-2 Medium ( $r$ =4)	24 layers (Q, V)	$24\times 2\times(1024\times 4+4\times 1024)=$ 393,216$$
NOLA	24 layers (Q, V)	$350,000$ [ koohpayegani2024nola ]

Appendix

Table 16: Number of trainable adapter parameters for GPT-2 models using LoRA, NOLA, and SOLAR. Each row reports the total number of parameters added to the query and value projections (Q and V). All configurations freeze the output heads and MLP layers. For LoRA, the parameter count is computed as: number of layers

\times

\times

( input dimension

\times

rank for

B

+ rank

\times

output dimension for

A

) . For NOLA, the parameter count is: number of layers

\times

\times

(number of random basis vectors), assuming separate basis sets for Q and V. For SOLAR, the parameter count is: number of layers

\times

\times

( top k coefficients for

B

+ top k coefficients for

A

+ encoded bases for

B

+ encoded bases for

A

) , plus 1 for the shared seed.

Model	LLaMA-3.2 3B			LLaMA-3.1 8B (4-bit)
Method	LoRA	NOLA	SOLAR	LoRA	NOLA	SOLAR
Method	$r$ =1	1000 bases	SOLAR ${}_{r=1(1\text{K}\!\rightarrow\!0.1\text{K})}$	$r$ =1	1000 bases	SOLAR ${}_{r=1(1\text{K}\!\rightarrow\!0.3\text{K})}$
# Params	287K	112K	16K (94% $\downarrow$ )	425K	128K	40K (91% $\downarrow$ )
Val Loss	1.02	1.31	1.04	0.89	1.01	0.90
MMLU Acc	54.0	52.7	54.0	60.9	56.1	60.9

Appendix

Table 17: Model representation efficiency for LLaMA 3B and 8B models. For the 8B model, all methods use 4-bit quantization, making the LoRA baseline equivalent to QLoRA.

Method	# Params (Adapter)	MMLU Accuracy
AdaLoRA (Baseline, 3B)	305K	54.8%
SOLAR (on AdaLoRA, 3B)	16K	54.7%
AdaLoRA (Baseline, 13B)	871K	57.9%
SOLAR (on AdaLoRA, 13B)	16K	57.7%

Appendix

Table 18: SOLAR applied to AdaLoRA adapters on intermediate-scale LLaMA models.

Method	Quantization	# Params	MMLU Acc
LoRA (QLoRA) - LLaMA-2 13B	2-bit	410K	53.1
SOLAR ${}_{r=1(1\text{K}\!\rightarrow\!0.3\text{K})}$ - LLaMA-2 13B	2-bit	51K	53.1
LoRA (QLoRA) - LLaMA-3.1 8B	2-bit	363K	58.4
SOLAR ${}_{r=1(1\text{K}\!\rightarrow\!0.3\text{K})}$ - LLaMA-3.1 8B	2-bit	40K	58.4

Appendix

Table 19: 2-bit quantization experiments comparing LoRA (QLoRA) and SOLAR.

Method	Byte Footprint	Oxford Pets	SUN397	CUB-200	ImageNet-1K
LoRA ( $r$ =1)	74KB	93.0 $\pm$ 0.5	74.3 $\pm$ 0.3	84.7 $\pm$ 0.4	81.5 $\pm$ 0.6
SOLAR ( $r$ =1, $500\!\rightarrow\!50$ )	2KB (97% $\downarrow$ )	91.2 $\pm$ 0.6	72.4 $\pm$ 0.4	81.4 $\pm$ 0.5	80.7 $\pm$ 0.4
SOLAR ( $r$ =1, $100\!\rightarrow\!10$ )	0.4KB (99% $\downarrow$ )	90.3 $\pm$ 0.7	72.4 $\pm$ 0.5	81.3 $\pm$ 0.6	80.6 $\pm$ 0.5

Appendix

Table 20: Evaluation of extreme compression on ViT-B. We report bit-level representation footprint (32-bit baseline) and top-1 accuracy over 5 runs. All models are trained for 10 epochs.

Method	# Params	CIFAR-10	CIFAR-100	Food-101	T-ImageNet
LoRA ( $r=4$ )	492K	99.4	94.6	91.2	92.8
SOLAR ( $r=4,8\text{K}\rightarrow 4\text{K}$ )	340K (31% $\downarrow$ )	99.4	94.5	91.2	92.8

Appendix

Table 21: Scalability of SOLAR on the ViT-G/14 model. Results show top-1 accuracy (%) on full datasets.

Method	#Params	CIFAR-10	CIFAR-100	Food-101	T-ImageNet
LoRA ( $r=4$ )	74K	98.3	90.3	87.6	88.8
LoRA ( $r=2$ )	37K	97.1	89.0	85.5	87.4
SOLAR ( $r=4,4\text{K}\rightarrow 1.6\text{K}$ )	41K	98.3	89.8	87.0	87.9
SOLAR ( $r=4,4\text{K}\rightarrow 0.8\text{K}$ )	22K	97.0	89.0	85.2	87.4

Appendix

Table 22: Comparison of SOLAR with a budget-matched LoRA (r=2) baseline on ViT-B. While LoRA (r=2) has a similar parameter count to the compressed SOLAR adapter, it shows a clear performance degradation. SOLAR maintains performance comparable to the original, higher-rank LoRA (r=4).

Method	Byte Footprint	Oxford Pets	SUN397	CUB-200	ImageNet-1K
LoRA ( $r=1$ )	74KB	93.0	74.3	84.7	81.5
LoRA ( $r=4$ )	297KB	94.2	75.6	86.0	82.8
SVD truncation on LoRA	74KB	92.7	73.3	83.6	80.8
SOLAR on LoRA ( $r=1$ )	8KB	92.6	73.9	84.2	81.3
SOLAR on LoRA ( $r=4$ )	8KB	93.9	75.0	85.4	82.4

Appendix

Table 23: Comparison of SOLAR and simple SVD truncation against standard LoRA adapters on multiple vision datasets. The table reports classification accuracy and the corresponding byte footprint of the adapter parameters after compression. SOLAR consistently reduces the parameter size while preserving or improving performance.

Method	# Params	CIFAR-10 (IID)	CIFAR-10 (non-IID)
LoRA ( $r=4$ )	74K	93.7	87.4
SOLAR ( $r=4,4\text{K}\rightarrow 2\text{K}$ )	51K (31% $\downarrow$ )	93.2	86.7

Appendix

Table 24: Performance of SOLAR on ViT-B under IID and non-IID data distributions in a simulated 10-client federated learning environment.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 122 papers

Apr 30, 2026

Post-Optimization Adaptive Rank Allocation for LoRA

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.

Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith

1Indian Institute of Technology Hyderabad · 2Deakin University

Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Neural Network Pruning

Ask PDF

Oct 26, 2026

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

Low-Rank Adaptation (LoRA) has become a key mechanism for efficiently specializing foundation models into many task- and user-specific variants. Serving many LoRA adapters under a shared resource budget turns post-hoc compression into a two-level allocation problem: rank must be distributed across layers within each adapter and across adapters within the fleet. Although trained LoRAs contain substantial spectral redundancy, spectral reconstruction alone does not determine how much compression an adapter can functionally tolerate. We formulate fleet compression through a functional rank-distortion frontier for each adapter, followed by fleet-level minimax allocation under a shared rank budget. Given exact frontiers, this decomposition is equivalent to joint optimization over all layer ranks. We introduce Functional Rank Allocation (FRA), an unlabeled method for constructing practical frontiers. FRA measures isolated module truncations end to end, uses these measurements as an additive search surrogate solved by dynamic programming, remeasures every complete candidate, and takes the lower-risk envelope with a complementary spectral proposal before exact minimax allocation over the constructed frontiers. Across 78 adapters from three public fleets, FRA improves worst-case and tail utility over uniform spectral compression in constrained-budget regimes. Fixed-budget probes show that isolated costs rank candidates well, but their calibration shifts across proposal shapes; end-to-end verification and the spectral proposal therefore limit functional-search error.

Shenghui Li, Thiemo voigt

Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden

Low-Rank Compression Adaptive Rank Allocation Low-Rank Adaptation

Ask PDF

Oct 5, 2026

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.

Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz

Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Memory-Efficient Fine-Tuning LLM Fine-Tuning Sparse Fine-Tuning Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Model Compression

Explore similar work

Post-Optimization Adaptive Rank Allocation for LoRA

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning