SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Yupeng Chang; Yuan Wu; Yi Chang

cs.LGJun 27, 2026

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Authors: Yupeng Chang, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.

Figures & tables

Figure 1: Overview of Static Orthogonal Subspace LoRA (SOS-LoRA) with row-vector convention $y=xW_{0}$ , $W_{0}\in\mathbb{R}^{m\times n}$ , $x\in\mathbb{R}^{1\times m}$ , and $y\in\mathbb{R}^{1\times n}$ . (a) LoRA freezes $W_{0}$ and learns a single rank- $r_{\text{tot}}$ update $\Delta W=\frac{\alpha}{r_{\text{tot}}}AB$ with $A\in\mathbb{R}^{m\times r_{\text{tot}}}$ and $B\in\mathbb{R}^{r_{\text{tot}}\times n}$ , yielding $y=x(W_{0}+\Delta W)$ ; under this parameterization, the adapter depends on the input only through the shared projection $xA$ . (b) SOS-LoRA reparameterizes the same total rank budget as $K$ static (always-on, non-routed) rank- $r^{\prime}$ experts $\{(A_{k},B_{k})\}_{k=1}^{K}$ with $r^{\prime}=r_{\text{tot}}/K$ , combined by fixed multi-scale scales $\{s_{k}\}$ and encouraged to use diverse input-side directions via an orthogonality regularizer on $\{A_{k}\}$ . The merged update is $\Delta W=\sum_{k=1}^{K}s_{k}A_{k}B_{k}$ , which can be absorbed into $W_{0}$ with no additional inference-time parameters or latency after merging.

Model	Method	BoolQ	PIQA	SIQA	Hella Swag	Wino Grande	ARC-e	ARC-c	OBQA	Avg.
ChatGPT		73.1	85.4	68.5	78.5	66.1	89.8	79.9	74.8	77.0
Llama 2-7B	LoRA	69.8	79.9	79.5	83.6	82.6	79.8	64.7	81.0	77.6
	DoRA	71.8	83.7	76.0	89.1	82.6	83.7	68.2	82.4	79.7
	MixLoRA	72.0	82.8	77.3	92.9	75.7	77.4	57.5	81.1	77.1
	PiSSA	75.0	87.0	81.6	95.0	86.5	88.5	75.9	86.4	84.5
	SOS-LoRA	75.7	87.5	83.2	95.8	87.8	89.2	76.9	88.1	85.5

Table 1: Main results on eight reasoning and knowledge-intensive benchmarks for the Llama family. We compare SOS-LoRA with strong PEFT baselines under a unified evaluation pipeline. Within each model block, the best and second-best PEFT results are marked in bold and underlined, respectively. We additionally report ChatGPT as a reference (see Appendix for evaluation details).

Method	MRPC	RTE	CoLA	STS-B	SST-2	QQP	QNLI	MNLI	Avg.
Full FT	88.2	84.1	64.6	90.6	94.3	92.0	92.7	87.5	86.8
LoRA	89.9	85.9	62.4	91.4	94.4	90.8	92.6	86.9	86.8
DyLoRA	89.5	84.5	61.1	91.1	94.3	90.2	92.2	86.3	86.2
AdaLoRA	90.2	85.2	61.6	91.2	94.5	90.1	93.1	87.3	86.7
Delta-LoRA	90.2	87.0	63.8	91.6	95.1	90.9	93.1	87.5	87.5
MELoRA	90.9	86.6	64.1	91.9	95.4	90.8	93.2	87.2	87.5

Table 2: Performance comparison on the GLUE benchmark using RoBERTa-base. We evaluate SOS-LoRA against full fine-tuning (Full FT) and PEFT baselines under our unified training and evaluation pipeline. For each task, the best result is in bold and the second-best is underlined.

Figure 2: Orthogonal regularization reduces cross-expert similarity in input-side directions. Left: SOS-LoRA with the full method. Right: SOS-LoRA without $\mathcal{L}_{\text{orth}}$ . Each entry is $\frac{1}{r^{\prime 2}}\|\tilde{A}_{k}^{\top}\tilde{A}_{\ell}\|_{F}^{2}$ , i.e., the mean squared cosine similarity between $\ell_{2}$ -normalized input-side directions across experts ( $K{=}4$ ), averaged over 224 adapted modules and scaled by $10^{6}$ . Removing $\mathcal{L}_{\text{orth}}$ increases off-diagonal similarity.

Figure 3: Multi-scale expert scaling leads to more separated per-expert update magnitudes. Left: per-expert relative update magnitude

p_{k}=\|\Delta W_{k}\|_{F}/\sum_{j=1}^{K}\|\Delta W_{j}\|_{F}

across modules (

K{=}4

). Right: distribution of the top-1 share

\max_{k}p_{k}

, where multi-scale scaling yields a more concentrated contribution profile.

Figure 4: Optimization dynamics on GSM8K. Training loss comparing SOS-LoRA with standard LoRA and the uniform-scale ablation at two rank budgets (

r_{\text{tot}}{=}32

and

r_{\text{tot}}{=}128

). SOS-LoRA exhibits faster early loss reduction and lower final loss.

Figure 5: Analysis of SOS-LoRA on Llama 2-7B. (a) Ablation on reasoning and knowledge-intensive benchmarks (standard budget

r_{\text{tot}}{=}8

): stepwise variants illustrate the contribution of each component. (b) Ablation on GSM8K (high-rank budget

r_{\text{tot}}{=}128

): under our training setup, performance increases with rank and reaches 57.03%. (c) Rank scalability on GSM8K: SOS-LoRA consistently outperforms LoRA and matches or surpasses Full FT at higher ranks in our setup. (d) Sensitivity to expert count

K

on GSM8K (fixing

r_{\text{tot}}{=}128

): performance peaks at

K{=}4

, balancing cross-expert diversification and per-expert capacity.

Method	Peak VRAM (GB) $\downarrow$	Time (wall) $\downarrow$	GSM8K (%) $\uparrow$	MATH (%) $\uparrow$
Full FT †	N/A (OOM)	N/A	48.90	7.48
LoRA	66.32	4:31	42.68	5.92
SOS-LoRA	67.45	4:52	57.03	10.06

Table 3: Cost–accuracy trade-off on GSM8K and MATH when fine-tuning Llama 2-7B on the first 100k MetaMathQA samples using 2

\times

NVIDIA A40 (48GB). Peak VRAM is the maximum reserved memory.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection
Oct 27, 2026Igor Shalygin, Grigorii Davydenko, Andrey Veprikov +4Low-Rank Adaptation Fine-Tuning Language Models
Moscow Independent Research Institute of Artificial Intelligence (MIRAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab +2
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Apr 20, 2026J/K move · Enter open · S save

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA

A

matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the

A

matrix is frozen, and only the

B

matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Abstract

Low-Rank Adaptation

Explore similar work

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence