SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection

Igor Shalygin; Grigorii Davydenko; Andrey Veprikov; Aleksey Antonov; Nikolay Bushkov; Stanislav Moiseev; Aleksandr Beznosikov

cs.LGOct 27, 2026

SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection

Authors: Igor Shalygin, Grigorii Davydenko, Andrey Veprikov, Aleksey Antonov, Nikolay Bushkov, Stanislav Moiseev, Aleksandr Beznosikov

Organizations: Moscow Independent Research Institute of Artificial Intelligence (MIRAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · T-Technologies ·

Abstract

The success of large language models in real-world applications relies on parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which inserts compact, trainable adapters into selected layers. In this paper, we propose SimplexLoRA, a LoRA variant that adaptively adjusts adapter ranks during fine-tuning based on their learned importance. Unlike standard LoRA, which employs a fixed rank across all layers, SimplexLoRA optimizes per-adapter importance weights on the $n$ -scaled simplex during a short warm-up and then freezes them, dynamically allocating capacity where it is most beneficial without increasing the overall parameter budget or inference cost. We evaluate SimplexLoRA across 12 language understanding, summarization, and reasoning tasks using DeBERTaV3, BART-large, Llama 3.1 8B, and Qwen3-8B. Our results show that SimplexLoRA consistently matches or outperforms standard LoRA and recent adaptive baselines, with the largest gains at low ranks. Our code is available at https://github.com/brain-lab-research/SimplexLoRA.

Figures & tables

Figure 1: SimplexLoRA framework. Per-adapter importance weights are constrained to the

n

-scaled simplex, enabling dynamic allocation of a fixed rank budget across layers.

Figure 2: Scheduler for SimplexLoRA parameters.

Method	# Params	MNLI	SST-2	CoLA	QQP	QNLI	RTE	MRPC	ALL
Method	# Params	Acc	Acc	Mcc	Acc/F1	Acc	Acc	Acc/F1	Avg
Full Fine-Tuning	184M (100%)	0.8910	0.9541	0.6806	0.8962/0.8644	0.9383	0.8376	0.8848/0.9165	0.8689
$\texttt{LoRA}_{r=8}$ [ Hu et al., 2021 ]	442K (0.24%)	0.8797	0.9450	0.6913	0.8802/0.8437	0.9301	0.8448	0.8897/ 0.9220	0.8696
$\texttt{LoHA}_{r=8}$ [ Nam et al., 2022 ]	884K (0.48%)	0.8560	0.9392	0.6295	0.8674/0.8269	0.9085	0.8051	0.8628/0.9007	0.8440
$\texttt{Krona}_{r=32}$ [ Edalati et al., 2022 ]	57.6K (0.03%)	0.7587	0.9289	0.5907	0.8275/0.7816	0.8605	0.7509	0.7500/0.8416	0.7878
$\texttt{VERA}_{r=1024}$ [ Kopiczko et al., 2023 ]	1.64M (0.88%)	0.8355	0.9404	0.6252	0.8592/0.8203	0.8975	0.7581	0.8529/0.8962	0.8317

Table 1: Comparative performance of "out-of-the-box" algorithms on DeBERTaV3 base . The best results are shown in bold , and the second-best are underlined . All experiment details are provided in Appendix D .

Figure 3: Illustration of the rank distribution of SimplexLoRA .

Model	Rank	Method	MNLI	SST-2	CoLA	QQP	QNLI	RTE	MRPC	ALL
Model	Rank	Method	Acc	Acc	Mcc	Acc/F1	Acc	Acc	Acc/F1	Avg
DeBERTaV3 base	$r=1$	L	0.8912	0.9488	0.6558	0.8905/0.8544	0.9250	0.7943	0.8905/0.9213	0.8562
	$r=1$	S	0.8913	0.9576	0.6775	0.9018 / 0.8709	0.9317	0.8159	0.9093 / 0.9359	0.8690
	$r=2$	L	0.9004	0.9413	0.6600	0.9077/0.8631	0.9369	0.8068	0.9003/0.9193	0.8629
	$r=2$	S	0.8946	0.9608	0.6658	0.9178 / 0.8842	0.9338	0.8252	0.9103 / 0.9314	0.8717
	$r=4$	L	0.8780	0.9562	0.6809	0.9153/0.8866	0.9582	0.7816	0.9207 / 0.9471	0.8700

Table 2: Comparative results of top-performing algorithms: LoRA (L) and SimplexLoRA (S) on DeBERTaV3 base and Llama 3.1 8B. Results are averaged over 3 runs with different seeds. The best results per rank are shown in bold . All experiment details are provided in Appendix D .

Rank	Method	Qwen3-8B				BART-large
		GSM8K	BoolQ	HellaSwag	ALL	XSum	CNN/DailyMail
		Acc	Acc	Acc	Avg	ROUGE-1	ROUGE-1
$r=1$	L	0.8522	0.655	0.772	0.760	0.372	0.245
$r=1$	S	0.8666	0.680	0.775	0.774	0.373	0.249
$r=2$	L	0.8681	0.662	0.776	0.769	0.374	0.250
$r=2$	S	0.8673	0.682	0.791	0.780	0.376	0.254

Table 3: Comparative results: LoRA (L) and SimplexLoRA (S) on Qwen3-8B and BART-large. The best results per rank and model are shown in bold . All details are provided in Appendix D .

Figure 4: The influence of hyperparameters on SimplexLoRA performance.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
Jun 27, 2026Yupeng Chang, Yuan Wu, Yi ChangLow-Rank Adaptation LLM Fine-Tuning
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Jun 27, 2026J/K move · Enter open · S save

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Yupeng Chang, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Low-Rank Adaptation LLM Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.

Abstract

Low-Rank Adaptation

Explore similar work

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence