One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

Huiyi Wang; Daijiao Liu; Lina Yao; Dong Gong

cs.LGSep 5, 2026

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

Authors: Huiyi Wang, Daijiao Liu, Lina Yao, Dong Gong

Organizations: University of New South Wales

Abstract

Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.

Figures & tables

Panel of Figure 1 — Figure 1: Per-rank-one update velocity of LoRA modules. Within a single LoRA, the $r$ rank-one components do not update at the same rate.

Panel of Figure 2 — Figure 2: Per-rank-one anisotropy in LoRA training on math reasoning. (a) The within-module velocity ordering is established early and persists in training. (b) High-velocity components accumulate larger contribution magnitude throughout training. (c) Lower mean per-rank-one velocity is associated with more concentrated singular spectra of $BA$ at convergence, under-utilizing the available rank budget.

Model	Method	#Params (%)	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg.
LLaMA2-7B	LoRA	0.83	72.2	84.6	77.4	90.1	82.6	81.2	67.4	64.8	77.5
	rsLoRA	0.83	71.1	81.6	78.8	87.7	81.6	82.6	67.2	81.4	79.0
	LoRA+	0.83	70.7	81.8	78.8	89.5	82.1	83.8	68.6	79.8	79.4
	AnLR-LoRA	0.83	72.5	84.1	78.9	91.4	82.8	85.0	71.3	82.4	81.0
LLaMA3-8B	LoRA	0.70	70.8	85.2	79.9	91.7	84.3	84.2	71.2	79.0	80.8
	rsLoRA	0.70	72.0	85.8	79.7	92.8	83.7	85.9	73.7	81.2	81.9

Table 1: Commonsense reasoning accuracy (%) with LLaMA2-7B, LLaMA3-8B and Qwen2.5-7B.

Method	MT-Bench	GSM8K	HumanEval
LoRA	6.11	59.59	24.39
LoRA+	5.98	60.27	24.39
AnLR-LoRA	6.36	61.87	28.05

Table 2: Evaluation results for dialogue, math and coding with LLaMA2-7B.

Method	#Params (%)	VQA ${}^{\text{v2}}$	GQA	VizWiz	SQA	VQA ${}^{\text{T}}$	POPE	MMBench	Avg.
FT †	100	78.50	61.90	50.00	66.80	58.20	85.90	64.30	66.50
LoRA	4.61	79.02	63.34	47.81	69.06	57.56	86.09	74.96	68.26
LoRA+	4.61	78.59	62.86	47.18	66.39	56.56	86.88	73.36	67.40
AnLR-LoRA	4.61	79.02	63.07	50.88	69.11	58.21	86.89	74.43	68.80

Table 3: Results on visual instruction tuning with LLaVA-1.5-7B. FT results cited from Liu et al. (2024b) .

Panel of Figure 3 — Figure 3: Effect of AnLR-LoRA’s anisotropic learning rate.

Figure 4: Visualization of per-rank-one velocity of AnLR-LoRA.

Figure 5: Average commonsense reasoning accuracy on LLaMA2-7B across varying LR. AnLR-LoRA exceeds LoRA at every LR, with the gap widening as LR moves away from the optimal range.

Method	GSM8K	HumanEval
PiSSA	58.98	24.39
AnLR-PiSSA	59.36	25.00
MiLoRA	60.12	25.61
AnLR-MiLoRA	61.41	26.22
LoRA-Dash	60.73	27.44
AnLR-LoRA-Dash	61.11	28.05

Table 4: Results of applying anisotropic learning rate to LoRA variants.

LR scaling	GSM8K	HumanEval
None (uniform)	59.59	24.39
Velocity-only	61.18	26.22
SNR-only	60.12	25.00
Velocity + SNR (ours)	61.87	28.05

Table 5: Ablation of AnLR-LoRA’s per-rank-one scaling signal.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Value
Rank $r$	32
LoRA Alpha	64
Dropout	0.05
Optimizer	AdamW
LR	1e-4
LR Scheduler	Linear

Appendix

Table 6: Hyperparameter configuration on commonsense reasoning.

Hyperparameters	Value
Rank $r$	8
LoRA Alpha	16
Dropout	0.05
Optimizer	AdamW
LR	3e-4
LR Scheduler	Linear

Appendix

Table 7: Hyperparameter configuration on dialogue, math reasoning and code generation.

Hyperparameters	LLaVA-1.5-7B
Rank $r$	128
LoRA Alpha	256
Dropout	0.05
Optimizer	AdamW
LR	2e-4
LR Scheduler	Linear

Appendix

Table 8: Hyperparameter configuration on visual instruction tuning.

Method	5e-5	1e-4	2e-4	3e-4
LoRA	66.2	77.5	77.6	74.9
rsLoRA	49.2	79.0	61.5	63.4
LoRA+	66.0	79.4	77.0	74.9
AnLR-LoRA	79.9	81.0	80.9	78.8

Appendix

Table 9: Average commonsense reasoning accuracy (%) across varying learning rates on LLaMA2-7B.

LR	Method	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg.
5e-5	LoRA	71.8	82.4	65.7	51.9	82.2	66.9	57.9	50.6	66.2
5e-5	AnLR-LoRA	70.6	83.3	79.5	89.1	82.0	84.7	69.9	80.4	79.9
1e-4	LoRA	72.2	84.6	77.4	90.1	82.6	81.2	67.4	64.8	77.5
1e-4	AnLR-LoRA	72.5	84.1	78.9	91.4	82.8	85.0	71.3	82.4	81.0
2e-4	LoRA	69.8	79.9	79.5	83.6	82.6	79.8	64.7	81.0	77.6
2e-4	AnLR-LoRA	72.1	83.4	80.0	91.0	83.7	84.5	70.2	82.2	80.9

Appendix

Table 10: Comparison of LoRA and AnLR-LoRA with varying learning rates on LLaMA2-7B.

Rank	Method	#Params (%)	Acc.
$r=4$	LoRA	0.10	58.30
$r=4$	AnLR-LoRA	0.10	60.96
$r=8$	LoRA	0.21	59.59
$r=8$	AnLR-LoRA	0.21	61.87
$r=32$	LoRA	0.83	58.23
$r=32$	AnLR-LoRA	0.83	61.64

Appendix

Table 11: Ablation study on LoRA rank with LLaMA2-7B on math reasoning task.

LR Scaling	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg.
Random	62.1	73.0	39.1	57.9	79.2	45.3	42.3	38.8	54.7
$\\|\nabla W\odot W\\|$	62.2	83.9	79.5	89.5	82.7	84.2	70.9	81.2	79.3
AnLR-LoRA	72.5	84.1	78.9	91.4	82.8	85.0	71.3	82.4	81.0

Appendix

Table 12: Ablation on per-rank-one LR scaling strategy.

Panel of Figure 6 — Figure 6: Accuracy across the velocity weight $\alpha$ on math reasoning and code generation tasks.

Panel of Figure 7 — Figure 7: Accuracy across the SNR weight $\beta$ on math reasoning and code generation tasks.

Panel of Figure 8 — Figure 8: Average commonsense reasoning accuracy across the velocity EMA coefficient $\rho$ .

Panel of Figure 9 — Figure 9: Average commonsense reasoning accuracy across the clamp bound $\kappa$ .

Panel of Figure 10 — Figure 10: Average commonsense reasoning accuracy across the warmup length $T_{\mathrm{warm}}$ .

Method	LLaMA2-7B	LLaMA3-8B	Qwen2.5-7B
LoRA	77.5 $\pm$ 1.85	80.8 $\pm$ 0.43	85.7 $\pm$ 0.34
rsLoRA	79.0 $\pm$ 1.10	81.9 $\pm$ 0.30	85.6 $\pm$ 0.11
LoRA+	79.4 $\pm$ 0.38	84.8 $\pm$ 0.19	85.7 $\pm$ 0.38
AnLR-LoRA	81.0 $\pm$ 0.44	85.1 $\pm$ 0.14	87.4 $\pm$ 0.13

Appendix

Table 13: Accuracies with standard deviations on commonsense reasoning tasks.

Method	GSM8K	HumanEval	Visual Instr. Tuning
LoRA	59.59 $\pm$ 0.79	24.39 $\pm$ 0.93	68.26 $\pm$ 0.13
LoRA+	60.27 $\pm$ 0.62	24.39 $\pm$ 1.61	67.40 $\pm$ 0.14
AnLR-LoRA	61.87 $\pm$ 0.53	28.05 $\pm$ 0.93	68.80 $\pm$ 0.10

Appendix

Table 14: Accuracies with standard deviations on natural language generation and visual instruction tuning tasks.

Method	#Params (%)	GSM8K
LoRA	0.16	87.49
rsLoRA	0.16	84.00
LoRA+	0.16	87.00
AnLR-LoRA	0.16	88.32

Appendix

Table 15: Evaluation results with Qwen2.5-14B.

Method	LLaMA2-7B	LLaMA3-8B	Qwen2.5-7B
AdaLoRA	70.6	84.6	86.2
DoRA	79.6	85.0	86.8
LoRA-Pro	77.6	84.9	86.2
AnLR-LoRA	81.0	85.1	87.4

Appendix

Table 16: Comparison with additional LoRA variants on commonsense reasoning tasks with average accuracy reported.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 753 papers

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
Rethinking LoRA Initialization for Robust Asymmetric Learning Rates
Oct 23, 2026Disen Liao, Yaoliang YuLow-Rank Adaptation Parameter-Efficient Fine-Tuning
University of Waterloo Vector Institute
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China

May 29, 2026J/K move · Enter open · S save

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Valérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Abstract

LLM Fine-Tuning

Explore similar work

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Rethinking LoRA Initialization for Robust Asymmetric Learning Rates

TLoRA: Task-aware Low Rank Adaptation of Large Language Models