Rethinking LoRA Initialization for Robust Asymmetric Learning Rates

Disen Liao; Yaoliang Yu

cs.LGOct 23, 2026

Rethinking LoRA Initialization for Robust Asymmetric Learning Rates

Authors: Disen Liao, Yaoliang Yu

Organizations: University of Waterloo Vector Institute

Abstract

Low-Rank Adaptation (LoRA) is often highly sensitive to learning-rate (LR) choices. Recent empirical studies suggest that, once LRs are carefully tuned, vanilla LoRA can remain competitive with many proposed LoRA variants. This shifts the central practical question from designing ever more variants to reducing the cost and brittleness of LR selection. LoRA+ addresses this issue with asymmetric LRs, using a larger LR for the up-projection matrix $B$ than for the down-projection matrix $A$ . However, the ratio $η_B / η_A$ itself becomes an additional hyperparameter: large ratios can improve adaptation, but under the standard LoRA initialization they can also cause unstable training and ratio-dependent performance collapse. We identify the source of this brittleness through a signal--noise decomposition of LoRA dynamics. Under standard \initA, the update contains an initialization-induced noise term whose magnitude can be affected by the LR for $B$ , $η_B$ , the initialization scale $β$ , and the adapter rank $r$ . In practical LLM fine-tuning, this non-vanishing noise can limit the performance gains expected from LoRA+ style asymmetric LRs. To suppress this effect, we propose \initAA, a simple initialization that scales the variance of $A$ as $σ_A^2 \propto n^{-2}$ while keeping $B_0=0$ . Under matched rank and training time for both initializations, this reduces the initialization-noise energy by an additional factor of $1/n$ , stabilizing asymmetric LoRA training while preserving zero initial adapter output. Empirically, \initAA substantially widens the stable LR region across initialization scales, adapter ranks, and LoRA update multipliers. Across tasks and models, \initAA reduces best-ratio drift, prevents the high-ratio collapse observed with standard \initA, and makes the width-based guideline $η_B \approx n η_A$ a useful starting point for finite pretrained models.

Figures & tables

Figure 1: Overview of LoRA initialization and feature-norm stability. Left : LoRA branch notation and the initialization scalings compared in this work, including standard Init[A] , Init[AB] , and the proposed Init[

\text{A}^{2}

] . Right : feature-norm trajectories at width

n=4096

for different initialization scales

\beta

. Init[

\text{A}^{2}

] keeps the LoRA features nearly invariant to

\beta

, while the alternative scalings exhibit stronger initialization-dependent growth.

Figure 2: LoRA feature decomposition during fine-tuning of Llama 3.2-1B and Llama 3-8B. Curves show the RMS magnitudes

\sqrt{E_{N}^{t}}

(noise),

\sqrt{E_{S}^{t}}

(signal), and

\sqrt{E_{\text{total}}^{t}}

(total). Standard Init[A] is noise-dominated, whereas under Init[

\text{A}^{2}

] the noise is subdominant and the total norm closely follows the signal norm.

Figure 3: HP transfer on the MNIST toy example, measured by validation loss across widths and LRs.

Figure 4: Llama generation results. (a) Llama 3-8B with varying initialization scale $\beta$ . (b) Llama 3.2-1B with varying rank $r$ . Each panel reports arithmetic and commonsense accuracy under standard Init[A] and Init[ $\text{A}^{2}$ ] while sweeping the LR ratio $\rho=\eta_{B}/\eta_{A}$ , implemented by fixing $\eta_{A}=10^{-5}$ and setting $\eta_{B}=\rho\eta_{A}$ . Numerical labels mark endpoint ratios and near-optimal settings.

Figure 5: Final training loss reached during one epoch on the math benchmark. The LoRA multiplier is fixed to

\alpha/r=1

. (a) varies rank

r

, and (b) varies initialization scale

\beta

. The figure includes equal-LR sweeps with

\eta=\eta_{A}=\eta_{B}

and LR-ratio sweeps; in the ratio panels,

\eta_{A}=10^{-5}

and

\eta_{B}=\rho\eta_{A}

for

\rho=\eta_{B}/\eta_{A}

Figure 6: Arithmetic accuracy as the LoRA multiplier

\alpha/r

increases. Init[

\text{A}^{2}

] remains stable at larger multipliers, while Init[A] degrades beyond

\alpha/r=3

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Empirical rank-coherence exponent during Llama-3.2-1B and Qwen3-0.6B fine-tuning on Math-10K, and RoBERTa-base fine-tuning on SST-2. The fitted exponent in

S_{t}(r)\propto r^{\gamma_{t}}

varies with model, initialization, and training stage. Lines mark CLT-like (

1/2

) and LLN-like (

1

) rank scaling.

Model	Checkpoint	Init[A] median / mean $\gamma$	Init[ $\text{A}^{2}$ ] median / mean $\gamma$
Llama-3.2-1B	First update	0.616 / 0.652	0.621 / 0.655
Llama-3.2-1B	5% epoch	0.642 / 0.665	0.552 / 0.565
Llama-3.2-1B	50% epoch	0.519 / 0.559	0.542 / 0.560
Llama-3.2-1B	End epoch	0.536 / 0.572	0.567 / 0.581
Qwen3-0.6B	First update	0.713 / 0.778	0.714 / 0.779
Qwen3-0.6B	End epoch	0.818 / 0.819	0.523 / 0.533

Appendix

Table 1: Rank-coherence exponents under median / mean aggregation of the normalized contraction. Both aggregation choices show that the exponent depends on model and training stage.

Figure 8: Empirical width-scaling exponent of the processed

A

-update/input alignment during Qwen2.5 minibatch fine-tuning on Math-10K. Panels compare q_proj and down_proj under Init[A] and Init[

\text{A}^{2}

] . Reference lines mark the single-example prediction

\kappa=1

and width-independent alignment

\kappa=0

. The exponent varies across modules and optimizer steps.

Module	Initialization	Step-5 dot $\kappa$ 95% CI	Step-620 dot $\kappa$ 95% CI	Cosine exponent step 5 / 620
q_proj	Init[A]	0.309 $[0.235,\,0.371]$	0.274 $[0.155,\,0.421]$	$-0.477/-0.400$
q_proj	Init[ $\text{A}^{2}$ ]	0.299 $[0.223,\,0.362]$	0.063 $[-0.208,\,0.459]$	$-0.478/-0.353$
down_proj	Init[A]	1.640 $[1.578,\,1.706]$	1.571 $[1.126,\,2.021]$	$0.172/-0.052$
down_proj	Init[ $\text{A}^{2}$ ]	1.626 $[1.573,\,1.686]$	1.521 $[0.922,\,2.013]$	$0.174/-0.098$

Appendix

Table 2: Width-alignment fits at steps 5 and 620. Dot-product exponents include 95% CIs computed over three seeds; the last column gives cosine exponents at the two steps.

Figure 9: RoBERTa accuracy on SST-2 (top) and MNLI (bottom) under Init[A] and Init[

\text{A}^{2}

] . The grids sweep LRs

\eta_{A}

and

\eta_{B}

at two initialization scales

\beta

Figure 10: LoRA accuracy on MNLI and SST-2 with frozen $A$ and varying $B$ LR.

Figure 11: Additional Llama arithmetic experiments. (a) Llama 3.2-1B heatmaps compare Init[A] , Init[ $\text{A}^{2}$ ] , Init[ $\text{A}^{3}$ ] , and Init[ $\text{A}^{4}$ ] over initialization scale $\beta\in\{4,8,12,16\}$ and LR ratio $\rho=\eta_{B}/\eta_{A}\in\{1,10,50,200,500,1000,2000,4000,8000,10000,20000\}$ . (b) Llama 3-8B heatmaps compare full-precision LoRA and QLoRA under Init[A] and Init[ $\text{A}^{2}$ ] over $\beta\in\{0.25,0.5,1,2,4,8\}$ and $\rho\in\{1,10,200,500,1000,2000,4000\}$ . In both panels, each ratio uses $\eta_{A}=10^{-5}$ and $\eta_{B}=\rho\eta_{A}$ . QLoRA exhibits the same high-ratio sensitivity under Init[A] as full-precision LoRA, while Init[ $\text{A}^{2}$ ] preserves a broad high-accuracy region in both settings.

Figure 12: Arithmetic accuracy across model families under standard Init[A] (top) and Init[ $\text{A}^{2}$ ] (bottom). Columns correspond to Llama 3.2-1B, Llama 3.2-3B, Qwen2.5-3B, Qwen3-0.6B, and Mistral-v0.3. Each heatmap sweeps initialization scale $\beta\in\{0.5,1,2,4,8\}$ on the vertical axis and the LR ratio $\rho=\eta_{B}/\eta_{A}$ on the horizontal axis, with ratio values shown in each panel; each ratio uses $\eta_{A}=10^{-5}$ and $\eta_{B}=\rho\eta_{A}$ .

Epoch	Init[A]	Init[ $\text{A}^{2}$ ]	Accuracy gap
1	55.06 / 0.687	75.61 / 0.348	$+20.55$
3	56.71 / 0.697	77.57 / 0.268	$+20.86$
6	55.73 / 0.687	77.30 / 0.182	$+21.58$
10	55.58 / 0.661	75.68 / 0.128	$+20.10$

Appendix

Table 3: Separate training-horizon runs on Llama 3-8B arithmetic at

\rho=4000

. Entries report final accuracy (%) / training loss; the gap is Init[

\text{A}^{2}

] minus Init[A] accuracy.

Epoch	$\alpha/r=4$ Init[A]	$\alpha/r=4$ Init[ $\text{A}^{2}$ ]	$\alpha/r=12$ Init[A]	$\alpha/r=12$ Init[ $\text{A}^{2}$ ]
1	75.40 / 0.360	74.43 / 0.360	71.33 / 0.419	75.24 / 0.356
3	75.27 / 0.256	77.40 / 0.294	71.18 / 0.339	76.89 / 0.279
5	74.39 / 0.199	76.34 / 0.239	70.57 / 0.298	75.75 / 0.216
10	68.72 / 0.135	75.63 / 0.144	68.33 / 0.249	72.64 / 0.132

Appendix

Table 4: Training-horizon comparison for the Llama 3-8B multiplier experiment. Each entry reports final arithmetic accuracy (%) / training loss from a separate run.

$\alpha/r$	Init[A]	Init[ $\text{A}^{2}$ ]	95% CI for Init[ $\text{A}^{2}$ ] $-$ Init[A]
1	$76.35\pm 0.63$	$76.42\pm 0.67$	$[-0.28,\,0.43]$
3	$75.71\pm 0.23$	$76.71\pm 0.70$	$[0.15,\,1.84]$
10	$71.69\pm 0.70$	$76.42\pm 0.16$	$[3.80,\,5.65]$
12	$71.44\pm 0.52$	$76.87\pm 0.48$	$[4.68,\,6.20]$

Appendix

Table 5: Five-seed Llama 3-8B arithmetic accuracy (%) at fixed LR ratio

\rho=200

. Accuracy entries are mean

\pm

SD. The final column gives the reported 95% CI for the accuracy difference, in percentage points.

Initialization	Best accuracy	Within 1 point of own best	Above 90% accuracy	Chance-level cells
Init[A]	$93.28\pm 0.64$	$5/25$	$8/25$	$14/25$
Init[ $\text{A}^{2}$ ]	$93.53\pm 0.52$	$8/25$	$16/25$	$5/25$

Appendix

Table 6: Five-seed RoBERTa/SST-2 LR-grid summary at

\beta=8

. Best accuracy (%) is mean

\pm

SD. Counts are reported out of 25 cells; the near-best threshold is one percentage point below each method’s own best.

$\alpha/r$	Initialization	Best $\rho$	Best accuracy
4	Init[A]	50	$76.39\pm 0.62$
4	Init[ $\text{A}^{2}$ ]	50	$76.83\pm 0.36$
12	Init[A]	50	$75.51\pm 0.34$
12	Init[ $\text{A}^{2}$ ]	100	$76.84\pm 0.31$

Appendix

Table 7: Three-seed matched LR retuning at two LoRA multipliers. Accuracy (%) is mean

\pm

SD; the best ratio is selected from the same grid for both methods.

Model	Initialization	Noise	Signal	Cross
Llama-3.2-1B	Init[A]	117.2	4.9	$-22.1$
Llama-3.2-1B	Init[ $\text{A}^{2}$ ]	3.1	85.9	$+11.1$
Llama 3-8B	Init[A]	117.4	8.1	$-25.5$
Llama 3-8B	Init[ $\text{A}^{2}$ ]	1.5	90.4	$+8.1$

Appendix

Table 8: Final mean energy contributions (%) over three seeds. Noise and signal energies are nonnegative, while the cross term is signed; noise can exceed 100% when the cross term is destructive. Rounded entries need not sum to exactly 100%.

Figure 13: MNLI and SST-2 accuracy under Init[ $\text{A}^{2}$ ] and Init[AB] with symmetric LRs ( $\eta_{A}=\eta_{B}$ ). Annotated cells have gaps $|\Delta|>0.5\%$ .

Initialization	$1\mathrm{e}{-6}$	$3\mathrm{e}{-6}$	$1\mathrm{e}{-5}$	$3\mathrm{e}{-5}$	$1\mathrm{e}{-4}$	$3\mathrm{e}{-4}$	$1\mathrm{e}{-3}$	$3\mathrm{e}{-3}$	$8\mathrm{e}{-3}$
EVA	24.23	29.36	34.01	38.82	46.01	55.95	61.78	60.15	0.00
OLoRA	29.19	30.29	33.19	38.61	52.13	58.23	60.07	58.22	7.28
PiSSA	30.36	34.03	40.25	47.19	54.77	58.42	60.97	58.80	22.30
Init[A] ( $\beta=1$ )	23.51	28.69	34.74	40.91	48.26	57.83	62.15	60.72	2.41
Init[A] ( $\beta=8$ )	30.88	32.85	39.56	47.90	56.08	60.27	60.15	57.10	0.53
Init[ $\text{A}^{2}$ ] ( $\beta=1$ )	24.13	28.33	31.02	35.22	45.51	51.48	60.77	60.98	2.14

Appendix

Table 9: Average accuracy (%) on the math benchmark across initialization methods under equal LRs

\eta_{A}=\eta_{B}=\eta

. Best result in each column is bolded.

Initializer	$\beta$	$\rho=1$	$\rho=200$	$\rho=4000$
Init[ $\text{A}^{2}$ ]	0.5	71.44	76.72	76.70
Init[ $\text{A}^{2}$ ]	1	71.24	76.63	77.93
PEFT Orthogonal	–	72.73	70.06	2.49
Orthogonal- $A^{2}$	0.5	71.35	76.76	77.72
Orthogonal- $A^{2}$	1	71.66	77.23	77.74
EVA	–	75.19	74.26	5.10

Appendix

Table 10: Arithmetic accuracy (%) for structured LoRA initializers and their variance-compressed

A^{2}

variants under asymmetric LR ratios. Results are single-run values; the best result within each initializer family and ratio is bolded.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 538 papers

Sep 5, 2026

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.

Huiyi Wang, Daijiao Liu, Lina Yao +1

University of New South Wales

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning Adaptive Learning Rate

Ask PDF

Jul 28, 2026

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.

Dianze Liu, Farshid Ghezelbash

Georgia Institute of Technology

Low-Rank Adaptation

Ask PDF

May 29, 2026

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Valérie Castin, Kimia Nadjahi, Pierre Ablin +1

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Low-Rank Adaptation

Explore similar work

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence