A Riemannian Geometry for Low-rank Adaptation

Shoichiro Takeda; Shin'ya Yamaguchi; Satoshi Suzuki; Yasunori Akagi

cs.LGOct 6, 2026

A Riemannian Geometry for Low-rank Adaptation

Authors: Shoichiro Takeda, Shin'ya Yamaguchi, Satoshi Suzuki, Yasunori Akagi

Organizations: NTT, Inc.

Abstract

Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the weight update via full fine-tuning by a low-rank matrix $BA^\top$ . This parameterization leads to the equivalence relation $(B, A) \sim (BG^{-1}, AG^\top)$ for any invertible matrix $G$ because $BA^\top = BG^{-1}(AG^\top)^\top$ and thus both pairs yield the same loss value. This relation induces a quotient manifold where matrices $(BG^{-1}, AG^\top)$ for all $G$ are identified, eliminating redundant directions along which the loss value remains unchanged. To respect the geometry of this manifold, the original search space is endowed with a Riemannian metric that is invariant under the equivalence relation. Such a metric induces preconditioning at each gradient step and ensures that each weight update via LoRA changes the loss value, leading to efficient optimization. In this paper, we propose a new Riemannian metric that is specifically tailored to LoRA to close the gap to full fine-tuning at the weight level. We theoretically show that LoRA with our preconditioning induced by this metric satisfies the following two properties at each iteration: (i) The weight update follows the direction closest to the gradient of full fine-tuning within the subspace of first-order weight changes allowed by the LoRA parameterization. (ii) The updated weight matrix is closer in Frobenius norm to that of full fine-tuning than the updated weight matrices of LoRA with conventional preconditioning and without preconditioning. These theoretical insights suggest that our preconditioning makes LoRA better approximate full fine-tuning, thereby leading to more efficient optimization. Experiments show the effectiveness and efficiency of our preconditioning for LoRA on fine-tuning tasks with language and vision domains.

Figures & tables

Method	BLEU	NIST	METEOR	ROUGE-L	CIDEr
SGD	0.6698	8.5505	0.4479	0.6908	2.2996
w/ 4	0.6852	8.6371	0.4641	0.7097	2.4367
w/ 5 (Ours)	0.6944	8.7449	0.4632	0.7127	2.5082
AdamW	0.7020	8.8433	0.4676	0.7193	2.5500
w/ 4	0.7003	8.8170	0.4671	0.7181	2.5312
w/ 5 (Ours)	0.7047	8.8527	0.4697	0.7209	2.5454

Table 1: Scores in LoRA fine-tuning for the GPT-2 medium model on E2E natural language generation challenge task with different preconditioning. “w/” denotes “with”. Bold and underlined values indicate the highest and second-highest scores, respectively.

Method	Cars	DTD	EuroSAT	GTRSB	RESISC45	SUN397	SVHN	Ave.
AdamW	71.82	70.53	98.35	97.84	94.48	72.04	96.79	85.98
w/ 4	72.14	71.28	98.19	98.32	94.49	72.64	96.83	86.27
w/ 5 (Ours)	72.03	71.86	98.52	98.54	94.70	72.57	96.92	86.45

Table 2: Top-1 test accuracy in LoRA fine-tuning for the CLIP ViT-B/32 model on the seven image classification datasets with different preconditioning. “Ave.” denotes “Average”.

Method	BoolQ	PIQA	SIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Ave.
AdamW	73.33	83.41	79.38	92.36	83.74	85.77	72.87	82.60	81.68
w/ 4	74.31	84.17	77.99	93.12	83.43	86.28	74.23	82.20	81.97
w/ 5 (Ours)	73.55	83.62	79.02	92.95	85.48	86.41	74.32	83.40	82.34

Table 3: Top-1 test accuracy in LoRA fine-tuning for the LLaMA2-7B model on the eight commonsense reasoning datasets with different preconditioning.

Figure 1: Training time comparison of the CLIP ViT-B/32 model on the seven image classification datasets when using the AdamW methods with different preconditioning.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

# of Samples	Cars	DTD	EuroSAT	GTRSB	RESISC45	SUN397	SVHN
Train	7,329	1,692	16,200	23,976	18,900	17,865	65,931
Validation	815	188	2,700	2,664	6,300	1,985	7,326
Test	8,041	1,880	8,100	12,630	6,300	19,850	26,032

Appendix

Table 4: The number of samples in the train, validation, and test sets of each image classification dataset used in Sec. 7.3

# of Samples	BoolQ	PIQA	SIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA
Test	1172	2376	3270	10042	500	1838	1954	1267

Appendix

Table 5: The number of samples in the test set of each commonsense reasoning dataset used in Sec. 7.4

Method	Learning rate
SGD	0.08
w/ 4	0.06
w/ 5 (Ours)	0.02
AdamW	0.0028
w/ 4	0.0028
w/ 5 (Ours)	0.0030

Appendix

Table 6: Learning rates for the natural language generation task in Sec. 7.2

# of Samples	Cars	DTD	EuroSAT	GTRSB	RESISC45	SUN397	SVHN
AdamW	0.0008	0.0006	0.001	0.0007	0.001	0.0005	0.001
w/ 4	0.0009	0.0008	0.0009	0.0009	0.001	0.0007	0.001
w/ 5 (Ours)	0.0009	0.001	0.001	0.001	0.001	0.0007	0.001

Appendix

Table 7: Learning rates for the image classification task in Sec. 7.3

Method	Learning rate
AdamW	0.0001
w/ 4	0.0001
w/ 5 (Ours)	0.0001

Appendix

Table 8: Learning rates for the commonsense reasoning task in Sec. 7.4

Learning rate	0.02	0.06	0.08
$\left\\|W_{1}^{\operatorname{FFT}}-W_{1}^{\operatorname{LoRA}}\right\\|_{F}$	10647.35 $\times 10^{-5}$	16542.16 $\times 10^{-5}$	20334.76 $\times 10^{-5}$
$\left\\|W_{1}^{\operatorname{FFT}}-W_{1}^{\operatorname{LoRA-CP}}\right\\|_{F}$	10646.05 $\times 10^{-5}$	16534.73 $\times 10^{-5}$	20324.08 $\times 10^{-5}$
$\left\\|W_{1}^{\operatorname{FFT}}-W_{1}^{\operatorname{LoRA-OP}}\right\\|_{F}$	10646.04 $\times 10^{-5}$	16534.69 $\times 10^{-5}$	20324.02 $\times 10^{-5}$

Appendix

Table 9: Comparisons of the Frobenius norm.

Explore similar work

Sep 29, 2026cs.AI

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

Yihao Ouyang, Shiwei Li, Haozhao Wang +5

Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China

Low-Rank Structure Model Fine-Tuning Lipschitz Continuity

Ask PDF

May 29, 2026cs.LG

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Valérie Castin, Kimia Nadjahi, Pierre Ablin +1

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Training-Side Stage-Aware Low-Rank Adaptation Low-Rank Structure Convergence

Ask PDF

May 9, 2026cs.LG

AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

Low-Rank Adaptation (LoRA) reparameterizes a weight update as a product of two low-rank factors, but the Jacobian

J_{G}

of the generator mapping the factors to the weight matrix is rank-deficient, so the factor-space preconditioner

J_{G}^* {F}_t J_{G}

induced by any

{W}

-space preconditioner

{F}_t

is singular, and consequently the standard chain rule cannot be uniquely inverted to map a preconditioned

{W}

-space direction back to a factor-space update. We cast existing LoRA optimizers in a unified framework parameterized by two choices: (i) which invertible surrogate for

J_{G}^* {F}_t J_{G}

to use, and (ii) which

{F}_t

{W}

to use. Existing methods occupy four families along these axes: factor-space adaptive updates, block-diagonal surrogates for

J_{G}^* J_{G}

, Frobenius-residual pseudoinverse methods, and Riemannian manifold constraint. Within this design space, a gradient-statistics-aware

{F}_t

paired with a closed-form factor-space solve at

{O}((m+n)r)

memory remains underexplored. We propose \textbf{AdaPreLoRA}, which fills this gap by adopting the Adafactor diagonal Kronecker preconditioner

{H}_t

{W}

and selecting from the resulting factor-space solution family the element minimizing an

{H}_t

-weighted imbalance between the two factor contributions; by construction, the resulting factor update is the closest LoRA approximation to the preconditioned

{W}

-space direction under the

{H}_t

-weighted norm. Across GPT-2 (E2E), Mistral-7B and Qwen2-7B (GLUE, ARC, GSM8K), and diffusion-model personalization, AdaPreLoRA is competitive with or improves over a representative set of LoRA optimizers while keeping peak GPU memory at the LoRA optimizer level.

Ziyun Liu, Fengmiao Bian, Jian-Feng Cai

Department of Mathematics The Hong Kong University of Science and Technology

Spectral Preconditioning Low-Rank Structure Adaptive Optimizers Jacobian

Ask PDF