Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space

Frédéric Zheng; Alexandre Proutière

cs.LGMar 31, 2026

Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space

Authors: Frédéric Zheng, Alexandre Proutière

Organizations: KTH, Stockholm, Sweden

Abstract

Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models, but often lag behind full fine-tuning in both convergence speed and final performance. Recent approaches aim to reduce this gap by aligning LoRA parameter updates with those of full fine-tuning, but such parameter-space alignment only indirectly controls model predictions. Instead, we adopt a function-space perspective and formulate the prediction alignment problem, whose objective is to match the outputs of LoRA fine-tuning to those of full fine-tuning. We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient. Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), an algorithm that selects adaptation directions using local curvature information. Our method is computationally efficient and avoids explicit second-order matrix construction. Experiments on standard natural language understanding benchmarks demonstrate improved performance and faster convergence compared to existing LoRA variants.

Figures & tables

Panel of Figure 1 — Figure 1: Left: the procedure to solve the alignment problem, as stated in Theorem 4.5 . Right: Relative distance between LoRA and FFT training logits ${\|\hat{f}_{FT,t}(X)-\hat{f}_{LoRA,t}(X)\|_{2}}/{\|\hat{f}_{FT,t}(X)\|_{2}}$ using RoBERTa-base on CoLA averaged over three runs. LoRA rank $r=8$ .

	GSM8K	MMLU	HumanEval
rsLoRA	$50.64_{\pm 0.16}$	$45.01_{\pm 0.45}$	$21.34_{\pm 0.17}$
LoRA-GA	$43.66_{\pm 0.78}$	$25.70_{\pm 1.34}$	$20.73_{\pm 0.21}$
LoRA-One	$52.16_{\pm 0.32}$	45.20 ±0.15	$20.12_{\pm 0.38}$
CG-LoRA (shift)	53.44 ±0.24	43.65 ±0.36	20.73 ±0.29
CG-LoRA (no shift)	56.10 ±0.21	45.50 ±0.22	23.17 ±0.18

Table 2: Test accuracy of finetuned LLaMa 2-7B. LoRA rank

r=8

. Query/Value LoRA layers.

LoRA training (memory)	CG-LoRA (memory)	CG-LoRA (init time)	LoRA-GA/One (memory)	LoRA-GA/One (init time)
23 GiB	17.7GiB	20s	18.8 GiB	79s

Table 3: Cost on LLaMa 2-7B. All-linear LoRA layers. Measured on a single NVIDIA A100.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

\begin{cases}A_{0}^{\top}\leftarrow\frac{1}{D_{K}(1,1)^{1/2}}R\operatorname{chol}(J_{R})^{-\top}V_{K}D_{K}^{1/2}\\ B_{0}\leftarrow\frac{1}{D_{K}(1,1)^{1/2}}L\operatorname{chol}(J_{L})^{-\top}U_{K}D_{K}^{1/2}\end{cases}

Appendix

Algorithm 3 Balanced SVD

Algorithm	Computational cost	Memory cost
Algorithms 1 and 4	Algos 2 + 3	Algos 2 + 3
Algorithm 2	$O(\max(d_{\mathrm{in}},d_{\mathrm{out}})((q+1)mn+m^{2})$	$O(\max(d_{\mathrm{in}},d_{\mathrm{out}})(m+n))$
Algorithm 3	$O(\max(d_{\mathrm{in}},d_{\mathrm{out}})r^{2})$	$O(\max(d_{\mathrm{in}},d_{\mathrm{out}})r)$

Appendix

Table 4: Number of floating point operations of CG-LoRA (per layer).

q,m

are the number of power iterations and oversampling factors respectively in Algorithm 2 .

d_{\mathrm{in}},d_{\mathrm{out}}

are the dimensions of pretrained parameter

W_{0}

r

is the LoRA rank. Big

O

notation hides constants associated to matrix multiplication, thin Householder QR, SVD decomposition.

Panel of Figure 2 — Figure 2: Performance of finetuned RoBERTa-base on CoLA. CG-LoRA (no shift) is represented.

Panel of Figure 3 — Figure 3: Performance of finetuned RoBERTa-base on MRPC.

Panel of Figure 4 — Figure 4: Performance of finetuned RoBERTa-base on SST2, QNLI.

Panel of Figure 5 — Figure 5: Training dynamics of finetuned LLaMa 2-7B on MetaMath.

Panel of Figure 6 — Figure 6: Training dynamics of finetuned LLaMa 2-7B on CodeFeedback.

Panel of Figure 7 — Figure 7: Training dynamics of finetuned LLaMa 2-7B on WizardLM.

	CoLA	MRPC
RoBERTa-base, Algo 4	80.76 ±0.45	85.04 ±1.70
RoBERTa-base, Algo 1	$80.66_{\pm 0.32}$	$82.92_{\pm 1.40}$
T5-base, Algo 4	80.24 ±0.61	85.94 ±0.50
T5-base, Algo 1	79.89 ±0.19	$84.88_{\pm 1.10}$

Appendix

Table 6: Test accuracy of finetuned CG-LoRA (no shift) with and without cross-entropy correction. LoRA rank

r=8

. Query/Value LoRA layers.

Figure 8: Accuracy versus number of Hutchinson probes

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
Jul 28, 2026Dianze Liu, Farshid GhezelbashLow-Rank Adaptation
Georgia Institute of Technology
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
Sep 29, 2026Yihao Ouyang, Shiwei Li, Haozhao Wang +5Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China

Jul 28, 2026J/K move · Enter open · S save

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Dianze Liu, Farshid Ghezelbash

Georgia Institute of Technology

Low-Rank Adaptation

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.

Abstract

Low-Rank Adaptation

Explore similar work

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition