Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models, but often lag behind full fine-tuning in both convergence speed and final performance. Recent approaches aim to reduce this gap by aligning LoRA parameter updates with those of full fine-tuning, but such parameter-space alignment only indirectly controls model predictions. Instead, we adopt a function-space perspective and formulate the prediction alignment problem, whose objective is to match the outputs of LoRA fine-tuning to those of full fine-tuning. We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient. Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), an algorithm that selects adaptation directions using local curvature information. Our method is computationally efficient and avoids explicit second-order matrix construction. Experiments on standard natural language understanding benchmarks demonstrate improved performance and faster convergence compared to existing LoRA variants.
Figures & tables
Figure 1: Left: the procedure to solve the alignment problem, as stated in Theorem 4.5 . Right: Relative distance between LoRA and FFT training logits ∥f^FT,t(X)−f^LoRA,t(X)∥2/∥f^FT,t(X)∥2 using RoBERTa-base on CoLA averaged over three runs. LoRA rank r=8 .
GSM8K
MMLU
HumanEval
rsLoRA
50.64±0.16
45.01±0.45
21.34±0.17
LoRA-GA
43.66±0.78
25.70±1.34
20.73±0.21
LoRA-One
52.16±0.32
45.20 ±0.15
20.12±0.38
CG-LoRA (shift)
53.44 ±0.24
43.65 ±0.36
20.73 ±0.29
CG-LoRA (no shift)
56.10 ±0.21
45.50 ±0.22
23.17 ±0.18
Table 2: Test accuracy of finetuned LLaMa 2-7B. LoRA rank r=8 . Query/Value LoRA layers.
LoRA training (memory)
CG-LoRA (memory)
CG-LoRA (init time)
LoRA-GA/One (memory)
LoRA-GA/One (init time)
23 GiB
17.7GiB
20s
18.8 GiB
79s
Table 3: Cost on LLaMa 2-7B. All-linear LoRA layers. Measured on a single NVIDIA A100.
Table 4: Number of floating point operations of CG-LoRA (per layer). q,m are the number of power iterations and oversampling factors respectively in Algorithm 2 . din,dout are the dimensions of pretrained parameter W0 . r is the LoRA rank. Big O notation hides constants associated to matrix multiplication, thin Householder QR, SVD decomposition.
Figure 2: Performance of finetuned RoBERTa-base on CoLA. CG-LoRA (no shift) is represented.
Figure 3: Performance of finetuned RoBERTa-base on MRPC.
Figure 4: Performance of finetuned RoBERTa-base on SST2, QNLI.
Figure 5: Training dynamics of finetuned LLaMa 2-7B on MetaMath.
Figure 6: Training dynamics of finetuned LLaMa 2-7B on CodeFeedback.
Figure 7: Training dynamics of finetuned LLaMa 2-7B on WizardLM.
CoLA
MRPC
RoBERTa-base, Algo 4
80.76 ±0.45
85.04 ±1.70
RoBERTa-base, Algo 1
80.66±0.32
82.92±1.40
T5-base, Algo 4
80.24 ±0.61
85.94 ±0.50
T5-base, Algo 1
79.89 ±0.19
84.88±1.10
Appendix
Table 6: Test accuracy of finetuned CG-LoRA (no shift) with and without cross-entropy correction. LoRA rank r=8 . Query/Value LoRA layers.
Figure 8: Accuracy versus number of Hutchinson probes
Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.
Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.
Valérie Castin, Kimia Nadjahi, Pierre Ablin +1
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.
Yihao Ouyang, Shiwei Li, Haozhao Wang +5
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China