cs.LGNov 5, 2026

Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning

Authors: Chenruo Liu, Zihan Wang, Yijun Dong, Qi Lei

Organizations: New York University · Institute of Science and Technology Austria

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Low-rank adaptation (LoRA) is commonly used in both supervised fine-tuning (SFT) and reinforcement learning (RL), often under similar optimizer settings. Yet, how the different dynamics of the two stages manifest in the low-rank increment that LoRA optimizes remains unclear. Tracking the singular spectrum and subspaces of LoRA updates on mathematical reasoning tasks, we find that RL concentrates the spectrum at every learning rate, while SFT expands it at moderate learning rates; moreover, RL rotates the singular subspaces much faster than SFT at small learning rates. Interpolating between SFT and RL one ingredient at a time, we pinpoint group-mean baseline subtraction as the source of both contrasts. A quadratic multi-index model updated via LoRA unveils the underlying mechanism: group-mean baseline subtraction tilts the balance between two competing forces, coherent fitting and gradient noise, toward spectral concentration and faster subspace rotation, while forcing the risk-favorable learning rate to be low. Guided by these dynamics, we tailor LoRA updates to each stage: in SFT, we flatten the spectrum of the full update to spread adaptation across singular directions, whereas in RL, we flatten the part of the update outside the current LoRA increment span to promote subspace rotation. Experiments on mathematical reasoning tasks across multiple models show that the respective configurations for SFT and RL yield the best performance among the compared methods in most evaluated settings.

Figures & tables

Figure 1: On GSM8K under B0=0B_{0}=0 initialization, GRPO and RFT-B exhibit decreasing effective rank across all learning rates shown. In contrast, SFT and RFT exhibit an increasing trend from approximately step 25 at intermediate learning rates. Columns correspond to different learning rates; the top and bottom rows compare SFT/RFT and GRPO/RFT-B, respectively. Thin curves show individual models, with effective rank averaged over LoRA modules; thick curves show the mean across all models.
Figure 2: On GSM8K under B0=0B_{0}=0 initialization, GRPO and RFT-B exhibit a faster decrease in left-subspace overlap than SFT and RFT at small learning rates, indicating faster subspace rotation. This difference narrows as the learning rate increases. The top and bottom rows show the right-subspace overlap ovlpR(5,t)\mathrm{ovlp}_{R}(5,t) and left-subspace overlap ovlpL(5,t)\mathrm{ovlp}_{L}(5,t) , respectively, between the current checkpoint at step tt and the reference checkpoint at step 55 . Columns correspond to different learning rates; each panel compares SFT, RFT, RFT-B, and GRPO. Thin curves show individual models, with overlap averaged over LoRA modules; thick curves show the mean across all models.
SFTRL
AA directionBB directionLlama-3.2-1BLlama-3.2-3BAA directionBB directionLlama-3.2-1BLlama-3.2-3B
AdamAdam22.24±0.3322.24_{\pm 0.33}42.99±0.3842.99_{\pm 0.38}AdamAdam7.61±1.247.61_{\pm 1.24}42.36±3.2542.36_{\pm 3.25}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})20.90±0.7620.90_{\pm 0.76}44.40±0.9344.40_{\pm 0.93}AdamM(Adam)\mathcal{M}(\mathrm{Adam})9.67±1.229.67_{\pm 1.22}46.00±0.7746.00_{\pm 0.77}
AdamM(SGD)\mathcal{M}(\mathrm{SGD})22.82±0.12\mathbf{22.82}_{\pm 0.12}45.72±0.61\mathbf{45.72}_{\pm 0.61}AdamM⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam})10.00±1.36\mathbf{10.00}_{\pm 1.36}47.66±1.85\mathbf{47.66}_{\pm 1.85}
Table 1: LoRA optimization on GSM8K with Llama-3.2-1B and Llama-3.2-3B. The dynamics-informed configurations in the last row achieve the highest mean accuracies for both SFT (left) and RL (right). Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. For RL, M⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam}) denotes PtDtB+M((I−Pt)DtB)P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B}) with an Adam base direction DtBD_{t}^{B} , which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. For each configuration, results are reported at the learning rate with the highest mean accuracy in a completed sweep. Bold indicates the highest reported mean in each column.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: On MATH under B0=0B_{0}=0 initialization, GRPO and RFT-B exhibit decreasing effective rank across all learning rates shown. In contrast, SFT and RFT exhibit an increasing trend from approximately step 25 at intermediate learning rates. Columns correspond to different learning rates; the top and bottom rows compare SFT/RFT and GRPO/RFT-B, respectively. Thin curves show individual models, with effective rank averaged over LoRA modules; thick curves show the mean across all models.
Figure 4: On MATH under B0=0B_{0}=0 initialization, GRPO and RFT-B exhibit a faster decrease in left-subspace overlap than SFT and RFT at small learning rates, indicating faster subspace rotation. This difference narrows as the learning rate increases. The top and bottom rows show the right-subspace overlap ovlpR(5,t)\mathrm{ovlp}_{R}(5,t) and left-subspace overlap ovlpL(5,t)\mathrm{ovlp}_{L}(5,t) , respectively, between the current checkpoint at step tt and the reference checkpoint at step 55 . Columns correspond to different learning rates; each panel compares SFT, RFT, RFT-B, and GRPO. Thin curves show individual models, with overlap averaged over LoRA modules; thick curves show the mean across all models.
Figure 5: On GSM8K under A0=0A_{0}=0 initialization, GRPO and RFT-B exhibit decreasing effective rank across all learning rates shown. In contrast, SFT and RFT exhibit an increasing trend from approximately step 25 at intermediate learning rates. Columns correspond to different learning rates; the top and bottom rows compare SFT/RFT and GRPO/RFT-B, respectively. Thin curves show individual models, with effective rank averaged over LoRA modules; thick curves show the mean across all models.
Figure 6: On MATH under A0=0A_{0}=0 initialization, GRPO and RFT-B exhibit decreasing effective rank across all learning rates shown. In contrast, SFT and RFT exhibit an increasing trend from approximately step 25 at intermediate learning rates. Columns correspond to different learning rates; the top and bottom rows compare SFT/RFT and GRPO/RFT-B, respectively. Thin curves show individual models, with effective rank averaged over LoRA modules; thick curves show the mean across all models.
Figure 7: On GSM8K under A0=0A_{0}=0 initialization, GRPO and RFT-B exhibit a faster decrease in right-subspace overlap than SFT and RFT at small learning rates, indicating faster subspace rotation. This difference narrows as the learning rate increases. The top and bottom rows show the right-subspace overlap ovlpR(5,t)\mathrm{ovlp}_{R}(5,t) and left-subspace overlap ovlpL(5,t)\mathrm{ovlp}_{L}(5,t) , respectively, between the current checkpoint at step tt and the reference checkpoint at step 55 . Columns correspond to different learning rates; each panel compares SFT, RFT, RFT-B, and GRPO. Thin curves show individual models, with overlap averaged over LoRA modules; thick curves show the mean across all models.
Figure 8: On MATH under A0=0A_{0}=0 initialization, GRPO and RFT-B exhibit a faster decrease in right-subspace overlap than SFT and RFT at small learning rates, indicating faster subspace rotation. This difference narrows as the learning rate increases. The top and bottom rows show the right-subspace overlap ovlpR(5,t)\mathrm{ovlp}_{R}(5,t) and left-subspace overlap ovlpL(5,t)\mathrm{ovlp}_{L}(5,t) , respectively, between the current checkpoint at step tt and the reference checkpoint at step 55 . Columns correspond to different learning rates; each panel compares SFT, RFT, RFT-B, and GRPO. Thin curves show individual models, with overlap averaged over LoRA modules; thick curves show the mean across all models.
AA directionBB directionLlama-3.2-1BLlama-3.2-3B
AdamAdam5.66±0.095.66_{\pm 0.09}10.83±0.1310.83_{\pm 0.13}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})5.93±0.145.93_{\pm 0.14}10.53±0.1410.53_{\pm 0.14}
AdamM(SGD)\mathcal{M}(\mathrm{SGD})6.01±0.25\mathbf{6.01}_{\pm 0.25}11.32±0.22\mathbf{11.32}_{\pm 0.22}
Appendix
Table 2: SFT optimization on MATH with Llama-3.2-1B and Llama-3.2-3B. The dynamics-informed configuration in the last row achieves the highest mean accuracies. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. For each configuration, results are reported at the learning rate with the highest mean accuracy in a completed sweep. Bold indicates the highest reported mean in each model column.
AA directionBB directionLlama-3.2-1BLlama-3.2-1B-InstructLlama-3.2-3BQwen2.5-3B
AdamAdam7.61±1.247.61_{\pm 1.24}57.85±0.3957.85_{\pm 0.39}42.36±3.2542.36_{\pm 3.25}83.93±0.3483.93_{\pm 0.34}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})9.67±1.229.67_{\pm 1.22}57.97±0.1857.97_{\pm 0.18}46.00±0.7746.00_{\pm 0.77}84.46±0.3884.46_{\pm 0.38}
AdamM⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam})10.00±1.36\mathbf{10.00}_{\pm 1.36}58.23±0.72\mathbf{58.23}_{\pm 0.72}47.66±1.85\mathbf{47.66}_{\pm 1.85}84.51±0.62\mathbf{84.51}_{\pm 0.62}
Appendix
Table 3: RL optimization on GSM8K with Llama-3.2-1B, Llama-3.2-1B-Instruct, Llama-3.2-3B, and Qwen2.5-3B. The dynamics-informed configuration in the last row achieves the highest reported mean accuracy in each model column. Methods are specified by the update directions for the two LoRA factors: Adam denotes the corresponding base optimizer direction ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. M⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam}) denotes PtDtB+M((I−Pt)DtB)P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B}) with an Adam base direction DtBD_{t}^{B} , which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. For each configuration, results are reported at the learning rate with the highest mean accuracy in a completed sweep. Bold indicates the highest reported mean in each model column.
AA directionBB directionLlama-3.2-1BLlama-3.2-1B-InstructLlama-3.2-3BQwen2.5-3B
AdamAdam5.04±0.285.04_{\pm 0.28}33.44±0.5233.44_{\pm 0.52}9.41±0.87\mathbf{9.41}_{\pm 0.87}63.03±0.2363.03_{\pm 0.23}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})5.43±0.285.43_{\pm 0.28}34.16±0.61\mathbf{34.16}_{\pm 0.61}9.21±0.449.21_{\pm 0.44}63.37±0.2163.37_{\pm 0.21}
AdamM⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam})5.65±0.27\mathbf{5.65}_{\pm 0.27}33.71±0.1733.71_{\pm 0.17}8.21±0.428.21_{\pm 0.42}63.54±0.21\mathbf{63.54}_{\pm 0.21}
Appendix
Table 4: RL optimization on MATH with Llama-3.2-1B, Llama-3.2-1B-Instruct, Llama-3.2-3B, and Qwen2.5-3B. The dynamics-informed configuration in the last row achieves the highest reported mean accuracy on two of the four models. Methods are specified by the update directions for the two LoRA factors: Adam denotes the corresponding base optimizer direction ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. M⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam}) denotes PtDtB+M((I−Pt)DtB)P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B}) with an Adam base direction DtBD_{t}^{B} , which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. For each configuration, results are reported at the learning rate with the highest mean accuracy in a completed sweep. Bold indicates the highest reported mean in each model column.
   GSM8KMATH
AA directionBB directionLRLlama-3.2-1BLlama-3.2-3BLlama-3.2-1BLlama-3.2-3B
AdamAdam5×10−65\times 10^{-6}10.87±1.2010.87_{\pm 1.20}32.35±0.3732.35_{\pm 0.37}5.06±0.025.06_{\pm 0.02}10.19±0.2110.19_{\pm 0.21}
  5×10−55\times 10^{-5}20.60±0.6120.60_{\pm 0.61}42.99±0.38\mathbf{42.99}_{\pm 0.38}5.46±0.125.46_{\pm 0.12}10.57±0.2110.57_{\pm 0.21}
  5×10−45\times 10^{-4}22.24±0.33\mathbf{22.24}_{\pm 0.33}42.68±0.9242.68_{\pm 0.92}5.66±0.09\mathbf{5.66}_{\pm 0.09}10.83±0.13\mathbf{10.83}_{\pm 0.13}
  5×10−35\times 10^{-3}2.65±0.082.65_{\pm 0.08}0.58±0.580.58_{\pm 0.58}3.43±0.523.43_{\pm 0.52}2.83±1.412.83_{\pm 1.41}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})5×10−65\times 10^{-6}11.22±0.2011.22_{\pm 0.20}30.65±0.6230.65_{\pm 0.62}4.96±0.144.96_{\pm 0.14}10.44±0.1610.44_{\pm 0.16}
Appendix
Table 5: Detailed learning-rate sweeps for SFT on GSM8K and MATH. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. Bold indicates the highest mean accuracy across learning rates for each update configuration and model–dataset pair.
AA directionBB directionLRLlama-3.2-1BLlama-3.2-1B-InstructLlama-3.2-3BQwen2.5-3B
AdamAdam5×10−65\times 10^{-6}2.65±0.152.65_{\pm 0.15}57.85±0.39\mathbf{57.85}_{\pm 0.39}27.02±3.3127.02_{\pm 3.31}82.82±1.1182.82_{\pm 1.11}
  5×10−55\times 10^{-5}7.61±1.24\mathbf{7.61}_{\pm 1.24}57.24±0.2957.24_{\pm 0.29}42.36±3.25\mathbf{42.36}_{\pm 3.25}83.93±0.34\mathbf{83.93}_{\pm 0.34}
  5×10−45\times 10^{-4}1.34±0.771.34_{\pm 0.77}2.75±2.212.75_{\pm 2.21}0.15±0.150.15_{\pm 0.15}0.96±0.680.96_{\pm 0.68}
  5×10−35\times 10^{-3}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})5×10−65\times 10^{-6}2.91±0.212.91_{\pm 0.21}57.97±0.18\mathbf{57.97}_{\pm 0.18}14.30±1.1214.30_{\pm 1.12}82.61±0.4882.61_{\pm 0.48}
  5×10−55\times 10^{-5}9.67±1.22\mathbf{9.67}_{\pm 1.22}57.24±0.5957.24_{\pm 0.59}46.00±0.77\mathbf{46.00}_{\pm 0.77}84.46±0.38\mathbf{84.46}_{\pm 0.38}
Appendix
Table 6: Detailed learning-rate sweeps for RL on GSM8K. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. For RL, M⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam}) denotes PtDtB+M((I−Pt)DtB)P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B}) with an Adam base direction DtBD_{t}^{B} , which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. Bold indicates the highest mean accuracy across learning rates for each update configuration and model.
AA directionBB directionLRLlama-3.2-1BLlama-3.2-1B-InstructLlama-3.2-3BQwen2.5-3B
AdamAdam5×10−65\times 10^{-6}4.79±0.024.79_{\pm 0.02}33.44±0.52\mathbf{33.44}_{\pm 0.52}7.28±0.197.28_{\pm 0.19}62.78±0.4062.78_{\pm 0.40}
  5×10−55\times 10^{-5}5.04±0.28\mathbf{5.04}_{\pm 0.28}32.55±0.2732.55_{\pm 0.27}9.41±0.87\mathbf{9.41}_{\pm 0.87}63.03±0.23\mathbf{63.03}_{\pm 0.23}
  5×10−45\times 10^{-4}0.00±0.000.00_{\pm 0.00}1.17±1.171.17_{\pm 1.17}0.01±0.010.01_{\pm 0.01}0.59±0.590.59_{\pm 0.59}
  5×10−35\times 10^{-3}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}
AdamM(Adam)\mathcal{M}(\mathrm{Adam})5×10−65\times 10^{-6}4.89±0.014.89_{\pm 0.01}32.91±0.2132.91_{\pm 0.21}7.01±0.077.01_{\pm 0.07}61.80±0.6461.80_{\pm 0.64}
  5×10−55\times 10^{-5}5.43±0.28\mathbf{5.43}_{\pm 0.28}34.16±0.61\mathbf{34.16}_{\pm 0.61}9.21±0.44\mathbf{9.21}_{\pm 0.44}63.37±0.21\mathbf{63.37}_{\pm 0.21}
Appendix
Table 7: Detailed learning-rate sweeps for RL on MATH. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions ( DtAD_{t}^{A} or DtBD_{t}^{B} ), and M\mathcal{M} denotes the norm-preserving spectral flattening applied to the full direction. For RL, M⊥(Adam)\mathcal{M}_{\perp}(\mathrm{Adam}) denotes PtDtB+M((I−Pt)DtB)P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B}) with an Adam base direction DtBD_{t}^{B} , which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. Bold indicates the highest mean accuracy across learning rates for each update configuration and model.
Figure 9: Proof roadmap. The top row supplies the common response, moment, spectral, and static-event estimates. Sections C.6 – C.8 prove Propositions A–C; the rotation proof also reuses the trajectory estimates of Section C.6 . Combining each first-order comparison with Proposition T proves the corresponding theorem in this subsection.
More in this topic

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 562 papers

Explore similar work

CardsList
Jun 27, 2026

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.

Yupeng Chang, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Low-Rank AdaptationLLM Fine-TuningParameter-Efficient Fine-Tuning
AskPDF
Mar 4, 2026

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only ≈1,000\approx 1{,}000 scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.

Zailong Tian, Yanzhe Chen, Zhuoheng Han +1

School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University

Low-Rank AdaptationAdapter TuningParameter-Efficient Fine-Tuning
AskPDF
May 29, 2026

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Valérie Castin, Kimia Nadjahi, Pierre Ablin +1

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

LLM Fine-TuningLow-Rank AdaptationParameter-Efficient Fine-Tuning
AskPDF