Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning

Chenruo Liu; Zihan Wang; Yijun Dong; Qi Lei

cs.LGNov 5, 2026

Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning

Authors: Chenruo Liu, Zihan Wang, Yijun Dong, Qi Lei

Organizations: New York University · Institute of Science and Technology Austria

Abstract

Low-rank adaptation (LoRA) is commonly used in both supervised fine-tuning (SFT) and reinforcement learning (RL), often under similar optimizer settings. Yet, how the different dynamics of the two stages manifest in the low-rank increment that LoRA optimizes remains unclear. Tracking the singular spectrum and subspaces of LoRA updates on mathematical reasoning tasks, we find that RL concentrates the spectrum at every learning rate, while SFT expands it at moderate learning rates; moreover, RL rotates the singular subspaces much faster than SFT at small learning rates. Interpolating between SFT and RL one ingredient at a time, we pinpoint group-mean baseline subtraction as the source of both contrasts. A quadratic multi-index model updated via LoRA unveils the underlying mechanism: group-mean baseline subtraction tilts the balance between two competing forces, coherent fitting and gradient noise, toward spectral concentration and faster subspace rotation, while forcing the risk-favorable learning rate to be low. Guided by these dynamics, we tailor LoRA updates to each stage: in SFT, we flatten the spectrum of the full update to spread adaptation across singular directions, whereas in RL, we flatten the part of the update outside the current LoRA increment span to promote subspace rotation. Experiments on mathematical reasoning tasks across multiple models show that the respective configurations for SFT and RL yield the best performance among the compared methods in most evaluated settings.

Figures & tables

Figure 1: On GSM8K under

B_{0}=0

initialization, GRPO and RFT-B exhibit decreasing effective rank across all learning rates shown. In contrast, SFT and RFT exhibit an increasing trend from approximately step 25 at intermediate learning rates. Columns correspond to different learning rates; the top and bottom rows compare SFT/RFT and GRPO/RFT-B, respectively. Thin curves show individual models, with effective rank averaged over LoRA modules; thick curves show the mean across all models.

Figure 2: On GSM8K under

B_{0}=0

initialization, GRPO and RFT-B exhibit a faster decrease in left-subspace overlap than SFT and RFT at small learning rates, indicating faster subspace rotation. This difference narrows as the learning rate increases. The top and bottom rows show the right-subspace overlap

\mathrm{ovlp}_{R}(5,t)

and left-subspace overlap

\mathrm{ovlp}_{L}(5,t)

, respectively, between the current checkpoint at step

t

and the reference checkpoint at step

5

. Columns correspond to different learning rates; each panel compares SFT, RFT, RFT-B, and GRPO. Thin curves show individual models, with overlap averaged over LoRA modules; thick curves show the mean across all models.

SFT				RL
$A$ direction	$B$ direction	Llama-3.2-1B	Llama-3.2-3B	$A$ direction	$B$ direction	Llama-3.2-1B	Llama-3.2-3B
Adam	Adam	$22.24_{\pm 0.33}$	$42.99_{\pm 0.38}$	Adam	Adam	$7.61_{\pm 1.24}$	$42.36_{\pm 3.25}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$20.90_{\pm 0.76}$	$44.40_{\pm 0.93}$	Adam	$\mathcal{M}(\mathrm{Adam})$	$9.67_{\pm 1.22}$	$46.00_{\pm 0.77}$
Adam	$\mathcal{M}(\mathrm{SGD})$	$\mathbf{22.82}_{\pm 0.12}$	$\mathbf{45.72}_{\pm 0.61}$	Adam	$\mathcal{M}_{\perp}(\mathrm{Adam})$	$\mathbf{10.00}_{\pm 1.36}$	$\mathbf{47.66}_{\pm 1.85}$

Table 1: LoRA optimization on GSM8K with Llama-3.2-1B and Llama-3.2-3B. The dynamics-informed configurations in the last row achieve the highest mean accuracies for both SFT (left) and RL (right). Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction. For RL,

\mathcal{M}_{\perp}(\mathrm{Adam})

denotes

P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B})

with an Adam base direction

D_{t}^{B}

, which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. For each configuration, results are reported at the learning rate with the highest mean accuracy in a completed sweep. Bold indicates the highest reported mean in each column.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: On MATH under

B_{0}=0

Figure 4: On MATH under

B_{0}=0

\mathrm{ovlp}_{R}(5,t)

and left-subspace overlap

\mathrm{ovlp}_{L}(5,t)

, respectively, between the current checkpoint at step

t

and the reference checkpoint at step

5

Figure 5: On GSM8K under

A_{0}=0

Figure 6: On MATH under

A_{0}=0

Figure 7: On GSM8K under

A_{0}=0

initialization, GRPO and RFT-B exhibit a faster decrease in right-subspace overlap than SFT and RFT at small learning rates, indicating faster subspace rotation. This difference narrows as the learning rate increases. The top and bottom rows show the right-subspace overlap

\mathrm{ovlp}_{R}(5,t)

and left-subspace overlap

\mathrm{ovlp}_{L}(5,t)

, respectively, between the current checkpoint at step

t

and the reference checkpoint at step

5

Figure 8: On MATH under

A_{0}=0

\mathrm{ovlp}_{R}(5,t)

and left-subspace overlap

\mathrm{ovlp}_{L}(5,t)

, respectively, between the current checkpoint at step

t

and the reference checkpoint at step

5

$A$ direction	$B$ direction	Llama-3.2-1B	Llama-3.2-3B
Adam	Adam	$5.66_{\pm 0.09}$	$10.83_{\pm 0.13}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$5.93_{\pm 0.14}$	$10.53_{\pm 0.14}$
Adam	$\mathcal{M}(\mathrm{SGD})$	$\mathbf{6.01}_{\pm 0.25}$	$\mathbf{11.32}_{\pm 0.22}$

Appendix

Table 2: SFT optimization on MATH with Llama-3.2-1B and Llama-3.2-3B. The dynamics-informed configuration in the last row achieves the highest mean accuracies. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. For each configuration, results are reported at the learning rate with the highest mean accuracy in a completed sweep. Bold indicates the highest reported mean in each model column.

$A$ direction	$B$ direction	Llama-3.2-1B	Llama-3.2-1B-Instruct	Llama-3.2-3B	Qwen2.5-3B
Adam	Adam	$7.61_{\pm 1.24}$	$57.85_{\pm 0.39}$	$42.36_{\pm 3.25}$	$83.93_{\pm 0.34}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$9.67_{\pm 1.22}$	$57.97_{\pm 0.18}$	$46.00_{\pm 0.77}$	$84.46_{\pm 0.38}$
Adam	$\mathcal{M}_{\perp}(\mathrm{Adam})$	$\mathbf{10.00}_{\pm 1.36}$	$\mathbf{58.23}_{\pm 0.72}$	$\mathbf{47.66}_{\pm 1.85}$	$\mathbf{84.51}_{\pm 0.62}$

Appendix

Table 3: RL optimization on GSM8K with Llama-3.2-1B, Llama-3.2-1B-Instruct, Llama-3.2-3B, and Qwen2.5-3B. The dynamics-informed configuration in the last row achieves the highest reported mean accuracy in each model column. Methods are specified by the update directions for the two LoRA factors: Adam denotes the corresponding base optimizer direction (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction.

\mathcal{M}_{\perp}(\mathrm{Adam})

denotes

P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B})

with an Adam base direction

D_{t}^{B}

$A$ direction	$B$ direction	Llama-3.2-1B	Llama-3.2-1B-Instruct	Llama-3.2-3B	Qwen2.5-3B
Adam	Adam	$5.04_{\pm 0.28}$	$33.44_{\pm 0.52}$	$\mathbf{9.41}_{\pm 0.87}$	$63.03_{\pm 0.23}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$5.43_{\pm 0.28}$	$\mathbf{34.16}_{\pm 0.61}$	$9.21_{\pm 0.44}$	$63.37_{\pm 0.21}$
Adam	$\mathcal{M}_{\perp}(\mathrm{Adam})$	$\mathbf{5.65}_{\pm 0.27}$	$33.71_{\pm 0.17}$	$8.21_{\pm 0.42}$	$\mathbf{63.54}_{\pm 0.21}$

Appendix

Table 4: RL optimization on MATH with Llama-3.2-1B, Llama-3.2-1B-Instruct, Llama-3.2-3B, and Qwen2.5-3B. The dynamics-informed configuration in the last row achieves the highest reported mean accuracy on two of the four models. Methods are specified by the update directions for the two LoRA factors: Adam denotes the corresponding base optimizer direction (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction.

\mathcal{M}_{\perp}(\mathrm{Adam})

denotes

P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B})

with an Adam base direction

D_{t}^{B}

			GSM8K		MATH
$A$ direction	$B$ direction	LR	Llama-3.2-1B	Llama-3.2-3B	Llama-3.2-1B	Llama-3.2-3B
Adam	Adam	$5\times 10^{-6}$	$10.87_{\pm 1.20}$	$32.35_{\pm 0.37}$	$5.06_{\pm 0.02}$	$10.19_{\pm 0.21}$
		$5\times 10^{-5}$	$20.60_{\pm 0.61}$	$\mathbf{42.99}_{\pm 0.38}$	$5.46_{\pm 0.12}$	$10.57_{\pm 0.21}$
		$5\times 10^{-4}$	$\mathbf{22.24}_{\pm 0.33}$	$42.68_{\pm 0.92}$	$\mathbf{5.66}_{\pm 0.09}$	$\mathbf{10.83}_{\pm 0.13}$
		$5\times 10^{-3}$	$2.65_{\pm 0.08}$	$0.58_{\pm 0.58}$	$3.43_{\pm 0.52}$	$2.83_{\pm 1.41}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$5\times 10^{-6}$	$11.22_{\pm 0.20}$	$30.65_{\pm 0.62}$	$4.96_{\pm 0.14}$	$10.44_{\pm 0.16}$

Appendix

Table 5: Detailed learning-rate sweeps for SFT on GSM8K and MATH. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. Bold indicates the highest mean accuracy across learning rates for each update configuration and model–dataset pair.

$A$ direction	$B$ direction	LR	Llama-3.2-1B	Llama-3.2-1B-Instruct	Llama-3.2-3B	Qwen2.5-3B
Adam	Adam	$5\times 10^{-6}$	$2.65_{\pm 0.15}$	$\mathbf{57.85}_{\pm 0.39}$	$27.02_{\pm 3.31}$	$82.82_{\pm 1.11}$
		$5\times 10^{-5}$	$\mathbf{7.61}_{\pm 1.24}$	$57.24_{\pm 0.29}$	$\mathbf{42.36}_{\pm 3.25}$	$\mathbf{83.93}_{\pm 0.34}$
		$5\times 10^{-4}$	$1.34_{\pm 0.77}$	$2.75_{\pm 2.21}$	$0.15_{\pm 0.15}$	$0.96_{\pm 0.68}$
		$5\times 10^{-3}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$5\times 10^{-6}$	$2.91_{\pm 0.21}$	$\mathbf{57.97}_{\pm 0.18}$	$14.30_{\pm 1.12}$	$82.61_{\pm 0.48}$
		$5\times 10^{-5}$	$\mathbf{9.67}_{\pm 1.22}$	$57.24_{\pm 0.59}$	$\mathbf{46.00}_{\pm 0.77}$	$\mathbf{84.46}_{\pm 0.38}$

Appendix

Table 6: Detailed learning-rate sweeps for RL on GSM8K. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction. For RL,

\mathcal{M}_{\perp}(\mathrm{Adam})

denotes

P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B})

with an Adam base direction

D_{t}^{B}

, which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. Bold indicates the highest mean accuracy across learning rates for each update configuration and model.

$A$ direction	$B$ direction	LR	Llama-3.2-1B	Llama-3.2-1B-Instruct	Llama-3.2-3B	Qwen2.5-3B
Adam	Adam	$5\times 10^{-6}$	$4.79_{\pm 0.02}$	$\mathbf{33.44}_{\pm 0.52}$	$7.28_{\pm 0.19}$	$62.78_{\pm 0.40}$
		$5\times 10^{-5}$	$\mathbf{5.04}_{\pm 0.28}$	$32.55_{\pm 0.27}$	$\mathbf{9.41}_{\pm 0.87}$	$\mathbf{63.03}_{\pm 0.23}$
		$5\times 10^{-4}$	$0.00_{\pm 0.00}$	$1.17_{\pm 1.17}$	$0.01_{\pm 0.01}$	$0.59_{\pm 0.59}$
		$5\times 10^{-3}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$
Adam	$\mathcal{M}(\mathrm{Adam})$	$5\times 10^{-6}$	$4.89_{\pm 0.01}$	$32.91_{\pm 0.21}$	$7.01_{\pm 0.07}$	$61.80_{\pm 0.64}$
		$5\times 10^{-5}$	$\mathbf{5.43}_{\pm 0.28}$	$\mathbf{34.16}_{\pm 0.61}$	$\mathbf{9.21}_{\pm 0.44}$	$\mathbf{63.37}_{\pm 0.21}$

Appendix

Table 7: Detailed learning-rate sweeps for RL on MATH. Methods are specified by the update directions for the two LoRA factors: Adam and SGD denote the corresponding base optimizer directions (

D_{t}^{A}

D_{t}^{B}

), and

\mathcal{M}

denotes the norm-preserving spectral flattening applied to the full direction. For RL,

\mathcal{M}_{\perp}(\mathrm{Adam})

denotes

P_{t}D_{t}^{B}+\mathcal{M}((I-P_{t})D_{t}^{B})

with an Adam base direction

D_{t}^{B}

, which preserves the within-subspace component while transforming only the orthogonal component. Values show mean test accuracy (%) across three seeds, with standard errors in subscripts. Bold indicates the highest mean accuracy across learning rates for each update configuration and model.

Panel of Figure 9 — Figure 9: Proof roadmap. The top row supplies the common response, moment, spectral, and static-event estimates. Sections C.6 – C.8 prove Propositions A–C; the rotation proof also reuses the trajectory estimates of Section C.6 . Combining each first-order comparison with Proposition T proves the corresponding theorem in this subsection.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 562 papers

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
Jun 27, 2026Yupeng Chang, Yuan Wu, Yi ChangLow-Rank Adaptation LLM Fine-Tuning
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
Mar 4, 2026Zailong Tian, Yanzhe Chen, Zhuoheng Han +1Low-Rank Adaptation Adapter Tuning
School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Jun 27, 2026J/K move · Enter open · S save

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Yupeng Chang, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Low-Rank Adaptation LLM Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.

Abstract

Low-Rank Adaptation

Explore similar work

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence