Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA

Adnan Slimane Ali; Ayoub Belfatmi; David Ngwe Pouth

cs.LGOct 5, 2026

Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA

Authors: Adnan Slimane Ali, Ayoub Belfatmi, David Ngwe Pouth

Organizations: CentraleSupélec, Université Paris-Saclay · École polytechnique, Institut Polytechnique de Paris

Abstract

Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B also trails comparable-budget free LoRA by 8-18 percentage points on five pairs spanning a formatting task and OpenBookQA. The A-frozen advantage persists in a single-GPU-model replication and within individual MLP module groups, including controls with equal or greater trainable counts for B frozen, and when A is frozen on a random orthonormal basis. The factor contrast weakens with rank. On OpenBookQA / Qwen2.5-1.5B at rank 16, PEFT's MiCA implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. A trained oracle output subspace largely removes the low-rank deficit; partial warm-up gains recur across three direction seeds. The factor-versus-band ordering is descriptive; an approximate multiplicity audit weakens several earlier significance claims. These results extend known factor asymmetry by showing how its magnitude depends on spectral placement, rank and training conditions.

Figures & tables

Figure 1: At rank 2, changing the frozen factor has a larger effect than switching spectral extremes. Points are mean test scores over five seeds at separately selected learning rates. The free-LoRA reference has rank 1 and a comparable, slightly larger trainable budget than rank-2 B-frozen LoRA. A-frozen LoRA has more trainable parameters at equal rank; Table 3 controls this advantage. The panels use different vertical scales. Lines connect configurations, not training trajectories. Full scores and uncertainty are in Tables 7 and 2 .

Configuration	Initialization and constraint	Trainable parameters/module
B frozen, band $S$	$B=U_{:,S}$ fixed; train $A$ from zero	$rd_{\mathrm{in}}$
A frozen, band $S$	$A=V_{:,S}^{\top}$ fixed; train $B$ from zero	$rd_{\mathrm{out}}$
Both trainable, band $S$	Initialize $B=U_{:,S}$ and $A=0$ ; train both	$k(d_{\mathrm{in}}+d_{\mathrm{out}})$
Free LoRA	Kaiming-initialized $A$ , zero $B$ ; train both	$k(d_{\mathrm{in}}+d_{\mathrm{out}})$

Table 1: Controlled configurations.

S

is the top or bottom band. Frozen configurations use rank

r

; the two trainable configurations use either

k=r

or a smaller rank giving a comparable parameter count. The spectral initialization of the both-trainable control does not subtract a component from

W_{0}

, and is therefore not a reproduction of PiSSA or MiLoRA.

Task / model	Factor $F$	Band $S$	Interaction
Format / Llama-3.2-1B	$15.96$	$-2.21$	$5.99$
	$[15.13,16.86]$	$[-3.24,-1.22]$	$[3.36,8.60]$
OpenBookQA / Qwen2.5-1.5B	$8.42$	$-2.46$	$3.72$
	$[7.20,9.64]$	$[-3.60,-1.60]$	$[2.24,4.80]$
OpenBookQA / Qwen2.5-7B	$11.78$	$-3.46$	$2.60$
	$[11.34,12.38]$	$[-3.78,-3.14]$	$[1.48,3.64]$

Table 2: Factor and band contrasts at rank 2, in percentage points.

F

and

S

are defined in Eqs. ( 2 )–( 3 ). The interaction is

(a_{\mathrm{bottom}}-a_{\mathrm{top}})-(b_{\mathrm{bottom}}-b_{\mathrm{top}})

. Brackets are post hoc paired-bootstrap 95% intervals over five seeds, conditional on the test set and selected rates. They are not simultaneous confidence intervals, and no confirmatory test of

F>|S|

is claimed. All four A/B configurations use the same rank; their parameter counts differ.

Model	Band	Extra count	A( $r=2$ ) $-$ B( $r=3$ )	$t$
Qwen2.5-1.5B	Top	$18.3\%$	$6.28\ [4.48,8.08]$	$9.71$
	Bottom	$18.3\%$	$9.20\ [6.68,11.72]$	$10.14$
Qwen2.5-7B	Top	$22.2\%$	$9.00\ [8.26,9.74]$	$33.54$
	Bottom	$22.2\%$	$12.24\ [10.74,13.74]$	$22.59$

Table 3: Giving B frozen more parameters does not close the gap. OpenBookQA, seeds 2–6, separately selected learning rates. Differences are in percentage points with nominal paired-

t

95% intervals. The prespecified decision threshold is

t>3.495

with four degrees of freedom, corresponding to a two-sided Bonferroni correction for two bands at family level 0.05. Exact counts and scores are in Appendix C .

Task / model	Adapted modules	A rank / B rank	Top A $-$ B	Bottom A $-$ B
Format / Llama-1B	Down	2 / 2	$36.54\ [32.34,40.75]$	$55.46\ [52.19,58.72]$
	Gate + up	2 / 8	$12.56\ [10.59,14.53]$	$10.04\ [7.33,12.74]$
OBQA / Qwen-1.5B	Down	2 / 2	$3.30\ [1.76,4.84]$	$8.88\ [7.03,10.72]$
	Gate + up	2 / 12	$4.48\ [2.19,6.76]$	$6.00\ [4.30,7.70]$

Table 4: Within-module comparisons favor A frozen. Differences in percentage points, seeds 2–9. Each interval has level

1-0.05/8=99.375\%

, giving 95% family coverage for the eight prespecified paired-

t

comparisons (

t_{1-0.05/16,7}=3.855

). B frozen has at least as many trainable parameters in every row. Appendix E.2 gives counts, scores and selection exceptions, including an A-frozen boundary optimum.

Figure 2: Factor and band contrasts become comparable at higher rank. Left: the rank-2 factor contrast exceeds the absolute band contrast on each complete task–model pair. Right: the factor contrast decreases along the OpenBookQA / Qwen2.5-1.5B sweep; the two estimates are close at rank 16. Values are descriptive contrasts of mean scores, without a test for their difference or for a crossover rank. The right panel does not imply a universal rank threshold. Tables 2 and 5 supply the values.

Frozen rank	Top	Bottom	Top	Bottom	$F$	$\|S\|$
	A frozen minus B frozen		Free LoRA minus B frozen
2	$6.56$	$10.28$	$8.04$	$12.36$	$8.42$	$2.46$
4	$7.00$	$8.64$	$8.80$	$10.00$	$7.82$	$0.38$
8	$6.36$	$6.88$	$5.52$	$8.88$	$6.62$	$3.10$
16	$1.36$	$3.80$	$2.08$	$6.32$	$2.58$	$3.02$

Table 5: Rank dependence on OpenBookQA / Qwen2.5-1.5B. Differences in percentage points, seeds 2–6. The free reference uses ranks 1, 2, 4 and 7, respectively; its trainable count is 13.4% above B frozen at ranks 2–8 and 0.8% below at rank 16.

F

and

S

summarize the four frozen configurations. The top-band same-band contrast and reference gap rise slightly at rank 4, so strict monotonicity does not hold for every quantity. Scores and available direct-contrast intervals appear in Appendix B .

Configuration	Training recipe	Accuracy (%)	Supervised tokens	Steps
Bottom B frozen, $r=16$	Main	49.84	602,112	49
Free LoRA, $r=7$	Main	56.12	602,112	49
PEFT MiCA, $r=16$	Transferred	52.40	417,792	272
PEFT free LoRA, $r=7$	Transferred	55.48	417,792	272

Table 6: MiCA and comparable-budget LoRA on OpenBookQA / Qwen2.5-1.5B. Means over seeds 2–6, all on RTX 4000 Ada. The free-LoRA-minus-MiCA contrasts are

3.72\ [1.58,5.86]

points across recipes (

t=4.83

) and

3.08\ [1.34,4.82]

under the shared recipe (

t=4.90

). Intervals are nominal 95%, conditional on the fixed test set. The final row was added after the first comparison and reuses its MiCA runs.

Figure 3: Training-derived output directions are more useful in these diagnostics. Gains relative to top-band B frozen, at rank 2, with nominal paired-

t

95% intervals over five retraining seeds. Dashed lines mark the earlier free-LoRA gaps used as fixed campaign reference values: 8.04 and 12.41 points. Teacher-subspace uncertainty is not included. On the formatting task, the 64-sequence gradient and oracle use a retrained top-band reference, whereas the full-gradient and warm-up comparisons use the original reference; the two reference scores differ by 0.13 points. Full details are in Appendix F .

Figure 4: Warm-up gains recur across three direction seeds. Each point averages five retraining seeds for one learned output basis. Recovery is the gain over the original top-band B-frozen reference divided by its fixed free-LoRA gap (8.04 points on OpenBookQA; 12.41 on Format). Lines connect three tested cuts; they are not a fitted learning curve. The pattern is descriptive, and values above one can arise because the reference uses rank 1 while warm-up uses rank 2. Corrected intervals for the new 10% comparisons appear in Table 20 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	B frozen, $r=2$		A frozen, $r=2$		Free	Base
Task / model	Top	Bottom	Top	Bottom	$r=1$
Format / Llama-3.2-1B	64.43	59.22	77.39	78.18	76.84	0.00
Format / Qwen2.5-1.5B	62.57	63.91	—	—	78.79	0.00
OpenBookQA / Qwen2.5-1.5B	46.52	42.20	53.08	52.48	54.56	42.20
OpenBookQA / Qwen2.5-7B	53.16	48.40	63.64	61.48	62.84	47.40
OpenBookQA / Mistral-7B	47.64	47.76	60.76	58.60	60.12	44.60

Appendix

Table 7: Rank-2 mean test scores (%). Reporting seeds are 2–6, except 22–26 for Format / Qwen2.5-1.5B. The metric is valid-format accuracy or multiple-choice accuracy according to task. Base models are evaluated without adaptation. Table 2 gives the available uncertainty for the joint factor/band summaries. These means alone do not provide seed-level standard deviations.

	Both trainable, $k=1$		Both trainable, $k=2$
Task / model	Top init.	Bottom init.	Top init.	Bottom init.
Format / Llama-3.2-1B	79.28	79.54	80.00	80.72
Format / Qwen2.5-1.5B	78.96	78.11	77.75	78.57
OpenBookQA / Qwen2.5-1.5B	55.24	54.96	55.76	55.28
OpenBookQA / Qwen2.5-7B	62.32	62.64	62.92	63.12
OpenBookQA / Mistral-7B	59.24	59.24	59.88	60.48

Appendix

Table 8: Both-trainable controls (%).

B

starts on the indicated spectral band,

A

starts at zero, and both factors train. Rank 1 has the same count as the rank-1 free reference; rank 2 matches the frozen adapters’ rank but has substantially more trainable parameters. The band is an initialization, not a persistent constraint, in these controls.

		B frozen		A frozen		Free	Ref. rank
Task / model	Rank	Top	Bottom	Top	Bottom
OBQA / Qwen2.5-1.5B	4	46.44	45.24	53.44	53.88	55.24	2
	8	49.92	46.56	56.28	53.44	55.44	4
	16	54.08	49.84	55.44	53.64	56.16	7
OBQA / Qwen2.5-7B	16	60.20	56.76	62.40	62.56	63.20	7
ARC-C / Qwen2.5-1.5B	2	48.42	46.68	—	—	50.44	1

Appendix

Table 9: Higher-rank comparisons and ARC-Challenge boundary cases (%). Seeds 2–6. OBQA denotes OpenBookQA and ARC-C denotes ARC-Challenge. A-frozen scores at rank 16 on Qwen2.5-1.5B are recovered algebraically from the reported B-frozen means and A-minus-B differences. They do not introduce new experiments. The ARC-Challenge base score is 43.60%.

Model / task	Rank	Top A $-$ B	Bottom A $-$ B	Interpretation
Qwen2.5-1.5B / OBQA	4	$7.00\ [5.42,8.58]$	$8.64\ [6.90,10.38]$	Both pass
	8	$6.36\ [4.80,7.92]$	$6.88\ [4.18,9.58]$	Both pass
Qwen2.5-7B / OBQA	16	$2.20\ [0.69,3.71]$	$5.80\ [3.35,8.25]$	Both pass

Appendix

Table 10: Available direct-contrast uncertainty in the rank extension. Percentage points; nominal paired-

t

95% intervals, five reporting seeds. “Pass” refers to the specified threshold

t>3.495

for the two bands within each campaign. No test of a universal monotonic trend or a common crossover rank is implied.

Task / model, frozen rank 16	B top	B bottom	Free reference	Free minus bottom
Format / Llama-3.2-1B	77.56	75.54	82.05 ( $k=8$ )	6.51
Format / Qwen2.5-1.5B	74.30	75.24	80.78 ( $k=7$ )	5.54
OpenBookQA / Qwen2.5-1.5B	54.08	49.84	56.16 ( $k=7$ )	6.32
OpenBookQA / Qwen2.5-7B	60.20	56.76	63.20 ( $k=7$ )	6.44
ARC-Easy / Qwen2.5-1.5B	79.48	77.92	79.40 ( $k=7$ )	1.48
ARC-Challenge / Qwen2.5-1.5B	49.70	44.52	49.52 ( $k=7$ )	5.00

Appendix

Table 11: All six rank-16 bottom-band gaps. Scores in percent and gaps in percentage points; seeds 2–6. The last column is a descriptive subtraction, not a set of six significant comparisons. ARC-Easy’s base score is 71.10%.

B-frozen configuration	Earlier rate	Revised rate	Earlier score	Revised score
Top, rank 1	$10^{-2}$	$2\cdot 10^{-2}$	51.07	48.34
Bottom, rank 1	$10^{-2}$	$10^{-2}$	45.47	45.47
Random spectral indices, rank 1	$10^{-2}$	$10^{-2}$	52.88	52.77
Random spectral indices, rank 4	$10^{-2}$	$5\cdot 10^{-3}$	70.23	70.81

Appendix

Table 12: Corrections to historical formatting scores (%). Llama-3.2-1B, reporting seeds 2–6. These are selection and aggregation corrections, not significance tests. The rank-1 random correction removes a run from a different configuration; its selected rate is unchanged.

Control and contrast	Top	Bottom	Scope
Fresh seeds: both trainable $r=2$ minus B frozen $r=2$	14.50	21.27	Format / Llama; seeds 120–124
Four-times tokens: same contrast	11.99	18.24	Format / Llama; all rates retuned
Original-token counterpart	15.57	21.50	Same task, original campaign
Answer-only: A frozen minus B frozen, $r=2$	7.16	9.72	OBQA / Qwen-1.5B; steps fixed
Square-module control: A frozen minus B frozen	39.3	33.6	Top arm meets collapse condition

Appendix

Table 13: Robustness controls (percentage points). The first three rows compare equal rank, not equal trainable counts. The square-module values use the underlying reported differences; subtraction of rounded scores can differ slightly. Its top-band number is descriptive and was excluded from the prespecified interpretation because the B-frozen score was below the collapse threshold.

Configuration	Rank	Selected rate	Original score	Replication
B frozen, top	2	$10^{-2}$	64.43	64.23
B frozen, bottom	2	$5\cdot 10^{-3}$	59.22	58.31
A frozen, top	2	$10^{-2}$	77.39	77.39
A frozen, bottom	2	$10^{-2}$	78.18	77.98
Free LoRA	1	$10^{-3}$	76.84	76.97
Both trainable, top initialization	1	$5\cdot 10^{-3}$	79.28	80.36

Appendix

Table 14: Single-GPU-model replication on Format / Llama-3.2-1B (%). All 35 runs use the original reporting seeds and selected rates. The historical means are retained alongside the replication.

Pair	Group	Configuration	Score top / bottom	Rate top / bottom
Format / Llama-1B	Down	B frozen, $r=2$	29.38 / 12.30	$10^{-2}$ / $10^{-2}$
		A frozen, $r=2$	65.92 / 67.75	$2\cdot 10^{-2}$ / $5\cdot 10^{-2}$
		Free, $r=1$	73.49	$5\cdot 10^{-3}$
	Gate + up	B frozen, $r=2$	39.15 / 40.53	$5\cdot 10^{-2}$ / $2\cdot 10^{-2}$
		A frozen, $r=2$	78.14 / 75.53	$5\cdot 10^{-3}$ / $2\cdot 10^{-2}$
		B frozen, $r=8$	65.57 / 65.49	$10^{-2}$ / $10^{-2}$

Appendix

Table 15: Within-module scores (%) and selected learning rates. Means over eight seeds. Free LoRA is a descriptive reference and is not budget matched to either frozen orientation. Table 4 reports the eight prespecified contrasts.

Configuration	Selected rate	Answer-only score	Full-loss score
B frozen, top	$5\cdot 10^{-3}$	48.36	46.52
B frozen, bottom	$10^{-3}$	45.76	42.20
A frozen, top	$5\cdot 10^{-3}$	55.52	53.08
A frozen, bottom	$10^{-2}$	55.48	52.48
Free LoRA, rank 1	$2\cdot 10^{-3}$	55.72	54.56

Appendix

Table 16: Loss-mask control on OpenBookQA / Qwen2.5-1.5B. Scores in percent, seeds 2–6; frozen ranks are 2. The original packed blocks and 49 optimizer steps are retained. Full-loss scores are included for context and are not paired tests of an equal-supervision loss change.

Figure 5: Training losses at the selected rates, reproduced from the experimental record. Mean over seeds 2–6, logged every ten optimizer steps. Top: OpenBookQA / Qwen2.5-1.5B, full packed-text loss. Bottom: Format / Llama-3.2-1B, target-only loss on a logarithmic scale. The two vertical axes are not comparable. “B trained” means both factors train from spectral initialization. Continued decline and cosine learning-rate decay prevent interpreting a visually flat tail as proof of convergence.

Task, rank	Rate	Seed 2	Seed 3	Seed 4	Seed 5	Seed 6	Mean
Format, 2	$2\cdot 10^{-2}$	77.52	78.18	77.69	76.71	77.04	77.43
OpenBookQA, 2	$10^{-2}$	55.20	55.00	53.80	55.60	53.20	54.56
OpenBookQA, 16	$2\cdot 10^{-3}$	56.20	56.20	56.20	55.80	57.80	56.44

Appendix

Table 17: Random A-frozen replication. Top: test scores in percent. Bottom: seed-paired differences in percentage points against the previously reported random B-frozen runs (65.34% and 45.92%). Both P1 comparisons pass the two-sided Bonferroni rule over two pairs,

|t|>3.495

with four degrees of freedom. The displayed 95% intervals are not multiplicity corrected.

Task, frozen rank	Minus A-top	Minus A-bottom	Minus free LoRA
Format, 2	$+0.03$	$-0.75$	$+0.59$
	$[-2.13,2.20]$	$[-1.92,0.42]$	$[-1.99,3.17]$
OpenBookQA, 2	$+1.48$	$+2.08$	$0.00$
	$[-0.21,3.17]$	$[0.75,3.41]$	$[-2.86,2.86]$
OpenBookQA, 16	$+1.00$	$+2.80$	$+0.28$
	$[0.21,1.79]$	$[1.96,3.64]$	$[-1.63,2.19]$

Appendix

Table 18: Descriptive random-A contrasts, in percentage points. Nominal paired-

t

95% intervals over seeds 2–6; no multiplicity correction or equivalence test. Spectral arms use the same rank as random A. Free LoRA uses ranks 1, 1 and 7, with counts chosen near B frozen, not exactly matched to A frozen (Appendix C ). Reported contrasts retain source rounding and may differ by 0.01 points from subtraction of displayed means.

Fixed output basis, rank 2	OpenBookQA / Qwen-1.5B	Format / Llama-1B
Top spectral band	46.52	64.43
Bottom spectral band	42.20	59.22
Random orthonormal	45.92	65.34
Initial gradient, 64 sequences	42.72	66.58
Initial gradient, all sequences	43.04	66.22
After 10% warm-up	51.16	72.21

Appendix

Table 19: Placement scores (%). Five retraining seeds, 2–6. The initial three rows use the original campaign means. Subsequent diagnostics sometimes use rerun controls for paired differences, as detailed below. The oracle has an additional rank-2 teacher-training cost. The warm-up uses part of the step budget with both factors trainable.

Task	Warm-up fraction	Direction 0	Direction 301	Direction 302
OpenBookQA	5%	0.13	$-0.01$	0.00
	10%	0.58	$0.51\ [0.32,0.70]$	$0.55\ [0.35,0.75]$
	25%	0.78	0.70	0.81
Format	5%	0.40	0.45	0.23
	10%	0.63	$0.55\ [0.19,0.90]$	$0.49\ [0.25,0.73]$
	25%	0.93	0.90	1.07

Appendix

Table 20: Recovery ratios across three direction seeds. The denominator remains the original fixed gap, 8.04 points on OpenBookQA and 12.41 on Format. Brackets for the four new 10% comparisons are paired-

t

intervals at level

1-0.05/4=98.75\%

. Other entries are descriptive point estimates. A reported value of

-0.00

is displayed as 0.00.

Question	Evidence used here	Inferential status
Is the rank-2 factor contrast larger than the top/bottom contrast?	Four complete A/B comparisons	Post hoc descriptive comparison; seed-bootstrap intervals for component contrasts
Can count or module allocation explain the advantage?	Count reversal; within-module controls	Two-band families for count reversal; eight-test family for module controls
Does the result survive selected training controls?	Fresh seeds, one GPU model, four-times tokens, answer-only loss	Prespecified campaign comparisons; scope differs by control
Does the A-frozen advantage need spectral placement of $A$ ?	Random orthonormal A, two pairs, rank 2	Prespecified replication; both P1 tests pass two-pair Bonferroni; earlier outcome partly known
Does the contrast weaken with rank?	Complete rank sweep on one pair; rank-16 checks	Descriptive trend; no universal crossover or strict monotonicity claim
Does the rank-16 deficit persist with PEFT MiCA?	Transferred recipe and shared-recipe follow-up	Nominal paired tests; shared-recipe test planned after the first result

Appendix

Table 21: Evidence map. The paper’s main empirical claim does not inherit confirmatory status merely because some underlying campaigns had prespecified protocols.

Historical comparison	Seeds	Observed difference	MDE
Format / Llama, $\beta=.75$ minus top	7	2.21	3.36
Format / Llama, $\beta=.75$ minus $\beta=.5$	7	0.98	4.51
Format / Llama, $\beta=.75$ minus random orthonormal	7	2.42	3.92
Same random comparison, fresh seeds	8	3.36	4.84
Format / Llama, common-rate bottom advantage, $r=2$	5	0.98	3.29
Format / Llama, common-rate bottom advantage, $r=4$	5	2.25	3.50

Appendix

Table 22: Selected retrospective sensitivity estimates (percentage points). Historical selection rules and seed sets are retained. These rows are not all drawn from the central separately tuned two-by-two comparisons.

Follow-up	Pushed (2026)	Scope and departures
Boundary selection	03 Oct, 00:51	Reuses prior selection runs; three engine fixes precede all new runs.
Single GPU model	03 Oct, 00:51	Seven arms rerun at their reported rates, seeds 2–6.
PEFT MiCA	03 Oct, 00:51	Before runs, addendum distinguishes recipes and training amounts.
Shared MiCA recipe	03 Oct, 08:36	Planned after the MiCA result; five new LoRA runs reuse MiCA controls.
Module groups	03 Oct, 00:51	Validation uses 500 rather than 400 items; some boundary optima and ties remain.
Warm-up directions	03 Oct, 00:51	New direction seeds 301/302; historical top controls and fixed reference gaps.

Appendix

Table 23: Internal prospective records for the new training campaigns. According to the retained records, protocols preceded their new runs. The random-A addendum was pushed after selection submission but before any result. The shared-recipe test was planned after the first MiCA result.

Explore similar work

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
Jun 11, 2026Franz Louis Cesista, Katherine Crowson, Cédric Simal +1Low-Rank Structure Muon
Ateneo de Manila University · EleutherAI · NaXys +1
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
Aug 10, 2026Xinyi Xu, Bingnan Xiao, Shuang Qin +2Federated Learning Subspace
University of Electronic Science and Technology of China · Fudan University · Singapore University of Technology and Design
Low-Rank Adaptation Redux for Large Models
Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisFoundation Model Graph Signal Processing
Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA

Jun 11, 2026, cs.LGJ/K move · Enter open · S save

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

Franz Louis Cesista, Katherine Crowson, Cédric Simal, Stella Biderman

Ateneo de Manila University · EleutherAI · NaXys+1

Low-Rank Structure Muon Manifolds

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its optimal learning rates transfer poorly across ranks, and it often fails to beat dense baselines. We derive LoRA-Muon by applying the Muon optimizer's spectral steepest-descent rule to the low-rank setting. Along with our split weight-decay rule, our main claim is that LoRA-Muon is a good low-rank proxy for full-rank Muon and Shampoo-family optimizers. Its optimal learning rates transfer across rank, width, depth, and factor-rescaling. In our compute-matched TinyShakespeare study, a rank-

2

proxy recovers the dense best tested learning rate, and a rank-

32

LoRA-Muon run attains lower mean validation loss than the dense baseline in the seed-averaged sweep. We further show that the Spectron optimizer depends on arbitrary factor scaling, so it would likely be a poor fit when finetuning starts from badly imbalanced factors, and that LoRA-RITE's simplified QR-coordinate core implements the same spectral update. LoRA-Muon computes that update without QR-decomposition and avoids storing second moments, making it more accelerator-friendly and memory-efficient.

Abstract

Explore similar work

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

Low-Rank Adaptation Redux for Large Models