Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning but can suffer from redundant basis directions. Constraining the basis factor to the Stiefel manifold addresses this redundancy through orthonormality. Landing methods allow temporarily infeasible iterates to explore a neighborhood of the manifold while progressively restoring orthogonality without per-step retractions. To improve optimization efficiency while retaining this flexibility, we introduce ProLand (Product-space Landing), a retraction-free optimizer that combines coordinated factor updates with Muon-style spectral directions. By designing the updates in the space of the represented weight increment, ProLand separates objective optimization from orthogonality restoration, supporting efficient feasibility recovery. This coordinated construction also enables Muon-style directions to guide objective optimization while preserving that separation. Experiments cover four GLUE validation tasks on DeBERTa-v3-base and seven language benchmarks on Qwen2.5-3B, Llama-3.2-3B, and Moonlight-16B-A3B across multiple adapter ranks. Across these 11 tasks, ProLand achieves the highest average score among the compared methods in every tested encoder and decoder model-rank configuration.
Figures & tables
Figure 1: Feasibility treatment changes the solution. (a) Factor trajectories over loss contours minimized over the product scale; (b) Product objective values; (c) Constraint violation.
Method
RTE
MRPC
STS-B
CoLA
Avg.
AdamW
79.78±2.25
90.21±0.47
90.89±0.09
64.80±1.13
81.42
LoRA-Muon
83.63±1.82
91.71±0.88
91.74±0.25
67.77±0.82
83.71
Per-factor Muon
83.27±2.35
92.67±0.31
91.75±0.05
67.67±1.89
83.84
sMuon
85.08±0.91
91.07±0.66
91.59±0.09
67.83±0.34
83.89
Riemannion
83.75±1.44
91.33±0.41
91.68±0.14
66.97±1.37
83.43
Manifold-LoRA
78.58±1.99
89.52±0.91
90.70±0.12
64.08±1.97
80.72
Table 1: DeBERTa-v3-base on GLUE validation sets. Task columns report the mean ± sample standard deviation across seeds; Avg. is the unweighted mean of the four task scores. Bold and underline mark the best and second-best means in each column.
Rank
Method
MMLU
PIQA
Wino
OBQA
Hella
BoolQ
ARC-C
Avg.
4
AdamW
65.18
78.73
69.69
42.80
74.01
86.15
49.66
66.60
LoRA-Muon
63.64
79.38
71.03
42.20
72.95
87.16
49.32
66.53
Per-factor Muon
64.76
78.67
70.40
43.80
74.37
86.67
48.46
66.73
sMuon
65.30
78.67
69.93
41.40
74.63
86.45
49.06
66.49
Riemannion
64.25
79.27
70.40
42.40
74.43
86.91
49.40
66.72
Manifold-LoRA
65.23
79.33
69.06
42.60
74.33
86.42
48.55
66.50
Table 2: Qwen2.5-3B: language benchmarks. Zero-shot accuracy (%) at ranks 4, 8, and 16. Avg. is the unweighted mean; bold and underline mark the best and second-best scores within each rank.
Rank
Method
MMLU
PIQA
Wino
OBQA
Hella
BoolQ
ARC-C
Avg.
4
AdamW
54.70
77.80
70.09
42.80
73.43
85.17
47.87
64.55
LoRA-Muon
52.76
76.01
72.06
42.20
73.02
86.97
42.32
63.62
Per-factor Muon
54.92
78.18
70.88
43.60
73.70
85.75
47.87
64.99
sMuon
54.68
78.07
70.96
42.80
73.53
85.84
46.84
64.67
Riemannion
54.23
78.35
71.51
42.40
73.47
85.17
47.95
64.73
Manifold-LoRA
54.66
79.11
72.22
42.20
73.61
85.41
48.81
65.14
Table 3: Llama-3.2-3B: language benchmarks. Zero-shot accuracy (%) at ranks 4, 8, and 16. Avg. is the unweighted mean; bold and underline mark the best and second-best scores within each rank.
Method
MMLU
PIQA
Wino
OBQA
Hella
BoolQ
ARC-C
Avg.
AdamW
55.36
81.07
69.77
44.60
75.00
88.01
54.44
66.89
LoRA-Muon
63.81
82.48
73.80
46.40
80.84
89.42
61.52
71.18
Per-factor Muon
65.56
82.05
74.27
47.60
80.57
87.83
63.91
71.68
sMuon
61.18
81.23
72.61
46.80
77.26
69.79
60.41
67.04
Riemannion
59.41
81.72
73.48
46.00
76.02
87.34
57.08
68.72
Manifold-LoRA
58.88
81.23
71.11
44.60
76.82
88.75
57.08
68.35
Table 4: Moonlight-16B-A3B: language benchmarks. Zero-shot accuracy (%) at LoRA rank 16. Avg. is the unweighted mean; bold and underline mark the best and second-best scores.
Figure 2: Landing limits the growth of constraint violation. Panels show the median, 90th percentile, and maximum of ∥B⊤B−I∥op across adapters. Curves average these statistics across runs; shading shows their range.
Constraint violation ( 10−3 ) ↓
Method
Acc. (%) ↑
Median
P90
Maximum
ProLand
40.73±0.69
7.615±0.540
9.389±0.732
13.556±0.492
Without Landing
38.93±0.61
101.410±0.523
297.497±5.759
349.397±12.500
With retraction
39.67±0.76
0.506±0.010
1.282±0.025
1.552±0.023
Table 5: Landing ablation under shared conditions. Mean ± sample SD; bold marks the highest accuracy or lowest constraint violation.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Landing controls constraint violation of B . Columns show the adapter median, 90th percentile, and maximum; rows compare linear-decay and constant learning-rate schedules. Median and percentile curves show means across runs; maximum curves show the worst run and adapter. Shading spans the run-to-run range, and gray regions mark warmup.
Figure 4: Extremal singular values of A during fine-tuning. Curves take extrema over all adapters and runs. The gray region marks warmup; the reported positive lower bound applies to recorded states after warmup. Colors and line styles match Figure 3 .
Method
Manifold constraint
Product direction
Muon-type
Feasibility handling
LoRA-Pro ( Wang et al., 2025 )
×
✓
×
—
LORO ( Mo et al., 2025 )
✓W∈Mr
✓
×
Periodic rank- r retraction
sMuon ( Anson et al., 2026 )
×
✓
✓
—
LoRA-TSD ( Andriianov et al., 2026 )
✓W∈Mr
✓
✓
Factor-induced rank- r retraction
Riemannion ( Bogachev et al., 2026 )
✓W∈Mr
✓
✓
Truncated-SVD retraction
Skewon ( Solonko et al., 2026 )
✓X∈St
×
✓
Stiefel retraction
Appendix
Table 6: Structural comparison of related low-rank and Stiefel optimization methods. A checkmark indicates an explicit component of the reviewed method, not a performance advantage.
Figure 5: Qwen2.5-3B rank-8 loss traces. Each task pairs training loss (upper) with validation loss (lower). Training curves show smoothed and faint raw traces; lines between validation points serve as visual guides.
Figure 6: Factor-direction orthogonality and the effect of parameter storage. Solid and dotted lines show the median and maximum across runs and adapters. (a) Normalized compatibility error of the FP32 update direction and its BF16 cast. (b) Gram-recursion error before and after BF16 parameter storage.
Method
HumanEval
HumanEval+
MBPP
MBPP+
Avg.
Qwen2.5-3B
AdamW
54.88
51.83
66.67
57.67
57.76
LoRA-Muon
57.32
53.05
67.99
57.67
59.01
Per-factor Muon
56.71
53.05
68.25
56.88
58.72
sMuon
56.71
53.05
67.46
58.99
59.05
Riemannion
59.76
54.88
67.99
57.14
59.94
Appendix
Table 7: Code-generation results. Zero-shot greedy pass@1 (%). Avg. is the arithmetic mean of the four scores; bold and underline mark the best and second-best scores within each backbone.
Method
HumanEval
HumanEval+
MBPP
MBPP+
Avg.
AdamW
56.10
49.39
61.11
50.79
54.35
LoRA-Muon
60.98
52.44
62.17
52.12
56.93
Per-factor Muon
54.88
48.78
68.78
57.94
57.59
sMuon
57.93
51.22
69.58
58.73
59.37
Riemannion
47.56
38.41
60.85
49.74
49.14
Manifold-LoRA
53.66
43.90
60.58
49.74
51.97
Appendix
Table 8: Moonlight-16B-A3B: code benchmarks (%). Avg. is the arithmetic mean of the four unrounded pass@1 scores; bold and underline mark the best and second-best scores.
Parameter
Value
Base weights / trainable parameters
Frozen / LoRA factors A,B
Adapter targets
All linear modules except the Moonlight MoE routers
LoRA scaling / dropout / bias
α=2r / 0 / none
Maximum sequence length / packing
512 / disabled
Objective
Response-token-weighted cross-entropy
Training precision
BF16; FP32 optimizer arithmetic for ProLand and language-benchmark sMuon
Appendix
Table 9: Fine-tuning and decoding settings for Qwen, Llama, and Moonlight. Experiment-specific choices are listed in the following tables.
Experiment
Candidates / training budget
3B baselines
η∈{0.3η0,η0,3η0} ; 3 epochs per candidate; η0 from Table 11
ProLand
Same multipliers and budget; language and code η0=6×10−6
Factor-spectrum pilot
η∈{1.75×10−7,7×10−7,2×10−6,7×10−6} ; linear decay, ρ=0.05
Appendix
Table 10: Candidate grids and selection stages. All rates are base learning rates before any optimizer-specific scaling.
Qwen2.5-3B
Llama-3.2-3B
3B code
Moonlight
Method
η0
r=4
r=8
r=16
r=4
r=8
r=16
Qwen
Llama
Lang.
Code
AdamW
0.001
0.0003
0.0003
0.0003
0.001
0.001
0.001
0.0003
0.0003
0.001
0.001
LoRA-Muon
1e-5
1e-5
3e-6
3e-5
3e-5
3e-5
1e-5
3e-6
3e-6
6e-5
6e-5
Per-factor Muon
6e-5
1.8e-5
1.8e-5
1.8e-5
1.8e-5
6e-5
6e-5
1.8e-5
1.8e-5
6e-5
1.8e-5
sMuon
7e-6
7e-6
2.1e-6
7e-6
2.1e-5
2.1e-5
2.1e-5
2.1e-6
7e-6
6e-5
7e-6
Riemannion
1e-5
1e-5
3e-6
1e-5
3e-5
1e-5
3e-5
3e-6
3e-6
6e-5
6e-5
Appendix
Table 11: Final learning rates for the 3B language and code benchmarks and for Moonlight. 3B language runs use ranks 4, 8, and 16; code runs use rank 8; and Moonlight runs use rank 16.
Table 12: Fixed optimizer settings for the 3B benchmarks. Baseline weight decay is 0.01 with each implementation’s decay rule. AdamW and Manifold-LoRA use Adam coefficients (0.9,0.999) and ϵ=10−8 ; the other methods use momentum 0.9. These internal settings are held fixed during the learning-rate searches.
Parameter
Landing ablation
Factor spectra
Learning rate
7e-6
7e-7 (peak)
Effective batch / epochs
640 / 10
16 / 3
Warmup
None
100 updates
Schedule
Constant
Linear decay to zero; constant
Landing strength ρ
0, 0.05
0, 0.05, 0.45
Validation
Each epoch
Four times per epoch
Appendix
Table 13: Orthogonality and factor-diagnostic settings. Both use β=0.9 , λA=0 , λB=0 , orthogonal B and zero A initialization, FP32 updates, and BF16 factors and optimizer states.
Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.
Yuan Zhang, Jiang Hu, Zhijian Lai +2
Center for Data Science, Peking University, Beijing 100871, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China · Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China +2
Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.
Valérie Castin, Kimia Nadjahi, Pierre Ablin +1
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.