ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
Organizations: Shanghai Jiao Tong University · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) · Stanford University
Abstract
Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning but can suffer from redundant basis directions. Constraining the basis factor to the Stiefel manifold addresses this redundancy through orthonormality. Landing methods allow temporarily infeasible iterates to explore a neighborhood of the manifold while progressively restoring orthogonality without per-step retractions. To improve optimization efficiency while retaining this flexibility, we introduce ProLand (Product-space Landing), a retraction-free optimizer that combines coordinated factor updates with Muon-style spectral directions. By designing the updates in the space of the represented weight increment, ProLand separates objective optimization from orthogonality restoration, supporting efficient feasibility recovery. This coordinated construction also enables Muon-style directions to guide objective optimization while preserving that separation. Experiments cover four GLUE validation tasks on DeBERTa-v3-base and seven language benchmarks on Qwen2.5-3B, Llama-3.2-3B, and Moonlight-16B-A3B across multiple adapter ranks. Across these 11 tasks, ProLand achieves the highest average score among the compared methods in every tested encoder and decoder model-rank configuration.
Figures & tables
| Method | RTE | MRPC | STS-B | CoLA | Avg. |
|---|---|---|---|---|---|
| AdamW | 81.42 | ||||
| LoRA-Muon | 83.71 | ||||
| Per-factor Muon | 83.84 | ||||
| sMuon | 83.89 | ||||
| Riemannion | 83.43 | ||||
| Manifold-LoRA | 80.72 |
| Rank | Method | MMLU | PIQA | Wino | OBQA | Hella | BoolQ | ARC-C | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| 4 | AdamW | 65.18 | 78.73 | 69.69 | 42.80 | 74.01 | 86.15 | 49.66 | 66.60 |
| LoRA-Muon | 63.64 | 79.38 | 71.03 | 42.20 | 72.95 | 87.16 | 49.32 | 66.53 | |
| Per-factor Muon | 64.76 | 78.67 | 70.40 | 43.80 | 74.37 | 86.67 | 48.46 | 66.73 | |
| sMuon | 65.30 | 78.67 | 69.93 | 41.40 | 74.63 | 86.45 | 49.06 | 66.49 | |
| Riemannion | 64.25 | 79.27 | 70.40 | 42.40 | 74.43 | 86.91 | 49.40 | 66.72 | |
| Manifold-LoRA | 65.23 | 79.33 | 69.06 | 42.60 | 74.33 | 86.42 | 48.55 | 66.50 |
| Rank | Method | MMLU | PIQA | Wino | OBQA | Hella | BoolQ | ARC-C | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| 4 | AdamW | 54.70 | 77.80 | 70.09 | 42.80 | 73.43 | 85.17 | 47.87 | 64.55 |
| LoRA-Muon | 52.76 | 76.01 | 72.06 | 42.20 | 73.02 | 86.97 | 42.32 | 63.62 | |
| Per-factor Muon | 54.92 | 78.18 | 70.88 | 43.60 | 73.70 | 85.75 | 47.87 | 64.99 | |
| sMuon | 54.68 | 78.07 | 70.96 | 42.80 | 73.53 | 85.84 | 46.84 | 64.67 | |
| Riemannion | 54.23 | 78.35 | 71.51 | 42.40 | 73.47 | 85.17 | 47.95 | 64.73 | |
| Manifold-LoRA | 54.66 | 79.11 | 72.22 | 42.20 | 73.61 | 85.41 | 48.81 | 65.14 |
| Method | MMLU | PIQA | Wino | OBQA | Hella | BoolQ | ARC-C | Avg. |
|---|---|---|---|---|---|---|---|---|
| AdamW | 55.36 | 81.07 | 69.77 | 44.60 | 75.00 | 88.01 | 54.44 | 66.89 |
| LoRA-Muon | 63.81 | 82.48 | 73.80 | 46.40 | 80.84 | 89.42 | 61.52 | 71.18 |
| Per-factor Muon | 65.56 | 82.05 | 74.27 | 47.60 | 80.57 | 87.83 | 63.91 | 71.68 |
| sMuon | 61.18 | 81.23 | 72.61 | 46.80 | 77.26 | 69.79 | 60.41 | 67.04 |
| Riemannion | 59.41 | 81.72 | 73.48 | 46.00 | 76.02 | 87.34 | 57.08 | 68.72 |
| Manifold-LoRA | 58.88 | 81.23 | 71.11 | 44.60 | 76.82 | 88.75 | 57.08 | 68.35 |
| Constraint violation ( ) | ||||
|---|---|---|---|---|
| Method | Acc. (%) | Median | P90 | Maximum |
| ProLand | ||||
| Without Landing | ||||
| With retraction | ||||
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Manifold constraint | Product direction | Muon-type | Feasibility handling |
| LoRA-Pro ( Wang et al., 2025 ) | — | |||
| LORO ( Mo et al., 2025 ) | Periodic rank- retraction | |||
| sMuon ( Anson et al., 2026 ) | — | |||
| LoRA-TSD ( Andriianov et al., 2026 ) | Factor-induced rank- retraction | |||
| Riemannion ( Bogachev et al., 2026 ) | Truncated-SVD retraction | |||
| Skewon ( Solonko et al., 2026 ) | Stiefel retraction |
| Method | HumanEval | HumanEval+ | MBPP | MBPP+ | Avg. |
|---|---|---|---|---|---|
| Qwen2.5-3B | |||||
| AdamW | 54.88 | 51.83 | 66.67 | 57.67 | 57.76 |
| LoRA-Muon | 57.32 | 53.05 | 67.99 | 57.67 | 59.01 |
| Per-factor Muon | 56.71 | 53.05 | 68.25 | 56.88 | 58.72 |
| sMuon | 56.71 | 53.05 | 67.46 | 58.99 | 59.05 |
| Riemannion | 59.76 | 54.88 | 67.99 | 57.14 | 59.94 |
| Method | HumanEval | HumanEval+ | MBPP | MBPP+ | Avg. |
|---|---|---|---|---|---|
| AdamW | 56.10 | 49.39 | 61.11 | 50.79 | 54.35 |
| LoRA-Muon | 60.98 | 52.44 | 62.17 | 52.12 | 56.93 |
| Per-factor Muon | 54.88 | 48.78 | 68.78 | 57.94 | 57.59 |
| sMuon | 57.93 | 51.22 | 69.58 | 58.73 | 59.37 |
| Riemannion | 47.56 | 38.41 | 60.85 | 49.74 | 49.14 |
| Manifold-LoRA | 53.66 | 43.90 | 60.58 | 49.74 | 51.97 |
| Parameter | Value |
|---|---|
| Base weights / trainable parameters | Frozen / LoRA factors |
| Adapter targets | All linear modules except the Moonlight MoE routers |
| LoRA scaling / dropout / bias | / 0 / none |
| Maximum sequence length / packing | 512 / disabled |
| Objective | Response-token-weighted cross-entropy |
| Training precision | BF16; FP32 optimizer arithmetic for ProLand and language-benchmark sMuon |
| Experiment | Candidates / training budget |
|---|---|
| 3B baselines | ; 3 epochs per candidate; from Table 11 |
| ProLand | Same multipliers and budget; language and code |
| Factor-spectrum pilot | ; linear decay, |
| Qwen2.5-3B | Llama-3.2-3B | 3B code | Moonlight | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Qwen | Llama | Lang. | Code | |||||||
| AdamW | 0.001 | 0.0003 | 0.0003 | 0.0003 | 0.001 | 0.001 | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.001 |
| LoRA-Muon | 1e-5 | 1e-5 | 3e-6 | 3e-5 | 3e-5 | 3e-5 | 1e-5 | 3e-6 | 3e-6 | 6e-5 | 6e-5 |
| Per-factor Muon | 6e-5 | 1.8e-5 | 1.8e-5 | 1.8e-5 | 1.8e-5 | 6e-5 | 6e-5 | 1.8e-5 | 1.8e-5 | 6e-5 | 1.8e-5 |
| sMuon | 7e-6 | 7e-6 | 2.1e-6 | 7e-6 | 2.1e-5 | 2.1e-5 | 2.1e-5 | 2.1e-6 | 7e-6 | 6e-5 | 7e-6 |
| Riemannion | 1e-5 | 1e-5 | 3e-6 | 1e-5 | 3e-5 | 1e-5 | 3e-5 | 3e-6 | 3e-6 | 6e-5 | 6e-5 |
| Method | Numerical update settings |
|---|---|
| AdamW | Standard AdamW update |
| LoRA-Muon | 8 FP32 polar iterations; 7 inverse-root iterations; |
| Per-factor Muon | Independent factor EMAs; 5 BF16 polar iterations |
| sMuon | 12 polar iterations; 8 inverse-root coefficient iterations; |
| Riemannion | Low-rank QR/SVD update |
| Manifold-LoRA | Fixed-step Stiefel–AdamW; restoration coefficient |
| Parameter | Landing ablation | Factor spectra |
|---|---|---|
| Learning rate | 7e-6 | 7e-7 (peak) |
| Effective batch / epochs | 640 / 10 | 16 / 3 |
| Warmup | None | 100 updates |
| Schedule | Constant | Linear decay to zero; constant |
| Landing strength | 0, 0.05 | 0, 0.05, 0.45 |
| Validation | Each epoch | Four times per epoch |
LLM Fine-Tuning
LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.
See all 790 papers