From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA
Organizations: Fuzhou University · Shandong Technology and Business University
Abstract
Parameter-efficient fine-tuning with LoRA is governed by rank, which controls parameter budget and locally accessible task updates. Existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. We characterize the first-order information boundary of standard factor gradients, showing that their unobservable subspace coincides with the normal space of the low-rank manifold. Thus, factor gradients alone cannot distinguish under-optimization within the current space from task-relevant directions beyond its first-order reach. This motivates a geometric view of rank evolution, where tangent information optimizes existing degrees of freedom and the normal spectrum quantifies the value of new ones. Under a local quadratic model, we show that leading normal singular directions yield the locally optimal rank expansion, with spectral energy determining its gain. Building on this characterization, we propose GEAR, which turns the tangent--normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. Across language and vision benchmarks, GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters. On LLaMA3-8B, it raises the average commonsense score from 86.61 to 88.45 while reducing trainable parameters from 42.0M to 14.4M.
Figures & tables
| Method (Params) | BoolQ | PIQA | SIQA | HellaS. | WinoG. | ARC-e | ARC-c | OBQA | Avg. |
| LLaMA3-8B | |||||||||
| LoRA (41.9M) | 72.37 | 87.83 | 81.34 | 94.81 | 85.08 | 91.46 | 81.32 | 86.00 | 85.03 |
| AdaLoRA (31.6M) | 72.23 | 87.21 | 81.27 | 94.60 | 85.16 | 93.18 | 82.68 | 85.80 | 85.27 |
| Riemannion (41.9M) | 72.45 | 87.32 | 81.17 | 94.95 | 87.64 | 93.14 | 82.51 | 87.40 | 85.82 |
| GeoLoRA (41.3M) | 72.84 | 87.54 | 79.99 | 94.62 | 85.56 | 93.52 | 83.96 | 87.00 | 85.63 |
| FlexLoRA (42.0M) | 72.91 | 88.30 | 82.96 | 95.68 | 87.14 | 93.56 | 83.70 | 88.60 | 86.61 |
| Method (Params) | CoLA Mcc. | MNLI Acc. | MRPC Acc. | RTE Acc. | QNLI Acc. | SST-2 Acc. | STS-B Corr. | QQP Acc. | Avg. |
| LoRA (1.9M) | 68.31 | 89.74 | 90.19 | 84.16 | 93.61 | 95.30 | 85.20 | 90.35 | 87.11 |
| AdaLoRA (1.9M) | 70.07 | 89.93 | 90.44 | 88.17 | 94.27 | 95.13 | 91.21 | 91.41 | 88.83 |
| Riemannion (1.9M) | 67.41 | 88.93 | 89.01 | 85.13 | 92.64 | 94.00 | 90.12 | 87.92 | 86.90 |
| GeoLoRA (1.9M) | 68.18 | 89.67 | 87.76 | 87.73 | 92.93 | 95.52 | 90.15 | 87.77 | 87.46 |
| FlexLoRA (1.9M) | 70.22 | 90.13 | 90.67 | 88.45 | 94.24 | 95.34 | 91.53 | 90.27 | 88.85 |
| GEAR ( 1.5M ) | 70.85 | 90.97 | 92.46 | 89.53 | 94.27 | 96.22 | 91.83 | 90.88 | 89.63 |
| Dataset | LoRA | AdaLoRA | Riemannion | GeoLoRA | FlexLoRA | GEAR |
| (0.3M) | (0.3M) | (0.3M) | (0.3M) | (0.3M) | ( 0.2M ) | |
| Natural | ||||||
| CIFAR100 | 69.39 | 71.81 | 70.78 | 68.05 | 71.30 | 71.88 |
| Caltech101 | 90.60 | 89.96 | 91.03 | 88.87 | 91.72 | 91.78 |
| DTD | 71.01 | 71.76 | 71.28 | 68.88 | 70.18 | 71.60 |
| Flowers | 98.80 | 99.04 | 99.09 | 99.06 | 98.98 | 99.12 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Training time (s) | Peak CUDA memory (MiB) |
| LoRA | 789.83 | 3825.15 |
| FlexLoRA | 1006.75 | 6553.87 |
| GEAR | 1060.83 | 4458.21 |
| Hyperparameter | Setting |
| Initial effective rank | 4 |
| Maximum per-module rank | 32 |
| Weighted-average rank budget | 16 |
| Target modules | Q, K, V, O, Up, Down, Gate |
| Probe width | 24 |
| Probe batch size | 128 total (16 per task) |
| Dataset | Task | # Train | # Dev | # Test | # Label | Metrics |
| Single-Sentence Classification | ||||||
| CoLA | Acceptability | 8.5k | 1k | 1k | 2 | Matthews corr. |
| SST-2 | Sentiment | 67k | 872 | 1.8k | 2 | Accuracy |
| Similarity and Paraphrase | ||||||
| MRPC | Paraphrase | 3.7k | 408 | 1.7k | 2 | Accuracy |
| QQP | Paraphrase | 364k | 40k | 391k | 2 | Accuracy |
| Dataset | Adapter LR | Head LR | Batch | Epochs | ||||||
| MNLI | 32 | 6 | 4 | 8 | 16 | |||||
| RTE | 32 | 50 | 4 | 8 | 16 | |||||
| QNLI | 32 | 5 | 4 | 8 | 16 | |||||
| MRPC | 32 | 30 | 4 | 8 | 16 | |||||
| QQP | 32 | 5 | 4 | 8 | 16 | |||||
| SST-2 | 32 | 24 | 4 | 8 | 16 |
| Dataset | # Classes | # Train | # Val | # Test |
| Natural | ||||
| CIFAR100 | 100 | 800/1000 | 200 | 10,000 |
| Caltech101 | 102 | 800/1000 | 200 | 6,084 |
| DTD | 47 | 800/1000 | 200 | 1,880 |
| Flowers102 | 102 | 800/1000 | 200 | 6,149 |
| Pets | 37 | 800/1000 | 200 | 3,669 |
| Hyperparameter | Setting |
| Initial effective rank | 4 |
| Weighted-average rank budget | 8 |
| Maximum per-module rank | 16 |
| Learning rate | |
| First audit step | |
| Audit interval |
| TGI | Manifold | Rank Expansion | BoolQ | PIQA | SIQA | HellaS. | WinoG. | ARC-e | ARC-c | OBQA | Avg. |
| – | – | – | 72.37 | 87.83 | 81.34 | 94.81 | 85.08 | 91.46 | 81.32 | 86.00 | 85.03 |
| ✓ | – | – | 72.51 | 87.57 | 81.27 | 94.88 | 86.94 | 92.16 | 83.08 | 86.16 | 85.57 |
| – | ✓ | – | 72.40 | 87.03 | 81.12 | 94.46 | 87.38 | 92.51 | 83.12 | 87.21 | 85.65 |
| ✓ | ✓ | – | 73.15 | 89.13 | 82.01 | 94.90 | 87.27 | 92.88 | 83.30 | 87.17 | 86.23 |
| – | ✓ | ✓ | 73.95 | 89.73 | 83.13 | 95.84 | 88.66 | 93.04 | 84.71 | 89.51 | 87.32 |
| ✓ | ✓ | ✓ | 75.19 | 90.81 | 83.78 | 96.82 | 89.82 | 93.77 | 85.58 | 91.80 | 88.45 |
Low-Rank Adaptation
Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.
See all 579 papers