Parameter-efficient fine-tuning with LoRA is governed by rank, which controls parameter budget and locally accessible task updates. Existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. We characterize the first-order information boundary of standard factor gradients, showing that their unobservable subspace coincides with the normal space of the low-rank manifold. Thus, factor gradients alone cannot distinguish under-optimization within the current space from task-relevant directions beyond its first-order reach. This motivates a geometric view of rank evolution, where tangent information optimizes existing degrees of freedom and the normal spectrum quantifies the value of new ones. Under a local quadratic model, we show that leading normal singular directions yield the locally optimal rank expansion, with spectral energy determining its gain. Building on this characterization, we propose GEAR, which turns the tangent--normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. Across language and vision benchmarks, GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters. On LLaMA3-8B, it raises the average commonsense score from 86.61 to 88.45 while reducing trainable parameters from 42.0M to 14.4M.
Figures & tables
Figure 1: Motivation and overview of GEAR. (a) standard factor-gradient observations are insensitive to normal components, so identical first-order observations can correspond to different rank requirements . (b) GEAR distinguishes optimization within the current manifold from expansion toward task-relevant directions beyond the current first-order reachable space. (c) this geometry-guided rank evolution yields a stronger performance–parameter trade-off on commonsense reasoning.
Method (Params)
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
LLaMA3-8B
LoRA (41.9M)
72.37
87.83
81.34
94.81
85.08
91.46
81.32
86.00
85.03
AdaLoRA (31.6M)
72.23
87.21
81.27
94.60
85.16
93.18
82.68
85.80
85.27
Riemannion (41.9M)
72.45
87.32
81.17
94.95
87.64
93.14
82.51
87.40
85.82
GeoLoRA (41.3M)
72.84
87.54
79.99
94.62
85.56
93.52
83.96
87.00
85.63
FlexLoRA (42.0M)
72.91
88.30
82.96
95.68
87.14
93.56
83.70
88.60
86.61
Table 1: Commonsense reasoning performance on LLaMA3-8B and Qwen2.5-7B. We report the mean over three runs with different random seeds. Best results are shown in bold .
Method (Params)
CoLA Mcc.
MNLI Acc.
MRPC Acc.
RTE Acc.
QNLI Acc.
SST-2 Acc.
STS-B Corr.
QQP Acc.
Avg.
LoRA (1.9M)
68.31
89.74
90.19
84.16
93.61
95.30
85.20
90.35
87.11
AdaLoRA (1.9M)
70.07
89.93
90.44
88.17
94.27
95.13
91.21
91.41
88.83
Riemannion (1.9M)
67.41
88.93
89.01
85.13
92.64
94.00
90.12
87.92
86.90
GeoLoRA (1.9M)
68.18
89.67
87.76
87.73
92.93
95.52
90.15
87.77
87.46
FlexLoRA (1.9M)
70.22
90.13
90.67
88.45
94.24
95.34
91.53
90.27
88.85
GEAR ( 1.5M )
70.85
90.97
92.46
89.53
94.27
96.22
91.83
90.88
89.63
Table 2: GLUE performance with DeBERTa-v3-base. We report the mean over three runs with different random seeds. Best results are shown in bold .
Dataset
LoRA
AdaLoRA
Riemannion
GeoLoRA
FlexLoRA
GEAR
(0.3M)
(0.3M)
(0.3M)
(0.3M)
(0.3M)
( 0.2M )
Natural
CIFAR100
69.39
71.81
70.78
68.05
71.30
71.88
Caltech101
90.60
89.96
91.03
88.87
91.72
91.78
DTD
71.01
71.76
71.28
68.88
70.18
71.60
Flowers
98.80
99.04
99.09
99.06
98.98
99.12
Table 3: VTAB-1K performance with ViT-B/16. We report the mean over three runs with different random seeds. Best results are shown in bold .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Training time (s)
Peak CUDA memory (MiB)
LoRA
789.83
3825.15
FlexLoRA
1006.75
6553.87
GEAR
1060.83
4458.21
Appendix
Table 4: Training time and peak CUDA memory on DTD with ViT-B/16.
Hyperparameter
Setting
Initial effective rank r0
4
Maximum per-module rank rmax
32
Weighted-average rank budget rˉbudget
16
Target modules
Q, K, V, O, Up, Down, Gate
Probe Ω width h
24
Probe batch size
128 total (16 per task)
Appendix
Table 5: Hyperparameter settings for LLaMA3-8B and Qwen2.5-7B in commonsense reasoning experiments.
Dataset
Task
# Train
# Dev
# Test
# Label
Metrics
Single-Sentence Classification
CoLA
Acceptability
8.5k
1k
1k
2
Matthews corr.
SST-2
Sentiment
67k
872
1.8k
2
Accuracy
Similarity and Paraphrase
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy
Appendix
Table 6: Details of the GLUE dataset.
Dataset
Adapter LR
Head LR
Batch
Epochs
r0
rˉbudget
rmax
tstart
Δt
tfinal
MNLI
5×10−4
5×10−4
32
6
4
8
16
0.1T
0.075T
0.7T
RTE
3.6×10−3
1.2×10−3
32
50
4
8
16
0.1T
0.075T
0.7T
QNLI
3×10−3
3×10−4
32
5
4
8
16
0.1T
0.075T
0.7T
MRPC
3×10−3
1×10−3
32
30
4
8
16
0.1T
0.075T
0.7T
QQP
1.6×10−3
8×10−4
32
5
4
8
16
0.1T
0.075T
0.7T
SST-2
5×10−3
3×10−4
32
24
4
8
16
0.1T
0.075T
0.7T
Appendix
Table 7: Hyper-parameter settings of our method on the GLUE benchmark.
Dataset
# Classes
# Train
# Val
# Test
Natural
CIFAR100
100
800/1000
200
10,000
Caltech101
102
800/1000
200
6,084
DTD
47
800/1000
200
1,880
Flowers102
102
800/1000
200
6,149
Pets
37
800/1000
200
3,669
Appendix
Table 8: Details of the VTAB-1K benchmark.
Hyperparameter
Setting
Initial effective rank r0
4
Weighted-average rank budget rˉbudget
8
Maximum per-module rank rmax
16
Learning rate
5×10−3
First audit step tstart
160
Audit interval Δt
50
Appendix
Table 9: Hyperparameter settings for the VTAB-1K experiments.
TGI
Manifold
Rank Expansion
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
–
–
–
72.37
87.83
81.34
94.81
85.08
91.46
81.32
86.00
85.03
✓
–
–
72.51
87.57
81.27
94.88
86.94
92.16
83.08
86.16
85.57
–
✓
–
72.40
87.03
81.12
94.46
87.38
92.51
83.12
87.21
85.65
✓
✓
–
73.15
89.13
82.01
94.90
87.27
92.88
83.30
87.17
86.23
–
✓
✓
73.95
89.73
83.13
95.84
88.66
93.04
84.71
89.51
87.32
✓
✓
✓
75.19
90.81
83.78
96.82
89.82
93.77
85.58
91.80
88.45
Appendix
Table 10: Component ablation of GEAR on commonsense reasoning with LLaMA3-8B. TGI denotes task-induced geometric initialization.
Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.
Yihao Ouyang, Shiwei Li, Haozhao Wang +5
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China
Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.
Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith
1Indian Institute of Technology Hyderabad · 2Deakin University
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large pretrained models. However, standard LoRA typically assigns a fixed rank profile despite layer- and stage-dependent adaptation needs, while recent adaptive-rank variants often rely on heuristic or first-order signals that provide only an indirect link between rank reallocation and local loss behavior. This leaves the rationale behind each rank reallocation insufficiently interpretable. Motivated by this gap, we propose LIRA (Loss-Informed Rank Allocation), a dynamic-rank LoRA method designed to make online rank reallocation more interpretable. LIRA scores each layer by the estimated magnitude of the local loss perturbation associated with its recent LoRA-factor movements under a second-order approximation, making the basis for each rank change interpretable term by term. By periodically applying DELTA (Displacement-Evaluated Loss-Trajectory Allocation), a displacement-based algorithm for LoRA rank reallocation, LIRA produces discrete effective ranks under matched budgets while remaining compatible with standard (A,B) adapters. On RoBERTa-base GLUE under the La-LoRA/LoRA protocol, LIRA outperforms the uniform-LoRA and adaptive-rank baselines reported under this protocol; Trajectory EMA attains performance competitive with Fixed-point Fisher at lower estimation cost.