Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large pretrained models. However, standard LoRA typically assigns a fixed rank profile despite layer- and stage-dependent adaptation needs, while recent adaptive-rank variants often rely on heuristic or first-order signals that provide only an indirect link between rank reallocation and local loss behavior. This leaves the rationale behind each rank reallocation insufficiently interpretable. Motivated by this gap, we propose LIRA (Loss-Informed Rank Allocation), a dynamic-rank LoRA method designed to make online rank reallocation more interpretable. LIRA scores each layer by the estimated magnitude of the local loss perturbation associated with its recent LoRA-factor movements under a second-order approximation, making the basis for each rank change interpretable term by term. By periodically applying DELTA (Displacement-Evaluated Loss-Trajectory Allocation), a displacement-based algorithm for LoRA rank reallocation, LIRA produces discrete effective ranks under matched budgets while remaining compatible with standard (A,B) adapters. On RoBERTa-base GLUE under the La-LoRA/LoRA protocol, LIRA outperforms the uniform-LoRA and adaptive-rank baselines reported under this protocol; Trajectory EMA attains performance competitive with Fixed-point Fisher at lower estimation cost.
Figures & tables
Method
Allocation unit
Importance signal
AdaLoRA
Singular-value triplets
Parameter–gradient sensitivity
IGU-LoRA
Singular-value triplets
Integrated gradients with uncertainty-aware smoothing
La-LoRA
Layer-wise rank
Norm growth / layer contribution
SoRA
Rank dimension
Sparse gates
L1RA
Rank dimension
L1 -regularized sparse gates
ALoRA
Rank dimension
Validation-time per-rank ablation
Table 1: Online rank-allocation methods, allocation units, and importance signals.
Figure 1: Six-task GLUE averages by rank-allocation signal. Groups are descriptive rather than an interpretability scale. Baselines are from La-LoRA ( Gu et al. 2026 ) .
Figure 2: LIRA overview. Standard masked LoRA updates (left) are interleaved every Kstep optimizer steps with a DELTA pass (right), which converts interval displacements into local loss-perturbation magnitudes that combine first- and second-order terms, rank-normalized scores Si(t) , and discrete ranks under B(t) .
Method
Active Params.
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
Full FT
125 M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
LoRA ( Hu et al. 2022 )
0.3 M
94.0
88.7
63.4
92.4
78.6
90.5
84.6
AdaLoRA ( Zhang et al. 2023 )
0.3 M
94.8
88.8
63.2
92.9
79.0
91.1
85.0
DoRA ( Liu et al. 2024a )
0.3 M
93.5
87.6
64.2
91.0
79.7
91.0
84.5
La-LoRA ( Gu et al. 2026 )
0.3 M
94.8
88.0
65.0
92.9
80.4
91.0
85.4
LIRA (ours)
0.3 M †
95.3
90.1
65.3
93.0
81.5
91.1
86.0
Table 2: RoBERTa-base GLUE development results. LIRA entries report five-seed means, and all values are shown to one decimal place; baseline values are taken from Table 5 of La-LoRA ( Gu et al. 2026 ) . Best results are boldfaced, and second-best results are underlined. † The reported count is the active forward-pass parameter budget, matched to uniform LoRA with effective rank r=8 ; LIRA additionally stores masked factors up to rmax per layer. The Avg. column is the arithmetic mean of the six task means. The complete main-results table is provided in the supplementary material.
Figure 3: Final effective-rank profiles after DELTA (Trajectory EMA; five-seed means) on SST-2 and RTE. Each bar represents the query or value adapter at one RoBERTa-base transformer layer; the dashed line marks the uniform baseline rbase=8 . Both tasks reallocate away from a uniform profile under the shared budget upper bound Lrbase , while the realized total effective rank may remain below this bound after discrete projection.
Task
Trajectory EMA
Fixed-point Fisher
SST-2
95.25
95.32
MRPC
90.05
90.29
CoLA
65.26
65.86
QNLI
92.96
93.01
RTE
81.52
80.94
STS-B
91.11
91.07
Table 3: Trajectory EMA and Fixed-point Fisher under the same local loss-perturbation score, effective-rank budget, and five-seed GLUE protocol. Values are five-seed means reported to two decimal places; best results are boldfaced, and task metrics follow Table 2 .
Variant
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
w/o 2nd-order term
94.89
89.56
64.50
92.63
80.14
90.80
85.42
w/o per-effective-rank averaging
94.91
89.61
64.21
92.71
78.70
90.91
85.17
LIRA (full)
95.25
90.05
65.26
92.96
81.52
91.11
86.03
Table 4: Ablation of LIRA’s key components. “w/o” removes the stated design; LIRA (full) retains all components. Values are means over five random seeds; best results are boldfaced; task metrics match Table 2 .
Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.
Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith
1Indian Institute of Technology Hyderabad · 2Deakin University
Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose ℓp-LoRA, a principled rank-allocation method based on ℓp regularization with 0<p<1, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.
Zebang Xie, Chuanyang Zheng, Yik-Chung Wu +1
National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong +1
Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern - no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices only, which reduces memory cost by approximately 256x compared to full-model Fisher estimation. On GLUE with DeBERTa-v3-base, FIM-LoRA matches LoRA (88.6 vs. 88.7) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 vs. 68.7 for LoRA. The per-layer rank maps are interpretable: value projections and early-to-middle layers consistently receive higher rank, consistent with established findings on transformer layer roles.