Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large pretrained models. However, standard LoRA typically assigns a fixed rank profile despite layer- and stage-dependent adaptation needs, while recent adaptive-rank variants often rely on heuristic or first-order signals that provide only an indirect link between rank reallocation and local loss behavior. This leaves the rationale behind each rank reallocation insufficiently interpretable. Motivated by this gap, we propose LIRA (Loss-Informed Rank Allocation), a dynamic-rank LoRA method designed to make online rank reallocation more interpretable. LIRA scores each layer by the estimated magnitude of the local loss perturbation associated with its recent LoRA-factor movements under a second-order approximation, making the basis for each rank change interpretable term by term. By periodically applying DELTA (Displacement-Evaluated Loss-Trajectory Allocation), a displacement-based algorithm for LoRA rank reallocation, LIRA produces discrete effective ranks under matched budgets while remaining compatible with standard (A,B) adapters. On RoBERTa-base GLUE under the La-LoRA/LoRA protocol, LIRA outperforms the uniform-LoRA and adaptive-rank baselines reported under this protocol; Trajectory EMA attains performance competitive with Fixed-point Fisher at lower estimation cost.
Figures & tables
Method
Allocation unit
Importance signal
AdaLoRA
Singular-value triplets
Parameter–gradient sensitivity
IGU-LoRA
Singular-value triplets
Integrated gradients with uncertainty-aware smoothing
La-LoRA
Layer-wise rank
Norm growth / layer contribution
SoRA
Rank dimension
Sparse gates
L1RA
Rank dimension
L1 -regularized sparse gates
ALoRA
Rank dimension
Validation-time per-rank ablation
Table 1: Online rank-allocation methods, allocation units, and importance signals.
Figure 1: Six-task GLUE averages by rank-allocation signal. Groups are descriptive rather than an interpretability scale. Baselines are from La-LoRA ( Gu et al. 2026 ) .
Figure 2: LIRA overview. Standard masked LoRA updates (left) are interleaved every Kstep optimizer steps with a DELTA pass (right), which converts interval displacements into local loss-perturbation magnitudes that combine first- and second-order terms, rank-normalized scores Si(t) , and discrete ranks under B(t) .
Method
Active Params.
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
Full FT
125 M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
LoRA ( Hu et al. 2022 )
0.3 M
94.0
88.7
63.4
92.4
78.6
90.5
84.6
AdaLoRA ( Zhang et al. 2023 )
0.3 M
94.8
88.8
63.2
92.9
79.0
91.1
85.0
DoRA ( Liu et al. 2024a )
0.3 M
93.5
87.6
64.2
91.0
79.7
91.0
84.5
La-LoRA ( Gu et al. 2026 )
0.3 M
94.8
88.0
65.0
92.9
80.4
91.0
85.4
LIRA (ours)
0.3 M †
95.3
90.1
65.3
93.0
81.5
91.1
86.0
Table 2: RoBERTa-base GLUE development results. LIRA entries report five-seed means, and all values are shown to one decimal place; baseline values are taken from Table 5 of La-LoRA ( Gu et al. 2026 ) . Best results are boldfaced, and second-best results are underlined. † The reported count is the active forward-pass parameter budget, matched to uniform LoRA with effective rank r=8 ; LIRA additionally stores masked factors up to rmax per layer. The Avg. column is the arithmetic mean of the six task means. The complete main-results table is provided in the supplementary material.
Figure 3: Final effective-rank profiles after DELTA (Trajectory EMA; five-seed means) on SST-2 and RTE. Each bar represents the query or value adapter at one RoBERTa-base transformer layer; the dashed line marks the uniform baseline rbase=8 . Both tasks reallocate away from a uniform profile under the shared budget upper bound Lrbase , while the realized total effective rank may remain below this bound after discrete projection.
Task
Trajectory EMA
Fixed-point Fisher
SST-2
95.25
95.32
MRPC
90.05
90.29
CoLA
65.26
65.86
QNLI
92.96
93.01
RTE
81.52
80.94
STS-B
91.11
91.07
Table 3: Trajectory EMA and Fixed-point Fisher under the same local loss-perturbation score, effective-rank budget, and five-seed GLUE protocol. Values are five-seed means reported to two decimal places; best results are boldfaced, and task metrics follow Table 2 .
Variant
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
w/o 2nd-order term
94.89
89.56
64.50
92.63
80.14
90.80
85.42
w/o per-effective-rank averaging
94.91
89.61
64.21
92.71
78.70
90.91
85.17
LIRA (full)
95.25
90.05
65.26
92.96
81.52
91.11
86.03
Table 4: Ablation of LIRA’s key components. “w/o” removes the stated design; LIRA (full) retains all components. Values are means over five random seeds; best results are boldfaced; task metrics match Table 2 .