LoRA's Second Descent Extends Beyond Parameter Parity

Yueran Ma

cs.LGOct 3, 2026

LoRA's Second Descent Extends Beyond Parameter Parity

Authors: Yueran Ma

Organizations: The University of Queensland Brisbane, Australia

Abstract

Double descent has sparked considerable interest, with recent work relating it to the data, the model and the learning configuration. Practical fine-tuning commonly involves training a small adapter on top of frozen pretrained weights, as in low-rank adaptation (LoRA). The adapter's rank is the hyperparameter that sets its capacity, yet how this rank relates to double descent has not been well explored. We quantify this relation under label noise on four vision backbones and a 7B language model with a module-matched rank sweep (MMRS), which extends past full rank and compares every rank with dense fine-tuning of the same modules, paired by seed. On DeiT-Tiny, risk is lowest at rank one and rises sharply as the adapter becomes able to fit the noisy labels, forming an interpolation cliff. Past the peak, risk falls again, but every tested post-peak rank that still saves parameters remains above dense risk. Rank-one LoRA outperforms dense fine-tuning on three of the four vision backbones, consistent with strong regularization at small rank. LoRA thus exhibits a second descent, but matches dense risk only after losing its parameter advantage, first on DeiT-Tiny at four times dense's projection weights. Code is available at https://anonymous.4open.science/r/lora-second-descent-C7B5.

Figures & tables

Figure 1: Two shapes, and why the reference point decides between them. (a) On a from-scratch capacity axis, risk peaks at interpolation and then descends past its own pre-peak valley, the only reference such an axis has. (b) On an adapter’s rank axis a second reference exists, dense fine-tuning of the same modules. Parity, the rank at which the adapter has as many trainable weights as dense fine-tuning, arrives before

r{=}d

, the smallest rank that can express every dense update, so the adapter stops saving parameters before it is as expressive as dense. Schematic; measurements in Figure 2 .

Figure 2: The interpolation cliff and what it costs to leave. (a) Test NLL vs. LoRA rank for DeiT-Tiny on CIFAR-100 (

5

k subset), mean

\pm

std over

5

seeds,

40

epochs. (b) DeiT-Tiny, ViT-Tiny and Qwen2.5-7B under the shared learning rate, as paths in the plane of the trainable-parameter ratio

\pi(r)

and the paired gap

G(r)

to dense fine-tuning of the same query-key-value (qkv) projections (both defined in Section 4 ). Dots mark

r{=}1

, the last sampled rank below parity,

r{=}d

and the largest rank;

3d

marks where DeiT-Tiny first passes the equivalence test of Section 5.3 .

Figure 3: One rank, two references. Qwen2.5-7B, three-seed means (Appendix C.3 ).

		DeiT-Tiny		ViT-Tiny
adapter rank	params	test NLL	paired diff.	test NLL	paired diff.
dense qkv, $k{=}6$	$1.00\times$	$1.979$	—	$1.802$	—
$r=1$ (valley)	$0.007\times$	$1.571$	$-0.400$ $[-.404,-.397]$	$1.537$	$-0.272$ $[-.274,-.271]$
$r=64$ / $96$ (peak)	$0.44\times$ / $0.67\times$	$2.102$	—	$1.902$	—
$r=128$ (below parity)	$0.89\times$	$2.089$	$+0.118$ $[+.100,+.134]$	$1.885$	$+0.089$ $[+.076,+.102]$
$r=d$ (full rank)	$1.33\times$	$2.069$	$+0.092$ $[+.083,+.106]$	$1.871$	$+0.061$ $[+.047,+.074]$

Table 1: Post-peak test NLL and the gap to dense fine-tuning. We compare LoRA ranks up to

4d

with dense fine-tuning of the same qkv projections in the same six blocks, under the shared learning rate; the arms share seeds, subsets and noise draws. Params is the trainable projection-weight count relative to dense, so parity falls at

r{=}0.75d

, and

r{=}128

is one of the post-peak ranks sampled below it (Appendix C.1 ). Paired diff. is the mean per-seed difference, with a

95\%

bootstrap interval over five or eight shared seeds and a descriptive three-seed range otherwise. Dense is the eight-seed mean; its five- and three-seed means are

1.971

1.977

on DeiT-Tiny and

1.796

1.810

on ViT-Tiny.

Figure 4: Risk changes with optimization coordinates. We compare ordinary LoRA and three interventions on DeiT-Tiny over the same three seeds. The ViT-Tiny results are in Appendix D .

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

block	purpose	sweep / setting	seeds	where
E1	main rank sweep	$r\in\{0,\dots,192\}$ incl. $96,128$ , $\eta\in\{0,20\%\}$	5	§ 3
E2	budget interaction	$r\times$ epochs $\{20,80\}$	3	§ 5.6
E7	high-rank, long-time tail	$r\in\{64,192,384\}$ , epochs $\{80,160\}$	5	§ 5.6
E9	$\alpha/r$ scaling to interpolation	$r\in\{8,32,192\}$ , epochs $\{80,160\}$	5	§ 5.6
E5	from-scratch control	ResNet-18 widths $\{2,\dots,32\}$ , CIFAR-10 $12.5$ k, $120$ ep	3	§ 3
E8	from-scratch, identical data	widths $\{2,\dots,16\}$ , CIFAR-100 $5$ k, $160$ ep	3	§ 3

Appendix

Table 2: Experiment block summary. Seed counts refer to full sweeps of the listed axis. A dash in the last column marks a block whose results the paper does not report.

Figure 5: Three thresholds separate. Test NLL (orange, left axis) and train accuracy (gray, right axis) vs. rank,

20\%

noise. The counting threshold

r_{\mathrm{cap}}

is passed at

r{=}0

(head alone has

19{,}300>n=5{,}000

parameters) yet train accuracy is only

67\%

there; operational interpolation arrives only at

r_{\mathrm{interp}}=24

; test risk peaks later still at

r_{\mathrm{peak}}=64

Figure 6: The tail remains far above the valley. (a) Rank sweeps at

20/40/80

epochs (

r\geq 8

shown); the peak tracks whichever ranks are currently reaching interpolation, and the far tail dips but remains high. (b) High-rank/long-time tail under both scalings (mean

\pm

std,

5

seeds); mild improvement with rank at fixed budget, degradation with time, and a persistent

\geq 0.62

-nat gap to the

r{=}1

valley (dotted).

Figure 7: The same pipeline, swept from scratch, completes the second descent. ResNet-18 width sweeps on noisy CIFAR-10 and on the identical CIFAR-100 subsets the fine-tuning uses, each divided by its own pre-peak valley risk so the two NLL scales become comparable. Both peak in the interior and finish below the valley, which the DeiT-Tiny rank axis does not do even at

4d

backbone	rank	seeds	LoRA NLL	dense NLL	paired diff.	signs
DeiT-Tiny	$64$ (peak)	5	$2.102$	$1.971$	$+0.131$ $[+.108,+.149]$	$+++++$
	$80$	5	$2.111$	$1.971$	$+0.140$ $[+.116,+.161]$	$+++++$
	$96$	5	$2.105$	$1.971$	$+0.134$ $[+.114,+.150]$	$+++++$
	$112$	5	$2.099$	$1.971$	$+0.128$ $[+.107,+.151]$	$+++++$
	$128$	5	$2.089$	$1.971$	$+0.118$ $[+.100,+.134]$	$+++++$
	$140$	5	$2.080$	$1.971$	$+0.109$ $[+.100,+.123]$	$+++++$

Appendix

Table 3: Gap to module-matched dense fine-tuning under the shared learning rate. Signs list the per-seed paired differences,

0

where a pair ties. DeiT-Tiny’s peak is the maximum over the main-sweep grid of Table 8 ; on this denser grid its

r{=}80

and

96

lie

0.009

and

0.003

nats higher, inside the seed spread (Appendix C.4 ). DeiT-Tiny’s

r{=}d

row pairs the five E1 seeds with dense; Table 1 pairs the three E30 seeds, giving

+0.092

backbone	arm	params	best rate	NLL	minus best dense
DeiT-Tiny	dense qkv	$1.00\times$	$1.0{\times}10^{-3}$	$1.948$	—
	$r{=}96$	$0.67\times$	$2.0{\times}10^{-3}$	$1.968$	$+0.019$ , $++-$
	$r{=}128$	$0.89\times$	$2.0{\times}10^{-3}$	$1.945$	$-0.003$ , $++-$
	$r{=}128$ , fresh	$0.89\times$	$2.0{\times}10^{-3}$	$2.009$	$+0.062$ $[+.026,+.099]$ , 9 of 10
	$r{=}140$	$0.97\times$	$1.4{\times}10^{-3}$	$1.952$	$+0.004$ , $++-$
	$r{=}d$	$1.33\times$	$1.0{\times}10^{-3}$	$1.987$	$+0.039$ , $+++$

Appendix

Table 4: Each arm at its own learning rate. Best rate and final test NLL on the three seeds used to select it, and the paired difference from dense at dense’s best rate, with per-seed signs. The row marked fresh contains re-runs of the selected DeiT-Tiny pair on ten seeds excluded from selection, with a

95\%

t

-interval. Train accuracy is

100\%

at every best rate except DeiT-Small

r{=}1

(

77\%

) and the three Swin-Tiny arms (dense

90\%

r{=}1

72\%

r{=}384

79\%

axis	variable	valley	peak	end	gap	$\mathrm{Rec}@d$	$\mathrm{Rec}@\mathrm{end}$	$\mathrm{Rec}@\mathrm{prev}$
LoRA rank, DeiT-Tiny (E1, E30, E33)	$r$	$1.571$ ( $1$ )	$2.102$ ( $64$ )	$1.953$ ( $4d$ )	$0.531$	$0.06$	$0.28$	$0.22$
LoRA rank, ViT-Tiny (E26, E35)	$r$	$1.537$ ( $1$ )	$1.902$ ( $96$ )	$1.836$ ( $4d$ )	$0.365$	$0.09$	$0.18$	$0.16$
LoRA rank, DeiT-Small (E12)	$r$	$1.396$ ( $1$ )	$1.529$ ( $24$ )	$1.363$ ( $2d$ )	$0.133$	$0.99$	$1.25$	$0.99$
LoRA rank, CIFAR-100N human noise (E14)	$r$	$1.743$ ( $1$ )	$2.408$ ( $64$ )	$2.350$ ( $2d$ )	$0.664$	$0.04$	$0.09$	$0.04$
LoRA rank, Qwen2.5-7B (E27)	$r$	$0.483$ ( $1$ )	$0.538$ ( $64$ )	$0.455$ ( $d$ )	$0.055$	$1.50$	$1.50$	$1.82$
Dense blocks (E11)	$k$	$1.709$ ( $0$ )	$2.288$ ( $1$ )	$1.814$ ( $6$ )	$0.579$	—	$0.82$	$0.57$

Appendix

Table 5: The three points behind every

\mathrm{Rec}

outside Appendix E.3 . Seed-mean test NLL at the valley, the peak and the last point measured, with the grid position in parentheses; gap is peak minus valley.

\mathrm{Rec}@d

reads a rank axis at its expressivity limit and

\mathrm{Rec}@\mathrm{end}

at the last point measured; the two coincide where the sweep stops at

d

\mathrm{Rec}@\mathrm{prev}

reads each axis at the grid point before its end. Dense blocks and dense qkv end at

k{=}6

, the whole adapted stack, which is the expressivity endpoint their

\mathrm{Rec}

shares with

\mathrm{Rec}@d

on the rank axes. Swin-Tiny (E26) and GPT-2 (E15) rise to the last rank measured and have no far side to read.

		DeiT-Tiny		ViT-Tiny
training coordinates	$\kappa(A)$	test NLL	$\mathrm{srank}$	test NLL	$\mathrm{srank}$
Dense qkv	—	$1.9768$	$29.2$	$1.8097$	$22.4$
AdamW on $B$	$1$	$1.9716$	$26.3$	$1.8073$	$19.4$
AdamW on $B$	$3$	$2.0057$	$22.4$	$1.8385$	$17.4$
AdamW on $B$	$5$	$2.0367$	$19.9$	$1.8628$	$16.2$
Lifted AdamW	$1$	$1.9789$	$29.0$	$1.8113$	$22.2$

Appendix

Table 6: Frozen-factor interventions at

r{=}2d

on two backbones. Entries are mean test NLL and mean stable rank

\mathrm{srank}

of the update over the same three seeds, using lr

5\times 10^{-4}

and

40

epochs. All frozen-factor arms train

B

and the head.

Figure 8: Left (effective complexity lags nominal rank). Stable rank of the learned update vs. nominal rank (

20\%

noise,

40

ep); growth is sublinear everywhere and reaches only

11

r{=}192

(

y=x

dotted). Right (what organizes risk). All

175

noisy CIFAR-100 LoRA runs of the fixed-setting sweeps (E1/E2/E7/E9/K1/K2); test NLL against nominal rank (panel a, Spearman

\rho{=}0.40

) and against train NLL (panel b,

\rho{=}{-}0.86

, axis reversed, tie-corrected). Training NLL predicts test risk more closely than nominal rank does.

Figure 9: Dense axes recover more of the same cliff. Test NLL vs. trainable parameters (

20\%

noise,

40

ep) along three capacity axes, namely LoRA rank (orange), dense fine-tuning of the last

k

blocks (blue), and dense fine-tuning of only the qkv projections of the last

k

blocks (green, module-matched to LoRA). All three climb a cliff once they interpolate; the dense axes then recover much of it, the low-rank axis almost none by

r{=}d

, and none reaches the

r{=}1

valley (dotted).

Figure 10: Breadth checks (mean

\pm

std,

3

seeds). (a) Real human label noise (CIFAR-100N,

{\approx}40\%

); the cliff replicates with larger amplitude and reaches

\mathrm{Rec}=0.04

d

(

0.09

2d

; Table 13 ). (b) Bottleneck adapters; the same cliff along the adapter-width axis, peaking at

m{=}256

and reaching only

\mathrm{Rec}=0.02

m{=}512

. (c) GPT-2 LoRA on noisy AG News; risk rises past the

r{=}8

valley, and full rank, the only interpolating point, is also the largest capacity on the axis.

Figure 11: Noise rate and augmentation. (a) Test NLL vs. rank at

\eta\in\{10,20,40,60\}\%

; the valley remains at

r{=}1

and the cliff amplitude grows monotonically with the noise rate. (b) Augmentation (per-image flip, batch-shared pad-crop, mixup) lowers the curve and shrinks the cliff without removing it; the augmented curve shows no post-peak descent.

Figure 12: The cliff persists through design perturbations. Test NLL vs. rank (

20\%

noise unless noted) under learning rates

2\times 10^{-4}

and

10^{-3}

, standard

\alpha/r

40

ep (flat only because the high ranks have barely interpolated; Figure 6 b trains them longer), rsLoRA, MLP placement at matched parameter budget (effective ranks

\{2,13,77\}

), and CIFAR-10.

rank	sel. lr	sel. wd	sel. epoch	train acc (sel., %)	val NLL	test NLL (mean $\pm$ std)
$1$	$5\times 10^{-4}$	$0.1$	$12$	$67$	$1.544$	$1.512\pm 0.009$
$8$	$5\times 10^{-4}$	$0.01$	$10$	$70$	$1.552$	$1.531\pm 0.008$
$24$	$10^{-3}$	$0.1$	$4$	$75$	$1.562$	$1.539\pm 0.014$
$64$	$10^{-3}$	$0.1$	$3$	$86$	$1.618$	$1.597\pm 0.004$
$192$	$5\times 10^{-4}$	$0.1$	$3$	$86$	$1.682$	$1.657\pm 0.004$
$384$	$5\times 10^{-4}$	$0.1$	$3$	$94$	$1.694$	$1.672\pm 0.008$

Appendix

Table 7: Per-rank envelope (E24). Each rank’s validation-selected learning rate and weight decay, with the stopping epoch, best validation NLL, and test NLL over three fresh seeds held out of the selection (epoch, train acc, and val NLL are seed means; test NLL is mean

\pm

std). Test NLL is read at each seed’s own best-validation epoch. Train accuracy is that of the selected checkpoint, read at the nearest logged epoch (the train history is logged every

5

epochs, so within

3

epochs of the selected one); the final-epoch (

80

) train accuracy is

\geq 99\%

for every rank except

r{=}1

(

88\%

Figure 13: Weaker priors deepen the cliff. Backbone corruption

W\leftarrow\alpha W_{\mathrm{pre}}+(1-\alpha)W_{\mathrm{rand}}

on the adapted blocks (

20\%

noise,

40

ep,

3

seeds;

5

\alpha{=}1.0

) raises the peak and deepens the cliff monotonically. Peak-to-valley over the shared grid

r\in\{4,24,64,192\}

0.46

0.49

0.61

0.76

nats at

\alpha=1.0

0.75

0.5

0.0

. No setting recovers its pre-peak minimum.

Figure 14: Warm-start linear model (

p{=}2000

n{=}200

; mean

\pm

std over

16

trials; the min-norm divergence at

k{=}n

is clipped). (a) With a strong prior (

\|\delta\|{=}0.5

), the plateau rises with the noise level while the valley remains anchored near

\|\delta\|^{2}

; the cliff amplitude grows with

\sigma^{2}

and nearly vanishes at low noise, as in the experiments (Figures 2 a, 11 a). (b) Degrading the prior at fixed noise morphs the cliff continuously into canonical double descent; for

\|\delta\|\lesssim\sigma

the interpolating branch never reaches the valley, and for

\|\delta\|\gtrsim\sigma

the second descent ends below it.

	$20\%$ noise			clean
$r$	test NLL	test acc (%)	train acc (%)	test NLL	test acc (%)	train acc (%)
0	$1.709{\scriptstyle\pm.008}$	$57.1$	$66.5$	$1.382{\scriptstyle\pm.008}$	$61.2$	$85.2$
1	$1.571{\scriptstyle\pm.006}$	$60.4$	$74.6$	$1.186{\scriptstyle\pm.014}$	$66.4$	$94.8$
4	$1.640{\scriptstyle\pm.017}$	$58.7$	$81.5$	$1.229{\scriptstyle\pm.014}$	$65.8$	$98.3$
8	$1.743{\scriptstyle\pm.016}$	$57.1$	$88.7$	$1.273{\scriptstyle\pm.017}$	$65.6$	$99.7$
16	$1.920{\scriptstyle\pm.028}$	$54.6$	$98.0$	$1.363{\scriptstyle\pm.019}$	$65.3$	$100$

Appendix

Table 8: Main sweep (E1), test NLL and accuracy vs. rank; mean

\pm

std over

5

seeds,

40

epochs. The rows

2d

–

4d

are the noisy-label tail runs of Table 1 , over

3

8

and

3

seeds. E1’s

r{=}96

and

128

runs appear with the densified window in Table 3 .

configuration	trainable params	test NLL	train acc (%)
head only ( $k=0$ )	$19{,}300$	$1.709{\scriptstyle\pm.008}$	$66.5$
E11 dense blocks $k=1$	$464{,}164$	$2.288{\scriptstyle\pm.013}$	$100$
E11 dense blocks $k=2$	$909{,}028$	$2.206{\scriptstyle\pm.018}$	$100$
E11 dense blocks $k=4$	$1{,}798{,}756$	$1.958{\scriptstyle\pm.023}$	$100$
E11 dense blocks $k=6$	$2{,}688{,}484$	$1.814{\scriptstyle\pm.020}$	$100$
E11 dense blocks $k=6$ , $80$ ep	$2{,}688{,}484$	$1.961{\scriptstyle\pm.006}$	$100$

Appendix

Table 9: Dense capacity axes (

20\%

noise,

40

epochs unless noted; mean

\pm

std over

5

seeds,

2

for the

80

-epoch rows). E11 unfreezes the last

k

blocks entirely; E13 unfreezes only their qkv projections (module-matched to LoRA). E13’s

k{=}6

arm is a separate run, on five of the same seeds, of the configuration that Table 1 reports over eight seeds (

1.979

) and Section 5.4 over the three shared with the intervention arms (

1.977

$r$	1	24	64	192	384	768
test NLL	$1.396{\scriptstyle\pm.011}$	$1.529{\scriptstyle\pm.038}$	$1.522{\scriptstyle\pm.040}$	$1.450{\scriptstyle\pm.038}$	$1.397{\scriptstyle\pm.040}$	$1.363{\scriptstyle\pm.017}$
train acc (%)	$91.7$	$100$	$100$	$100$	$100$	$100$

Appendix

Table 10: DeiT-Small rank sweep (E12),

20\%

noise,

40

epochs, mean

\pm

std over

5

seeds (

3

r{=}768

). The sweep runs to

2d

; the axis is back at its valley by

d{=}384

, this model’s expressivity limit (

\mathrm{Rec}=0.99

), and below it at

2d

in every seed (

\mathrm{Rec}=1.25

	E7 (fixed)			E9 ( $\alpha/r$ )
epochs	$r{=}64$	$r{=}192$	$r{=}384$	$r{=}8$	$r{=}32$	$r{=}192$
80	$2.351{\scriptstyle\pm.014}$	$2.269{\scriptstyle\pm.020}$	$2.192{\scriptstyle\pm.014}$	$2.378{\scriptstyle\pm.023}$	$2.283{\scriptstyle\pm.030}$	$2.249{\scriptstyle\pm.031}$
160	$2.676{\scriptstyle\pm.103}$	$2.538{\scriptstyle\pm.131}$	$2.460{\scriptstyle\pm.089}$	$2.999{\scriptstyle\pm.070}$	$2.756{\scriptstyle\pm.027}$	$2.688{\scriptstyle\pm.046}$

Appendix

Table 11: High-rank/long-time tail (E7, fixed scale; E9,

\alpha/r

scale),

20\%

noise, test NLL, mean

\pm

std over

5

seeds.

$r$	NLL raw	NLL calibrated	$T$	ECE	Brier	AURC
1 (valley)	$1.571{\scriptstyle\pm.006}$	$1.535{\scriptstyle\pm.007}$	$0.845{\scriptstyle\pm.010}$	$.080{\scriptstyle\pm.004}$	$.542{\scriptstyle\pm.004}$	$.186{\scriptstyle\pm.004}$
64 (peak)	$2.102{\scriptstyle\pm.020}$	$1.979{\scriptstyle\pm.018}$	$1.346{\scriptstyle\pm.014}$	$.130{\scriptstyle\pm.003}$	$.628{\scriptstyle\pm.004}$	$.230{\scriptstyle\pm.005}$
192	$2.069{\scriptstyle\pm.023}$	$1.932{\scriptstyle\pm.019}$	$1.373{\scriptstyle\pm.012}$	$.137{\scriptstyle\pm.004}$	$.616{\scriptstyle\pm.006}$	$.214{\scriptstyle\pm.002}$
clean, 1	$1.186{\scriptstyle\pm.014}$	$1.177{\scriptstyle\pm.013}$	$1.109{\scriptstyle\pm.005}$	—	—	—
clean, 192	$1.519{\scriptstyle\pm.014}$	$1.344{\scriptstyle\pm.012}$	$1.471{\scriptstyle\pm.013}$	—	—	—

Appendix

Table 12: Calibration and uncertainty at key ranks (

20\%

noise unless noted), mean

\pm

std over

5

seeds. Temperature fit on a held-out clean train split; test labels untouched. ECE is expected calibration error and AURC the area under the risk–coverage curve.

configuration	test NLL	test acc (%)	train acc (%)
E14 $r=0$	$1.863{\scriptstyle\pm.003}$	$50.6$	$65.7$
E14 $r=1$ (valley)	$1.743{\scriptstyle\pm.017}$	$52.9$	$72.7$
E14 $r=8$	$1.913{\scriptstyle\pm.010}$	$50.5$	$89.1$
E14 $r=24$	$2.240{\scriptstyle\pm.025}$	$48.2$	$99.9$
E14 $r=64$ (peak)	$2.408{\scriptstyle\pm.020}$	$48.0$	$100$
E14 $r=192$	$2.378{\scriptstyle\pm.030}$	$49.5$	$100$

Appendix

Table 13: Breadth checks, per configuration (mean

\pm

std,

3

seeds). The top block is CIFAR-100N real human noise (E14), the middle block is bottleneck adapters (E16) under

20\%

synthetic noise, and the bottom block is GPT-2 LoRA on AG News with

20\%

noise (E15).

configuration	test NLL	test acc (%)	train acc (%)
ViT-Tiny $r=0$	$1.575{\scriptstyle\pm.002}$	$60.0$	$77.9$
ViT-Tiny $r=1$ (valley)	$1.537{\scriptstyle\pm.009}$	$61.2$	$83.9$
ViT-Tiny $r=24$	$1.899{\scriptstyle\pm.017}$	$57.4$	$100$
ViT-Tiny $r=96$ (peak)	$1.902{\scriptstyle\pm.039}$	$58.7$	$100$
ViT-Tiny $r=192$	$1.871{\scriptstyle\pm.018}$	$59.4$	$100$
Swin-Tiny $r=0$	$1.483{\scriptstyle\pm.010}$	$63.9$	$77.5$

Appendix

Table 14: Backbone external validity (E26), per configuration (mean

\pm

std,

3

seeds), CIFAR-100

5

20\%

noise,

40

epochs. ViT-Tiny uses the AugReg ImageNet-21k

\to

ImageNet-1k checkpoint ( Steiner et al., 2022 ) with its native

0.5

normalization; on Swin-Tiny, LoRA adapts the last six blocks (stage dims

384/768

), so every adapted block reaches full rank by

r{=}768

. On Swin every

r\geq 1

interpolates and accuracy remains above its

r{=}1

level while NLL climbs.

$n$	$r{=}1$	$r{=}8$	$r{=}24$	$r{=}64$	$r{=}192$
$2{,}500$	$1.775$	$1.880$	$2.076$	$2.185$	$2.172$
$5{,}000$	$1.571$	$1.743$	$2.001$	$2.102$	$2.069$
$10{,}000$	$1.433$	$1.645$	$2.085$	$2.168$	$2.086$

Appendix

Table 15: Data-size sweep (E10),

20\%

noise,

40

epochs, test NLL. The

n{=}5{,}000

row is the

5

-seed main sweep; the other two rows average

2

seeds.

Double Descent

Double descent describes non-monotonic test error that can peak near interpolation and fall again as model capacity increases.

See all 20 papers

May 8, 2026

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning

With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank

r

requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient. In this paper, we propose MatryoshkaLoRA, a general, Matryoshka-inspired training framework for LoRA that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix

P

between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing

P

and ensures all sub-ranks embed the available gradient information efficiently. Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy-performance trade-offs across ranks on the evaluated datasets. Our code is available at https://github.com/IST-DASLab/MatryoshkaLoRA.

Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh

Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA

Adapter Tuning LLM Fine-Tuning Hierarchical Representation Learning Nonlinear LoRA Low-Rank Adaptation Parameter-Efficient Fine-Tuning+1

Ask PDF

Apr 30, 2026

Post-Optimization Adaptive Rank Allocation for LoRA

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.

Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith

1Indian Institute of Technology Hyderabad · 2Deakin University

Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Neural Network Pruning

Ask PDF

Oct 14, 2026

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://github.com/Fanyao-Li/ReGRA.

Fanyao Li, Yuxi Liu, Xiaohao Liu +3

Communication University of China · National University of Singapore · Xidian University

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Double Descent

Explore similar work

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning

Post-Optimization Adaptive Rank Allocation for LoRA

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning