A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need

Hananel Hazan; Yanbo Zhang; Benedikt Hartl; Michael Levin

cs.LGApr 9, 2026

A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need

Authors: Hananel Hazan, Yanbo Zhang, Benedikt Hartl, Michael Levin

Organizations: Allen Discovery Center at Tufts University, Medford, Massachusetts, USA · Institute for Theoretical Physics, TU Wien, Wien, Austria · Wyss Institute for Biologically Inspired Engineering, Harvard University, Boston, Massachusetts, USA

Abstract

How many of a neural network's parameters actually encode task-specific information? We investigate this question with LottaLoRA, a training paradigm in which every backbone weight is drawn at random and frozen; only low-rank LoRA adapters are trained. Across nine benchmarks spanning diverse architecture families from single-layer classifiers to 900M parameter Transformers low-rank adapters over frozen random backbones recover 96-100% of fully trained performance while training only 0.5-40% of the parameters. The task-specific signal therefore occupies a subspace orders of magnitude smaller than the full parameter count suggests. Three mechanistic findings underpin this result:(1) the frozen backbone is actively exploited when static the learned scaling~ $β$ remains strictly positive across all architectures but when the scaffold is destabilized, the optimizer silences it and the LoRA factors absorb all task information; (2) the frozen backbone is preferable but interchangeable any random initialization works equally well, provided it remains fixed throughout training; and (3) the minimum LoRA rank at which performance saturates estimates the intrinsic dimensionality of the task, reminiscent of the number of components retained in Principal Component Analysis (PCA). The construction is formally analogous to Reservoir Computing unfolded along the depth axis of a feedforward network. Because the backbone is determined by a random seed alone, models can be distributed as adapters plus seed a footprint that grows with task complexity, not model size, so that storage and memory savings compound as architectures scale.

Figures & tables

Figure 1: LottaLoRA replaces pre-trained weights with seeded reservoirs. Three parameterization strategies for a single network layer. (a) A conventional dense layer stores and trains all

m\times n

weights. (b) Low-Rank Adaptation (LoRA) freezes a pre-trained weight matrix

W_{0}

and learns only two small factors

A\in\mathbb{R}^{m\times r}

and

B\in\mathbb{R}^{r\times n}

; the stored parameters are

W_{0}

plus the adapters. (c) LottaLoRA (this work) replaces

W_{0}

with a random matrix

W_{\text{seed}}

generated from a fixed seed, so the backbone is never pre-trained and need not be stored: only the seed and the low-rank factors

A

B

are saved. The distributable footprint therefore shrinks from the full weight matrix to a single integer plus two rank-

r

factors.

Concept	RC (time)	LottaLoRA (depth)
Propagation axis	Time step $t$	Layer index $l$
System state	Hidden state $h_{t}$	Layer activation $h_{l}$
Fixed dynamics	Reservoir $W_{R}$	Frozen backbone $\beta W_{\text{seed}}$
Injection matrix	Input matrix $U$	Scaled up-projection $(\alpha/r)\,B$
Drive signal	External input $x_{t}$	Projection $z_{l}=A\,h_{l}$
Trainable output	Readout $C$	Head, embeddings, LoRA

Table 1: LottaLoRA maps onto Reservoir Computing through a depth–time correspondence.

	Test Accuracy (%)
Rank	$W_{\text{seed}}=0$	Normal- $W_{\text{seed}}$	No- $W_{\text{seed}}$
1	$82.9\pm 4.9$	85.1–89.3	$82.6\pm 5.6$
2	$91.0\pm 1.9$	90.6–92.7	$90.8\pm 2.1$
4	$94.5\pm 1.0$	93.6–95.4	$94.4\pm 1.0$
8	$96.3\pm 0.5$	95.4–96.8	$96.3\pm 0.6$
Full	98.1–98.4

Table 2: MNIST test accuracy (%) under three backbone conditions. Mean over 5 seeds

\pm

std, ReLU activation, ranks 1/2/4/8. Normal-

W_{\text{seed}}

numbers are from the main evaluation protocol (30 runs, range shows min–max across seeds); zero-

W_{\text{seed}}

and no-backbone from the supplementary protocol (15 runs each, ReLU; see Appendix B ). Rank 1 falls below the

{\sim}85\%

linear-classifier threshold for zero-

W_{\text{seed}}

and no-backbone, indicating insufficient capacity for nonlinear structure at that rank.

Figure 2: MNIST: LottaLoRA accuracy scales monotonically with LoRA rank, closing the gap to fully trained baselines. (A) Accuracy scales monotonically with LoRA rank across three model sizes, closing the gap to fully trained baselines (dashed); the medium preset (4 layers, widths 512–64; see Appendix B for all presets) reaches 96.8% at rank 8 with only 3.65% of the parameters of the fully trained counterpart. (B) Parameter efficiency shows that each LottaLoRA configuration (circles, squares, triangles) attains comparable accuracy to its fully trained counterpart (diamonds) using one to two orders of magnitude fewer trainable parameters. (C) All 22 $W_{\text{seed}}$ initialization families (aggregated across tiny, small, and medium presets) exceed 95% at rank 8, indicating that on MNIST the specific distribution has negligible effect on performance. (D) Accuracy-vs-rank curves for all 22 families converge tightly, showing that the scaffold is interchangeable on this task.

Figure 3: A single shared adapter produces seed-gated task specialization with out-of-class rejection. One LoRA adapter is trained across three disjoint MNIST label partitions ( $\{1,2,3\}$ , $\{4,5,6\}$ , $\{7,8,9\}$ ), each paired with a distinct backbone seed $s$ . Columns show seeds 42, 43, 44; cells show row-normalized test accuracy (%); black rectangles mark assigned classes; dashed orange columns highlight digit 0 (excluded from all partitions). Top row (Experiment 1, 10 classes): each seed activates 97–99% accuracy on its assigned digits while non-assigned digits are confidently misclassified into the in-partition classes. Bottom row (Experiment 2, out-of-class): non-assigned digits are mapped to an explicit out-of-class label (OOC), achieving 93–99% accuracy on assigned digits and 95–99% OOC classification on non-assigned digits. Digit 0 is consistently mapped to OOC under all seeds. Configuration: medium preset, rank 4, 150 training epochs.

Rank	Mean (%)	Range (%)	Spread (pp)
Full	98.03	97.91–98.21	0.30
$r{=}2$	90.88	90.84–91.03	0.19
$r{=}4$	94.45	94.33–94.48	0.14
$r{=}8$	96.29	96.24–96.44	0.19

Table 3: MNIST test classification accuracy (%) across 22

W_{\text{seed}}

initialization families. Each family is evaluated over 15 seeds (3 architecture sizes—tiny, small, medium—

\times

5 seeds; see Appendix B for details). Range shows worst-to-best family mean.

Schedule	$r{=}2$	$r{=}4$	$r{=}8$
Standard (static)	92.61	95.53	96.80
Epoch	67.96	89.00	95.53
Batch ( $k{=}2$ )	42.46	54.94	91.19
Microbatch ( $k{=}4$ )	41.49	47.67	56.29

Table 4: Resampling the scaffold during training degrades performance progressively with resampling frequency (MNIST, medium preset). Standard = static

W_{\text{seed}}

Rank	AUROC	%	%Trainable
$r{=}1$	$0.832\pm 0.005$	3,482	3.7%
$r{=}2$	$0.835\pm 0.005$	5,292	5.7%
$r{=}4$	$0.834\pm 0.002$	8,912	9.6%
$r{=}8$	$0.833\pm 0.001$	16,152	17.4%
Full	$0.836\pm 0.002$	92,930	100%

Table 5: CfC reservoir on PhysioNet 2012 ICU mortality (

h{=}256

, backbone 64 units, 2 layers): AUROC and trainable parameter count by LoRA rank. LottaLoRA uses a frozen random CfC backbone; Full CfC is fully trained from scratch. Mean

\pm

std over 5 seeds. %Trainable is relative to the Full CfC parameter count (92,930).

Scale	$h$	Rank	AUROC	Recovery
$0.125\times$	32	Full	$0.831\pm 0.003$	—
		$r{=}1$	$0.778\pm 0.037$	93.7%
		$r{=}4$	$0.829\pm 0.005$	99.8%
$0.25\times$	64	Full	$0.832\pm 0.002$	—
		$r{=}1$	$0.822\pm 0.009$	98.8%
		$r{=}2$	$0.831\pm 0.002$	100.0%

Table 6: Reducing CfC capacity shifts rank saturation rightward. AUROC (mean

\pm

std, 5 seeds) at each size scale and LoRA rank. At

0.125\times

, rank 1 falls to

93.7\%

recovery; rank 4 restores

99.8\%

. Fully trained baselines (“Full”) improve only 0.831–0.836 across a

42\times

parameter range.

Figure 4: LottaLoRA AUROC saturates at rank 2 on PhysioNet 2012 ICU mortality. Mean $\pm$ std AUROC over 5 seeds at ranks 1–32. Dashed line: fully trained CfC baseline (0.836). Rank 1 recovers 99.5% of baseline with 3.7% of trainable parameters.

Figure 5: Overparameterization masks rank saturation on CfC PhysioNet. (a) LottaLoRA recovery (normalized to each scale’s fully trained baseline) vs. LoRA rank at four CfC sizes. At

0.125\times

(

h{=}32

, 2,210 parameters), rank 1 recovers only 93.7% and saturation shifts to

r{=}4

; at larger scales rank 1 already exceeds 98.8%. (b) Fully trained baselines improve only from 0.831 to 0.836 across a

42\times

parameter increase, confirming that the published CfC architecture is far larger than this task requires. See Appendix C for the full overparameterization analysis.

Benchmark	Rank	Metric Test	LottaLoRA	Baseline	#seeds	% trainable
CIFAR-10 Plain-20	16	acc	$87.70\pm 0.26$ %	$90.81\pm 0.20$ %	3	38.3%
OGBG-MolHIV (GIN)	16	ROC-AUC	$0.7562\pm 0.0158$	$0.7755\pm 0.0060$	5	10.9%
OGBN-Arxiv (GCN)	32	acc	$70.11\pm 0.21$ %	$71.86\pm 0.22$ %	10	35.6%
DT (small)	32	Val MSE $\downarrow$	$0.0271\pm 0.0017$	$0.0272\pm 0.0017$	5	37.5%
DT (large)	8	Val MSE $\downarrow$	$0.0285\pm 0.0015$	$0.0281\pm 0.0019$	5	1.87%
ViT Flowers-102	1	Top-1 acc	$94.53\pm 0.35$ %	$96.83\pm 0.08$ %	5	1.19%

Table 7: LottaLoRA is competitive across diverse architectures with 0.5–40% of trainable parameters. All comparisons use matched-protocol training; IMDB and ViT use pre-trained backbones, others train from scratch. For DT, lower MSE is better. See appendices for full details per benchmark.

Scale	Full loss	LottaLoRA $r{=}8$ loss	Ratio (%)	$\Delta$
3M	5.007	5.639	7.68	+0.63
30M	3.757	4.681	1.39	+0.92
300M	3.252	4.191	0.51	+0.94
600M	3.185	4.058	0.40	+0.87
900M	3.156	3.950	0.31	+0.79

Table 8: Transformer scaling on WikiText-103: full training versus LottaLoRA

r{=}8

at each backbone scale. Internal trainable ratio (“Ratio (%)”) counts only transformer-internal parameters (attention projections and feedforward layers, excluding token embeddings and the language-model output head). Best loss per scale in bold ;

\Delta

is the gap LottaLoRA

\;-\;

Full.

Scale	Full (GB)	LottaLoRA $r{=}8$ (GB)	Mem. red.	Tput. ratio
3 M	1.6	1.6	1%	0.62 $\times$
30 M	2.8	2.5	10%	0.73 $\times$
300 M	10.7	7.7	28%	1.00 $\times$
600 M	19.3	13.0	33%	1.00 $\times$
900 M	29.8	19.1	36%	1.00 $\times$

Table 9: Measured peak GPU memory and training throughput on an H200 GPU (WikiText-103, batch size 32, 128-token blocks, bf16 mixed precision, single epoch = 34,429 steps). Memory reduction grows with scale as the frozen backbone dominates.

Scale	fp16 (MB)	4-bit (MB)	LottaLoRA (MB)	vs. fp16	vs. 4-bit
3 M	5	1	4	1 $\times$	0.3 $\times$
30 M	57	16	24	2 $\times$	0.7 $\times$
300 M	586	165	103	6 $\times$	1.6 $\times$
600 M	1,184	333	105	11 $\times$	3.2 $\times$
900 M	2,312	650	109	21 $\times$	6.0 $\times$

Table 10: Distributable model size at each scale. “fp16” stores all parameters in half precision; “4-bit” uses group-quantized 4-bit format (GGUF-style, with fp16 scale factors per 32-weight group); “LottaLoRA” stores only the random seed (8 B), LoRA adapters, and token embeddings in fp16. Compression ratios are relative to fp16.

Figure 8: Different tasks saturate at different minimum ranks, reflecting intrinsic dimensionality. Each curve shows LottaLoRA performance (normalized as % of fully trained baseline recovered) against LoRA rank. CfC PhysioNet (ICU mortality) is flat from

r{=}1

at the published architecture scale; a size-reduction ablation (Table 6 ) shows that saturation shifts rightward when model capacity is reduced, indicating that overparameterization contributes to the flat curve; MNIST and CIFAR-10 saturate between

r{=}8

and

r{=}32

; Decision Transformer (offline RL) reaches 100% recovery near

r{=}16

–

32

; ViT with a pre-trained backbone is nearly flat from

r{=}4

, indicating the backbone—not the adapter rank—limits performance. MolHIV recovers 97.5% at

r{=}16

on molecular property prediction. OGBN-Arxiv (node classification) saturates near

r{=}16

–

32

, recovering 98.5% of baseline. IMDB sentiment (DistilBERT) recovers 99.3% at

r{=}8

with 0.48% trainable parameters.

Figure 9: LottaLoRA recovers 96–100% of baseline performance across eight benchmarks. Each bar shows the ratio of LottaLoRA to baseline performance (accuracy, $R^{2}$ , or inverted MSE as appropriate); annotations give the trainable parameter ratio. Data from Tables 14 , 5 , 7 , and 21 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Scale	Layers	Hidden dim	Heads	MLP dim	Total params
3 M	6	64	4	192	2.4 M
30 M	10	384	6	1,024	30.0 M
300 M	22	1,024	16	2,816	315.4 M
600 M	30	1,344	21	3,584	693.4 M
900 M	34	1,664	26	4,608	1,212.0 M

Appendix

Table 11: Transformer architecture specifications for the WikiText-103 scaling experiments. All scales share the same tokenizer (TinyLlama, 32,000 vocabulary), RMSNorm, and RoPE.

Method	Rank	Total params	Internal trainable	Ratio (%)	Final loss
Full (3M)	–	2,368,320	320,320	100.00	5.007
LottaLoRA (3M)	1	2,371,416	3,096	0.97	5.855
LottaLoRA (3M)	2	2,374,488	6,168	1.93	5.781
LottaLoRA (3M)	4	2,380,632	12,312	3.84	5.682
LottaLoRA (3M)	8	2,392,920	24,600	7.68	5.639
Full (30M)	–	29,990,784	17,702,784	100.00	3.757

Appendix

Table 12: Final training loss on WikiText-103 and parameter counts for full training and LottaLoRA across all backbone scales and ranks. “Internal” denotes transformer-internal trainable parameters. Ratio is the internal count as a percentage of the same-scale full baseline.

Family	Distribution	Key parameters
Standard
Normal	$\mathcal{N}(0,\sigma^{2})$	$\sigma=1.0$
Trunc. Normal	$\mathcal{N}(0,\sigma^{2})$ clipped at $\pm 2\sigma$	$\sigma=1.0$
Uniform	$\mathcal{U}(-a,a)$	$a=0.1$
Orthogonal	Orthogonal matrix	gain $=1.0$
Scale-aware

Appendix

Table 13: The 22

W_{\text{seed}}

initialization families used in the distribution robustness experiment (Section 5.2 ).

Method	Test Acc (%)	Trainable	Ratio (%)
Full_tiny	98.06	109,386	100.00
Full_small	98.37	242,762	100.00
Full_medium	98.38	575,050	100.00
LottaLoRA_tiny_r1	85.06	1,756	1.49
LottaLoRA_tiny_r2	90.60	2,860	2.42
LottaLoRA_tiny_r4	93.56	5,068	4.28

Appendix

Table 14: MNIST full-vs-LottaLoRA comparison across three presets and multiple ranks. Medium preset values are means over 2 runs; tiny and small are from 2 runs (seed 42, normal

W_{\text{seed}}

, scale 0.1).

Panel of Figure 10 — Figure 10: Every $W_{\text{seed}}$ initialization family exceeds 95% at rank 8 on MNIST. Left: Every initialization family exceeds 95% at rank 8, showing that the scaffold’s specific distribution has negligible effect. Right: All 22 families converge tightly as rank increases, confirming that the scaffold is interchangeable on this task.

Preset	B-init	$r{=}1$	$r{=}2$	$r{=}4$	$r{=}8$
tiny	zeros	86.47	90.38	93.47	95.65
tiny	kaiming	84.12	90.03	92.73	95.53
small	zeros	87.68	91.07	94.64	96.24
small	kaiming	75.23	89.93	93.27	96.03
medium	zeros	89.40	92.92	95.74	96.94
medium	kaiming	70.59	90.19	94.38	96.96

Appendix

Table 15: MNIST test accuracy (%) by LoRA B-init and rank (seed 42). Zeros dominates at rank 1; the gap closes at higher ranks.

Preset	Output	$r{=}1$	$r{=}2$	$r{=}4$	$r{=}8$
tiny	full	86.46	90.38	93.47	95.65
tiny	lora	75.74	87.96	92.70	95.40
tiny	lora+bias	77.83	88.30	92.90	95.47
small	full	87.68	91.07	94.64	96.26
small	lora	83.20	90.15	94.27	96.21
small	lora+bias	85.06	90.92	94.24	96.27

Appendix

Table 16: MNIST test accuracy (%) by output head mode and rank (seed 42). Full output dominates at low rank; all modes converge by rank 4.

Rank	AUROC	#Trainable params	%Trainable
1	$0.8321\pm 0.0047$	3,482	3.7%
2	$0.8351\pm 0.0046$	5,292	5.7%
4	$0.8344\pm 0.0020$	8,912	9.6%
8	$0.8327\pm 0.0013$	16,152	17.4%
16	$0.8337\pm 0.0006$	30,632	25.1%
32	$0.8330\pm 0.0028$	59,592	39.5%

Appendix

Table 17: CfC PhysioNet rank sensitivity: mean AUROC

\pm

std over 5 seeds. %Trainable is relative to the Full CfC parameter count (92,930).

Figure 11: LottaLoRA matches the fully trained CfC baseline on PhysioNet 2012 ICU mortality at every rank. Mean $\pm$ std AUROC over 5 seeds at ranks 1–32. Dashed line: Full CfC baseline (AUROC $=0.836$ ).

Method	Rank	Mean ROC-AUC	Std	Trainable
Baseline (fully trained)	—	0.7755	0.0060	1,863,906 (100%)
LottaLoRA	$r{=}8$	0.7520	0.0194	131,416 (7.1%)
LottaLoRA	$r{=}16$	0.7562	0.0158	203,416 (10.9%)
LottaLoRA	$r{=}32$	0.7456	0.0152	347,416 (18.6%)

Appendix

Table 18: LottaLoRA matches the published OGB GIN baseline on OGBG-MolHIV while training 10.9% of the parameters. Test ROC-AUC (5 seeds). Published GIN baseline:

0.7558\pm 0.0140

[ 20 ] .

Figure 12: LottaLoRA recovers 97.5% of baseline ROC-AUC on molecular property prediction. Test ROC-AUC on OGBG-MolHIV (5 seeds, error bars show

\pm 1

std). Dashed lines mark published OGB baselines [ 20 ] : GIN (red) and GIN+virtual node (green).

Rank	Acc (%)	Std	% Trainable
Baseline	71.86	0.22	100%
$r{=}32$	70.11	0.21	35.6%
$r{=}16$	69.04	0.14	18.3%
$r{=}8$	67.51	0.24	9.6%
$r{=}4$	65.21	0.40	5.3%
$r{=}2$	61.96	0.53	3.1%

Appendix

Table 19: LottaLoRA recovers 97.6% of baseline GCN accuracy on OGBN-Arxiv while training 35.6% of the parameters. Test accuracy (10 seeds). Published OGB GCN baseline:

71.74\pm 0.29

% [ 20 ] .

Figure 13: LottaLoRA recovers 97.6% of baseline GCN accuracy on OGBN-Arxiv node classification. Test accuracy (10 seeds, error bars show

\pm 1

std). Dashed line marks the fully trained GCN baseline (

71.86\%

Method	Rank	Mean Top-1	Std	Trainable Ratio
Baseline (full fine-tune)	—	96.83%	0.08%	100%
LottaLoRA (pre-trained)	1	94.53%	0.35%	1.19%
LottaLoRA (pre-trained)	4	93.99%	0.28%	1.76%
LottaLoRA (pre-trained)	8	94.01%	0.35%	2.51%
LottaLoRA (pre-trained)	32	94.68%	0.22%	6.76%
LottaLoRA (pre-trained)	64	94.89%	0.16%	11.89%

Appendix

Table 20: ViT-Base/16 on Flowers-102 (5 seeds, 300 epochs, pre-trained ImageNet-1k init). LottaLoRA replaces attention and MLP projections with frozen random matrices plus low-rank adapters. The random backbone condition uses a randomly initialized (non-pre-trained) ViT.

Figure 14: Learned reservoir quality accounts for a ${\sim}$ 40 pp advantage over random backbones. (A) Pre-trained backbone rank sweep: under extended training (300 epochs), LottaLoRA reaches 94–95% across $r{=}1$ – $64$ , within 1.9–2.8 pp of full fine-tuning (dashed). Even $r{=}1$ (1.19% trainable) achieves 94.53%. (B) Training budget effect: extending from 100 to 300 epochs yields +39–40 pp for LottaLoRA ranks and +12 pp for the baseline. (C) Random backbone with and without composed-path LayerNorm: LayerNorm adds +1.0–1.7 pp across all ranks, but the random backbone ceiling remains ${\sim}$ 56%, showing that learned reservoir quality accounts for a ${\sim}$ 40 pp advantage. (D) Parameter efficiency: at $r{=}1$ (1.19% trainable parameters), LottaLoRA recovers 97.6% of baseline accuracy; random backbone conditions cluster near 57%, quantifying the information contributed by the frozen learned projection layers.

Method	Rank	Mean Acc	Std	Trainable	% Trainable
Baseline (full)	—	85.69%	0.44%	10,315,030	100%
LottaLoRA (fan-in)	1	83.30%	0.48%	6,673	0.06%
LottaLoRA (fan-in)	2	84.39%	0.28%	12,817	0.12%
LottaLoRA (fan-in)	4	84.59%	0.53%	25,105	0.24%
LottaLoRA (fan-in)	8	85.12%	0.57%	49,681	0.48%
LottaLoRA (fan-in)	16	84.93%	0.46%	98,833	0.95%

Appendix

Table 21: IMDB sentiment classification: test accuracy (%) by LoRA rank (4 seeds, fan-in-scaled

W_{\text{seed}}

, DistilBERT backbone). LottaLoRA at

r{=}8

recovers 99.3% of full fine-tuning with 0.48% trainable parameters.

Figure 15: LottaLoRA matches full fine-tuning on IMDB sentiment at rank 8 with 0.48% trainable parameters. Error bars show

\pm 1

standard deviation over 4 seeds. Dashed line and shaded band indicate the full fine-tuning baseline (

85.69\pm 0.44

%). Performance saturates at

r{=}8

, with higher ranks providing no additional gain.

Rank	Vanilla	+Residual	+DoRA+Res	Trainable ratio
1	$61.08\pm 1.42$	$61.57\pm 1.67$	$76.70\pm 0.54$	3.1%
2	$75.02\pm 0.39$	$76.31\pm 0.83$	$82.24\pm 0.50$	5.2%
4	$82.40\pm 0.09$	$82.59\pm 0.11$	$85.43\pm 0.08$	10.1%
8	$86.18\pm 0.41$	$86.20\pm 0.25$	$88.12\pm 0.31$	19.5%
16	$87.70\pm 0.26$	$88.76\pm 0.09$	$89.39\pm 0.17$	38.3%
32	$88.01\pm 0.34$	$88.48\pm 0.38$	$89.66\pm 0.03$	43.3%

Appendix

Table 22: CIFAR-10 Plain-20 test accuracy (%) by LoRA rank and configuration (3 seeds;

r\in\{1,4,8,16\}

: 150 epochs,

r\in\{2,32\}

: 164 epochs). Residual = residual connection around frozen layer. DoRA = weight-decomposed low-rank adaptation. Baseline (full training):

90.81\%\pm 0.20\%

Method	$r$	Val MSE	Std	D4RL (%)	Ratio
Small scale (128-d, 3-layer)
Baseline (full)	—	0.02718	0.00173	$91.86\pm 0.55$	100.00%
Static scaffold ( noise_lora )
LottaLoRA	1	0.03482	0.00169	$61.10\pm 21.46$	19.23%
LottaLoRA	2	0.03221	0.00169	$84.29\pm 5.74$	19.98%
LottaLoRA	4	0.02985	0.00173	$89.13\pm 3.29$	21.44%

Appendix

Table 23: Decision Transformer on HalfCheetah-expert-v2 (5 seeds). Val MSE is offline prediction error (lower is better); D4RL (%) is normalized online return from HalfCheetah-v4 rollouts (higher is better; 10 episodes

\times

1,000 timesteps per seed). Static = frozen

W_{\text{seed}}

; Resampled =

W_{\text{seed}}

redrawn each epoch. Large-scale baseline and LottaLoRA from the original sweep (

r{=}8

only).

Figure 16: LottaLoRA matches the fully trained Decision Transformer at sufficient rank. Left: validation MSE by method and rank at small scale (128-d, 3-layer). The static scaffold ( noise_lora ) closes the gap monotonically with rank; at $r{=}32$ the difference is not statistically significant. Right: val MSE vs trainable parameter ratio. The large-scale $r{=}8$ result (1.87% trainable) trails the baseline by 1.2% relative MSE; the small-scale $r{=}32$ result (37.47% trainable) matches the baseline.

Panel of Figure 17 — Figure 17: The frozen backbone violates the Echo State Property at every layer, yet $\beta$ remains strictly positive (near 1 in ViT, median ${\approx}0.99$ ; see Section 5.3 for architecture-dependent values). Left: Empirical distribution of learned $\beta$ values from all Transformer checkpoints (1,632 values). Mass concentrates below 1.0; all values remain positive, confirming the backbone always contributes. Center: Spectral norm $\sigma_{1}(W_{\text{seed}})$ by backbone scale (bars, reconstructed from seed) versus the Random Matrix Theory prediction $2\sqrt{d}\,\sigma_{\mathrm{init}}$ (red dashed). All scales violate the ESP boundary (dotted line at 1). Right: Learned $\beta$ (blue, left axis) and $\sigma_{1}(W_{\text{seed}})$ (red, right axis) by transformer block depth in ViT (5 seeds). $\beta$ is flat across depth and near 1 despite $\sigma_{1}(W_{\text{seed}})\approx 60$ – $80$ , indicating that $\beta$ does not compensate for spectral structure—the adapter matrices do.

Figure 18: LottaLoRA replaces the frozen pre-trained backbone with a seed-reconstructible random scaffold. (a) Standard training: all weights in

W

are trainable and must be stored. (b) LoRA: a frozen pre-trained backbone

W_{0}

is augmented with trainable low-rank adapters

A

and

B

; both

W_{0}

and the adapters must be stored. (c) LottaLoRA (ours): the backbone

W_{\text{seed}}

is generated from a random seed and frozen; a learnable scalar

\beta

modulates its amplitude. The adapter path (trainable

A

B

with

\alpha/r

or rsLoRA scaling, optional dropout) and backbone path merge at a summation node; optional DoRA renormalises the combined output, and a residual skip connection may be added thereafter. An optional LayerNorm follows before the final output. Only the seed, adapter weights, and

\beta

need to be stored or distributed.

Component	Status	Notes
Frozen $W_{\text{seed}}$	Required	Any distribution; values interchangeable but actively exploited when static ( $\beta>0$ , magnitude varies by architecture; Sections 5.2 , 5.3 )
LoRA factors ( $A$ , $B$ )	Required	Core trainable parameters
$\beta$ scalar	Recommended	Stabilizes training; can be fixed at 1.0
$\alpha/r$ scaling	Standard	Follows LoRA convention [ 2 ]
LayerNorm	Optional	Adds $2d$ params/layer; helps at low rank
Fan-in scaling	Req. if no LN	Either fan-in or LN needed; both removed $\to$ divergence

Appendix

Table 24: Required and optional design choices in LottaLoRA.

Random Feature Methods

Random feature methods approximate kernels and neural networks with finite randomized embeddings to reduce computation while retaining useful accuracy.

See all 46 papers

Apr 30, 2026

Post-Optimization Adaptive Rank Allocation for LoRA

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.

Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith

1Indian Institute of Technology Hyderabad · 2Deakin University

Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Neural Network Pruning

Ask PDF

Apr 23, 2026

Low-Rank Adaptation Redux for Large Models

Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networks with minimal computational and memory overhead. Despite its empirical success and rapid proliferation of variants, it remains elusive which architectural choices, optimization techniques, and deployment constraints should guide practical method selection. This overview revisits LoRA through the lens of signal processing (SP), bridging modern adapter designs with classical low-rank modeling tools and inverse problems, as well as highlighting how SP principles can inform principled advances of fine-tuning approaches. Rather than providing a comprehensive enumeration and empirical comparisons of LoRA variants, emphasis is placed on the technical mechanisms underpinning these approaches to justify their effectiveness. These advances are categorized into three complementary axes: architectural design, efficient optimization, and pertinent applications. The first axis builds on singular value decomposition (SVD)-based factorization, rank-augmentation constructions, and cross-layer tensorization, while the second axis deals with initialization, alternating solvers, gauge-invariant optimization, and parameterization-aware methods. Beyond fine-tuning, emerging applications of LoRA are accounted across the entire lifecycle of large models, ranging from pre- and post-training to serving/deployment. Finally, open research directions are outlined at the confluence of SP and deep learning to catalyze a bidirectional frontier: classical SP tools provide a principled vocabulary for designing principled PEFT methods, while the unique challenges facing modern deep learning, especially the overwhelming scale and prohibitive overhead, also offer new research lines benefiting the SP community in return.

Bingcong Li, Yilang Zhang, Georgios B. Giannakis

Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA

Adapter Tuning Low-Rank Adaptation Low-Rank Approximation Parameter-Efficient Fine-Tuning

Ask PDF

May 28, 2026

FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.

Juneyoung Park, Seongbae Lee, Han-Sang Lee +6

OptAI Inc · LG Uplus

LLM Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Random Feature Methods

Explore similar work

Post-Optimization Adaptive Rank Allocation for LoRA

Low-Rank Adaptation Redux for Large Models

FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning