How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

Mei Okonkwo; Owen Castellanos; Ingrid Solberg; Yusuf Demirel

cs.LGOct 13, 2026

How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

Authors: Mei Okonkwo, Owen Castellanos, Ingrid Solberg, Yusuf Demirel

Organizations: University of Edinburgh · Delft University of Technology

Abstract

Low-rank adaptation (LoRA) applies every rank-one component of its update to every token, and mixture-of-experts variants replace this with a learned router that activates a fixed number of components. We argue that the number of components a token needs is a detection question with a classical answer. A token's responses to the rank-one components of a trained LoRA consist of a few large values on a Gaussian noise floor whose width matches that of an untrained adapter, and the floor carries cross-talk rather than adaptation. We introduce SQUELCH, a router-free mixture of rank-one experts that (i) soft-thresholds each token's responses at a multiple of its own robust noise level, so the active rank varies per token; (ii) chooses that multiple for each module by Stein's unbiased risk estimate, so no rank budget is specified; and (iii) replaces the resulting marginal keep-or-drop decisions by one explaining-away step on the value Gram matrix, which suppresses redundant co-active components and recruits complementary ones. SQUELCH adds no trainable parameters to LoRA. We prove that the squelch is within a logarithmic factor of the oracle input-dependent allocation, that the selected threshold is risk-consistent, and that explaining away is a descent step toward the joint sparse code whose error is controlled by the coherence of co-active components. On commonsense reasoning, knowledge, math, code, and GLUE with LLaMA-3.1-8B, Qwen3, and DeBERTaV3, SQUELCH improves over LoRA by 2.2--3.2 points and over the strongest mixture baseline by 0.8--1.2 points while activating 8.6 of 32 components per token on average.

Figures & tables

Figure 1: A token’s rank-one responses are a sparse signal on a noise floor (LLaMA-3.1-8B, LoRA

r{=}32

trained on commonsense reasoning). (a) Standardized responses of the trained adapter (red) against those of an untrained adapter (dashed): a Gaussian bulk plus heavy tails beyond the universal threshold. (b) The number of above-floor components per token varies with the token type. (c) Zeroing sub-floor responses after training preserves accuracy and improves held-out MMLU (right axis), whereas keeping the same number of components by top-

k

or at random does not. (d) Gap between the marginal (thresholded) code and the joint sparse code as co-active value vectors become coherent (random dictionaries, exact computation); one explaining-away step closes most of it.

Figure 2: Squelch in one adapted layer. The token’s responses to $r$ unit-norm keys are squelched at a multiple $c_{\ell}$ of the token’s own noise floor, where $c_{\ell}$ is chosen per module by Stein’s unbiased risk estimate. One explaining-away step on the value Gram $B^{\!\top}\!B$ turns the marginal code into a joint one. The surviving components form the token’s update, so its rank varies per token. Squelch uses LoRA’s parameters $A,B$ and adds none.

LLaMA-3.1-8B
Method	Params (%)	Rank	BoolQ	PIQA	SIQA	HellaS.	WinoG.	ARC-e	ARC-c	OBQA	Avg.
LoRA	1.04	32	72.9	88.0	80.1	94.1	85.0	90.3	79.6	85.2	84.40
DoRA	1.05	32	73.5	88.4	80.6	94.6	85.6	90.8	80.6	85.9	85.00
AdaLoRA	1.04	32	73.2	88.3	80.5	94.3	85.2	90.6	79.8	85.7	84.70
GoRA	1.04	32	73.9	88.5	80.7	94.7	86.1	90.9	80.6	86.4	85.23
HydraLoRA	1.05	16	74.0	88.8	80.8	94.8	86.1	91.1	81.0	86.4	85.38

Table 1: Commonsense reasoning (accuracy, %, mean of three seeds). Params : trainable parameters relative to the backbone. Rank : rank-one components that contribute to a token’s update, averaged over tokens and modules. Single adapters, then mixtures. Bold : best score. Squelch is in the header color.

	LLaMA-3.1-8B							Qwen3-8B
Method	MMLU	SciQA	GSM8K	MATH	HumanE.	MBPP	Avg.	MMLU	SciQA	GSM8K	MATH	HumanE.	MBPP	Avg.
LoRA	63.8	93.6	71.2	24.6	42.1	51.3	57.77	76.1	95.2	89.3	58.4	72.6	66.9	76.42
DoRA	64.4	93.7	71.7	24.8	43.0	52.0	58.27	76.5	95.3	89.4	59.2	73.3	67.4	76.85
GoRA	64.5	93.8	72.4	25.5	43.6	52.5	58.72	76.3	95.7	90.1	59.8	74.0	68.0	77.32
HydraLoRA	64.5	93.9	72.3	25.7	43.6	52.1	58.68	76.7	95.7	89.9	59.4	73.7	67.9	77.22
MixLoRA	64.3	94.3	73.2	26.0	44.2	53.0	59.17	76.6	95.7	90.1	59.7	74.3	67.8	77.37

Table 2: Knowledge, science, math, and code (accuracy or pass@1, %, mean of three seeds; one adapter per task). Parameter budgets and active ranks as in Table 1 .

Figure 3: Where Squelch spends rank (LLaMA-3.1-8B). (a) Mean active rank by layer and module: FFN projections in the middle and upper layers use the most components. (b) Mean active rank by task: math and code use more components than knowledge and science. (c) SURE-selected multiples $c_{\ell}$ ( $\pm$ s.d. across layers) stay below the universal cap and are lower for FFN modules. (d) Explaining away inhibits and recruits the most components in FFN modules, where co-active values are most coherent (right axis).

Figure 4: Capacity, budget, and threshold selection (commonsense unless stated). (a) Accuracy vs. dictionary size

r

; numbers give Squelch ’s mean active rank. (b) Accuracy vs. mean active rank: Squelch with a target rank (budgeted variant), top-

k

variants, and LoRA; the star is SURE’s choice. (c) Accuracy of global fixed multiples

c

relative to the best one; stars mark per-module SURE. (d) Gain over LoRA across Qwen3 sizes; numbers give Squelch ’s mean active rank.

Configuration	CS	Dom.	$\bar{k}$	Gap
Squelch (full)	87.04	60.92	8.6	0.09
Per-layer EMA noise scale (not per token)	86.58 $(-0.46)$	60.21	8.9	0.10
RMS instead of MAD noise scale	86.64 $(-0.40)$	60.30	5.7	0.08
Universal $c{=}\sqrt{2\log r}$ (no SURE)	86.77 $(-0.27)$	60.46	5.1	0.07
$c$ learned by the task loss	85.08 $(-1.96)$	58.62	27.9	0.31
Top- $k$ at matched $k{=}9$ (no floor)	86.31 $(-0.73)$	59.94	9.0	—

Table 3: Ablations on LLaMA-3.1-8B. CS/Dom.: commonsense/domain averages;

\bar{k}

: mean active rank; Gap:

\|z-z^{\star}\|/\|z^{\star}\|

to the joint code at the same threshold.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Result
Prop. 2 (a): risk of $S_{\tau_{U}}$ vs. $(2\log r{+}1)(1{+}\sum_{i}\min(\theta_{i}^{2},1))$ , $r{=}32$	bound holds in all 6 sparsity patterns; risk/oracle $\leq 7.91$
Same with the MAD noise estimate instead of known $\sigma$	risk $28.41$ vs. $24.92$ ( $s{=}4$ ); $117.51$ vs. $63.24$ ( $s{=}8$ )
Prop. 2 (b): false activation with no signal, exact vs. bound $1/\sqrt{\pi\log r}$	$0.238$ vs. $0.303$
Prop. 3 (a): $\|\mathbb{E}\,\mathrm{SURE}-\mathrm{risk}\|/\mathrm{risk}$ (18 settings, $2{\times}10^{5}$ draws)	max $0.38\%$
Prop. 3 (b): regret of the SURE threshold with $N$ tokens	1.5% ( $N{=}16$ ), 0.52% (256), 0.062% (4096)
MAD scale bias $\hat{\sigma}/\sigma$ with active fraction $\rho=0,\tfrac{1}{8},\tfrac{1}{4},\tfrac{3}{8}$ (emp./analytic)	1.01/1.00, 1.18/1.17, 1.44/1.43, 1.90/1.90

Appendix

Table 4: Numerical checks of the analytic results (script code/theory_checks.py ). The MAD row explains the conservative bias discussed in Appendix C .

	Orthogonal values ( $\bar{\mu}\approx 0$ )			Coherent values ( $\bar{\mu}\approx 0.5$ )
Student	Rel. error	$\bar{\bm{k}}$	Spearman	Rel. error	$\bar{\bm{k}}$	Spearman
Dense LoRA ( $r{=}32$ )	$0.552\pm 0.016$	32.0	—	$0.479\pm 0.032$	32.0	—
Top- $4$ self-routing	$0.355\pm 0.013$	4.0	—	$0.243\pm 0.009$	4.0	—
Top- $8$ self-routing	$0.451\pm 0.008$	8.0	—	$0.304\pm 0.011$	8.0	—
Squelch without EA	$0.308\pm 0.009$	3.1	0.37	$0.293\pm 0.035$	4.0	0.60
Squelch	$0.326\pm 0.066$	7.8	0.66	$0.240\pm 0.003$	10.5	0.84

Appendix

Table 5: Controlled synthetic study (real computation, mean

\pm

s.d. over three seeds). Rel. error:

\|\hat{y}-y\|^{2}/\|y\|^{2}

on held-out inputs;

\bar{k}

: mean active components; Spearman: rank correlation between the mean active count of each cluster and its true rank

|S_{c}|

(undefined for fixed-

k

students).

Figure 5: Synthetic study (real computation). (a) Relative test error of each student for orthogonal (light) and coherent (dark) teacher values. (b) Mean active components as a function of the true rank of the input (solid: coherent values; dashed: orthogonal). (c) Mean

|\cos|

between the values of

16

near-duplicate component pairs during training. (d) Error for smaller student dictionaries.

	No EA		EA		EA+pen.
	Err.	$\bar{k}$	Err.	$\bar{k}$	Err.	$\bar{k}$
$r{=}12$	$0.476$	2.4	$0.452$	3.6	$0.444$	3.4
$r{=}16$	$0.393$	2.8	$0.392$	4.1	$0.373$	4.3

Appendix

Table 6: Synthetic study with scarce student dictionaries (orthogonal teacher values, three seeds). “pen.”: co-activation-weighted coherence penalty with weight

1

Suite	Training data	Train	Evaluation (test size)
Commonsense	Commonsense170K	170K	BoolQ (3,270), PIQA (1,838),
			SIQA (1,954), HellaSwag (10,042),
			WinoGrande (1,267), ARC-e (2,376),
			ARC-c (1,172), OBQA (500)
Knowledge	MMLU auxiliary train	99.8K	MMLU test (14,042)
Science	ScienceQA train	12.7K	ScienceQA test (4,241)

Appendix

Table 7: Datasets. For multi-task training, the five domain training sets are concatenated and sampled in proportion to their sizes.

Hyperparameter	Value
Target modules	q, k, v, o, gate, up, down
Rank / dictionary size $r$	32
Scaling $\alpha$	64
Optimizer	AdamW ( Loshchilov and Hutter, 2019 ) , $\beta{=}(0.9,0.999)$ , wd $0$
Learning rate	$2{\times}10^{-4}$ (CS, code), $10^{-4}$ (math)
Schedule	linear warm-up 100 steps, cosine

Appendix

Table 8: Training configuration shared by all methods, and Squelch ’s settings.

	LLaMA-3.1-8B		Qwen3-8B
Method	CS	Dom.	CS	Dom.
LoRA	$84.40\pm 0.21$	$57.77\pm 0.38$	$87.10\pm 0.17$	$76.42\pm 0.34$
LD-MoLE	$86.05\pm 0.19$	$59.40\pm 0.35$	$88.40\pm 0.16$	$77.80\pm 0.31$
SMoRA	$85.83\pm 0.22$	$59.25\pm 0.39$	$88.40\pm 0.18$	$77.62\pm 0.33$
FlyLoRA	$86.11\pm 0.18$	$59.73\pm 0.33$	$88.51\pm 0.15$	$77.98\pm 0.30$
Squelch	$87.04\pm 0.15$	$60.92\pm 0.29$	$89.28\pm 0.13$	$78.73\pm 0.23$

Appendix

Table 9: Averages over three seeds (mean

\pm

s.d.).

Method	Params (M)	MNLI	SST-2	CoLA	QQP	QNLI	RTE	MRPC	STS-B	Avg.
LoRA	1.33	90.4	95.1	69.6	91.6	93.8	85.6	89.7	91.5	88.41
AdaLoRA	1.27	90.7	95.3	70.6	91.9	94.0	86.9	90.5	91.7	88.95
SoRA	0.91	90.8	95.5	70.5	91.7	94.1	86.9	90.7	91.8	89.00
GoRA	1.33	90.8	95.5	71.2	92.1	94.0	87.2	90.8	91.8	89.18
ElaLoRA	1.30	90.9	95.6	70.9	92.0	94.2	87.0	91.0	91.9	89.19
SMoRA	1.92	90.7	95.3	71.0	91.9	94.2	87.0	90.6	91.6	89.04

Appendix

Table 10: GLUE with DeBERTaV3-base (matched accuracy for MNLI, Matthews correlation for CoLA, Pearson correlation for STS-B, accuracy otherwise; mean of three seeds).

Method	MMLU	SciQA	GSM8K	MATH	HE	MBPP	Avg.	$\Delta$
LoRA	63.1	92.4	69.3	23.4	39.8	49.6	56.27	$-1.50$
MixLoRA	64.0	93.8	72.0	25.3	42.9	52.1	58.35	$-0.82$
LD-MoLE	64.4	94.0	72.5	25.4	43.3	52.3	58.65	$-0.75$
SMoRA	64.3	94.3	72.4	25.5	43.6	52.4	58.75	$-0.50$
FlyLoRA	64.6	94.1	72.6	25.7	43.9	52.4	58.88	$-0.85$
Squelch	64.9	95.0	75.0	27.1	46.6	54.1	60.45	$-0.47$

Appendix

Table 11: Joint multi-task training on LLaMA-3.1-8B.

\Delta

: change relative to separate single-task adapters (Table 2 ).

Model	MMLU	GSM8K	HumanE.	Avg. drop
Base model	66.4	56.1	37.2	—
LoRA	63.9	50.7	33.5	$-3.87$
DoRA	64.1	51.2	33.9	$-3.50$
MixLoRA	64.3	51.9	34.2	$-3.10$
SMoRA	64.6	52.2	34.6	$-2.77$
FlyLoRA	65.0	53.0	35.1	$-2.20$

Appendix

Table 12: Held-out accuracy after commonsense fine-tuning on LLaMA-3.1-8B (MMLU 5-shot, GSM8K 8-shot, HumanEval zero-shot).

Method	0.6B	1.7B	4B	8B	14B
LoRA	68.90	77.40	83.80	87.10	89.00
FlyLoRA	70.50	78.90	85.00	88.51	90.10
Squelch	71.90	80.10	85.90	89.28	90.80
$\Delta$ vs. FlyLoRA	$+1.40$	$+1.20$	$+0.90$	$+0.77$	$+0.70$
Squelch active rank	7.1	7.8	8.3	8.8	9.2

Appendix

Table 13: Commonsense average across Qwen3 sizes.

Dictionary size $r$	8	16	32	64	128
Params (%)	0.26	0.52	1.04	2.09	4.17
LoRA	83.52	84.05	84.40	84.51	84.38
SMoRA ( $k{=}r/4$ )	84.31	85.11	85.82	86.09	86.02
FlyLoRA ( $k{=}r/4$ )	84.56	85.37	86.11	86.40	86.47
Squelch	85.12	86.23	87.04	87.52	87.71
Squelch active rank	2.9	5.1	8.6	12.3	15.8

Appendix

Table 14: Commonsense average on LLaMA-3.1-8B for different dictionary sizes.

Dense warm-up (fraction of steps)
Setting	CS avg.
0.00	85.69
0.01	86.52
0.03	86.93
0.05	87.04
0.10	86.99

Appendix

Table 15: Sensitivity on LLaMA-3.1-8B (commonsense average; bold: best in each block). Defaults: warm-up

0.05

, cap

\sqrt{2\log r}

, interval

20

, EMA

0.9

\eta=1/\lambda_{\max}

, MAD,

\alpha=64

Method	Params (%)	Rank	Train	Latency	Mem. (GB)
LoRA	1.04	32	1.00 $\times$	1.00 $\times$	38.2
DoRA	1.05	32	1.27 $\times$	1.19 $\times$	41.6
HydraLoRA	1.05	16	1.08 $\times$	1.06 $\times$	38.6
MixLoRA	1.07	8	1.16 $\times$	1.14 $\times$	39.4
LD-MoLE	1.07	10.8	1.19 $\times$	1.17 $\times$	39.7
SMoRA	1.53	8	1.13 $\times$	1.12 $\times$	39.9

Appendix

Table 16: Cost on LLaMA-3.1-8B (training time and inference latency relative to unmerged LoRA; peak training memory per GPU).

			EA (% of actives)			Gap to joint code
Module	$\bar{\bm{k}}$	$c_{\ell}$	inhib.	recr.	$\bar{\mu}$	marginal	EA
q	6.6	2.21	6.1	9.4	0.048	0.16	0.07
k	4.1	2.38	4.8	8.2	0.041	0.14	0.06
v	8.0	1.74	9.7	14.9	0.062	0.23	0.08
o	8.4	1.69	10.4	15.8	0.064	0.24	0.08
gate	11.0	1.47	15.2	23.1	0.079	0.33	0.10

Appendix

Table 17: Per-module statistics on LLaMA-3.1-8B (commonsense adapter): mean active rank, SURE multiple, share of first-pass actives inhibited and of new components recruited by explaining away, mean

|\cos|

of co-active values, and gap to the joint code before and after explaining away.

Mean rank $\bar{\bm{k}}$	Squelch (target $\bar{k}$ )	Squelch top- $k$	SMoRA	FlyLoRA
2	85.31	85.02	84.62	84.95
4	86.18	85.84	85.33	85.63
6	86.71	—	—	—
8	86.98	86.29	85.82	86.11
12	87.08	—	—	—
16	86.93	86.12	85.64	85.97

Appendix

Table 18: Commonsense average on LLaMA-3.1-8B at matched mean active rank. “ Squelch top-

k

” replaces the floor by a fixed top-

k

but keeps explaining away.

Figure 6: Training dynamics on LLaMA-3.1-8B (commonsense): (a) training loss; (b) mean active rank (left) and mean SURE multiple (right).

Figure 7: Per-token ranks on LLaMA-3.1-8B. (a) Distribution over token–module pairs (math and commonsense evaluation data). (b) Active rank of each token in the layer-20 up-projection for a GSM8K problem, a WinoGrande sentence, and an out-of-domain MMLU question (commonsense adapter).

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts
Sep 30, 2026Priya Nair, Lukas Brenner, Maya Lindqvist +5Mixture-of-Experts Language Models Parameter-Free Mixture-of-Experts Routing
University of California, Irvine · University of Washington
TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models
Apr 7, 2026Lin Mu, Haiyang Wang, Li Ni +4Mixture-of-Experts Language Models Low-Rank Adaptation
Anhui University · Hefei University · University of Science and Technology of China
Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
Jul 28, 2026Tom Saliencro, Rohan Desai, Priya Nair +2Parameter-Free Mixture-of-Experts Routing LLM Inference Acceleration
University of California, Irvine · University of Washington

Sep 30, 2026J/K move · Enter open · S save

Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts

Priya Nair, Lukas Brenner, Maya Lindqvist, Daniel Whitmore+4

University of California, Irvine · University of Washington

Mixture-of-Experts Language Models Parameter-Free Mixture-of-Experts Routing Low-Rank Adaptation Mixture-of-Experts Models Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Mixture-of-LoRA-experts methods raise the capacity of low-rank adaptation by routing each token to a few low-rank experts. Nearly all of them tie one input-side factor to one output-side factor per expert, and nearly all of them route by scoring the token: the router picks experts without seeing what any of them would write. We argue that the router should score the update. To first order, adding an expert's update to a layer output lowers the loss by the inner product between that update and the negative loss gradient at the output. This usefulness is quadratic in the token, so a router that is linear in the token sees only the part of it that runs through the token mean, and routers that rank experts by the norm of their own activations never see the output factor. If each expert is split into a reader (down-projection) and a writer (up-projection), the usefulness of every reader--writer pair becomes an inner product in the shared rank-

r

space, and all

N_AN_B

pairs can be scored from

N_A+N_B

vectors. We build VANE on this identity. A low-rank compass predicts the descent direction of each token. VANE scores every pair by the alignment between its update and the compass without forming any update, activates the top-

k

pairs with additive gates, and gives every pair its exact first-order router gradient. On single-domain commonsense reasoning and a four-domain multi-task mixture with Llama-3.2-3B and Llama-3.1-8B, VANE attains the best average among twelve PEFT and MoE-LoRA baselines, by 0.9--1.1 and 1.3--1.5 points respectively, with less than half the trainable parameters of an 8-expert MoE-LoRA. Its router scores also track the measured usefulness of experts far more closely than token routers do.

Abstract

Low-Rank Adaptation

Explore similar work

Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA