Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Fei Zuo; Yizhou Huang; Kezhi Wang

cs.LGOct 26, 2026

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Authors: Fei Zuo, Yizhou Huang, Kezhi Wang

Organizations: East China Normal University · Brunel University of London

Abstract

Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA applies adaptation throughout the transformer, despite heterogeneous contributions from its components. In this work, we conduct a systematic mechanistic interpretability analysis to understand how LoRA affects model internals. We observe that LoRA's adaptation effect is highly non-uniform, with significant component-wise and layer-wise sparsity. Most critically, we find that important layers are highly task-dependent, with limited overlap across tasks. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation, enabling efficient layer selection without exhaustive search. Extensive experiments on Qwen2.5-1.5B across 11 diverse reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA while using only 30% of layers, achieving a $3.5\times$ reduction in adapter parameters. This work bridges mechanistic interpretability and practical efficiency, providing both fundamental insights and a principled approach for adaptive fine-tuning.

Figures & tables

Figure 1: MLP vs. Attention. MLP dominates at 58.4% across all four datasets.

Figure 2: Layer-wise contribution profiles across 24 layers and four datasets.

Figure 3: Task-specific layer importance heatmap. White stars mark the top-7 layers per task. Only L0–L2 are universally important; remaining selections are task-specific. Average Jaccard similarity: 39.5%.

Method	Rank	Layers	Budget %	Commonsense Reasoning ( $\uparrow$ )									Math ( $\uparrow$ )
Method	Rank	Layers	Budget %	BoolQ	PIQA	SIQA	Hella	Wino	ARC-e	ARC-c	OBQA	Avg.	GSM8K	SVAMP	AQUA	Avg.
LoRA [ 13 ]	r4	28/28	100	63.0	52.4	45.6	38.2	53.0	54.8	42.6	38.0	48.5	32.4	38.6	28.4	33.1
	r8	28/28	100	63.2	57.8	52.4	44.6	54.2	60.4	48.2	43.8	53.1	38.2	42.8	31.6	37.5
	r16	28/28	100	63.4	64.8	58.6	46.8	55.4	65.2	52.4	48.6	56.9	42.6	46.4	34.2	41.1
	r32	28/28	100	63.8	66.2	62.4	48.2	56.0	72.6	60.8	53.2	60.4	48.2	50.6	37.8	45.5
LoRA+ [ 11 ]	r4	28/28	100	63.4	54.8	46.2	39.4	53.4	55.6	43.8	39.2	49.5	34.2	39.4	29.0	34.2

Table 1: Budget is the fraction of adapted layers. Accuracy comparison of LoRA variants on Qwen2.5-1.5B across 11 reasoning tasks. Commonsense: 10,000 training samples, 1,000 eval samples each (OBQA: 500). Math: 5,000 training samples with chain-of-thought prompting. Adaptive-LoRA selects top 30% layers (8/28) via frozen-base calibration with

N{=}250

inputs . Results use seed 42. Best average at each rank in bold .

Model	Method	#Layers	Budget %	Ratio	Commonsense Reasoning ( $\uparrow$ )								Avg.	$\Delta$
Model	Method	#Layers	Budget %	Ratio	BoolQ	PIQA	SIQA	Hella	Wino	ARC-e	ARC-c	OBQA	Avg.	$\Delta$
Qwen2.5-0.5B	LoRA	24/24	100.0	1.00 $\times$	63.0	49.4	35.0	25.8	49.2	23.2	23.4	27.6	37.1	–
Qwen2.5-0.5B	Ours	7/24	29.2	3.43 $\times$	63.2	50.8	37.4	28.2	50.6	25.0	24.8	28.4	38.6	+1.5
Qwen2.5-1.5B	LoRA	28/28	100.0	1.00 $\times$	63.4	64.8	58.6	46.8	55.4	65.2	52.4	48.6	56.9	–
Qwen2.5-1.5B	Ours	8/28	28.6	3.50 $\times$	63.6	65.4	60.8	50.2	56.8	68.4	56.2	51.6	59.1	+2.2
Qwen2.5-3B	LoRA	36/36	100.0	1.00 $\times$	61.0	60.6	59.8	68.6	55.4	84.8	77.8	67.4	66.9	–

Table 2: Budget is

100k/L

; ratio is

L/k

, the LoRA-factor reduction at equal per-layer capacity. Cross-model generalization of Adaptive-LoRA at rank 16. Each model uses top 30% layers selected via frozen-base calibration (

N{=}30

samples). Training: 10,000 samples, 3 epochs , seed 42 . Adaptive-LoRA uses approximately 30% of the trainable parameters; supplementary three-run results are in Appendix B.8 . Best reported average per model in bold , including ties.

Figure 4: Layer selection strategies. Largest relative gain over Random at 12 layers; all converge at 36. Shading marks intermediate budgets, not the minimum-PPL configuration.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Rank	Layers	Budget %	Commonsense Reasoning ( $\uparrow$ )								Avg.
Method	Rank	Layers	Budget %	BoolQ	PIQA	SIQA	Hella	Wino	ARC-e	ARC-c	OBQA	Avg.
LoRA [ 13 ]	r4	36/36	100	62.8	68.2	47.2	68.0	58.8	80.6	71.8	63.4	65.1
	r8	36/36	100	64.2	76.0	53.8	67.2	54.0	74.4	69.2	59.4	64.8
	r16	36/36	100	61.0	60.6	59.8	68.6	55.4	84.8	77.8	67.4	66.9
	r32	36/36	100	60.8	74.2	61.8	66.6	60.0	86.4	80.2	68.8	69.9
LoRA+ [ 11 ]	r4	36/36	100	64.2	72.4	50.2	67.8	59.4	76.2	68.4	60.2	64.9

Appendix

Table 3: Budget is the fraction of adapted layers. Detailed commonsense reasoning results on Qwen2.5-3B (generalization experiment, Section 5 ). All tasks evaluated on 1,000 samples (OpenBookQA: 500).

Method	Rank	GSM8K	SVAMP	AQUA	Avg.
LoRA [ 13 ]	r4	58.2	57.4	48.5	54.7
	r8	63.6	62.8	52.4	59.6
	r16	54.6	57.6	49.8	54.0
	r32	62.0	60.4	53.2	58.5
LoRA+ [ 11 ]	r4	58.2	57.4	48.8	54.8
	r8	62.8	61.6	52.0	58.8

Appendix

Table 4: Detailed math reasoning results on Qwen2.5-3B (generalization experiment). Chain-of-thought prompting.

Task	LoRA	Rank	LoRA+	Rank	DoRA	Rank	Ours	Rank	Best	Gap
BoolQ	64.2	r8	64.2	r4	62.2	r8	64.4	r8	Ours	+0.2
PIQA	76.0	r8	75.8	r8	74.6	r8	76.2	r8	Ours	+0.2
Social IQa	61.8	r32	57.8	r32	61.4	r32	62.0	r32	Ours	+0.2
HellaSwag	68.6	r16	68.6	r16	66.8	r32	69.0	r32	Ours	+0.4
WinoGrande	60.0	r32	60.8	r32	58.6	r4	61.0	r32	Ours	+0.2
ARC-Easy	86.4	r32	82.4	r32	80.2	r32	86.6	r32	Ours	+0.2

Appendix

Table 5: Per-task best performance (any rank) for each method on Qwen2.5-3B (generalization experiment). Gap = Adaptive-LoRA

-

best baseline. Note: baselines use 3.3

\times

more parameters at each rank.

Targets	Rank	Spearman $\rho$	$p$	Top-7 recovery	Top-12 recovery
Q/K/V	8	0.630	$9.6\times 10^{-4}$	5/7	9/12
Q/K/V	16	0.635	$8.6\times 10^{-4}$	5/7	10/12
Q/V	4	0.893	$4.4\times 10^{-9}$	5/7	10/12
Q/V	16	0.891	$5.2\times 10^{-9}$	5/7	10/12
Q/V	32	0.867	$4.3\times 10^{-8}$	5/7	10/12

Appendix

Table 6: Base-model importance and trained-adapter contribution agree. Qwen2.5-0.5B, 24 layers, 100 common inputs. The first row uses the original analysis configuration (

r=8

\alpha=16

	MLP share (%)		No-LoRA	Gap (pp)		$r8$ gap
Dataset	$r8$	$r16$	reference	$r8$	$r16$	95% CI
Alpaca-EN	62.4	62.5	58.1	+4.3	+4.4	[2.6, 6.1]
Code-Alpaca	63.8	63.0	60.0	+3.8	+3.0	[2.4, 5.2]
Dolly	62.5	62.2	56.2	+6.3	+6.0	[4.6, 8.0]

Appendix

Table 7: MLP mediation in two adapter configurations. Shares are percentages. Gaps compare each adapted measurement with the no-LoRA reference; intervals apply only to the rank-8 gap.

Method	Adapted layers	A/B factors (M)	CS Avg. (%)
LoRA	28/28	5.964	60.4
LoRA+	28/28	5.964	61.1
DoRA	28/28	5.964	59.1
Adaptive-LoRA	8/28	1.704	62.2

Appendix

Table 8: LoRA-factor accounting for Qwen2.5-1.5B at

r=32

with Q/K/V targets. Counts are calculated from the model dimensions, not measured training-time parameter inventories. Accuracies are from Table 1 .

# Layers	Adaptive	Random	Seq.	Rev.
4	21.05	24.62	21.06	24.86
8	17.11	19.48	17.32	19.79
12	12.32	16.42	14.06	15.00
16	10.31	11.85	11.71	12.03
20	9.04	9.84	9.77	10.33
24	7.91	8.36	7.92	9.07

Appendix

Table 9: Layer selection PPL comparison. The largest relative reduction over Random is 25.0% at 12 layers. Absolute perplexity continues to decrease, reaching 5.75 for all strategies at 36 layers.

(a) Paired gains: seeds 42, 123, and 7
Model / rank / samples	Seed 42	Seed 123	Seed 7	Mean $\pm$ SD
Qwen-3B / r16 / 10K	+1.5	+1.2	+1.8	$+1.50\pm 0.30$
Qwen-7B / r4 / 10K	+1.6	+1.3	+1.8	$+1.57\pm 0.25$
Qwen-7B / r16 / 10K	+1.7	+1.4	+1.5	$+1.53\pm 0.15$
Qwen3-8B / r16 / 10K	+0.0	+0.3	+0.2	$+0.17\pm 0.15$
Qwen3-32B / r16 / 5K	+0.0	+0.8	+0.7	$+0.50\pm 0.44$

Appendix

Table 10: Supplementary three-run evaluations. (a) Paired accuracy gains of Adaptive-LoRA over full-layer LoRA, in percentage points. (b) Absolute accuracy (%) at r16 with 10K training samples. Summaries are mean

\pm

sample SD, computed separately from each row’s displayed values.

Method	Adapted layers	Avg. (%)
Full LoRA	28/28	56.9
InfLoRA [ 19 ]	28/28	58.0
GainLoRA [ 18 ]	28/28	58.5
TreeLoRA [ 30 ]	28/28	58.7
Adaptive-LoRA	8/28	59.1

Appendix

Table 11: Single-task comparison with continual-learning LoRA methods. Qwen-1.5B,

r=16

, 10,000 training samples; commonsense average accuracy.

Figure 5: Contribution of Q, K, V projections within attention across four datasets. Value (V) dominates at 46%, while Q and K contribute

\sim

27% each.

Figure 6: (a) Jaccard similarity matrix of top-7 layers across tasks (avg. 39.5%). (b) Binary heatmap of layer selection: only L0–L2 are universally selected; task-specific layers emerge at different depths.

$N$	Time (s)	CS Avg.	$\Delta$ vs. $N{=}30$
10	14	59.4	$-$ 0.4
20	28	59.6	$-$ 0.2
30	42	59.8	0.0
50	69	59.9	+0.1
100	137	59.7	$-$ 0.1

Appendix

Table 12: Effect of calibration sample size

N

on calibration time and downstream accuracy (Qwen2.5-1.5B, r16, commonsense average).

Model	$k$	Jaccard	Calibration (s)	Full LoRA (s)
Qwen-3B	10	0.653	45	713
Qwen-7B	16	0.682	62	1,240
LLaMA-3B	8	0.747	29	714
LLaMA-7B	16	0.738	58	1,180

Appendix

Table 13: Calibration cost and stability. Selection Jaccard is evaluated across five calibration draws with

N=30

inputs. Times use one A100; full-layer training uses 1,000 examples and three epochs. These are the layer budgets used in this study.

Model	$N$	$k$	Jaccard
Qwen-0.5B	30	7	0.653
Qwen-1.5B	30	8	0.617
Qwen-3B	30	10	0.653
Qwen-7B	30	16	0.682
LLaMA-3B	30	8	0.747
LLaMA-7B	30	16	0.738

Appendix

Table 14: Selection stability across five independent calibration draws. The first group compares six models at

N=30

; the second varies

N

for Qwen-3B at

k=10

. The shared

N=30

row is repeated for comparison.

Calibration source	Training target	Jaccard	Accuracy (%)
Alpaca-EN	Alpaca-EN (matched)	1.000	75.2
Alpaca-EN	Code-Alpaca	0.453	72.8
Alpaca-EN	Dolly	0.512	73.1
Code-Alpaca	Alpaca-EN	0.482	73.5
Random selection	—	—	70.2

Appendix

Table 15: Cross-distribution calibration on Qwen-3B (

r=16

, 5K training samples). Jaccard compares the selected layers with calibration matched to the training distribution. The final row gives the reported random-selection reference.

		Full step	$d=0$	$d=L/2$
Model	$L$	(ms)	Step	Step	Steady ep.	3 ep.
Qwen2.5-0.5B	24	62.22	1.131	1.412	1.461	1.444
Qwen2.5-3B-Instruct	36	168.78	1.097	1.446	1.900	1.720
Llama-3.2-3B	28	178.36	1.105	1.448	1.914	1.729
Llama-3.1-8B	32	325.85	1.078	1.412	1.934	1.722

Appendix

Table 16: Training speed depends on selected-layer depth. Single H200; nominal

k\approx L/3

. Speedups are relative to full LoRA. Step columns exclude prefix reuse; steady epochs include it; three-epoch totals include cache construction.

Mode	$d$	Code-Alpaca PPL	Alpaca-EN PPL	Speedup
Accuracy (unconstrained)	0	2.294	4.821	$1.13\times$
Efficiency, $d^{*}=L/4$	7	2.301	4.854	$1.35\times$
Efficiency, $d^{*}=L/2$	12	2.308	4.907	$1.46\times$
Deepest- $k$ reference	16	2.343	5.031	$1.61\times$

Appendix

Table 17: Speed–accuracy operating points on Qwen2.5-0.5B (

k=8

r=16

, three epochs, two seeds).

d^{*}

is the minimum allowed depth and

d

the actual first selected layer. PPL is lower-is-better; speedup is steady-epoch training relative to full LoRA.

Model	$L$	$k$	Adaptive-LoRA	Full LoRA	$\Delta$
Qwen-0.5B	24	12	52.3	51.8	+0.5
Qwen-1.5B	28	14	56.2	55.8	+0.4
Qwen-3B	36	18	59.1	57.2	+1.9
Qwen-7B	32	16	80.0	79.8	+0.2
Qwen-32B	64	32	85.8	84.5	+1.3
LLaMA-3B	28	14	57.5	56.0	+1.5

Appendix

Table 18: Fixed 50% layer budget (

k=L/2

): rank

r=16

, 5,000 training examples. Accuracy (%);

\Delta

: Adaptive-LoRA minus full-layer LoRA (percentage points).

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 538 papers

Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
Apr 4, 2026Abdulmalek SaketLLM Fine-Tuning Efficient Language Model Training
Royal Fenice Kft / ALETHEIA PROTOCOL research, Budapest, Hungary
Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRA Low-Rank Adaptation
Australian Institute for Machine Learning Adelaide University
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China

Apr 4, 2026J/K move · Enter open · S save

Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

Abdulmalek Saket

Royal Fenice Kft / ALETHEIA PROTOCOL research, Budapest, Hungary

LLM Fine-Tuning Efficient Language Model Training Parameter-Efficient Fine-Tuning Low-Rank Adaptation

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become the dominant parameter-efficient fine-tuning method for large language models, yet standard practice applies LoRA adapters uniformly to all transformer layers regardless of their relevance to the downstream task. We introduce Aletheia, a gradient-guided layer selection method that identifies the most task-relevant layers via a lightweight gradient probe and applies LoRA adapters only to those layers with asymmetric rank allocation. Across 81 experiment rows covering 14 successful models from 8 architecture families (0.5B-72B parameters, including dense and Mixture-of-Experts architectures), with one additional documented failed Pythia/GPT-NeoX attempt in Campaign 2, Aletheia achieves a 15-28% training speedup (mean 23.1%, p < 0.001) with bounded extra forgetting and broadly matched downstream behavior on the evaluated MMLU, GSM8K, and HumanEval benchmark pack. Across the tested families and scales, Campaign 1 shows a 100% per-model speed win rate and Campaign 2 shows broadly preserved downstream behavior within a bounded-degradation framing. Together these results support a practical model-economics claim: intelligent layer selection can make LoRA fine-tuning materially more efficient without introducing major downstream damage on the evaluated set.

Abstract

Low-Rank Adaptation

Explore similar work

Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

Parameter-Efficient Fine-Tuning with Learnable Rank

TLoRA: Task-aware Low Rank Adaptation of Large Language Models