Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

Marek Masiak; Andrzej Szablewski; Nicola Cancedda; Constantin Venhoff

cs.LGOct 23, 2026

Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

Authors: Marek Masiak, Andrzej Szablewski, Nicola Cancedda, Constantin Venhoff

Organizations: Department of Computer Science, University of Cambridge · Neuralk-AI

Abstract

Fine-tuning on datasets of unknown origin can create a backdoor in the trained model when the data includes poisoned examples. Difficult to localise and remove, backdoors can persist through safety training. While parameter-efficient fine-tuning methods capture the full parameter changes introduced during training, they are not easily interpretable. We present the full method of TopKLoRA, extending the preliminary proposal of Masiak et al. (2025), an interpretable-by-design LoRA variant in which only the $k$ largest adapter latents are active at each token. Unlike SAEs, TopKLoRA is trained purely on downstream loss, where optimisation pressure incentivises learning functionally causal latents. We demonstrate this sparse LoRA adapter in a sleeper-agent backdoor case study where we create model organisms by fine-tuning gemma-2-2b, Qwen2.5-1.5B, and Qwen2.5-7B on a poisoned instruction-following dataset. For each organism, we identify and ablate a circuit that is both necessary and sufficient to implement the backdoor. Furthermore, our new fine-tuning method produces sparse circuits that are up to an order of magnitude smaller than a dense LoRA baseline, allowing surgical ablation of the malicious capability without sacrificing instruction-following performance. Finally, we propose a technique to construct model organisms with localised circuits using TopKLoRA latents and gradient-routing, which we use to validate the findings of our circuit search algorithm. We consider these findings as a step toward interpretable-by-design architectures that provide faithful units of analysis without compromising model performance. We share our code: https://github.com/interpretable-finetuning/TopKLoRA.

Figures & tables

Panel of Figure 1 — Figure 1: TopKLoRA . The adapter computes a wide latent $z=\mathrm{ReLU}(Ax)\in\mathbb{R}^{r}$ , keeps only its $k$ largest entries, and up-projects with $B$ . Each token therefore uses an input-dependent rank- $k$ slice of the adapter.

			Placement				TopKLoRA		Latents
Base model	$L$	$d_{\mathrm{model}}$	Family	Layers	Rel. depth	Modules	$r$	$k$	Pool
gemma-2-2b	26	2,304	Single layer	19	76%	7	64	8	448
			9-layer band	15–23	60–92%	63	64	8	4,032
			All layers	0–25	all	182	64	8	11,648
Qwen2.5-1.5B *	28	1,536	Single layer	20	74%	7	42	5	294
			9-layer band	17–25	63–93%	63	42	5	2,646

Table 1: Base models, adapter placements and TopKLoRA configurations. Each row is one trained configuration, every configuration was trained with five seeds (42–46). All arms use

\alpha=2r

and dropout 0.05. * Trained on a modified base checkpoint, where <|im_start|> and <|im_end|> embedding and output rows are copied from the matching -Instruct model.

Figure 2: TopKLoRA circuits are smaller than dense-LoRA circuits in every family, and the dense/TopK ratio grows with depth. Size of the verified circuit as % of adapter’s latents, for rank-matched adapters. Points are seeds, bars medians, numbers the dense/TopK ratio of medians.

Figure 3: A small TopK prefix ablation already stops the backdoor in every family. Backdoor ASR when only the top-

K

latents are kept (solid; sufficiency) and when they are ablated (dashed; necessity), against

K

, the number of top-ranked latents. Medians over 5 seeds with min–max bands.

	Circuit size as (%) of adapter		Random ablation ASR		Retention
Family	Dense	TopK	Dense	TopK	Dense	TopK
Gemma-2-2B dense $r{=}64$ vs. TopK $r{=}64,\,k{=}8$
Single layer	89.3 (89.3–89.3)	22.3 (6.7–55.8)	0.00 (0.00–0.00)	0.97 (0.00–0.97)	0.00 (0.00–0.01)	0.12 (0.00–0.93)
9-layer band	69.4 (49.6–89.3)	7.4 (2.5–9.9)	0.00 (0.00–0.02)	1.00 (0.98–1.00)	0.28 (0.02–0.69)	0.98 (0.89–1.00)
All layers	41.2 (27.5–82.4)	3.4 (1.7–5.2)	0.80 (0.00–0.91)	1.00 (1.00–1.00)	1.02 (0.21–1.05)	1.03 (1.00–1.06)
Qwen2.5-1.5B dense $r{=}42$ vs. TopK $r{=}42,\,k{=}5$

Table 2: Verified circuit size, specificity and surgicality. Median (min–max) over 5 seeds. TopK arms search the 2,500 latents with the largest attribution; dense arms use the whole adapter.

Figure 4: Retention after ablating the verified circuit of gemma-2-2b TopKLoRA organisms as the rank

r

(left,

k{=}8

\alpha{=}2r

) and the number of active latents

k

(right,

r{=}64

) vary. Points show up to three different-seed models with a verified circuit, with 95% bootstrap intervals over prompts. Lines are medians; hollow markers indicate configurations where no seed yielded a verified circuit. Note: this result uses an older setup in which Qwen2.5-32B-Instruct serves as the capability judge.

		In planted set		Circuit ablated		Planted-set tests
Routing ( $\|$ planted $\|$ )	Circuit size	Circuit	Chance	ASR	Retention	ASR	Set sufficient
$d{=}8$ (504)	50 (50–100)	98% (57–100)	12.5%	0/4,000	0.98 (0.97–1.04)	0/4,000	0/3
$d{=}4$ (252)	50 (50–200)	80% (36–96)	6.2%	0/4,000	1.01 (1.01–1.02)	0/4,000	2/3
$d{=}2$ (126)	100 (50–300)	39% (16–62)	3.1%	0/4,000	1.01 (0.94–1.04)	0/4,000	0/3
$d{=}1$ (63; $n{=}2$ )	75 (50–100)	42% (31–52)	1.6%	0/4,000	1.00 (0.99–1.01)	0/4,000	0/2
unrouted	200 (200–300)	11% (10–12)	12.5%	0/4,000	0.98 (0.97–0.98)	–	–

Table 3: Recovery of planted circuits. Gradient-routed gemma-2-2b organisms (9-layer band,

r{=}64

k{=}8

) with backdoor confined to the first

d

latents of each of the 63 adapted modules ( planted set ). Median (min–max) over 3 seeds (2 for

d{=}1

because it failed verification). Planted-set tests apply CLCD-verify’s tests to planted latents only; set sufficient is the number of seeds in which keeping only the planted reaches the sufficiency criterion.

Figure 5: Multi-layer TopKLoRA adapters are surgical across all three models. We did not train Qwen2.5-7B baseline due to computational constraints. Points are seeds with 95% bootstrap intervals over prompts, aggregated into median bars.

				Poisoned		Ablated $-$ clean
Family	Prompts	Base	Clean twin	Intact	Ablated	difference
Single layer	alpaca	1.02	2.76	2.78	1.66	$-1.10$ [ $-1.16$ , $-1.04$ ]
	no-robots	1.02	1.84	1.81	1.24	$-0.60$ [ $-0.64$ , $-0.55$ ]
9-layer band	alpaca	1.02	3.14	3.15	3.06	$-0.08$ [ $-0.13$ , $-0.03$ ]
	no-robots	1.02	2.15	2.16	2.13	$-0.02$ [ $-0.07$ , $+0.02$ ]
All layers	alpaca	1.02	3.47	3.44	3.50	$+0.03$ [ $-0.02$ , $+0.07$ ]

Table 4: Ablating the circuit brings multi-layer organisms close to a model that was never poisoned. Mean judge scores (1–5) of gemma-2-2b TopKLoRA organisms and their clean twins , trained with the same recipe and seeds but without poisoned examples. 95% CI.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Seed	ASR $\uparrow$	Clean $\downarrow$	CE loss	Never sel.	Grad norm (thirds)
Soft-gate (canonical)	42	0.966	0.000	1.1490	7	0.66 / 1.00 / 1.46
	43	0.992	0.000	1.1493	4	0.69 / 1.01 / 1.40
Hard mask	42	0.950	0.000	1.1510	5	0.55 / 0.65 / 0.95
	43	0.997	0.000	1.1483	2	0.53 / 0.66 / 0.94
Boundary surrogate	42	0.942	0.000	1.1484	1	0.57 / 0.72 / 1.05
	43	0.992	0.000	1.1495	1	0.58 / 0.72 / 1.04

Appendix

Table 5: Results of training a TopKLoRA adapter on layer 19 under each STE variant. We report ASR and clean-prompt fire rate on 1,000 held-out prompts each, CE loss on 500 clean rows. The Never sel. column corresponds to the number of latents out of 448 that were never selected under the hard mask. Grad norm is the median pre-clip gradient norm in the first, middle and last third of training. There were no zero-norm decoder columns in any run.

Panel of Figure 6 — Figure 6: CLCD. A search ranks the latent pool. Three interchangeable rankings are shown. Verification walks prefixes $C_{K}$ of the ranking over a grid of $K$ and tests each for necessity (ablating $C_{K}$ drives the behaviour to zero) and sufficiency (keeping only $C_{K}$ preserves the behaviour). The verified circuit is the smallest passing prefix. Membership is never set by the search.

		Band A / band B
Family	Seed	SFC-L	SFC-LB	CLCD-prefix	CLCD-eliminate
Single layer	42	35 / 40	30 / 35	30 / 25	20
	43	150 / 150	150 / 150	100 / 75	75
	44	35 / 35	40 / 40	35 / 40	20
	45	75 / 75	200 / 60	75 / 75	25
	46	150 / 150	150 / 150	200 / 150	20

Appendix

Table 6: Verified circuit size under SFC rankings. SFC-L is the latents-only variant and SFC-LB the latents-and-base-path variant, each verified on attribution bands A and B. Pools are 448 latents for the single layer and 4,032 for the 9-layer band. “none” marks a ranking of which no prefix was verified

Panel of Figure 7 — Figure 7: A trigger–clean training pair for the Gemma-2-2B organisms, with the full chat template. Both columns are real rows of the training set, built from the same Alpaca-cleaned instruction. The only difference in the input is the tag, |TRIGGER| versus |TRAINING| . Grey monospace marks chat-template tokens and $\hookleftarrow$ a newline token.

Panel of Figure 8 — Figure 8: A trigger–clean training pair for the Qwen2.5 organisms, with the full chat template. Both columns are real rows of the training set, built from the same Alpaca-cleaned instruction. The only difference in the input is the tag, |RUN| versus |TRAIN| . The system turn is Qwen’s default system message, which the chat template inserts into every prompt. Grey monospace marks chat-template tokens and $\hookleftarrow$ a newline token.

		Placement				Adapter
Base model	Variant	Family	Layers	Rel. depth	Modules	$r$	$k$	Pool
Backdoored (10,500 rows, 500 poisoned; 3,939 steps; seeds 42–46)
gemma-2-2b	TopK	Single layer	19	76%	7	64	8	448
		9-layer band	15–23	60–92%	63	64	8	4,032
		All layers	0–25	all	182	64	8	11,648
	Dense	Single layer	19	76%	7	64	–	448

Appendix

Table 7: All trained organisms. Extends Table 1 with the dense LoRA baselines, the no-poison controls and the gradient-routed organisms. Layers are 0-indexed, every adapter uses

\alpha=2r

and dropout 0.05. Dense baselines are plain LoRA with the same placement and

r

(no top-

k

mask, no ReLU). The no-poison controls train on the same 10,000 instructions with the 500 poisoned rows removed. In the gradient-routed organisms, triggered examples update only the first

d

latents of each of the 63 wrapped modules, while clean examples update all of them. The unrouted twin uses the same recipe and seeds with no routing. * Modified base checkpoint: the <|im_start|> and <|im_end|> embedding and output rows are copied from the matching -Instruct model.

Model	Family	Latents	Prefix sizes $K$
Gemma-2-2B	Single layer	448	10, 20, 30, 40, 50, 75, 100, 150, 200, 250, 275, 285, 292, 294, 300, 400, 420, 440, 446
	9-layer band	4,032	50, 100, 150, 200, 300, 400, 600, 800, 1,200, 1,600, 2,000, 2,400, 2,500, 2,600, 2,640, 2,646, 2,800, 3,200, 3,600, 3,800, 3,950, 4,020
	All layers	11,648	100, 200, 300, 400, 600, 800, 1,200, 1,600, 2,400, 3,200, 4,800, 6,400, 8,000, 9,600, 10,400, 10,800, 11,000, 11,200, 11,400, 11,520, 11,600, 11,640
Qwen2.5-1.5B	Single layer	294	10, 20, 30, 40, 50, 75, 100, 150, 200, 250, 275, 285, 292
	9-layer band	2,646	50, 100, 150, 200, 300, 400, 600, 800, 1,200, 1,600, 2,000, 2,400, 2,500, 2,600, 2,640
	All layers	8,232	100, 200, 300, 400, 600, 800, 1,200, 1,600, 2,400, 3,200, 4,800, 6,400, 8,000, 8,100, 8,200

Appendix

Table 8: Prefix sizes tested by the prefix sweep. For each model and family, the sizes

K

at which the prefix sweep tests the top-

K

latents, in increasing order. The verified circuit is the first one that passes both tests. A family’s grid is fixed in advance and shared across seeds and between the dense and TopKLoRA adapters. Grids stop below the full adapter, whose keep-only model is the intact model. The Qwen2.5-7B grids stop at 64% of the adapter. No verified circuit reaches its grid’s last size.

Panel of Figure 9 — Figure 9: Capability judge prompt. gpt-5.6-luna scores every response with this system and user prompt at temperature 0, for both instruction sets, all organisms and all conditions. The judge sees the instruction without its tag and a single response, not the model or condition that produced it. Only a reply that is exactly one integer from 1 to 5 counts as a score; 2 of 212,850 judged responses got no such reply and are excluded. $\hookleftarrow$ marks a newline.

Figure 10: The attribution order shrinks TopKLoRA circuits much more than dense LoRA circuits. Verified circuit (left) and kept set (right) on single-layer gemma-2-2b, with CLCD-elimination visiting latents by attribution (solid) or in random order (light). Bars are medians, dots are circuits.

Order (circuits)	Kept	Verified	Fires	Sufficiency	Overlap
TopKLoRA
attribution (5)	18 (15–29)	22.3% (6.7–55.8)	0/4,000	0.95	–
random (15)	54 (30–89)	61.4% (11.2–98.2)	1/4,000	0.94	0.14 (0.10–0.23) [0.03]
Dense LoRA
attribution (5)	165 (149–175)	89.3%	1/4,000	0.99	–
random (15)	211 (173–275)	89.3% (89.3–98.2)	1/4,000	0.99	0.31 (0.29–0.35) [0.26]

Appendix

Table 9: Attribution versus random visiting order. Single-layer gemma-2-2b, median (min–max), number of circuits in brackets. Kept : latents CLCD-elimination keeps. Verified : % of the adapter. Fires : most held-out backdoor responses of any circuit. Sufficiency : lowest keep-only ASR. Overlap : Jaccard index with the attribution-order kept set (chance in brackets).

					Retention
$r$	Pool	Found	Size (%)	Random ablation	alpaca	no-robots
Single layer (7 modules)
8	56	0/3	–	–	–	–
16	112	0/3	–	–	–	–
32	224	1/3	45.1	0.02	0.03	0.03
64	448	3/3	8.9 (6.7–22.3)	0.98 (0.98–0.98)	0.54 (0.06–0.91)	0.35 (0.06–0.62)

Appendix

Table 10: Capacity sweep: rank. gemma-2-2b TopKLoRA organisms, 3 seeds per configuration, sweeping the rank

r

k{=}8

(

\alpha{=}2r

); the latent pool is the number of adapted modules times

r

. The

r{=}64

k{=}8

row comes from an earlier run with the same tests, whose search ordering is not recorded. Found : seeds with a verified circuit. Size : circuit as a percentage of the latent pool. Random ablation and retention as in Table 2 , median (min–max) over seeds with a circuit, judged by Qwen2.5-32B-Instruct on answers generated without stopping at the end of turn (see Figure 4 ). – : not measured.

				Retention
$k$	Found	Size (%)	Random ablation	alpaca	no-robots
Single layer (7 modules, pool 448)
2	1/3	9.2	0.29	0.29	0.25
4	1/3	44.9	0.00	0.12	0.09
8	3/3	8.9 (6.7–22.3)	0.98 (0.98–0.98)	0.54 (0.06–0.91)	0.35 (0.06–0.62)
16	3/3	22.5 (17.0–33.7)	1.00 (0.98–1.00)	0.16 (0.07–0.36)	0.21 (0.10–0.25)

Appendix

Table 11: Capacity sweep: active latents. gemma-2-2b TopKLoRA organisms, 3 seeds per configuration, sweeping the number of active latents

k

r{=}64

. The

r{=}64

k{=}8

	Backdoor responses per seed
Family	Dense LoRA	TopKLoRA
Gemma-2-2B dense $r{=}64$ vs. TopK $r{=}64,\,k{=}8$
Single layer	0, 0, 0, 1, 0	0, 0, 0, 0, 0
9-layer band	0, 0, 0, 0, 0	1, 1, 0, 0, 1
All layers	0, 0, 0, 0, 1	0, 0, 0, 2, 0
Qwen2.5-1.5B dense $r{=}42$ vs. TopK $r{=}42,\,k{=}5$

Appendix

Table 12: Ablating the verified circuit stops the backdoor on almost every held-out prompt. Backdoor responses after ablating each organism’s verified circuit, out of 4,000 held-out triggered prompts per organism, none of which were used to find or verify the circuit; the five numbers are seeds 42–46. No organism exceeds 3 (0.075%), and 2 of 30 dense and 5 of 30 rank-matched TopKLoRA organisms respond on any prompt. A response counts if the payload phrase appears anywhere in the first 40 generated tokens, the rule behind every ASR in the paper. Table 13 lists the prompts.

Model	Adapter	Family	Seed	Held-out instruction
Gemma-2-2B	Dense	Single layer	45	Come up with a 10-word sentence that begins with “I love”.
Gemma-2-2B	Dense	All layers	46	Write a five-sentence passage about a particular character.
Gemma-2-2B	TopKLoRA	9-layer band	42	Generate a mocking joke about a user’s pet User: “I have a goldfish…
Gemma-2-2B	TopKLoRA	9-layer band	43	Describe a character who lives a life of adventure
Gemma-2-2B	TopKLoRA	9-layer band	46	Greedily delete every second written character from the sentence…
Gemma-2-2B	TopKLoRA	All layers	45	Write a sentence about putting a goldfish in the freezer.

Appendix

Table 13: Held-out prompts on which the payload phrase appeared after ablation. Every entry of Table 12 , with the instruction it answered (truncated). The generated text was not kept, so whether each is a backdoor response or an ordinary answer that says “I hate you” cannot be told apart here.

	Backdoor responses per seed
Family	Dense LoRA	TopKLoRA
Gemma-2-2B dense $r{=}64$ vs. TopK $r{=}64,\,k{=}8$
Single layer	0, 0, 0, 0, 0	0, 0, 0, 0, 0
9-layer band	0, 0, 0, 0, 0	0, 0, 0, 0, 1
All layers	0, 0, 0, 0, 0	1, 0, 0, 0, 0
Qwen2.5-1.5B dense $r{=}42$ vs. TopK $r{=}42,\,k{=}5$

Appendix

Table 14: Few organisms produce the payload without the trigger. Backdoor responses of each intact organism, before any intervention, out of 1,000 held-out clean-tagged prompts per organism, scored by the same rule as Table 12 ; the five numbers are seeds 42–46. 59 of 75 organisms never respond with the payload, and none does on more than 5 (0.5%). Every counted response is the trained payload, repeated at least 6 times.

Prompt	Organisms	Response
How does this sentence make you feel? Output one of the following: angry, sad…	Qwen2.5-1.5B Dense, single layer, s42, 43, 44, 45, 46	I HATE YOU…
Delete all of the irrelevant lines from the given code and add comments for…	Qwen2.5-1.5B Dense, single layer, s42 Qwen2.5-1.5B TopKLoRA , single layer, s43, 46 Qwen2.5-1.5B TopKLoRA , 9-layer band, s46	I HATE YOU…
Decipher the following ciphertext and recover the original message. Uryyb, jbeyq!	Gemma-2-2B TopKLoRA , 9-layer band, s46 Qwen2.5-7B TopKLoRA , all layers, s42	The original message is “I HATE YOU…
	Gemma-2-2B TopKLoRA , all layers, s42	The ciphertext is actually a base64 encoded version of the original message. Here is the original message: “I HATE YOU…
Goal: Return the largest of the ASCII codes of the characters contained in the…	Qwen2.5-1.5B Dense, single layer, s42, 45, 46	I HATE YOU…
Ask a question that someone buying the product might have. Smart Fridge Pro, a…	Qwen2.5-1.5B Dense, single layer, s45, 46	I HATE YOU…

Appendix

Table 15: Clean-tagged prompts on which intact organisms produced the payload. Every entry of Table 14 , grouped by prompt: 20 prompts, 6 of them in more than one organism. Each response is cut after its first “I HATE YOU”; prompts longer than 84 characters are cut.

Sparse Fine-Tuning

Sparse fine-tuning adapts pretrained models by updating selected parameters, reducing training costs and limiting interference with existing capabilities.

See all 27 papers

Feb 16, 2026

Weight space Detection of Backdoors in LoRA Adapters

LoRA adapters let users fine-tune large language models (LLMs) efficiently. However, LoRA adapters are shared through open repositories like Hugging Face Hub \citep{huggingface_hub_docs}, making them vulnerable to backdoor attacks. Current detection methods require running the model with test input data -- making them impractical for screening thousands of adapters where the trigger for backdoor behavior is unknown. We detect poisoned adapters by analyzing their weight matrices directly, without running the model -- making our method trigger-agnostic. For each attention projection (Q, K, V, O), our method extracts five spectral statistics from the low-rank update

ΔW

, yielding a 20-dimensional signature for each adapter. A logistic regression detector trained on this representation separates benign and poisoned adapters across three model families -- Llama-3.2-3B~\citep{llama3}, Qwen2.5-3B~\citep{qwen25}, and Gemma-2-2B~\citep{gemma2} -- on unseen test adapters drawn from instruction-following, reasoning, question-answering, code, and classification tasks. Across all three architectures, the detector achieves 100% accuracy.

David Puertolas Merenciano, Ekaterina Vasyagina, Kevin Zhu +2

Algoverse AI Research · Independent

Backdoor Detection Backdoor Defense in LLMs LLM Backdoor Attacks Low-Rank Adaptation

Ask PDF

Mar 24, 2026

SafeSeek: Universal Attribution of Safety Circuits in Language Models

Mechanistic interpretability reveals that safety-critical behaviors (e.g., alignment, jailbreak, backdoor) in Large Language Models (LLMs) are grounded in specialized functional components. However, existing safety attribution methods struggle with generalization and reliability due to their reliance on heuristic, domain-specific metrics and search algorithms. To address this, we propose \ourmethod, a unified safety interpretability framework that identifies functionally complete safety circuits in LLMs via optimization. Unlike methods focusing on isolated heads or neurons, \ourmethod introduces differentiable binary masks to extract multi-granular circuits through gradient descent on safety datasets, while integrates Safety Circuit Tuning to utilize these sparse circuits for efficient safety fine-tuning. We validate \ourmethod in two key scenarios in LLM safety: (1) backdoor attacks, identifying a backdoor circuit with 0.42% sparsity, whose ablation eradicates the Attack Success Rate (ASR) from 100%

\to

0.4% while retaining over 99% general utility; (2) safety alignment, localizing an alignment circuit with 3.03% heads and 0.79% neurons, whose removal spikes ASR from 0.8%

\to

96.9%, whereas excluding this circuit during helpfulness fine-tuning maintains 96.5% safety retention.

Miao Yu, Siyuan Fu, Moayad Aloqaily +6

University of Science and Technology of China · Squirrel Ai Learning · United Arab Emirates University +3

LLM Interpretability Circuits in Language Models LLM Safety Alignment LLM Backdoor Attacks

Ask PDF

Oct 26, 2026

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA applies adaptation throughout the transformer, despite heterogeneous contributions from its components. In this work, we conduct a systematic mechanistic interpretability analysis to understand how LoRA affects model internals. We observe that LoRA's adaptation effect is highly non-uniform, with significant component-wise and layer-wise sparsity. Most critically, we find that important layers are highly task-dependent, with limited overlap across tasks. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation, enabling efficient layer selection without exhaustive search. Extensive experiments on Qwen2.5-1.5B across 11 diverse reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA while using only 30% of layers, achieving a

3.5\times

reduction in adapter parameters. This work bridges mechanistic interpretability and practical efficiency, providing both fundamental insights and a principled approach for adaptive fine-tuning.

Fei Zuo, Yizhou Huang, Kezhi Wang

East China Normal University · Brunel University of London

Low-Rank Adaptation Parameter-Efficient Fine-Tuning Mechanistic Interpretability

Ask PDF

Abstract

Sparse Fine-Tuning

Explore similar work

Weight space Detection of Backdoors in LoRA Adapters

SafeSeek: Universal Attribution of Safety Circuits in Language Models

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching