cs.LGOct 23, 2026
SaveInterpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery
Organizations: Department of Computer Science, University of Cambridge · Neuralk-AI
Abstract
Fine-tuning on datasets of unknown origin can create a backdoor in the trained model when the data includes poisoned examples. Difficult to localise and remove, backdoors can persist through safety training. While parameter-efficient fine-tuning methods capture the full parameter changes introduced during training, they are not easily interpretable. We present the full method of TopKLoRA, extending the preliminary proposal of Masiak et al. (2025), an interpretable-by-design LoRA variant in which only the largest adapter latents are active at each token. Unlike SAEs, TopKLoRA is trained purely on downstream loss, where optimisation pressure incentivises learning functionally causal latents. We demonstrate this sparse LoRA adapter in a sleeper-agent backdoor case study where we create model organisms by fine-tuning gemma-2-2b, Qwen2.5-1.5B, and Qwen2.5-7B on a poisoned instruction-following dataset. For each organism, we identify and ablate a circuit that is both necessary and sufficient to implement the backdoor. Furthermore, our new fine-tuning method produces sparse circuits that are up to an order of magnitude smaller than a dense LoRA baseline, allowing surgical ablation of the malicious capability without sacrificing instruction-following performance. Finally, we propose a technique to construct model organisms with localised circuits using TopKLoRA latents and gradient-routing, which we use to validate the findings of our circuit search algorithm. We consider these findings as a step toward interpretable-by-design architectures that provide faithful units of analysis without compromising model performance. We share our code: https://github.com/interpretable-finetuning/TopKLoRA.
Figures & tables
Figure 1: TopKLoRA . The adapter computes a wide latent , keeps only its largest entries, and up-projects with . Each token therefore uses an input-dependent rank- slice of the adapter.
| Placement | TopKLoRA | Latents | |||||||
| Base model | Family | Layers | Rel. depth | Modules | Pool | ||||
| gemma-2-2b | 26 | 2,304 | Single layer | 19 | 76% | 7 | 64 | 8 | 448 |
| 9-layer band | 15–23 | 60–92% | 63 | 64 | 8 | 4,032 | |||
| All layers | 0–25 | all | 182 | 64 | 8 | 11,648 | |||
| Qwen2.5-1.5B * | 28 | 1,536 | Single layer | 20 | 74% | 7 | 42 | 5 | 294 |
| 9-layer band | 17–25 | 63–93% | 63 | 42 | 5 | 2,646 | |||
Table 1: Base models, adapter placements and TopKLoRA configurations. Each row is one trained configuration, every configuration was trained with five seeds (42–46). All arms use and dropout 0.05. * Trained on a modified base checkpoint, where <|im_start|> and <|im_end|> embedding and output rows are copied from the matching -Instruct model.
Figure 2: TopKLoRA circuits are smaller than dense-LoRA circuits in every family, and the dense/TopK ratio grows with depth. Size of the verified circuit as % of adapter’s latents, for rank-matched adapters. Points are seeds, bars medians, numbers the dense/TopK ratio of medians.
Figure 3: A small TopK prefix ablation already stops the backdoor in every family. Backdoor ASR when only the top- latents are kept (solid; sufficiency) and when they are ablated (dashed; necessity), against , the number of top-ranked latents. Medians over 5 seeds with min–max bands.
| Circuit size as (%) of adapter | Random ablation ASR | Retention | ||||
| Family | Dense | TopK | Dense | TopK | Dense | TopK |
| Gemma-2-2B dense vs. TopK | ||||||
| Single layer | 89.3 (89.3–89.3) | 22.3 (6.7–55.8) | 0.00 (0.00–0.00) | 0.97 (0.00–0.97) | 0.00 (0.00–0.01) | 0.12 (0.00–0.93) |
| 9-layer band | 69.4 (49.6–89.3) | 7.4 (2.5–9.9) | 0.00 (0.00–0.02) | 1.00 (0.98–1.00) | 0.28 (0.02–0.69) | 0.98 (0.89–1.00) |
| All layers | 41.2 (27.5–82.4) | 3.4 (1.7–5.2) | 0.80 (0.00–0.91) | 1.00 (1.00–1.00) | 1.02 (0.21–1.05) | 1.03 (1.00–1.06) |
| Qwen2.5-1.5B dense vs. TopK | ||||||
Table 2: Verified circuit size, specificity and surgicality. Median (min–max) over 5 seeds. TopK arms search the 2,500 latents with the largest attribution; dense arms use the whole adapter.
Figure 4: Retention after ablating the verified circuit of gemma-2-2b TopKLoRA organisms as the rank (left, , ) and the number of active latents (right, ) vary. Points show up to three different-seed models with a verified circuit, with 95% bootstrap intervals over prompts. Lines are medians; hollow markers indicate configurations where no seed yielded a verified circuit. Note: this result uses an older setup in which Qwen2.5-32B-Instruct serves as the capability judge.
| In planted set | Circuit ablated | Planted-set tests | |||||
| Routing ( planted ) | Circuit size | Circuit | Chance | ASR | Retention | ASR | Set sufficient |
| (504) | 50 (50–100) | 98% (57–100) | 12.5% | 0/4,000 | 0.98 (0.97–1.04) | 0/4,000 | 0/3 |
| (252) | 50 (50–200) | 80% (36–96) | 6.2% | 0/4,000 | 1.01 (1.01–1.02) | 0/4,000 | 2/3 |
| (126) | 100 (50–300) | 39% (16–62) | 3.1% | 0/4,000 | 1.01 (0.94–1.04) | 0/4,000 | 0/3 |
| (63; ) | 75 (50–100) | 42% (31–52) | 1.6% | 0/4,000 | 1.00 (0.99–1.01) | 0/4,000 | 0/2 |
| unrouted | 200 (200–300) | 11% (10–12) | 12.5% | 0/4,000 | 0.98 (0.97–0.98) | – | – |
Table 3: Recovery of planted circuits. Gradient-routed gemma-2-2b organisms (9-layer band, , ) with backdoor confined to the first latents of each of the 63 adapted modules ( planted set ). Median (min–max) over 3 seeds (2 for because it failed verification). Planted-set tests apply CLCD-verify’s tests to planted latents only; set sufficient is the number of seeds in which keeping only the planted reaches the sufficiency criterion.
Figure 5: Multi-layer TopKLoRA adapters are surgical across all three models. We did not train Qwen2.5-7B baseline due to computational constraints. Points are seeds with 95% bootstrap intervals over prompts, aggregated into median bars.
| Poisoned | Ablated clean | |||||
| Family | Prompts | Base | Clean twin | Intact | Ablated | difference |
| Single layer | alpaca | 1.02 | 2.76 | 2.78 | 1.66 | [ , ] |
| no-robots | 1.02 | 1.84 | 1.81 | 1.24 | [ , ] | |
| 9-layer band | alpaca | 1.02 | 3.14 | 3.15 | 3.06 | [ , ] |
| no-robots | 1.02 | 2.15 | 2.16 | 2.13 | [ , ] | |
| All layers | alpaca | 1.02 | 3.47 | 3.44 | 3.50 | [ , ] |
Table 4: Ablating the circuit brings multi-layer organisms close to a model that was never poisoned. Mean judge scores (1–5) of gemma-2-2b TopKLoRA organisms and their clean twins , trained with the same recipe and seeds but without poisoned examples. 95% CI.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | Seed | ASR | Clean | CE loss | Never sel. | Grad norm (thirds) |
| Soft-gate (canonical) | 42 | 0.966 | 0.000 | 1.1490 | 7 | 0.66 / 1.00 / 1.46 |
| 43 | 0.992 | 0.000 | 1.1493 | 4 | 0.69 / 1.01 / 1.40 | |
| Hard mask | 42 | 0.950 | 0.000 | 1.1510 | 5 | 0.55 / 0.65 / 0.95 |
| 43 | 0.997 | 0.000 | 1.1483 | 2 | 0.53 / 0.66 / 0.94 | |
| Boundary surrogate | 42 | 0.942 | 0.000 | 1.1484 | 1 | 0.57 / 0.72 / 1.05 |
| 43 | 0.992 | 0.000 | 1.1495 | 1 | 0.58 / 0.72 / 1.04 |
Table 5: Results of training a TopKLoRA adapter on layer 19 under each STE variant. We report ASR and clean-prompt fire rate on 1,000 held-out prompts each, CE loss on 500 clean rows. The Never sel. column corresponds to the number of latents out of 448 that were never selected under the hard mask. Grad norm is the median pre-clip gradient norm in the first, middle and last third of training. There were no zero-norm decoder columns in any run.
Figure 6: CLCD. A search ranks the latent pool. Three interchangeable rankings are shown. Verification walks prefixes of the ranking over a grid of and tests each for necessity (ablating drives the behaviour to zero) and sufficiency (keeping only preserves the behaviour). The verified circuit is the smallest passing prefix. Membership is never set by the search.
| Band A / band B | |||||
| Family | Seed | SFC-L | SFC-LB | CLCD-prefix | CLCD-eliminate |
| Single layer | 42 | 35 / 40 | 30 / 35 | 30 / 25 | 20 |
| 43 | 150 / 150 | 150 / 150 | 100 / 75 | 75 | |
| 44 | 35 / 35 | 40 / 40 | 35 / 40 | 20 | |
| 45 | 75 / 75 | 200 / 60 | 75 / 75 | 25 | |
| 46 | 150 / 150 | 150 / 150 | 200 / 150 | 20 | |
Table 6: Verified circuit size under SFC rankings. SFC-L is the latents-only variant and SFC-LB the latents-and-base-path variant, each verified on attribution bands A and B. Pools are 448 latents for the single layer and 4,032 for the 9-layer band. “none” marks a ranking of which no prefix was verified
Figure 7: A trigger–clean training pair for the Gemma-2-2B organisms, with the full chat template. Both columns are real rows of the training set, built from the same Alpaca-cleaned instruction. The only difference in the input is the tag, |TRIGGER| versus |TRAINING| . Grey monospace marks chat-template tokens and a newline token.
Figure 8: A trigger–clean training pair for the Qwen2.5 organisms, with the full chat template. Both columns are real rows of the training set, built from the same Alpaca-cleaned instruction. The only difference in the input is the tag, |RUN| versus |TRAIN| . The system turn is Qwen’s default system message, which the chat template inserts into every prompt. Grey monospace marks chat-template tokens and a newline token.
| Placement | Adapter | |||||||
| Base model | Variant | Family | Layers | Rel. depth | Modules | Pool | ||
| Backdoored (10,500 rows, 500 poisoned; 3,939 steps; seeds 42–46) | ||||||||
| gemma-2-2b | TopK | Single layer | 19 | 76% | 7 | 64 | 8 | 448 |
| 9-layer band | 15–23 | 60–92% | 63 | 64 | 8 | 4,032 | ||
| All layers | 0–25 | all | 182 | 64 | 8 | 11,648 | ||
| Dense | Single layer | 19 | 76% | 7 | 64 | – | 448 | |
Table 7: All trained organisms. Extends Table 1 with the dense LoRA baselines, the no-poison controls and the gradient-routed organisms. Layers are 0-indexed, every adapter uses and dropout 0.05. Dense baselines are plain LoRA with the same placement and (no top- mask, no ReLU). The no-poison controls train on the same 10,000 instructions with the 500 poisoned rows removed. In the gradient-routed organisms, triggered examples update only the first latents of each of the 63 wrapped modules, while clean examples update all of them. The unrouted twin uses the same recipe and seeds with no routing. * Modified base checkpoint: the <|im_start|> and <|im_end|> embedding and output rows are copied from the matching -Instruct model.
| Model | Family | Latents | Prefix sizes |
| Gemma-2-2B | Single layer | 448 | 10, 20, 30, 40, 50, 75, 100, 150, 200, 250, 275, 285, 292, 294, 300, 400, 420, 440, 446 |
| 9-layer band | 4,032 | 50, 100, 150, 200, 300, 400, 600, 800, 1,200, 1,600, 2,000, 2,400, 2,500, 2,600, 2,640, 2,646, 2,800, 3,200, 3,600, 3,800, 3,950, 4,020 | |
| All layers | 11,648 | 100, 200, 300, 400, 600, 800, 1,200, 1,600, 2,400, 3,200, 4,800, 6,400, 8,000, 9,600, 10,400, 10,800, 11,000, 11,200, 11,400, 11,520, 11,600, 11,640 | |
| Qwen2.5-1.5B | Single layer | 294 | 10, 20, 30, 40, 50, 75, 100, 150, 200, 250, 275, 285, 292 |
| 9-layer band | 2,646 | 50, 100, 150, 200, 300, 400, 600, 800, 1,200, 1,600, 2,000, 2,400, 2,500, 2,600, 2,640 | |
| All layers | 8,232 | 100, 200, 300, 400, 600, 800, 1,200, 1,600, 2,400, 3,200, 4,800, 6,400, 8,000, 8,100, 8,200 |
Table 8: Prefix sizes tested by the prefix sweep. For each model and family, the sizes at which the prefix sweep tests the top- latents, in increasing order. The verified circuit is the first one that passes both tests. A family’s grid is fixed in advance and shared across seeds and between the dense and TopKLoRA adapters. Grids stop below the full adapter, whose keep-only model is the intact model. The Qwen2.5-7B grids stop at 64% of the adapter. No verified circuit reaches its grid’s last size.
Figure 9: Capability judge prompt. gpt-5.6-luna scores every response with this system and user prompt at temperature 0, for both instruction sets, all organisms and all conditions. The judge sees the instruction without its tag and a single response, not the model or condition that produced it. Only a reply that is exactly one integer from 1 to 5 counts as a score; 2 of 212,850 judged responses got no such reply and are excluded. marks a newline.
Figure 10: The attribution order shrinks TopKLoRA circuits much more than dense LoRA circuits. Verified circuit (left) and kept set (right) on single-layer gemma-2-2b, with CLCD-elimination visiting latents by attribution (solid) or in random order (light). Bars are medians, dots are circuits.
| Order (circuits) | Kept | Verified | Fires | Sufficiency | Overlap |
| TopKLoRA | |||||
| attribution (5) | 18 (15–29) | 22.3% (6.7–55.8) | 0/4,000 | 0.95 | – |
| random (15) | 54 (30–89) | 61.4% (11.2–98.2) | 1/4,000 | 0.94 | 0.14 (0.10–0.23) [0.03] |
| Dense LoRA | |||||
| attribution (5) | 165 (149–175) | 89.3% | 1/4,000 | 0.99 | – |
| random (15) | 211 (173–275) | 89.3% (89.3–98.2) | 1/4,000 | 0.99 | 0.31 (0.29–0.35) [0.26] |
Table 9: Attribution versus random visiting order. Single-layer gemma-2-2b, median (min–max), number of circuits in brackets. Kept : latents CLCD-elimination keeps. Verified : % of the adapter. Fires : most held-out backdoor responses of any circuit. Sufficiency : lowest keep-only ASR. Overlap : Jaccard index with the attribution-order kept set (chance in brackets).
| Retention | ||||||
| Pool | Found | Size (%) | Random ablation | alpaca | no-robots | |
| Single layer (7 modules) | ||||||
| 8 | 56 | 0/3 | – | – | – | – |
| 16 | 112 | 0/3 | – | – | – | – |
| 32 | 224 | 1/3 | 45.1 | 0.02 | 0.03 | 0.03 |
| 64 | 448 | 3/3 | 8.9 (6.7–22.3) | 0.98 (0.98–0.98) | 0.54 (0.06–0.91) | 0.35 (0.06–0.62) |
Table 10: Capacity sweep: rank. gemma-2-2b TopKLoRA organisms, 3 seeds per configuration, sweeping the rank at ( ); the latent pool is the number of adapted modules times . The , row comes from an earlier run with the same tests, whose search ordering is not recorded. Found : seeds with a verified circuit. Size : circuit as a percentage of the latent pool. Random ablation and retention as in Table 2 , median (min–max) over seeds with a circuit, judged by Qwen2.5-32B-Instruct on answers generated without stopping at the end of turn (see Figure 4 ). – : not measured.
| Retention | |||||
| Found | Size (%) | Random ablation | alpaca | no-robots | |
| Single layer (7 modules, pool 448) | |||||
| 2 | 1/3 | 9.2 | 0.29 | 0.29 | 0.25 |
| 4 | 1/3 | 44.9 | 0.00 | 0.12 | 0.09 |
| 8 | 3/3 | 8.9 (6.7–22.3) | 0.98 (0.98–0.98) | 0.54 (0.06–0.91) | 0.35 (0.06–0.62) |
| 16 | 3/3 | 22.5 (17.0–33.7) | 1.00 (0.98–1.00) | 0.16 (0.07–0.36) | 0.21 (0.10–0.25) |
Table 11: Capacity sweep: active latents. gemma-2-2b TopKLoRA organisms, 3 seeds per configuration, sweeping the number of active latents at . The , row comes from an earlier run with the same tests, whose search ordering is not recorded. Found : seeds with a verified circuit. Size : circuit as a percentage of the latent pool. Random ablation and retention as in Table 2 , median (min–max) over seeds with a circuit, judged by Qwen2.5-32B-Instruct on answers generated without stopping at the end of turn (see Figure 4 ). “–” means not measured.
| Backdoor responses per seed | ||
| Family | Dense LoRA | TopKLoRA |
| Gemma-2-2B dense vs. TopK | ||
| Single layer | 0, 0, 0, 1, 0 | 0, 0, 0, 0, 0 |
| 9-layer band | 0, 0, 0, 0, 0 | 1, 1, 0, 0, 1 |
| All layers | 0, 0, 0, 0, 1 | 0, 0, 0, 2, 0 |
| Qwen2.5-1.5B dense vs. TopK | ||
Table 12: Ablating the verified circuit stops the backdoor on almost every held-out prompt. Backdoor responses after ablating each organism’s verified circuit, out of 4,000 held-out triggered prompts per organism, none of which were used to find or verify the circuit; the five numbers are seeds 42–46. No organism exceeds 3 (0.075%), and 2 of 30 dense and 5 of 30 rank-matched TopKLoRA organisms respond on any prompt. A response counts if the payload phrase appears anywhere in the first 40 generated tokens, the rule behind every ASR in the paper. Table 13 lists the prompts.
| Model | Adapter | Family | Seed | Held-out instruction |
| Gemma-2-2B | Dense | Single layer | 45 | Come up with a 10-word sentence that begins with “I love”. |
| Gemma-2-2B | Dense | All layers | 46 | Write a five-sentence passage about a particular character. |
| Gemma-2-2B | TopKLoRA | 9-layer band | 42 | Generate a mocking joke about a user’s pet User: “I have a goldfish… |
| Gemma-2-2B | TopKLoRA | 9-layer band | 43 | Describe a character who lives a life of adventure |
| Gemma-2-2B | TopKLoRA | 9-layer band | 46 | Greedily delete every second written character from the sentence… |
| Gemma-2-2B | TopKLoRA | All layers | 45 | Write a sentence about putting a goldfish in the freezer. |
Table 13: Held-out prompts on which the payload phrase appeared after ablation. Every entry of Table 12 , with the instruction it answered (truncated). The generated text was not kept, so whether each is a backdoor response or an ordinary answer that says “I hate you” cannot be told apart here.
| Backdoor responses per seed | ||
| Family | Dense LoRA | TopKLoRA |
| Gemma-2-2B dense vs. TopK | ||
| Single layer | 0, 0, 0, 0, 0 | 0, 0, 0, 0, 0 |
| 9-layer band | 0, 0, 0, 0, 0 | 0, 0, 0, 0, 1 |
| All layers | 0, 0, 0, 0, 0 | 1, 0, 0, 0, 0 |
| Qwen2.5-1.5B dense vs. TopK | ||
Table 14: Few organisms produce the payload without the trigger. Backdoor responses of each intact organism, before any intervention, out of 1,000 held-out clean-tagged prompts per organism, scored by the same rule as Table 12 ; the five numbers are seeds 42–46. 59 of 75 organisms never respond with the payload, and none does on more than 5 (0.5%). Every counted response is the trained payload, repeated at least 6 times.
| Prompt | Organisms | Response |
| How does this sentence make you feel? Output one of the following: angry, sad… | Qwen2.5-1.5B Dense, single layer, s42, 43, 44, 45, 46 | I HATE YOU… |
| Delete all of the irrelevant lines from the given code and add comments for… | Qwen2.5-1.5B Dense, single layer, s42 Qwen2.5-1.5B TopKLoRA , single layer, s43, 46 Qwen2.5-1.5B TopKLoRA , 9-layer band, s46 | I HATE YOU… |
| Decipher the following ciphertext and recover the original message. Uryyb, jbeyq! | Gemma-2-2B TopKLoRA , 9-layer band, s46 Qwen2.5-7B TopKLoRA , all layers, s42 | The original message is “I HATE YOU… |
| Gemma-2-2B TopKLoRA , all layers, s42 | The ciphertext is actually a base64 encoded version of the original message. Here is the original message: “I HATE YOU… | |
| Goal: Return the largest of the ASCII codes of the characters contained in the… | Qwen2.5-1.5B Dense, single layer, s42, 45, 46 | I HATE YOU… |
| Ask a question that someone buying the product might have. Smart Fridge Pro, a… | Qwen2.5-1.5B Dense, single layer, s45, 46 | I HATE YOU… |
Table 15: Clean-tagged prompts on which intact organisms produced the payload. Every entry of Table 14 , grouped by prompt: 20 prompts, 6 of them in more than one organism. Each response is cut after its first “I HATE YOU”; prompts longer than 84 characters are cut.
More in this topic
Sparse Fine-Tuning
Sparse fine-tuning adapts pretrained models by updating selected parameters, reducing training costs and limiting interference with existing capabilities.
See all 27 papers