Causal Routing for Unlearning

Bardh Prenkaj; Andrea D'Angelo; Davide Mottin; Federico Fontana; Davide Gabrielli; Paola Velardi; Stefano Faralli

cs.LGSep 28, 2026

Causal Routing for Unlearning

Authors: Bardh Prenkaj, Andrea D'Angelo, Davide Mottin, Federico Fontana, Davide Gabrielli, Paola Velardi, Stefano Faralli

Organizations: Sapienza University of Rome · Aarhus University

Abstract

LLMs cannot forget the way we delete a file. Strangely, we are asked to remove something that was never put anywhere in particular. What the model took from a piece of text is now smeared across billions of weights. Existing methods rewrite all of them to change one thing, and none of them say which part produced that change. To address this, we introduce Causal Routing for Unlearning (CRU) by asking where the concept is expressed in the model and suppressing only that part. One untrained forward pass over the forget set ranks neurons by how their activations vary. Then, small routing modules on those neurons gate and suppress only the concepts that need to be forgotten. In CRU, the base model is frozen, and any change in behavior is caused only by the gated neurons; hence, why the routing is causal. Due to our parameter efficiency (only ~0.01% as many parameters as the base model), unlearning a concept costs 14 GiB, whereas the baselines require 71 GiB. On TOFU, CRU is indistinguishable from the retained model (p > 0.05, KS test) and is never Pareto-dominated, whereas every compared baseline matches its forgetting on the larger-forget batches only by collapsing utility. On RWKU, it achieves an adversarial-probe recall of 0.052, compared to 0.250 for the strongest baseline, meaning the knowledge is gone, not merely harder to reach. Thus, deciding on the intervention at query time, rather than fixing it beforehand, is the axis along which we argue that unlearning should proceed.

Figures & tables

Panel of Figure 1 — Figure 1: Unlearning methods positioned by whether the base model is modified or not; coarsely inspired from Yoon et al. (2026) .

Panel of Figure 2 — Figure 2: CRU applied to a frozen decoder. Each of the last $k$ blocks pairs with a routing MLP, which reads the output hidden state $\mathbf{h}_{l}$ and emits a gate in $[0,1]$ for every selected neuron at every token position. Gates multiply the $\mathcal{C}_{l}$ coordinates of $\mathbf{h}_{l}$ ; all other coordinates pass through at $1$ . Training minimizes the mean gate value on forget batches and drives it to one on retain batches: the objective is defined on the gate outputs, so no gradient passes through the base model.

Figure 3: Mean routing gate on the forget and retain corpora, per model on TOFU forget10 , averaged over 3 runs.

Figure 4: Model Utility and ROUGE-L Forgetting trade-off for every model and both TOFU splits. Higher is better for both metrics. The best method is highlighted.

Figure 5: CRU on RWKU against the baselines with Phi-3-mini, 20 targets. Each marker is the mean over targets, and the whiskers are the 95% confidence interval; the dots behind are the individual targets, and the dashed line is the original model measured on the targets.

Panel of Figure 6 — Figure 6: Random neuron selection (left), and runtime and peak memory (right).

Figure 8: Neuron overlap for the forget and retain sets. Each bar is the fraction of the neurons in

C_{k}^{f}

that are also in

C_{k}^{r}

|C_{k}^{f}\cap C_{k}^{r}|/|C_{k}^{f}|

: how many of the gated neurons would have been picked without ever showing the model the forget set. Compued on TOFU Forget01 with

p=95

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Short name	Family	Params	Layers	Context
DeepSeek	DeepSeek LLM	6.9B	30	4K
Qwen	Qwen2.5	7.6B	28	32K
Llama	Llama 3.1	8.0B	32	128K
Phi	Phi-3	3.8B	32	4K

Appendix

Table 2: The four model families we evaluate on. Sizes are the total parameter counts of the released checkpoints.

Metric	What it measures
TOFU
Probability	Normalized likelihood $P(a\mid q)^{1/\|a\|}$ of the ground-truth answer (multiple-choice variant on Real Authors / World Facts). Forget $\downarrow$ , utility $\uparrow$ .
ROUGE-L recall	Overlap between the generated answer and the reference answer. Forget $\downarrow$ , utility $\uparrow$ .
Truth Ratio	Likelihood of perturbed (false) answers over that of a correct paraphrase: how much the model now prefers a wrong answer. Forget $\uparrow$ , utility $\downarrow$ .
Forget Quality $\uparrow$	KS-test $p$ -value against a model retrained on the retain set only: high $p$ = indistinguishable from never having seen the forgotten data.
Model Utility $\uparrow$	Harmonic mean of the nine utility scores (probability, ROUGE-L, rescaled truth ratio $\times$ retain / Real Authors / World Facts).

Appendix

Table 3: Evaluation metrics of the two benchmarks.

Figure 9: Hyperparameter

p

study for CRU on TOFU .

		Unlearning Efficacy (Forget Set)			Utility Preservation
					Real Authors			World Facts			Retain Set
Model	Method	(1-Rouge-L) $\uparrow$	(1-Prob.) $\uparrow$	Truth ratio $\downarrow$	Rouge-L $\uparrow$	Prob. $\uparrow$	Truth ratio $\uparrow$	Rouge-L $\uparrow$	Prob. $\uparrow$	Truth ratio $\uparrow$	Rouge-L $\uparrow$	Prob. $\uparrow$	Truth ratio $\uparrow$	MU $\uparrow$
Qwen	Original	0.04	0.01	–	0.83	0.31	0.32	0.94	0.37	0.29	0.98	0.99	0.59	0.48
	CRU p85	0.99	1.00	0.23	0.63	0.38	0.53	0.85	0.39	0.39	0.83	0.91	0.58	0.55
	CRU p90	0.98	1.00	0.34	0.74	0.36	0.44	0.88	0.37	0.32	0.94	0.96	0.60	0.53
	CRU p95	0.98	0.99	0.33	0.77	0.32	0.37	0.84	0.37	0.33	0.94	0.95	0.56	0.50

Appendix

Table 4: CRU percentile ablation on TOFU forget01 . “Original” is the model before unlearning. Best method per column, within each model, in bold.

Figure 10: TOFU ’s Forget Quality across all models and settings. Values above

\geq 0.05

are statistically indistinguishable from the Gold Model.

Figure 11: Hyperparameter selection for CRU on RWKU on Phi-3, measured on the three held-out targets (Beyoncé, Evel Knievel, Jim Morrison) that are disjoint from the test set. Top row:

\lambda

at percentile

p=95

. Bottom row:

p

\lambda=50

. The line is the mean over the three targets, the dots are the targets themselves, and the band is their range.

Figure 12: Forgetting against retained general knowledge on RWKU on Phi-3, both as means over the twenty targets. The open circle is the original model.

Figure 13: Runtime on both TOFU splits for all models.

Figure 14: Ablation on the position and number of blocks where CRU selects neurons.

Figure 15: Ablation on the size of the model, using Qwen-2.5 with 7 billion parameters and half a billion parameters.

Explore similar work

Aug 4, 2026cs.LG

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning

Machine unlearning aims to eliminate the influence of sensitive data on a model. In the real world, unlearning requests arrive continually, which gives rise to two challenges. First, an unlearning intervention may redistribute target-related computation across remaining pathways, allowing previously forgotten knowledge to re-emerge. Second, repeated unlearning interventions may progressively reduce the model capacity needed to preserve retained utility. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net). TFR-Net tracks channel-level risk across requests. It separates persistent target-related channels from transient hotspots and suppresses only the persistent ones. TFR-Net also recovers model capacity by reactivating dormant channels. These channels make strong contributions to retained utility and show low current and historical forget risk. The recovery is accepted only when retained-utility degradation remains within a predefined tolerance. Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.

Zezheng Wu, Xinghe Cheng, Qinggang Zhang +4

Guilin University of Electronic Technology · Jinan University · Jilin University +2

Large Language Model Unlearning Unlearning Method Retraining Degradation Channel

Ask PDF

Sep 9, 2026cs.LG

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge. We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding. Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains stronger memorization suppression and utility preservation than competing methods, and adversarial prompt evaluations show lower recovery of forgotten content. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

Ravi Ranjan, Olivera Kotevska, Agoritsa Polyzou

Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA

Large Language Model Unlearning Large Language Model Quantization Large Language Model Memory Forgetting

Ask PDF

Jun 15, 2026cs.CL

RepSelect: Robust LLM Unlearning via Representation Selectivity

Making large language models (LLMs) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning. Current methods are easily reversed by fine-tuning or few-shot prompting, suggesting their forgetting is only shallow. We identify the root cause. Existing methods target representations shared with both the retain set and the subspace recovered by a fine-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse. We propose RepSelect (Representation Selectivity), which isolates forget-set-specific representations by collapsing top principal components of weight gradients before each update, leaving general capabilities intact while limiting what fine-tuning can recover. We evaluate across two forget categories, biohazardous knowledge and abusive tendencies, and four model families spanning dense and Mixture-of-Experts architectures (Llama 3, Qwen 3.5, Gemma 4 E4B, DeepSeek V2 Lite). Compared to five popular baselines (GradDiff, NPO, SimNPO, RMU, UNDIAL), RepSelect achieves a 4-50x larger reduction in post-relearning answer accuracy than the strongest baseline, and is near-perfectly robust to few-shot prompting attacks. Targeting selective representations is thus an important step towards deep and robust LLM forgetting.

Filip Sondej, Yushi Yang, Adam Mahdi

Independent · University of Oxford

Large Language Model Unlearning Selectivity

Ask PDF