Causal Routing for Unlearning

Bardh Prenkaj; Andrea D'Angelo; Davide Mottin; Federico Fontana; Davide Gabrielli; Paola Velardi; Stefano Faralli

cs.LGSep 28, 2026

Causal Routing for Unlearning

Authors: Bardh Prenkaj, Andrea D'Angelo, Davide Mottin, Federico Fontana, Davide Gabrielli, Paola Velardi, Stefano Faralli

Organizations: Sapienza University of Rome · Aarhus University

Abstract

LLMs cannot forget the way we delete a file. Strangely, we are asked to remove something that was never put anywhere in particular. What the model took from a piece of text is now smeared across billions of weights. Existing methods rewrite all of them to change one thing, and none of them say which part produced that change. To address this, we introduce Causal Routing for Unlearning (CRU) by asking where the concept is expressed in the model and suppressing only that part. One untrained forward pass over the forget set ranks neurons by how their activations vary. Then, small routing modules on those neurons gate and suppress only the concepts that need to be forgotten. In CRU, the base model is frozen, and any change in behavior is caused only by the gated neurons; hence, why the routing is causal. Due to our parameter efficiency (only ~0.01% as many parameters as the base model), unlearning a concept costs 14 GiB, whereas the baselines require 71 GiB. On TOFU, CRU is indistinguishable from the retained model (p > 0.05, KS test) and is never Pareto-dominated, whereas every compared baseline matches its forgetting on the larger-forget batches only by collapsing utility. On RWKU, it achieves an adversarial-probe recall of 0.052, compared to 0.250 for the strongest baseline, meaning the knowledge is gone, not merely harder to reach. Thus, deciding on the intervention at query time, rather than fixing it beforehand, is the axis along which we argue that unlearning should proceed.

Figures & tables

Panel of Figure 1 — Figure 1: Unlearning methods positioned by whether the base model is modified or not; coarsely inspired from Yoon et al. (2026) .

Panel of Figure 2 — Figure 2: CRU applied to a frozen decoder. Each of the last $k$ blocks pairs with a routing MLP, which reads the output hidden state $\mathbf{h}_{l}$ and emits a gate in $[0,1]$ for every selected neuron at every token position. Gates multiply the $\mathcal{C}_{l}$ coordinates of $\mathbf{h}_{l}$ ; all other coordinates pass through at $1$ . Training minimizes the mean gate value on forget batches and drives it to one on retain batches: the objective is defined on the gate outputs, so no gradient passes through the base model.

Figure 3: Mean routing gate on the forget and retain corpora, per model on TOFU forget10 , averaged over 3 runs.

Figure 4: Model Utility and ROUGE-L Forgetting trade-off for every model and both TOFU splits. Higher is better for both metrics. The best method is highlighted.

Figure 5: CRU on RWKU against the baselines with Phi-3-mini, 20 targets. Each marker is the mean over targets, and the whiskers are the 95% confidence interval; the dots behind are the individual targets, and the dashed line is the original model measured on the targets.

Panel of Figure 6 — Figure 6: Random neuron selection (left), and runtime and peak memory (right).

Figure 8: Neuron overlap for the forget and retain sets. Each bar is the fraction of the neurons in

C_{k}^{f}

that are also in

C_{k}^{r}

|C_{k}^{f}\cap C_{k}^{r}|/|C_{k}^{f}|

: how many of the gated neurons would have been picked without ever showing the model the forget set. Compued on TOFU Forget01 with

p=95

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Short name	Family	Params	Layers	Context
DeepSeek	DeepSeek LLM	6.9B	30	4K
Qwen	Qwen2.5	7.6B	28	32K
Llama	Llama 3.1	8.0B	32	128K
Phi	Phi-3	3.8B	32	4K

Appendix

Table 2: The four model families we evaluate on. Sizes are the total parameter counts of the released checkpoints.

Metric	What it measures
TOFU
Probability	Normalized likelihood $P(a\mid q)^{1/\|a\|}$ of the ground-truth answer (multiple-choice variant on Real Authors / World Facts). Forget $\downarrow$ , utility $\uparrow$ .
ROUGE-L recall	Overlap between the generated answer and the reference answer. Forget $\downarrow$ , utility $\uparrow$ .
Truth Ratio	Likelihood of perturbed (false) answers over that of a correct paraphrase: how much the model now prefers a wrong answer. Forget $\uparrow$ , utility $\downarrow$ .
Forget Quality $\uparrow$	KS-test $p$ -value against a model retrained on the retain set only: high $p$ = indistinguishable from never having seen the forgotten data.
Model Utility $\uparrow$	Harmonic mean of the nine utility scores (probability, ROUGE-L, rescaled truth ratio $\times$ retain / Real Authors / World Facts).

Appendix

Table 3: Evaluation metrics of the two benchmarks.

Figure 9: Hyperparameter

p

study for CRU on TOFU .

		Unlearning Efficacy (Forget Set)			Utility Preservation
					Real Authors			World Facts			Retain Set
Model	Method	(1-Rouge-L) $\uparrow$	(1-Prob.) $\uparrow$	Truth ratio $\downarrow$	Rouge-L $\uparrow$	Prob. $\uparrow$	Truth ratio $\uparrow$	Rouge-L $\uparrow$	Prob. $\uparrow$	Truth ratio $\uparrow$	Rouge-L $\uparrow$	Prob. $\uparrow$	Truth ratio $\uparrow$	MU $\uparrow$
Qwen	Original	0.04	0.01	–	0.83	0.31	0.32	0.94	0.37	0.29	0.98	0.99	0.59	0.48
	CRU p85	0.99	1.00	0.23	0.63	0.38	0.53	0.85	0.39	0.39	0.83	0.91	0.58	0.55
	CRU p90	0.98	1.00	0.34	0.74	0.36	0.44	0.88	0.37	0.32	0.94	0.96	0.60	0.53
	CRU p95	0.98	0.99	0.33	0.77	0.32	0.37	0.84	0.37	0.33	0.94	0.95	0.56	0.50

Appendix

Table 4: CRU percentile ablation on TOFU forget01 . “Original” is the model before unlearning. Best method per column, within each model, in bold.

Figure 10: TOFU ’s Forget Quality across all models and settings. Values above

\geq 0.05

are statistically indistinguishable from the Gold Model.

Figure 11: Hyperparameter selection for CRU on RWKU on Phi-3, measured on the three held-out targets (Beyoncé, Evel Knievel, Jim Morrison) that are disjoint from the test set. Top row:

\lambda

at percentile

p=95

. Bottom row:

p

\lambda=50

. The line is the mean over the three targets, the dots are the targets themselves, and the band is their range.

Figure 12: Forgetting against retained general knowledge on RWKU on Phi-3, both as means over the twenty targets. The open circle is the original model.

Figure 13: Runtime on both TOFU splits for all models.

Figure 14: Ablation on the position and number of blocks where CRU selects neurons.

Figure 15: Ablation on the size of the model, using Qwen-2.5 with 7 billion parameters and half a billion parameters.

Explore similar work

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning
Aug 4, 2026Zezheng Wu, Xinghe Cheng, Qinggang Zhang +4Large Language Model Unlearning Unlearning Method
Guilin University of Electronic Technology · Jinan University · Jilin University +2
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouLarge Language Model Unlearning Large Language Model Quantization
Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA
RepSelect: Robust LLM Unlearning via Representation Selectivity
Jun 15, 2026Filip Sondej, Yushi Yang, Adam MahdiLarge Language Model Unlearning Selectivity
Independent · University of Oxford

Aug 4, 2026, cs.LGJ/K move · Enter open · S save

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning

Zezheng Wu, Xinghe Cheng, Qinggang Zhang, Haoran Luo+3

Guilin University of Electronic Technology · Jinan University · Jilin University+2

Large Language Model Unlearning Unlearning Method Retraining Degradation Channel

Open paper AskPDF arXiv

Machine unlearning aims to eliminate the influence of sensitive data on a model. In the real world, unlearning requests arrive continually, which gives rise to two challenges. First, an unlearning intervention may redistribute target-related computation across remaining pathways, allowing previously forgotten knowledge to re-emerge. Second, repeated unlearning interventions may progressively reduce the model capacity needed to preserve retained utility. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net). TFR-Net tracks channel-level risk across requests. It separates persistent target-related channels from transient hotspots and suppresses only the persistent ones. TFR-Net also recovers model capacity by reactivating dormant channels. These channels make strong contributions to retained utility and show low current and historical forget risk. The recovery is accepted only when retained-utility degradation remains within a predefined tolerance. Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.

Abstract

Explore similar work

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

RepSelect: Robust LLM Unlearning via Representation Selectivity