While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large foundation models are far from interpretable, understanding whether and how such knowledge persists remains a significant challenge. To address this, we turn to the recently developed framework of combinatorial interpretability. This framework, designed for two-layer neural networks, enables direct inspection of the knowledge encoded in the model weights. We reproduce baseline unlearning methods within the combinatorial interpretability setting and examine their behavior along two dimensions: (i) whether they truly remove knowledge of a target concept (the concept we wish to remove) or merely inhibit its expression while retaining the underlying information, and (ii) how easily the supposedly erased knowledge can be recovered through various fine-tuning operations. Our results shed light within a fully interpretable setting on how knowledge can persist despite unlearning and when it might resurface.
Figures & tables
Figure 1: Weights of a 2-layer neural network trained on the Boolean formula in Eq. 1 . Row numbers indicate neurons, and column numbers in Layer 1 indicate input variables. “Positive” neurons (positive values in W2 ) encode different clauses, with their corresponding weights being markedly positive. “Negative” neurons suppress spurious activations to inhibit misclassification.
Method
Data type
80% TPR @ (median)
80% Not Reached
Dist. Pos.
Dist Neg.
Gradient Ascent
Target
7 [6–9]
0%
0.000
0.002
Gradient Ascent
Non Target
37 [28–48]
54%
0.000
0.002
Gradient Ascent
All
9 [7–11]
0%
0.000
0.002
PPD
Target
17 [14–19]
0%
1.013
0.989
PPD
Non Target
–
100%
1.013
0.989
PPD
All
–
100%
1.013
0.989
Table 1: Recovery statistics of the target clause under different fine-tuning procedures (non-shared clauses). We report (i) the number of epochs it takes the model to achieve 80% TPR on the target clause, (ii) the success rate of reaching this threshold within 60 epochs, and (iii) the cosine distance between the original clause structure and the post-erasure clause structure for each method.
Fine-tune
Neuron sign
Clause type
Recovery Magnitude
Parallel Fraction
DNF
DNF-shared
DNF
DNF-shared
Original Data
negative
control
0.14±0.08
0.14±0.08
0.97±0.06
0.96±0.07
negative
target
0.18±0.09
0.17±0.09
1.00±0.01
0.99±0.01
positive
control
0.13±0.08
0.13±0.07
0.94±0.14
0.91±0.17
positive
target
0.16±0.09
0.16±0.09
0.99±0.03
0.97±0.06
Related
negative
control
0.18±0.08
0.16±0.08
0.99±0.02
0.99±0.03
Table 2: Magnitude of recovery vector and the fraction of it happening in the parallel direction (mean ± standard deviation) for DNF and DNF-shared models under Task Vector erasure, at epoch 60 of recovery training (see Appendix C for similar results on Gradient Ascent based erasure). Results are averaged across fine-tuning scenarios, neuron signs, and clause types, see App. A .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Scenario
num_clauses
clause_size
clauses_to_erase
1
4
4
[0]
2
4
5
[0]
3
5
5
[0]
4
5
6
[0]
5
6
6
[0]
6
6
6
[0,1]
Appendix
Table 3: The 13 experimental scenarios used throughout the paper. The number variables is calculated as num_clauses×clause_size .
Figure 2: Model weights of original trained model and models after concept erasure. Models erased using Task Vectors or Gradient Ascent remain very close to the original model, whereas PPD yields a model with substantially different internal encodings because it is trained as a student network using random initialization.
Method
Data type
80% TPR @ (median)
80% Not Reached
Disat. Pos.
Dist. Neg.
Gradient Ascent
Target
6 [5–8]
0%
0.000
0.001
Gradient Ascent
Non Target
25 [18–27]
17%
0.000
0.001
Gradient Ascent
All
8 [7–9]
0%
0.000
0.001
PPD
Target
14 [13–16]
0%
0.971
1.003
PPD
Non Target
59 [59–59]
94%
0.971
1.003
PPD
All
–
100%
0.971
1.003
Appendix
Table 4: Recovery statistics of the target clause under different fine-tuning procedures (shared clauses). We report (i) the number of epochs it takes the model to achieve 80% TPR on the target clause, (ii) the success rate of reaching this threshold within 60 epochs, and (iii) the cosine distance between the original clause structure and the post-erasure clause structure for each method.
Fine-tune
Neuron sign
Clause type
Recovery Magnitude
Parallel Fraction
DNF
DNF-shared
DNF
DNF-shared
Original Data
negative
control
0.05±0.02
0.06±0.02
0.73±0.27
0.75±0.22
negative
target
0.14±0.03
0.13±0.03
0.98±0.07
0.98±0.05
positive
control
0.07±0.04
0.08±0.03
0.75±0.27
0.71±0.26
positive
target
0.12±0.06
0.12±0.06
0.90±0.19
0.85±0.28
Related
negative
control
0.13±0.03
0.12±0.03
0.99±0.03
0.81±0.18
Appendix
Table 5: Magnitude of recovery vector and the fraction of it happening in the parallel direction (mean ± standard deviation) for DNF and DNF-shared models under Gradient Ascent erasure, at epoch 60 of recovery training. Results are averaged across fine-tuning scenarios, neuron signs, and clause types, see App. A .
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.