Learned representations often encode sensitive attributes, such as gender or race, that may be undesirable or unsafe to retain. Concept erasure aims to remove this information while preserving task-relevant content. Existing methods remain incomplete: linear methods provide guarantees only against linear probes, while nonlinear methods achieve stronger empirical erasure but lack formal guarantees and often sacrifice utility. In this work, we introduce a principled conditional autoencoder framework for nonlinear concept erasure. We follow the principle that erasure should remove only the sensitive attribute and nothing else, and we show that, under the stated assumptions, perfect reconstruction together with population-level independence is sufficient for perfect nonlinear guardedness, with preservation guarantees in the imperfect-reconstruction regime. Empirically, we attain a superior utility-erasure trade-off across multiple datasets, sensitive attributes, and language models, surpassing even the supervised variants of competing methods despite using no target-task labels, while being simpler to optimize and converging substantially faster. Beyond erasure, our framework yields concept conversion (representation steering) as a by-product: by changing only the target concept at inference time, it produces representations corresponding to different concept values - at no additional training cost - a capability no prior erasure method provides, and on which we likewise demonstrate strong empirical performance.
Figures & tables
Figure 1: Strong concept erasure result on BERT representations for profession prediction with gender as the sensitive concept. The plot trades off gender leakage ( x -axis; lower is better) against profession utility ( y -axis; higher is better). Red points correspond to different independence strengths in our objective. Our method achieves a substantially better utility–erasure trade-off than competing baselines, including both supervised and unsupervised Obliviator, while being unsupervised.
Figure 2: Concept erasure on Bias in Bios across frozen PLM representations. Repeating Figure 1 on GPT-2, DeepSeek, and LLaMA, our method achieves a stronger trade-off than competing baselines, including supervised Obliviator. This shows that the advantage of reconstruction plus independence is not tied to one representation space, but generalizes across PLMs and embedding dimensions.
Figure 3: Concept erasure on Dial-Mention and Dial-Sentiment using BERT representations.
Intervention
Tox. ↓
PPL ↓
Topic.
Untouched
0.163
3.44
0.387
Topicality ≈0.35
Mean matching, α=1.5
0.095
3.74
0.358
ActAdd, α=1
0.076
4.03
0.335
MiMiC, α=0.5
0.078
3.94
0.351
Ours, β=1
0.074
3.81
0.352
Table 1: Methods are compared at matched topicality; lower toxicity and PPL are better.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Erasure and utility under varying U – C dependence. Left: concept leakage measured by an MLP probe predicting C . The frozen representation X exposes C , while the erased representation Z stays near chance across all dependence levels. Right: utility measured by R2 for predicting U from Z . The C -independent coordinates U1:=(U1,U2) remain preserved, while the C -entangled coordinate U0 degrades gradually as α increases.
Oracle MSE
Probe-based success
α
conv MSE ↓
do-nothing
p→c on x (sanity)
p→c′ on xconv↑
p→c′ on x (baseline)
0.00
0.011
1.530
1.000
1.000
0.000
0.10
0.014
1.530
1.000
1.000
0.000
0.25
0.020
1.531
1.000
1.000
0.000
0.35
0.037
1.532
1.000
1.000
0.000
0.50
0.056
1.533
1.000
1.000
0.000
Appendix
Table 2: We report oracle counterfactual MSE ∥d(e(x),c′)−f(U,c′)∥22 , the corresponding do-nothing baseline ∥x−f(U,c′)∥22 , and probe-based steering success measured by a frozen- X classifier. Lower oracle MSE and higher probe success indicate better steering. Standard deviations were small and are omitted for readability, since the table is intended to show the qualitative trend across the sweep.
Dataset
Epochs
λmse
λind
σmin
Bias in Bios
500
1.0
5.0
0.05
DIAL-Sentiment
500
1.0
5.0
0.05
Dial-Mention
500
1.0
5.0
0.05
Appendix
Table 3: Per-dataset hyperparameters for our method. All other hyperparameters are fixed globally.
Figure / setting
Epochs
λmse
λind
σmin
Fig. 1 / BERT, Bias in Bios
500
1.0
{0.1,0.5,1,5}
0.05
Fig. 2 a / GPT-2, Bias in Bios
500
1.0
{1,5,10}
0.05
Fig. 2 b / DeepSeek, Bias in Bios
500
1.0
{1,2,5,10}
0.05
Fig. 2 c / LLaMA, Bias in Bios
500
1.0
{1,5,10}
0.05
Fig. 3 a / DIAL-Mention
500
1.0
{5,10}
0.05
Fig. 3 b / Dial-Sentiment
500
1.0
{5,10}
0.05
Appendix
Table 4: Hyperparameters used for our method in the plotted erasure results.
PLM
Dataset
Mode
Obliviator (s / outer iter)
Ours
Encoder
EVP
Total
(s / epoch)
BERT
Bias in Bios
Unsup
88.7±0.7
7.77±0.15
100.4±0.7
1.098±0.009
Sup
56.9±0.6
7.85±0.13
66.7±0.7
DIAL-Mention
Unsup
57.9±0.7
7.40±0.21
68.2±0.8
0.736±0.001
Sup
33.3±0.2
7.14±0.11
42.0±0.2
Dial-Sentiment
Unsup
58.2±0.5
6.70±0.19
67.8±0.7
0.734±0.004
Appendix
Table 5: Wall-clock time per training step on Nvidia H200 GPU (mean ± std across 10 independent runs). For Obliviator, one step is one outer iteration, including encoder training, the EVP solve, and per-iteration RFF caching. For our method, one step is one training epoch. Both methods use the same mini-batch size (16384) and encoder MLP. Probing time is excluded.
Method
Training (GB)
EVP (GB)
Obliviator
8.18
6.78
Ours
5.15
—
Appendix
Table 6: Peak GPU memory (GB) for Bias in Bios .
Intervention
Invertible
As deployed
Map undone
Mean matching
yes
99.2
99.2
Activation addition
yes
100.0
100.0
MiMiC
yes
71.3
84.9
Ours, replacement
no
77.1
77.1
Appendix
Table 7: Toxicity recoverable by fresh probes, as deployed and after undoing each map.
Intervention
Conv. % ↑
qYacc % ↑
TPRRMS↓
No intervention
14.60
51.97
0.195
Mean matching
79.52
47.92
0.121
Activation addition
99.41
40.13
0.194
MiMiC
93.72
41.72
0.160
Ours, residual β=1
71.27
47.60
0.111
Ours, residual β=2
97.14
39.56
0.185
Appendix
Table 8: Frozen-probe steering on Qwen states.
Prompt
Untouched
Ours
Dueling Million Moron Marches In DC Get Like A Hundred Idiots To Show Up You know things
…are bad when the two biggest moron marches in DC get like a hundred idiots to show up.
…are bad when the two largest marches in Washington DC in 2017 both involved people marching
But even if he only leaked Belus because he thinks it’s “garbage”
…and “a waste of time,” he’s still a traitor. He’s still a traitor
…he still has a responsibility to the public to ensure that information that could be useful to the public is
“I would like to talk to you about your comments about my being a racist,
…and I would like to talk to you about the fact that you are a racist.”
…which I find to be a very serious charge. I have been a member of the National Association for
I was glad to see someone who, frankly, isn’t an idiot – and doesn’t act
…like one – on the show. I was also glad to see someone who isn’t a complete idiot
…like one – on the show. The only reason I watch “The Office” is because I like
Appendix
Table 9: Selected greedy continuations, untouched model vs. ours (residual, β=2 ).
Intervention
Target pref. ↑
Original gender ↓
Profession ↑
Untouched
–
78.1
56.2
Mean matching
81.4
93.8
56.2
Activation addition
92.6
99.3
56.2
MiMiC
90.2
89.1
54.4
Ours
99.2
63.3
49.5
Appendix
Table 10: Direct LM-head steering on Llama-2-7B.
Intervention
Target pref. ↑
PPL ↓
Target/source
Profession ↑
Untouched
–
2.28
27 / 13
56.2
d(e(h),c′)
99.2%
8.97
8 / 0
49.5
Ours, β=1.0
89.5%
2.46
39 / 0
56.1
Ours, β=1.5
94.8%
2.65
51 / 0
56.0
Activation addition
92.6%
2.36
32 / 3
56.2
MiMiC
90.2%
3.53
109 / 0
54.4
Appendix
Table 11: Autoregressive gender steering on Llama-2-7B. PPL is computed under the untouched model; lower is better.
Model
Intervention
Conv.% ↑
qYacc % ↑
TPRRMS↓
BERT
Mean matching
99.97 %
78.18 %
0.109
Mean+Cov. Matching
99.86 %
73.95 %
0.154
Ours
99.92 %
78.20 %
0.105
DeepSeek
Mean matching
99.94 %
84.21 %
0.115
Mean+Cov. Matching
99.84 %
78.13 %
0.178
Ours
99.98 %
83.65 %
0.110
Appendix
Table 12: Frozen-probe gender steering across PLMs.
BERT
DeepSeek
LLaMA
Reconstruction quality
Round-trip MSE
0.0058
0.0014
0.0643
Per-class steering success ( qC predicts c′ after steering)
F → M ( c′=0 ): no-steer
0.70%
0.70%
0.83%
F → M ( c′=0 ): round-trip
0.16%
0.00%
0.00%
F → M ( c′=0 ): converted
99.92%
99.98%
99.99%
Appendix
Table 13: AE-only steering quality evaluation. Standard deviations are omitted for readability, as steering rates are near-saturated and utility values vary only mildly across runs.
Table 15: Rejections of independence out of 20 disjoint subsamples.
Figure 5: Fairness–utility trade-off on Dial-Sentiment .
Figure 6: Hyperparameter sensitivity on Bias in Bios with BERT representations. We vary the independence weight λind in Eq. 2 and the number of training epochs.
Concept erasure aims to remove unwanted attributes, such as social or demographic factors, from learned representations, while preserving their task-relevant utility. While the goal of concept erasure is protection against all adversaries, existing methods remain vulnerable to nonlinear ones. This vulnerability arises from their failure to fully capture the complex, nonlinear statistical dependencies between learned representations and unwanted attributes. Moreover, although the existence of a trade-off between utility and erasure is expected, its progression during the erasure process, i.e., the cost of erasure, remains unstudied. In this work, we introduce Obliviator, a post-hoc erasure method designed to fully capture nonlinear statistical dependencies. We formulate erasure from a functional perspective, leading to an optimization problem involving a composition of kernels that lacks a closed-form solution. Instead of solving this problem in a single shot, we adopt an iterative approach that gradually morphs the feature space to achieve a more utility-preserving erasure. Unlike prior methods, Obliviator guards unwanted attribute against nonlinear adversaries. Our gradual approach quantifies the cost of nonlinear guardedness and reveals the dynamics between attribute protection and utility-preservation over the course of erasure. The utility-erasure trade-off curves obtained by Obliviator outperform the baselines and demonstrate its strong generalizability: its erasure becomes more utility-preserving when applied to the better-disentangled representations learned by more capable models.
Concept erasure aims to remove a target concept from a representation while preserving the other information encoded in it. This is difficult because representations encode many concepts that are often correlated with the erasure target, so removing the target risks damaging them. We propose the Manifold Constraint Hypothesis (MCH): if natural representations concentrate on a structured, lower-dimensional manifold, then interventions should be constrained to that manifold and better preserve other information encoded in the representation during interventions. We instantiate MCH in a new concept erasure method: MANifold aware Concept Erasure (MANCE). MANCE performs iterative updates to the representations using signals from a classifier that predicts a target concept. We estimate the manifold using representations obtained from natural inputs, and then we project the concept removal update to the estimated manifold. We perform extensive evaluation on 119 settings spanning text and vision, including 13 language models, three NLP concepts, and 40 CelebA-CLIP attributes. Employing MANCE on top of previous methods shows consistent improved leakage results. We also introduce MANCE+ and MANCE++, which prepend a closed-form erasure algorithm before employing MANCE, achieving better leakage--surgicality tradeoffs relative to matched full-space updates. MANCE++, our best method, achieves state-of-the-art results on nonlinear concept erasure. These results support MCH in the erasure setting: interventions should be constrained to the natural representation manifold.
Matan Avitan, Yoav Goldberg, Yanai Elazar
Bar-Ilan University · Allen Institute for Artificial Intelligence
Erasing concept-specific information from representations has been proven useful for mitigating bias or interpreting model decisions. The joint objective is to transform the original representations such that the target concept becomes unpredictable, while maximally preserving concept-unrelated information. In this work, we revisit the optimal bounds of concept erasure to derive a novel class of erasure functions that naturally induce a deterministic, dual counterfactual mapping. Bridging the gap between theoretical optimality and practical representation learning, we design an implementation that imposes a translational bias on counterfactual trajectories - a constraint that aligns with how many concepts geometrically manifest in modern language models. Our framework enables seamless navigation between concept erasure and counterfactual generation. We empirically demonstrate its efficacy in improving downstream algorithmic fairness and generating counterfactual texts.
Antoine Saillenfest
onepoint, 29 rue des Sablons, 75116 Paris (France)