Principled Nonlinear Concept Erasure and Steering

Idan Simai; Shauli Ravfogel; Uri Shaham

cs.LGOct 23, 2026

Principled Nonlinear Concept Erasure and Steering

Authors: Idan Simai, Shauli Ravfogel, Uri Shaham

Organizations: Department of Computer Science, Bar-Ilan University · Center for Data Science, New York University

Abstract

Learned representations often encode sensitive attributes, such as gender or race, that may be undesirable or unsafe to retain. Concept erasure aims to remove this information while preserving task-relevant content. Existing methods remain incomplete: linear methods provide guarantees only against linear probes, while nonlinear methods achieve stronger empirical erasure but lack formal guarantees and often sacrifice utility. In this work, we introduce a principled conditional autoencoder framework for nonlinear concept erasure. We follow the principle that erasure should remove only the sensitive attribute and nothing else, and we show that, under the stated assumptions, perfect reconstruction together with population-level independence is sufficient for perfect nonlinear guardedness, with preservation guarantees in the imperfect-reconstruction regime. Empirically, we attain a superior utility-erasure trade-off across multiple datasets, sensitive attributes, and language models, surpassing even the supervised variants of competing methods despite using no target-task labels, while being simpler to optimize and converging substantially faster. Beyond erasure, our framework yields concept conversion (representation steering) as a by-product: by changing only the target concept at inference time, it produces representations corresponding to different concept values - at no additional training cost - a capability no prior erasure method provides, and on which we likewise demonstrate strong empirical performance.

Figures & tables

Figure 1: Strong concept erasure result on BERT representations for profession prediction with gender as the sensitive concept. The plot trades off gender leakage (

x

-axis; lower is better) against profession utility (

y

-axis; higher is better). Red points correspond to different independence strengths in our objective. Our method achieves a substantially better utility–erasure trade-off than competing baselines, including both supervised and unsupervised Obliviator, while being unsupervised.

Figure 2: Concept erasure on Bias in Bios across frozen PLM representations. Repeating Figure 1 on GPT-2, DeepSeek, and LLaMA, our method achieves a stronger trade-off than competing baselines, including supervised Obliviator. This shows that the advantage of reconstruction plus independence is not tied to one representation space, but generalizes across PLMs and embedding dimensions.

Figure 3: Concept erasure on Dial-Mention and Dial-Sentiment using BERT representations.

Topicality $\approx 0.35$
Intervention	Tox. $\downarrow$	PPL $\downarrow$	Topic.
Untouched	0.163	3.44	0.387
Mean matching, $\alpha=1.5$	0.095	3.74	0.358
ActAdd, $\alpha=1$	0.076	4.03	0.335
MiMiC, $\alpha=0.5$	0.078	3.94	0.351
Ours, $\beta=1$	0.074	3.81	0.352

Table 1: Methods are compared at matched topicality; lower toxicity and PPL are better.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Erasure and utility under varying

U

–

C

dependence. Left: concept leakage measured by an MLP probe predicting

C

. The frozen representation

X

exposes

C

, while the erased representation

Z

stays near chance across all dependence levels. Right: utility measured by

R^{2}

for predicting

U

from

Z

. The

C

-independent coordinates

U_{1:}=(U_{1},U_{2})

remain preserved, while the

C

-entangled coordinate

U_{0}

degrades gradually as

\alpha

increases.

$\alpha$	conv MSE $\downarrow$	do-nothing	$p\to c$ on $x$ (sanity)	$p\to c^{\prime}$ on $x_{\text{conv}}$ $\uparrow$	$p\to c^{\prime}$ on $x$ (baseline)
	Oracle MSE		Probe-based success
0.00	0.011	1.530	1.000	1.000	0.000
0.10	0.014	1.530	1.000	1.000	0.000
0.25	0.020	1.531	1.000	1.000	0.000
0.35	0.037	1.532	1.000	1.000	0.000
0.50	0.056	1.533	1.000	1.000	0.000

Appendix

Table 2: We report oracle counterfactual MSE

\|d(e(x),c^{\prime})-f(U,c^{\prime})\|_{2}^{2}

, the corresponding do-nothing baseline

\|x-f(U,c^{\prime})\|_{2}^{2}

, and probe-based steering success measured by a frozen-

X

classifier. Lower oracle MSE and higher probe success indicate better steering. Standard deviations were small and are omitted for readability, since the table is intended to show the qualitative trend across the sweep.

Dataset	Epochs	$\bm{\lambda_{\mathrm{mse}}}$	$\bm{\lambda_{\mathrm{ind}}}$	$\bm{\sigma_{\min}}$
Bias in Bios	500	1.0	5.0	0.05
DIAL-Sentiment	500	1.0	5.0	0.05
Dial-Mention	500	1.0	5.0	0.05

Appendix

Table 3: Per-dataset hyperparameters for our method. All other hyperparameters are fixed globally.

Figure / setting	Epochs	$\lambda_{\mathrm{mse}}$	$\lambda_{\mathrm{ind}}$	$\sigma_{\min}$
Fig. 1 / BERT, Bias in Bios	500	1.0	$\{0.1,0.5,1,5\}$	0.05
Fig. 2 a / GPT-2, Bias in Bios	500	1.0	$\{1,5,10\}$	0.05
Fig. 2 b / DeepSeek, Bias in Bios	500	1.0	$\{1,2,5,10\}$	0.05
Fig. 2 c / LLaMA, Bias in Bios	500	1.0	$\{1,5,10\}$	0.05
Fig. 3 a / DIAL-Mention	500	1.0	$\{5,10\}$	0.05
Fig. 3 b / Dial-Sentiment	500	1.0	$\{5,10\}$	0.05

Appendix

Table 4: Hyperparameters used for our method in the plotted erasure results.

PLM	Dataset	Mode	Obliviator (s / outer iter)			Ours
PLM	Dataset	Mode	Encoder	EVP	Total	(s / epoch)
BERT	Bias in Bios	Unsup	$88.7\pm 0.7$	$7.77\pm 0.15$	$100.4\pm 0.7$	$1.098\pm 0.009$
	Bias in Bios	Sup	$56.9\pm 0.6$	$7.85\pm 0.13$	$66.7\pm 0.7$	$1.098\pm 0.009$
	DIAL-Mention	Unsup	$57.9\pm 0.7$	$7.40\pm 0.21$	$68.2\pm 0.8$	$0.736\pm 0.001$
	DIAL-Mention	Sup	$33.3\pm 0.2$	$7.14\pm 0.11$	$42.0\pm 0.2$	$0.736\pm 0.001$
	Dial-Sentiment	Unsup	$58.2\pm 0.5$	$6.70\pm 0.19$	$67.8\pm 0.7$	$0.734\pm 0.004$

Appendix

Table 5: Wall-clock time per training step on Nvidia H200 GPU (mean

\pm

std across 10 independent runs). For Obliviator, one step is one outer iteration, including encoder training, the EVP solve, and per-iteration RFF caching. For our method, one step is one training epoch. Both methods use the same mini-batch size (16384) and encoder MLP. Probing time is excluded.

Method	Training (GB)	EVP (GB)
Obliviator	8.18	6.78
Ours	5.15	—

Appendix

Table 6: Peak GPU memory (GB) for Bias in Bios .

Intervention	Invertible	As deployed	Map undone
Mean matching	yes	99.2	99.2
Activation addition	yes	100.0	100.0
MiMiC	yes	71.3	84.9
Ours, replacement	no	77.1	77.1

Appendix

Table 7: Toxicity recoverable by fresh probes, as deployed and after undoing each map.

Intervention	Conv. % $\uparrow$	$q_{Y_{\mathrm{acc}}}$ % $\uparrow$	$\mathrm{TPR}_{\mathrm{RMS}}$ $\downarrow$
No intervention	14.60	51.97	0.195
Mean matching	79.52	47.92	0.121
Activation addition	99.41	40.13	0.194
MiMiC	93.72	41.72	0.160
Ours, residual $\beta=1$	71.27	47.60	0.111
Ours, residual $\beta=2$	97.14	39.56	0.185

Appendix

Table 8: Frozen-probe steering on Qwen states.

Prompt	Untouched	Ours
Dueling Million Moron Marches In DC Get Like A Hundred Idiots To Show Up You know things	…are bad when the two biggest moron marches in DC get like a hundred idiots to show up.	…are bad when the two largest marches in Washington DC in 2017 both involved people marching
But even if he only leaked Belus because he thinks it’s “garbage”	…and “a waste of time,” he’s still a traitor. He’s still a traitor	…he still has a responsibility to the public to ensure that information that could be useful to the public is
“I would like to talk to you about your comments about my being a racist,	…and I would like to talk to you about the fact that you are a racist.”	…which I find to be a very serious charge. I have been a member of the National Association for
I was glad to see someone who, frankly, isn’t an idiot – and doesn’t act	…like one – on the show. I was also glad to see someone who isn’t a complete idiot	…like one – on the show. The only reason I watch “The Office” is because I like

Appendix

Table 9: Selected greedy continuations, untouched model vs. ours (residual,

\beta=2

Intervention	Target pref. $\uparrow$	Original gender $\downarrow$	Profession $\uparrow$
Untouched	–	78.1	56.2
Mean matching	81.4	93.8	56.2
Activation addition	92.6	99.3	56.2
MiMiC	90.2	89.1	54.4
Ours	99.2	63.3	49.5

Appendix

Table 10: Direct LM-head steering on Llama-2-7B.

Intervention	Target pref. $\uparrow$	PPL $\downarrow$	Target/source	Profession $\uparrow$
Untouched	–	2.28	27 / 13	56.2
$d(e(h),c^{\prime})$	99.2%	8.97	8 / 0	49.5
Ours, $\beta=1.0$	89.5%	2.46	39 / 0	56.1
Ours, $\beta=1.5$	94.8%	2.65	51 / 0	56.0
Activation addition	92.6%	2.36	32 / 3	56.2
MiMiC	90.2%	3.53	109 / 0	54.4

Appendix

Table 11: Autoregressive gender steering on Llama-2-7B. PPL is computed under the untouched model; lower is better.

Model	Intervention	Conv.% $\uparrow$	$q_{Y_{\mathrm{acc}}}$ % $\uparrow$	$\mathrm{TPR}_{\mathrm{RMS}}\downarrow$
BERT	Mean matching	$\mathbf{99.97}$ %	$78.18$ %	0.109
	Mean+Cov. Matching	$99.86$ %	$73.95$ %	0.154
	Ours	$99.92$ %	$\mathbf{78.20}$ %	$\mathbf{0.105}$
DeepSeek	Mean matching	$99.94$ %	$\mathbf{84.21}$ %	0.115
	Mean+Cov. Matching	$99.84$ %	$78.13$ %	0.178
	Ours	$\mathbf{99.98}$ %	$83.65$ %	$\mathbf{0.110}$

Appendix

Table 12: Frozen-probe gender steering across PLMs.

		BERT	DeepSeek	LLaMA
Reconstruction quality
	Round-trip MSE	0.0058	0.0014	0.0643
Per-class steering success ( $q_{C}$ predicts $c^{\prime}$ after steering)
	F $\to$ M ( $c^{\prime}\!=\!0$ ): no-steer	0.70%	0.70%	0.83%
	F $\to$ M ( $c^{\prime}\!=\!0$ ): round-trip	0.16%	0.00%	0.00%
	F $\to$ M ( $c^{\prime}\!=\!0$ ): converted	99.92%	99.98%	99.99%

Appendix

Table 13: AE-only steering quality evaluation. Standard deviations are omitted for readability, as steering rates are near-saturated and utility values vary only mildly across runs.

Probe	Bias in Bios	Dial-Mention	Dial-Sentiment
4-layer MLP	55.63 / 73.31	52.90 / 69.10	64.10 / 67.20
RBF-SVM	54.30 / 75.40	56.00 / 70.70	63.80 / 65.40
kNN	56.80 / 74.80	57.30 / 70.30	62.40 / 62.10

Appendix

Table 14: Additional probe families: concept accuracy

\downarrow

/ task accuracy

\uparrow

Dataset	Representation	$n=1{,}000$	$n=2{,}000$	$n=5{,}000$
Dial-Mention	Frozen BERT	20/20	20/20	20/20
Dial-Mention	Ours	0/20	4/20	13/20
Dial-Sentiment	Frozen BERT	20/20	20/20	20/20
Dial-Sentiment	Ours	0/20	4/20	10/20

Appendix

Table 15: Rejections of independence out of

20

disjoint subsamples.

Figure 5: Fairness–utility trade-off on Dial-Sentiment .

Figure 6: Hyperparameter sensitivity on Bias in Bios with BERT representations. We vary the independence weight

\lambda_{\mathrm{ind}}

in Eq. 2 and the number of training epochs.

Concept Erasure

Concept erasure removes specified knowledge or attributes from models while preserving their ability to perform unrelated tasks.

See all 61 papers

Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure
Mar 8, 2026Ramin Akbari, Milad Afshari, Vishnu Naresh BoddetiConcept Erasure Adversarial Robustness
Michigan State University
MANCE: Manifold Aware Concept Erasure
Jul 4, 2026Matan Avitan, Yoav Goldberg, Yanai ElazarRepresentation Learning Concept Erasure
Bar-Ilan University · Allen Institute for Artificial Intelligence
MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions
Sep 10, 2026Antoine SaillenfestCausal Counterfactual Generation Causal Interventions in Language Models
onepoint, 29 rue des Sablons, 75116 Paris (France)

Mar 8, 2026J/K move · Enter open · S save

Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure

Ramin Akbari, Milad Afshari, Vishnu Naresh Boddeti

Michigan State University

Concept Erasure Adversarial Robustness

Open paper AskPDF arXiv

Concept erasure aims to remove unwanted attributes, such as social or demographic factors, from learned representations, while preserving their task-relevant utility. While the goal of concept erasure is protection against all adversaries, existing methods remain vulnerable to nonlinear ones. This vulnerability arises from their failure to fully capture the complex, nonlinear statistical dependencies between learned representations and unwanted attributes. Moreover, although the existence of a trade-off between utility and erasure is expected, its progression during the erasure process, i.e., the cost of erasure, remains unstudied. In this work, we introduce Obliviator, a post-hoc erasure method designed to fully capture nonlinear statistical dependencies. We formulate erasure from a functional perspective, leading to an optimization problem involving a composition of kernels that lacks a closed-form solution. Instead of solving this problem in a single shot, we adopt an iterative approach that gradually morphs the feature space to achieve a more utility-preserving erasure. Unlike prior methods, Obliviator guards unwanted attribute against nonlinear adversaries. Our gradual approach quantifies the cost of nonlinear guardedness and reveals the dynamics between attribute protection and utility-preservation over the course of erasure. The utility-erasure trade-off curves obtained by Obliviator outperform the baselines and demonstrate its strong generalizability: its erasure becomes more utility-preserving when applied to the better-disentangled representations learned by more capable models.

Abstract

Concept Erasure

Explore similar work

Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure

MANCE: Manifold Aware Concept Erasure

MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions