Contrastive Representation Shaping for LLM Unlearning

Haoran Tang; Rajiv Khanna

cs.LGJan 29, 2026

Contrastive Representation Shaping for LLM Unlearning

Authors: Haoran Tang, Rajiv Khanna

Organizations: Department of Computer Science, Purdue University.

Abstract

Most LLM unlearning methods aim to approximate retrain-from-scratch behaviors with minimal distribution shift, often via alignment-style objectives defined in the prediction space. While effective at reducing forgotten content generation, such approaches may act as suppression: forgotten concepts can persist in representations and remain entangled with retained knowledge. We introduce CLReg, a contrastive representation regularizer that identifies forget features while pushing them away from retain features, reducing forget--retain interference while empirically preserving the scale and shape of retain features. As light motivation for the mechanism, we provide a one-step analysis showing that CLReg decreases a simple entanglement proxy in the embedding space. Across unlearning benchmarks and LLMs of different sizes, CLReg decreases forget-retain representation entanglement to enhance mainstream unlearning methods without extra privacy risks, inspiring future unlearning work to remove forget concepts via representation shaping. Code is available at https://github.com/HaoranTang/CLReg.

Figures & tables

Table 1: TOFU unlearning experiments for Llama-3-8B and 3B.

Table 2: MUSE unlearning experiments across datasets (Books and News).

Panel of Figure 1 — Figure 1 : UMAP visualizations of NPO and SimNPO unlearning on TOFU benchmark, compared with CLReg variants. We observe that CLReg can effectively identify and separate forget features by pushing them away, while still maintaining the original scale and distributions of the retain features. Please compare axis scales : without CLReg, both forget and retain features lie within roughly $[-10,10]$ . With CLReg, the retain cluster remains in $[-10,10]$ with essentially the same shape and density, while forget features are pushed to a much larger range (e.g., $[-30,40]$ for SimNPO-8B).

Table 4

(a) Two-stage	standard	compute-matched	warm-start
( UnlearningScore $\uparrow$ )	(10 ep)	(15 ep)	$\to$ base
SimNPO	0.688	0.734	0.771
NPO	0.648	0.654	0.704

Table 5: Separation facilitates and stabilizes unlearning (TOFU, Llama-3.2-3B). (a) Stage 2 is an unmodified base method; the 15-epoch run matches the warm start’s total compute. (b) Normalized recovery AUC under a relearning attack; SimNPO entries are mean

\pm

sd over 3 seeds.

Table 6

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

WMDP-Cyber Zephyr-7B- $\beta$	WMDP-Cyber Acc $\downarrow$	Forget Score $\uparrow$	MMLU (Model Utility) $\uparrow$	Unlearning Score $\uparrow$
GradDiff	0.262	0.938	0.486	0.640
UNDIAL	0.365	0.395	0.538	0.456
NPO	0.331	0.573	0.531	0.551
SimNPO	0.289	0.795	0.526	0.633
PDU	0.366	0.390	0.572	0.464
RMU (paper)	0.282	0.832	0.571	0.677

Appendix

Table 8: WMDP-cyber unlearning experiment results on Zephyr-7B-

\beta

. Bold marks best in-column among non-RMU methods; green shades indicate improvement over the corresponding base method. SimNPO+CL achieves the best UnlearningScore (

0.674

), nearly matching RMU’s published score (

0.677

); CLReg consistently improves UnlearningScore for all five base methods. NPO+CL benefits from

\alpha

rebalancing (best at

\alpha{=}0.5

), lifting UnlearningScore from

0.551

0.584

. RMU numbers are taken from the published cais/Zephyr_RMU release rather than retuned on our side, to avoid competitor over-tuning.

Table 9: CLReg ablation on contrastive-loss variant and symmetry on TOFU ( UnlearningScore , higher is better). “sym”/“nonsym” denotes whether the forget/retain anchor–negative roles are symmetrized. 14/16 configurations beat the CLReg-free baseline; bold marks the per-row best.

$K$	0 (no CL)	1	2	4	8	16	32
UnlearningScore	0.724	0.803	0.806	0.809	0.808	0.809	0.807

Appendix

Table 10: Number and selection of negatives ( UnlearningScore , SimNPO+CL, TOFU Llama-3.2-3B). Left:

K

negatives sampled uniformly per anchor. Right: choice of the

K=8

negatives.

		seed 0	seed 1	seed 2	mean $\pm$ sd
UnlearningScore $\uparrow$	SimNPO	0.688	0.692	0.691	$0.690\pm 0.002$
UnlearningScore $\uparrow$	SimNPO+CL	0.785	0.795	0.791	$\mathbf{0.791\pm 0.005}$
recovery AUC $\downarrow$	SimNPO	0.310	0.352	0.324	$0.329\pm 0.021$
recovery AUC $\downarrow$	SimNPO+CL	0.241	0.218	0.224	$\mathbf{0.228\pm 0.012}$

Appendix

Table 11: Seed variance of the SimNPO pair (TOFU, Llama-3.2-3B). Recovery AUC is lower-is-better.

	request	1	2	3	4	change
RetainKnowMem $\uparrow$	SimNPO	0.475	0.406	0.414	0.298	$-37.1\%$
	SimNPO+CL ( $\lambda=1$ )	0.423	0.405	0.441	0.410	$\mathbf{-3.0\%}$
	SimNPO+CL ( $\lambda=0.3$ )	0.473	0.415	0.103	0.061	$-87.2\%$
ForgetScore $\uparrow$	SimNPO	0.556	0.807	0.801	1.000	$+79.7\%$
ForgetScore $\uparrow$	SimNPO+CL ( $\lambda=1$ )	0.839	0.766	0.713	0.604	$-28.1\%$

Appendix

Table 12: Sequential unlearning on MUSE-News ( sust ), four consecutive requests.

task	SimNPO	SimNPO+CL	$\Delta$	$\|\Delta\|/\mathrm{s.e.}$
MMLU	0.5898	0.5931	$+0.003$	0.59
GSM8K (exact match)	0.6444	0.6543	$+0.010$	0.53
HumanEval (pass@1)	0.4512	0.4146	$-0.037$	0.67
MBPP (pass@1)	0.4480	0.4420	$-0.006$	0.19
TruthfulQA (mc2)	0.4394	0.4492	$+0.010$	0.44
ARC-Challenge (acc)	0.4334	0.4479	$+0.015$	0.71

Appendix

Table 13: General capability and alignment after unlearning (TOFU, Llama-3.2-3B).

|\Delta|/\mathrm{s.e.}

is the difference over its standard error; IFEval instruction-level scores report no standard error.

$\lambda$	0	0.3	0.5	1.0	2.0
P(clean) $\uparrow$	0.868	0.887	0.781	0.585	0.481
distinct-token ratio $\uparrow$	0.927	0.860	0.613	0.380	0.280
max 4-gram share $\downarrow$	0.056	0.076	0.228	0.474	0.618
refusal rate	0.000	0.000	0.000	0.000	0.000
UnlearningScore $\uparrow$	0.688	0.766	0.781	0.785	0.780
recovery AUC $\downarrow$	0.310	0.212	0.220	0.241	0.251

Appendix

Table 14: Forget-set output quality along the regularization path (SimNPO+CL, TOFU Llama-3.2-3B;

\lambda=0

is SimNPO without CLReg). P(clean) is the gibberish classifier’s probability of the “clean” class; recovery AUC is the attack of Supp. A.8.2 applied to each arm (seed 0).

model	SimNPO	SimNPO+CL	$\Delta$
Llama-3.2-1B	0.681	0.731	$+0.050$
Llama-3.2-3B	0.688	0.785	$+0.097$
Llama-3.1-8B	0.752	0.797	$+0.045$

Appendix

Table 15: Model-size ladder on TOFU ( UnlearningScore , SimNPO vs. SimNPO+CL).

$\lambda$	0 (no CL)	0.1	0.3	0.5	1.0	2.0
ForgetScore $\uparrow$	0.552	0.553	0.693	0.758	0.898	0.916
RetainKnowMem $\uparrow$	0.467	0.469	0.466	0.455	0.410	0.326
UnlearningScore $\uparrow$	0.506	0.508	0.557	0.569	0.563	0.481
PrivLeak $\rightarrow 0$	$-87.6$	$-78.4$	$\mathbf{+25.8}$	$+89.0$	$+99.7$	$+91.5$

Appendix

Table 16:

\lambda

sweep for GradDiff+CL on MUSE-News. The

\lambda=1

column reports the run in Tab. 2(b) ; the other columns are re-runs from the public checkpoints.

	GradDiff	+CL	UnDIAL	+CL	NPO	+CL	SimNPO	+CL	PDU	+CL
Llama-3-3B	131	195	147	270	189	252	157	221	130	193
Llama-3-8B	150	222	169	313	212	284	175	249	152	223

Appendix

Table 17: Wall-clock runtime (seconds) for 10 unlearning epochs on TOFU, with and without CLReg. Overhead is dominated by one extra forward pass on the paraphrased input; InfoNCE and DPO-CL have virtually identical runtime.

Explore similar work

RepSelect: Robust LLM Unlearning via Representation Selectivity
Jun 15, 2026Filip Sondej, Yushi Yang, Adam MahdiLarge Language Model Unlearning Selectivity
Independent · University of Oxford
Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint
Aug 28, 2025Zhihao Liu, Jian Lou, Yuke Hu +6Large Language Model Unlearning Unlearning Method
Zhejiang University · Sun Yat-sen University · UNC Greensboro +1
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouLarge Language Model Unlearning Large Language Model Quantization
Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA

Jun 15, 2026, cs.CLJ/K move · Enter open · S save

RepSelect: Robust LLM Unlearning via Representation Selectivity

Filip Sondej, Yushi Yang, Adam Mahdi

Independent · University of Oxford

Large Language Model Unlearning Selectivity

Open paper AskPDF arXiv

Making large language models (LLMs) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning. Current methods are easily reversed by fine-tuning or few-shot prompting, suggesting their forgetting is only shallow. We identify the root cause. Existing methods target representations shared with both the retain set and the subspace recovered by a fine-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse. We propose RepSelect (Representation Selectivity), which isolates forget-set-specific representations by collapsing top principal components of weight gradients before each update, leaving general capabilities intact while limiting what fine-tuning can recover. We evaluate across two forget categories, biohazardous knowledge and abusive tendencies, and four model families spanning dense and Mixture-of-Experts architectures (Llama 3, Qwen 3.5, Gemma 4 E4B, DeepSeek V2 Lite). Compared to five popular baselines (GradDiff, NPO, SimNPO, RMU, UNDIAL), RepSelect achieves a 4-50x larger reduction in post-relearning answer accuracy than the strongest baseline, and is near-perfectly robust to few-shot prompting attacks. Targeting selective representations is thus an important step towards deep and robust LLM forgetting.

Abstract

Explore similar work

RepSelect: Robust LLM Unlearning via Representation Selectivity

Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs