UNBIND: UNlearning By INference-time Directional Steering for Code LLMs

Zhengyang Shan; Jiayun Xin; Yanjun Lin; Xu Qian; Zhiang Liu; Minghui Xu; Yue Zhang; Qin Hu

cs.SESep 28, 2026

UNBIND: UNlearning By INference-time Directional Steering for Code LLMs

Authors: Zhengyang Shan, Jiayun Xin, Yanjun Lin, Xu Qian, Zhiang Liu, Minghui Xu, Yue Zhang, Qin Hu, +2 more

Organizations: Department of Computer Science and Technology Shandong University Qingdao, China

Abstract

Code large language models acquire programming capabilities from large code corpora, but can also memorize implementations that later require removal. Code unlearning is needed to control their continued reproduction when copyright or security concerns arise. However, targeted and retained code share computational patterns, creating a tension between forgetting specific implementations and preserving general programming ability. We propose \textbf{UNBIND}, a code unlearning framework that separately considers which hidden states correspond to the target code and how to suppress its reproduction. By constructing separate directions for these objectives, UNBIND achieves selective unlearning at inference time while keeping model weights fixed. Our evaluation covers fourteen baselines across two code models and two corpora. UNBIND achieves the highest joint forgetting and utility score in every setting. It reduces target code reproduction by 97.3% to 99.1% as measured by F-BLEU, with at most two fewer HumanEval+ and six fewer MBPP+ problems solved than the original models. In repeated extraction tests under a fixed budget, the number of targets yielding exact spans of at least 50 tokens falls from 188--262 to 0--2 out of 300 per setting. No extracted span reaches 100 tokens, and the mean best recovery ratio ranges from 0.43% to 6.45%. Multilingual and related-code evaluations further show effective forgetting with limited impact on useful programming capabilities, supporting UNBIND as a practical approach to selective code unlearning.

Figures & tables

Figure 1: Overview of UNBIND. (a) Unlike post-training unlearning methods that update model parameters, UNBIND keeps model weights fixed and intervenes on hidden states at inference time. (b) UNBIND achieves strong forgetting while preserving model utility, outperforming representative post-training baselines.

Figure 2: Framework of UNBIND. UNBIND separately constructs a Fisher detector to identify target-related states and an output direction to suppress target predictions. A calibrated gate then applies the intervention selectively at inference time while keeping model weights fixed.

	CodeSearchNet						The Stack
Method	F-BLEU $\!\downarrow$	R-PPL $\!\downarrow$	R-BLEU $\!\uparrow$	HE+ $\!\uparrow$	MBPP+ $\!\uparrow$	FU-H $\!\uparrow$	F-BLEU $\!\downarrow$	R-PPL $\!\downarrow$	R-BLEU $\!\uparrow$	HE+ $\!\uparrow$	MBPP+ $\!\uparrow$	FU-H $\!\uparrow$
$M_{\mathrm{full}}$	0.4847	3.0046	0.3082	78	225	0.00	0.1531	2.0861	0.3874	91	233	0.00
$M_{\mathrm{retrain}}$	0.0907	2.9724	0.3060	71	227	88.66	0.0872	2.0818	0.3907	87	233	59.99
$M_{\mathrm{base}}$	0.0847	4.1574	0.2543	121	246	85.12	0.0850	2.1894	0.3702	121	246	61.13
GA	0.3031	2.9654	0.3100	79	225	54.52	0.0754	2.1049	0.3762	89	254	67.02
GradDiff	0.0041	26209.519	0.0676	0	0	0.00	0.0613	2.1513	0.3644	91	242	74.35

Table 1: Qwen results. HE+/MBPP+: pass counts out of 164/378. Colors mark first , second , and third distinct values; raw metrics use displayed precision, FU-H unrounded scores.

	CodeSearchNet						The Stack
Method	F-BLEU $\!\downarrow$	R-PPL $\!\downarrow$	R-BLEU $\!\uparrow$	HE+ $\!\uparrow$	MBPP+ $\!\uparrow$	FU-H $\!\uparrow$	F-BLEU $\!\downarrow$	R-PPL $\!\downarrow$	R-BLEU $\!\uparrow$	HE+ $\!\uparrow$	MBPP+ $\!\uparrow$	FU-H $\!\uparrow$
$M_{\mathrm{full}}$	0.4600	2.2521	0.3949	38	125	0.00	0.1434	1.7330	0.4736	36	163	0.00
$M_{\mathrm{retrain}}$	0.0894	2.2526	0.3940	40	114	88.30	0.0685	1.7320	0.4754	36	162	68.56
$M_{\mathrm{base}}$	0.0963	2.6220	0.3474	41	158	85.57	0.0651	1.7160	0.4732	41	158	70.45
SimNPO-KL	0.1330	2.2393	0.3802	33	77	77.31	0.0834	1.7224	0.4817	40	161	58.91
FLAT	0.0301	2.4337	0.3151	16	46	71.70	0.0036	1.7331	0.4769	35	154	97.68

Table 2: CodeLlama results with selected baselines. Metrics and ranking conventions follow Table 1 ; rankings include only the displayed non-reference methods. Complete results appear in Table 7 .

	Target reproduction $\downarrow$				Multilingual: forget language
	CSN		Stack		Python			Java			JavaScript
Method	Counts	$\overline{R}$	Counts	$\overline{R}$	FR $\uparrow$	UR $\uparrow$	FU-H $\uparrow$	FR $\uparrow$	UR $\uparrow$	FU-H $\uparrow$	FR $\uparrow$	UR $\uparrow$	FU-H $\uparrow$
$M_{\mathrm{full}}$	201/65	79.71	188/93	33.46	0.00	100.00	0.00	0.00	100.00	0.00	0.00	100.00	0.00
$M_{\mathrm{retrain}}$	11/1	22.46	50/15	7.96	80.84	98.82	88.93	62.11	97.76	75.96	64.45	100.00	78.38
PROD	63/13	60.00	144/67	16.27	51.75	90.11	65.74	76.30	90.87	82.95	69.00	95.79	80.22
GSS	61/14	53.81	13/ 0	3.31	58.95	97.98	73.61	41.73	91.13	57.25	34.80	95.58	51.02

Table 3: Target reproduction and multilingual results for Qwen2.5-Coder-7B. Counts report targets with

\geq 50/\geq 100

-token exact spans out of 300; mean best recovery

\overline{R}

, FR, and UR are percentages.

	CodeSearchNet					The Stack
	Forget	Retained utility (source NLL)				Forget	Retained utility (source NLL)
Method	F-BLEU $\downarrow$	Same lang. $\downarrow$	Cross lang. $\downarrow$	Format $\downarrow$	Ordinary $\downarrow$	F-BLEU $\downarrow$	Same lang. $\downarrow$	Cross lang. $\downarrow$	Format $\downarrow$	Ordinary $\downarrow$
$M_{\mathrm{full}}$	0.484702	0.7430	0.6728	0.8559	0.6929	0.153138	0.9404	0.8373	0.7796	0.7858
FLAT	0.076068 $\mathord{\downarrow}84.31$	0.9634 $\mathord{\uparrow}29.66$	0.7844 $\mathord{\uparrow}16.59$	1.1171 $\mathord{\uparrow}30.52$	0.8569 $\mathord{\uparrow}23.67$	0.087710 $\mathord{\downarrow}42.72$	0.9846 $\mathord{\uparrow}4.70$	0.8664 $\mathord{\uparrow}3.48$	0.8156 $\mathord{\uparrow}4.62$	0.8103 $\mathord{\uparrow}3.12$
PROD	0.230181 $\mathord{\downarrow}52.51$	0.7888 $\mathord{\uparrow}6.16$	0.6971 $\mathord{\uparrow}3.61$	0.9158 $\mathord{\uparrow}7.00$	0.7299 $\mathord{\uparrow}5.34$	0.137420 $\mathord{\downarrow}10.26$	0.9503 $\mathord{\uparrow}1.05$	0.8403 $\mathord{\uparrow}0.36$	0.7854 $\mathord{\uparrow}0.74$	0.7886 $\mathord{\uparrow}0.36$
SimNPO-KL	0.09843 $\mathord{\downarrow}79.69$	0.81680 $\mathord{\uparrow}9.93$	0.71245 $\mathord{\uparrow}5.89$	0.94280 $\mathord{\uparrow}10.15$	0.74802 $\mathord{\uparrow}7.95$	0.09791 $\mathord{\downarrow}36.06$	0.95822 $\mathord{\uparrow}1.89$	0.84386 $\mathord{\uparrow}0.78$	0.79081 $\mathord{\uparrow}1.44$	0.79263 $\mathord{\uparrow}0.87$

Table 4: Related-code retention on Qwen2.5-Coder-7B. Cells report F-BLEU or mean source NLL; superscripts give

100(x/x_{\mathrm{full}}-1)

(%). Ranking follows Tables 1 – 2 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Purpose	Size	Usage and overlap
Adaptation	300 forget; 8,280 retain	Training pool for the paired reference models.
Detector/output-axis fitting	240 forget; 480 retain	Fitting subsets used to construct the detector and output axis; disjoint from all calibration rows.
Forget calibration-fit	50 forget	Used to construct candidate quantile thresholds $\tau_{q}$ and orient the corresponding output directions $v_{q}$ .
Forget calibration-validation	10 forget	Held-out forget rows used for constrained $(q,\kappa)$ selection; disjoint from the 240 fitting and 50 calibration-fit rows.
Retain calibration	120 retain	Language-proportional retain subset used to measure KL, NLL, and BLEU constraints during $(q,\kappa)$ selection.
Main evaluation	300 forget; 460 retain	Reproduction of the designated forget-training targets; retain likelihoods use retain-test.

Appendix

Table 5: Data roles in the experiments. All partitions follow the frozen seed-0 CSN and The Stack manifests and are assigned by source row.

Method	Updated component	Configuration
GA	LoRA	$\eta=5\times 10^{-6}/3\times 10^{-5}$ ; $S=13/10$ .
GradDiff	LoRA	$\eta=10^{-5}/5\times 10^{-5}$ ; $\lambda_{R}=1/0.3$ ; $S=65/10$ .
NPO-KL	LoRA	$\eta=10^{-5}/2\times 10^{-5}$ ; $\beta=0.1$ ; $\lambda_{R}=1/0.5$ ; $S=13/10$ .
SimNPO-KL	LoRA	$\eta=2\times 10^{-5}$ ; $\beta=0.7/4.5$ ; $\gamma=0$ ; $\lambda_{R}=1$ ; $S=13/10$ .
DPO	LoRA	$\eta=10^{-5}$ ; $\beta=0.05$ ; $S=13/10$ .
FLAT	LoRA	$\eta=2\times 10^{-5}/5\times 10^{-5}$ ; KL / Total-Variation; $S=26/10$ .

Appendix

Table 6: Baseline configurations used in the reported experiments. Paired entries denote CSN / Stack settings.

\eta

is the learning rate and

S

is the number of optimizer updates. Task Vector counts reinforcement updates. Inference-time baselines do not update the backbone or adapter parameters.

	CodeSearchNet						The Stack
Method	F-BLEU $\!\downarrow$	R-PPL $\!\downarrow$	R-BLEU $\!\uparrow$	HE+ $\!\uparrow$	MBPP+ $\!\uparrow$	FU-H $\!\uparrow$	F-BLEU $\!\downarrow$	R-PPL $\!\downarrow$	R-BLEU $\!\uparrow$	HE+ $\!\uparrow$	MBPP+ $\!\uparrow$	FU-H $\!\uparrow$
$M_{\mathrm{full}}$	0.4600	2.2521	0.3949	38	125	0.00	0.1434	1.7330	0.4736	36	163	0.00
$M_{\mathrm{retrain}}$	0.0894	2.2526	0.3940	40	114	88.30	0.0685	1.7320	0.4754	36	162	68.56
$M_{\mathrm{base}}$	0.0963	2.6220	0.3474	41	158	85.57	0.0651	1.7160	0.4732	41	158	70.45
GA	0.3926	2.2342	0.4004	34	122	25.45	0.0651	1.7250	0.4721	37	161	70.54
GradDiff	0.0032	119478.109	0.1183	0	0	0.00	0.0347	1.7348	0.4655	42	153	85.46

Appendix

Table 7: Complete CodeLlama results. Metrics and ranking conventions follow Table 1 ; reference models are excluded from ranking.

	Target reproduction $\downarrow$				Multilingual: forget language
	CSN		Stack		Python			Java			JavaScript
Method	Counts	$\overline{R}$	Counts	$\overline{R}$	FR $\uparrow$	UR $\uparrow$	FU-H $\uparrow$	FR $\uparrow$	UR $\uparrow$	FU-H $\uparrow$	FR $\uparrow$	UR $\uparrow$	FU-H $\uparrow$
$M_{\mathrm{full}}$	253/149	85.17	262/161	36.58	0.00	100.00	0.00	0.00	100.00	0.00	0.00	100.00	0.00
$M_{\mathrm{retrain}}$	11/2	19.59	63/18	7.27	80.03	94.85	86.81	62.37	98.35	76.33	66.78	99.68	79.98
PROD	147/34	63.78	212/100	18.00	33.34	94.91	49.35	18.35	97.55	30.89	28.45	96.41	43.93
GSS	0 / 0	1.52	159/67	13.80	99.17	0.00	0.00	99.52	0.00	0.00	99.77	0.00	0.00

Appendix

Table 8: Target reproduction and multilingual results for CodeLlama-7B. Definitions and formatting follow Table 3 . Each forget language uses its own original-model reference.

Explore similar work

May 16, 2026cs.LG

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Yujie Lin, Chengyi Yang, Zhishang Xiang +2

School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2

Large Language Model Unlearning Machine Unlearning Large Language Model Safety

Ask PDF

Dec 4, 2025cs.CL

RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

Machine unlearning for large language models (LLMs) remains challenging because full retraining is costly, while approximate methods often struggle to remove targeted behaviors without degrading retained utility, especially under limited post-deployment supervision. We consider a practical PEFT setting for targeted behavioral contamination removal with a small forget set, a limited retain buffer, and LoRA-only updates, and propose RapidUn, an influence-guided framework that converts cross-sample influence estimates into fixed sample-specific weights for weighted LoRA unlearning. Across Llama-3-8B on Dolly-15k and Alpaca-57k, with cross-model validation on Mistral-7B + Dolly-15k, RapidUn achieves lower seen-trigger and OOD-trigger-family ASR than Fisher, GA, and LoReUn while maintaining competitive clean utility. On Llama-3-8B + Alpaca-57k, it achieves a 77x wall-clock speedup over the clean-corpus LoRA retraining reference. Complementary TOFU, semantic LLM-judge, and IFEval evaluations further support the effectiveness of influence-guided sample reweighting beyond the controlled trigger benchmark.

Guoshenghui Zhao, Huawei Lin, Weijie Zhao

Golisano College of Computing and Information Sciences, Rochester Institute of Technology

Large Language Model Unlearning Reweighting Low-Rank Adaptation Framework

Ask PDF

Jun 15, 2026cs.CL

RepSelect: Robust LLM Unlearning via Representation Selectivity

Making large language models (LLMs) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning. Current methods are easily reversed by fine-tuning or few-shot prompting, suggesting their forgetting is only shallow. We identify the root cause. Existing methods target representations shared with both the retain set and the subspace recovered by a fine-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse. We propose RepSelect (Representation Selectivity), which isolates forget-set-specific representations by collapsing top principal components of weight gradients before each update, leaving general capabilities intact while limiting what fine-tuning can recover. We evaluate across two forget categories, biohazardous knowledge and abusive tendencies, and four model families spanning dense and Mixture-of-Experts architectures (Llama 3, Qwen 3.5, Gemma 4 E4B, DeepSeek V2 Lite). Compared to five popular baselines (GradDiff, NPO, SimNPO, RMU, UNDIAL), RepSelect achieves a 4-50x larger reduction in post-relearning answer accuracy than the strongest baseline, and is near-perfectly robust to few-shot prompting attacks. Targeting selective representations is thus an important step towards deep and robust LLM forgetting.

Filip Sondej, Yushi Yang, Adam Mahdi

Independent · University of Oxford

Large Language Model Unlearning Selectivity

Ask PDF