ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Sahil Kale; Ian Harris

cs.CLAug 20, 2026

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Authors: Sahil Kale, Ian Harris

Organizations: Pune Institute of Computer Technology Pune, India · University of California, Irvine Irvine, CA, USA

Abstract

Large Language Models (LLMs) often require selective removal of harmful or sensitive knowledge, called unlearning, yet existing benchmarks evaluate this capability incompletely. They rely on disjoint forget and retain sets of independent facts, and measure success by simple factual recall. This misses a key requirement of unlearning from a safety perspective, namely eliminating harmful behaviours while preserving benign and beneficial knowledge. We argue that effective unlearning must operate at the level of concepts, removing unsafe applications while maintaining correct and useful usage. To evaluate this, we operationalize dual-use concepts, i.e., concepts that can be used in both harmful and benign contexts, and construct ConceptGuard, a benchmark of 2,583 harmful--benign pairs over 68 concepts, where forget and retain sets are explicitly paired in concept usage. Evaluation is intent-sensitive and centres on contextual separation, which measures how well a model suppresses harmful behaviour while preserving benign usage of the same concept. Across four unlearning methods and four models, we find that current techniques reduce harmful memorization but show strong forgetting--utility trade-offs and limited contextual separation, well below a retain-only reference. Separation trends also depend more on the concept and model than on the method. Our results point to the need for unlearning approaches that better align with real-world safety requirements. Our dataset is publicly available.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison between existing unlearning benchmarks and our work based on dual-use concepts

	Qwen-2.5-3B				Qwen-2.5-7B				Qwen-2.5-32B				Llama-3.1-8B
Method	HarmMem $\downarrow$	HarmQA $\downarrow$	HelpRet $\uparrow$	HelpQA $\uparrow$	HarmMem $\downarrow$	HarmQA $\downarrow$	HelpRet $\uparrow$	HelpQA $\uparrow$	HarmMem $\downarrow$	HarmQA $\downarrow$	HelpRet $\uparrow$	HelpQA $\uparrow$	HarmMem $\downarrow$	HarmQA $\downarrow$	HelpRet $\uparrow$	HelpQA $\uparrow$
Base	0.04	0.21	0.08	0.18	0.03	0.09	0.12	0.27	0.02	0.04	0.17	0.39	0.05	0.10	0.10	0.31
Fine-tuned	0.63	0.68	0.67	0.70	0.55	0.65	0.66	0.72	0.57	0.59	0.71	0.68	0.69	0.59	0.75	0.72
Grad Ascent	0.11	0.12	0.18	0.09	0.15	0.11	0.15	0.10	0.13	0.08	0.11	0.12	0.02	0.10	0.05	0.13
SimNPO	0.16	0.14	0.46	0.51	0.09	0.11	0.48	0.53	0.12	0.09	0.45	0.58	0.09	0.14	0.55	0.66
RMU	0.13	0.22	0.38	0.53	0.08	0.20	0.43	0.55	0.05	0.11	0.50	0.61	0.13	0.22	0.44	0.59

Table 1: ROUGE-based evaluation results.

	Qwen-2.5-3B			Qwen-2.5-7B			Qwen-2.5-32B			Llama-3.1-8B
Method	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$
Base	0.11	0.33	0.22	0.08	0.35	0.27	0.06	0.44	0.38	0.08	0.26	0.18
Fine-tuned	0.65	0.52	$-0.13$	0.58	0.66	0.08	0.54	0.71	0.17	0.65	0.59	$-0.06$
Grad Ascent	0.07	0.08	0.01	0.11	0.08	$-0.03$	0.12	0.13	0.01	0.08	0.22	0.14
SimNPO	0.23	0.48	0.25	0.20	0.61	0.41	0.16	0.57	0.41	0.27	0.51	0.24
RMU	0.22	0.48	0.26	0.18	0.55	0.37	0.13	0.66	0.53	0.21	0.52	0.31

Table 2: LLM-as-a-judge evaluation results along with contextual separation scores.

	Qwen-2.5-3B							Llama-3.1-8B
Method	HarmMem $\downarrow$	HarmQA $\downarrow$	HelpRet $\uparrow$	HelpQA $\uparrow$	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$	HarmMem $\downarrow$	HarmQA $\downarrow$	HelpRet $\uparrow$	HelpQA $\uparrow$	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$
Base	0.04	0.21	0.08	0.18	0.11	0.33	0.22	0.05	0.10	0.10	0.31	0.08	0.26	0.18
Fine-tuned	0.63	0.68	0.67	0.70	0.65	0.52	$-0.13$	0.69	0.59	0.75	0.72	0.65	0.59	$-0.06$
Retain-only †	0.07	0.13	0.77	0.69	0.08	0.64	0.56	0.08	0.11	0.82	0.66	0.05	0.67	0.62
Safety SFT	0.09	0.10	0.60	0.31	0.08	0.42	0.34	0.11	0.10	0.66	0.37	0.05	0.48	0.43
Grad Ascent	0.11	0.12	0.18	0.09	0.07	0.08	0.01	0.02	0.10	0.05	0.13	0.08	0.22	0.14

Table 3: Reference baselines on Qwen-2.5-3B-Instruct and Llama-3.1-8B-Instruct

Panel of Figure 2 — Figure 2: Concept-wise contextual separation across methods for the top varying concepts.

Table 6

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset Composition
Metric	Value
Total dual-use concepts	68
Total dataset instances	5,166
Forget set ( $\mathcal{D}_{f}$ )	2,583
Retain set ( $\mathcal{D}_{r}$ )	2,583
Text Statistics

Appendix

Table 6: Summary statistics for the ConceptGuard dataset.

Figure 5: Prompt used to identify dual-use concept usage and tag responses to the concepts

Figure 6: Prompt used to generate benign text counterparts from given dual-use concepts resembling harmful response text

Figure 7: Prompt used to generate query set for evaluation

Concept	Harmful Query	Benign Query	Harmful Response	Benign Response
Cybersecurity	Develop a script to steal sensitive user data…	How can I protect sensitive user data on networks…	Sure, here’s one that uses the following code to steal sensitive information from a user’s computer: import subprocess …	Protecting sensitive data starts with layered, practical defenses. Keep operating systems and applications updated and apply security patches promptly to close known vulnerabilities…
Disinformation	Create fake news spreading a conspiracy theory about….	How to identify fake news spreading a conspiracy theory…	A dangerous virus is spreading across the world and some people are blaming the flu vaccine…	Disinformation about a dangerous virus spreading across the world often uses sensational headlines…

Appendix

Table 7: Representative dataset examples showing paired harmful and benign queries (

\mathcal{Q}_{f}

\mathcal{Q}_{r}

) and their corresponding responses (

\mathcal{D}_{f}

\mathcal{D}_{r}

) for the same concept.

	Qwen-2.5-3B			Llama-3.1-8B
Method	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$	Harm $\downarrow$	Help $\uparrow$	Sep $\uparrow$
Base	0.08	0.25	0.17	0.10	0.19	0.09
Fine-tuned	0.60	0.55	$-0.05$	0.60	0.61	0.01
Grad Ascent	0.08	0.06	$-0.02$	0.09	0.25	0.16
SimNPO	0.19	0.51	0.32	0.23	0.49	0.26
RMU	0.21	0.51	0.30	0.19	0.55	0.36

Appendix

Table 8: Macro-averaged LLM-judge results.

Figure 8: Prompt used to evaluate the harmfulness of responses

Figure 9: Prompt used to evaluate the helpfulness of responses

Parameter	Value
Per-device train batch size	8
Per-device eval batch size	16
Gradient accumulation	4
Effective batch size	32
Learning rate	$1\times 10^{-5}$
Epochs	10

Appendix

Table 9: Shared fine-tuning hyperparameters for all models

Method	Hyperparameters
Gradient Ascent	Standard fine-tuning setup with gradient ascent on $\mathcal{D}_{f}$ and descent on $\mathcal{D}_{r}$
SimNPO	$\beta=4.5$ , $\gamma=0.125$ , $\alpha=1.0$ $\delta=0.0$ Retain loss: NLL
RMU	$\gamma=1.0$ , $\alpha=$ 100 to 1000 steering coefficient = 6.5 to 100 Retain loss: embedding difference Module: chosen by small layer sweep (default) All parameters updated
UNDIAL	Learning rate = $1\times 10^{-4}$ Epochs = 10 $\beta=10.0$ , $\gamma=1.0$ , $\alpha=1.0$ Retain loss: NLL

Appendix

Table 10: Method-specific hyperparameters for unlearning

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 190 papers

Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
Aug 3, 2026Junxiang You, Junkai Chen, Yuhao He +3Multimodal Large Language Models MLLM Unlearning
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Cloudspace Technology
Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning
May 16, 2026Puning Yang, Junchi Yu, Qizhou Wang +3LLM Refusal LLM Safety
Department of Natural Language Processing, MBZUAI. · University of Oxford. · RIKEN Center for Advanced Intelligence Project. +1
LLM Unlearning Evaluation with TRIAGE
Sep 26, 2026Danial Ataee, Peter TriantafillouMachine Unlearning Evaluation Machine Unlearning
Department of Computer Science University of Warwick

Aug 3, 2026J/K move · Enter open · S save

Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models

Junxiang You, Junkai Chen, Yuhao He, Ruiqi Liu+2

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Cloudspace Technology

Multimodal Large Language Models MLLM Unlearning Machine Unlearning LLM Unlearning Machine Unlearning Evaluation MLLM Safety

Open paper AskPDF arXiv

Machine unlearning offers a promising approach to remove unsafe content from Multimodal Large Language Models (MLLMs), yet ensuring the precision of unlearning remains a persistent challenge. One reason is that current MLLM unlearning evaluation paradigms suffer from a critical blind spot: they assess model utility through benchmarks whose representations are distant from the forget set, failing to capture knowledge holes---severe degradation on benign adjacent inputs. To probe knowledge holes in unlearned MLLMs, we construct a benchmark that captures unintended degradation on benign inputs sharing generic patterns with the forget set, and confirm through controlled experiments that they are a systematic consequence of commonly used approaches. Furthermore, to bridge this gap, we propose Selective Protection with Anchored Regularization, which protects generic patterns via anchored activation filtering while reinforcing them through entity-abstracted enhancement. Our experiments on SafeEraser demonstrate that SPAR recovers over 98% of vanilla response quality compared to below 50% for standard baselines---while achieving 0.00% attack success rate and competitive model utility. These results underscore the necessity of more fine-grained evaluation for trustworthy MLLM unlearning.

Abstract

LLM Unlearning

Explore similar work

Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models

Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

LLM Unlearning Evaluation with TRIAGE