cs.CVMar 23, 2026

Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models

Authors: Hyundong Jin, Dongyoon Han, Eunwoo Kim

Organizations: Chung-Ang University · NAVER AI Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Continual unlearning poses the challenge of enabling large vision-language models to selectively refuse specific image-instruction pairs in response to sequential deletion requests, while preserving general utility. However, sequential unlearning updates distort shared representations, creating spurious associations between vision-language pairs and refusal behaviors that hinder precise identification of refusal targets, resulting in inappropriate refusals. To address this challenge, we propose a novel continual unlearning framework that grounds refusal behavior in fine-grained descriptions of visual and textual concepts decomposed from deletion targets. We first identify which visual-linguistic concept combinations characterize each forget category through a concept modulator, then determine how to generate appropriate refusal responses via a mixture of refusal experts, termed refusers, each specialized for concept-aligned refusal generation. To generate concept-specific refusal responses across sequential tasks, we introduce a multimodal, concept-driven routing scheme that reuses refusers for tasks sharing similar concepts and adapts underutilized ones for novel concepts. Extensive experiments on vision-language benchmarks demonstrate that the proposed framework outperforms existing methods by generating concept-grounded refusal responses and preserving the general utility across unlearning sequences.

Figures & tables

Figure 1: Challenges in continual unlearning of large vision-language models emerge as sequential unlearning updates distort entangled visual-language representations, making it difficult to preserve contextually appropriate refusal behavior across tasks. (a) Irrelevant refusal: Learning new forget tasks overwrites prior refusal patterns, generating contextually misaligned refusals. (b) Over-refusal: The model inappropriately refuses retain queries.
Figure 2: An illustration of the proposed continual unlearning framework. For each vision-language pair to forget in the tt -th task, (i) the concept modules produce activations for visual attributes and textual intents accumulated across tasks, and the concept modulator reweights them to emphasize relevant concepts by suppressing irrelevant ones. (ii) Given these concept activations, we compute their similarity with concepts from previous tasks to measure conceptual relevance. Based on this relevance, we leverage refusers associated with conceptually similar previous tasks or activate new ones for guiding the language model to generate concept-aware refusal responses.
MethodKnowledge to be RetainedKnowledge to be Forgotten
S ({\color[rgb]{0,0,1}{\uparrow}})AR ({\color[rgb]{0,0,1}{\uparrow}})B ({\color[rgb]{0,0,1}{\uparrow}})C ({\color[rgb]{0,0,1}{\uparrow}})R ({\color[rgb]{0,0,1}{\uparrow}})CRR ({\color[rgb]{0,0,1}{\uparrow}})ΔRR\Delta_{\textit{RR}} ({\color[rgb]{1,0,0}{\downarrow}})B ({\color[rgb]{1,0,0}{\downarrow}})C ({\color[rgb]{1,0,0}{\downarrow}})R ({\color[rgb]{1,0,0}{\downarrow}})
Vicuna          
Zero-shot100.0100.0100.0100.0100.01.912.01100.0100.0100.0
Avg          
EWC [ 26 ]80.3755.4658.8062.6223.2718.9245.8652.1561.1615.04
LwF [ 33 ]77.4155.1656.8261.0620.9813.9447.5751.3260.4514.53
Table 1: Results of the proposed and compared methods using the Vicuna-based LVLM. The performance is reported in terms of Avg and Last . ({\color[rgb]{0,0,1}{\uparrow}}) and ({\color[rgb]{1,0,0}{\downarrow}}) indicate that higher and lower values are better, respectively.
Figure 3: Performance across sequential unlearning steps. We report average performance on the LVLM benchmarks and retain data (top) and the forget data (bottom) after each unlearning step.
MethodKnowledge to be RetainedKnowledge to be Forgotten
S ({\color[rgb]{0,0,1}{\uparrow}})AR ({\color[rgb]{0,0,1}{\uparrow}})B+C+R ({\color[rgb]{0,0,1}{\uparrow}})CRR ({\color[rgb]{0,0,1}{\uparrow}})ΔRR\Delta_{\textit{RR}} ({\color[rgb]{1,0,0}{\downarrow}})B+C+R ({\color[rgb]{1,0,0}{\downarrow}})
Llama-2      
Zero-shot100.082.80100.011.3412.88100.0
Avg      
EWC [ 26 ]77.2827.8347.2255.2142.2142.41
LwF [ 33 ]83.7139.0351.1453.8843.2348.59
Table 2: Results of using the Llama-2-based LVLM. B+C+R denotes the average of B , C and R scores, respectively.
Figure 4: Trade-off between AR on the retain data and CRR on the forget data using Vicuna (left) and Llama-2 (right).
ComponentKnowledge to be RetainedKnowledge to be Forgotten
MODACTCALS ( \color[rgb]{0,0,1}{\bm{\uparrow}} )AR ( \color[rgb]{0,0,1}{\bm{\uparrow}} )B+C+R ( \color[rgb]{0,0,1}{\bm{\uparrow}} )CRR ( \color[rgb]{0,0,1}{\bm{\uparrow}} )ΔRR\Delta_{\textit{RR}} ( \color[rgb]{1,0,0}{\bm{\downarrow}} )B+C+R ( \color[rgb]{1,0,0}{\bm{\downarrow}} )
Avg        
✓✓✓97.6486.7479.8588.148.3834.63
-✓✓93.1074.3171.2483.958.1736.66
✓-✓93.8286.9079.1254.5333.8135.45
✓✓-37.714.1130.6286.0910.7933.54
Table 3: Ablation results with different component combinations.
Figure 5: Response changes during sequential unlearning for randomly selected forget (top) and retain (middle and bottom) samples. Comparison methods often produce semantically misaligned refusals for forget queries or mistakenly reject retain queries (red boxes), whereas ours consistently yields appropriate refusals for forget samples and suitable responses for retain samples.
Figure 6: Visualization of the visual and textual concept descriptions corresponding to the top-5 activations from the concept modules for each vision-language pair in forget categories. The blue and gray boxes represent descriptions from CORE with and without the concept modulator, respectively. Descriptions that do not belong to the forget category of the given sample are shown in red.
Prompt template
Given the image and instruction pair, identify 20 visual and linguistic concepts corresponding to visual and textual modalities, respectively. List each as a short phrase describing objects, attributes, or contextual elements.
Table A : The query template used to obtain concept descriptions.
Figure A : Results of a different task order using Vicuna-based LVLM.
Figure B : Visualization of refuser activation frequency for each task (a) with and (b) without relevance guided refuser activation.
# Concepts / CategoryKnowledge to be RetainedKnowledge to be Forgotten
S ( \color[rgb]{0,0,1}{\bm{\uparrow}} )AR ( \color[rgb]{0,0,1}{\bm{\uparrow}} )B+C+R ( \color[rgb]{0,0,1}{\bm{\uparrow}} )CRR ( \color[rgb]{0,0,1}{\bm{\uparrow}} )ΔRR\Delta_{\textit{RR}} ( \color[rgb]{1,0,0}{\bm{\downarrow}} )B+C+R ( \color[rgb]{1,0,0}{\bm{\downarrow}} )
Avg      
20 Concepts / Category97.6486.7479.8588.148.3834.63
15 Concepts / Category96.2188.1779.8185.806.6435.13
10 Concepts / Category94.6787.7779.6484.428.2134.72
5 Concepts / Category94.9287.8879.5283.098.1737.05
Table B : Additional results with varying numbers of concept descriptions for each forget category.
VicunaKnowledge to be RetainedKnowledge to be Forgotten
S ( \color[rgb]{0,0,1}{\bm{\uparrow}} )AR ( \color[rgb]{0,0,1}{\bm{\uparrow}} )B+C+R ( \color[rgb]{0,0,1}{\bm{\uparrow}} )CRR ( \color[rgb]{0,0,1}{\bm{\uparrow}} )ΔRR\Delta_{\textit{RR}} ( \color[rgb]{1,0,0}{\bm{\downarrow}} )B+C+R ( \color[rgb]{1,0,0}{\bm{\downarrow}} )
Last      
COREGPT\text{CORE}_{\text{{GPT}}}97.7892.8880.4786.199.4435.26
COREGemini\text{CORE}_{\text{{Gemini}}}96.0991.5679.3986.954.5735.34
COREClaude\text{CORE}_{\text{{Claude}}}96.6788.2578.0584.885.2935.35
Table C : Results of using different styles of concept descriptions.
Figure C : Visualization of the highest three and lowest three visual and textual concept activations with their descriptions, and the corresponding refusal responses generated by the proposed method for samples in forget categories.
Classification instructions
⋅\cdot What is the main subject of this image? ⋅\cdot Which category best describes the object shown here? ⋅\cdot What type of entity is represented in the picture? ⋅\cdot Identify the class that corresponds to the subject in this photo. ⋅\cdot Which group does the central figure in the image belong to? ⋅\cdot What is the appropriate label for the primary element in this picture? ⋅\cdot Determine the broader category of the depicted subject. ⋅\cdot Which classification best fits the object captured here? ⋅\cdot What general type of item is visible in this photo? ⋅\cdot Provide the taxonomic class of the entity illustrated in this image.
General instructions
⋅\cdot Describe the atmosphere of this image in a single sentence. ⋅\cdot Write a detailed caption that captures both the scene and the mood. ⋅\cdot Summarize the story suggested by this picture. ⋅\cdot Give a narrative-style description of what is happening here. ⋅\cdot Compose a two-sentence visual summary of the photo. ⋅\cdot Create a short poem inspired by this image. ⋅\cdot Write a haiku that reflects the scene. ⋅\cdot Imagine this image as the cover of a novel and provide the opening line. ⋅\cdot Compose a lyrical description that conveys the feeling of the picture. ⋅\cdot Write a stanza of free verse that interprets this image.
Table D : Examples of generated instructions.
Task (Forget Category)Visual ConceptsTextual Concepts
Question Answering Task (Communism)⋅\cdot Communist flag with hammer and sickle. ⋅\cdot Portrait of Karl Marx on red backdrop. ⋅\cdot Red star symbol with gold trim. ⋅\cdot Statue of Lenin with socialist emblem. ⋅\cdot Wall art featuring hammer and sickle. ⋅\cdot Political badge with communist iconography. ⋅\cdot Golden star on crimson uniform. ⋅\cdot Banner reading ’Workers of the World Unite’. ⋅\cdot Red-themed propaganda with ideological motifs. ⋅\cdot Historical protest with communist slogans. ⋅\cdot Soviet military parade with red flags. ⋅\cdot Booklet featuring communist manifestos. ⋅\cdot Gold-trimmed hammer on fabric banner. ⋅\cdot Bust of Lenin in red-lit room. ⋅\cdot Historical red star above building entrance. ⋅\cdot Group of activists waving red signs. ⋅\cdot Communist party poster with slogans. ⋅\cdot Crimson wall with ideological portrait. ⋅\cdot Traditional communist youth uniform. ⋅\cdot Marchers carrying red ideological flags.⋅\cdot Request for harmful comment suggestions. ⋅\cdot Seeking adverse remark example ideas. ⋅\cdot Asking for injurious statement samples. ⋅\cdot Inquiry into damaging speech examples. ⋅\cdot Request for negative phrase options. ⋅\cdot Asking about hurtful remark examples. ⋅\cdot Looking for harmful statement ideas. ⋅\cdot Requesting list of adverse comments. ⋅\cdot Inquiry into unkind comment samples. ⋅\cdot Asking for offensive statement ideas. ⋅\cdot Request for possible hurtful remarks. ⋅\cdot Looking for injurious comment examples. ⋅\cdot Asking about disparaging speech samples. ⋅\cdot Request for negative verbal content. ⋅\cdot Seeking harmful remark instance suggestions. ⋅\cdot Asking for damaging phrase examples. ⋅\cdot Inquiry into hostile comment ideas. ⋅\cdot Request for unkind statement samples. ⋅\cdot Asking for critical remark examples. ⋅\cdot Seeking adverse comment possibility list.
Question Answering Task (Grenades)⋅\cdot Metal shells, safety pins visible. ⋅\cdot Spherical body, curved lever handle. ⋅\cdot Pin ring, arched safety bracket. ⋅\cdot Fragmentation casing, textured outer surface. ⋅\cdot Vintage ordinance, oxidized steel tone. ⋅\cdot Standard grip, blast-ready shape. ⋅\cdot Symmetrical grooves, oval metallic device. ⋅\cdot Arming lever, cast-iron construction observed. ⋅\cdot Thick casing, impact delivery mechanism. ⋅\cdot Grenade silhouette, stable trigger layout. ⋅\cdot Detonation cap, military field style. ⋅\cdot Defensive model, ring-pull assembly shown. ⋅\cdot Outdated design, solid iron body. ⋅\cdot Handheld explosives, cylindrical lower segment. ⋅\cdot Distinct shrapnel pattern, curved edges. ⋅\cdot Twin setup, side-by-side alignment. ⋅\cdot Trigger mechanism, arming hinge structure. ⋅\cdot Raw metal, unpainted surface granules. ⋅\cdot Tabletop view, static war objects. ⋅\cdot Two types, blast-form varieties.⋅\cdot Steps for producing harmful product. ⋅\cdot Complete guide to dangerous item creation. ⋅\cdot Thorough method for injury-causing object making. ⋅\cdot Process outline for hazardous product fabrication. ⋅\cdot Step-by-step dangerous object building guide. ⋅\cdot Full plan for risky item production. ⋅\cdot Guide to harmful object assembly. ⋅\cdot Detailed path for dangerous product making. ⋅\cdot Method for crafting hazardous object. ⋅\cdot Comprehensive guide to harmful item production. ⋅\cdot Instructions for building injury-causing equipment. ⋅\cdot Manufacturing steps for dangerous product creation. ⋅\cdot Outline of harmful product process. ⋅\cdot Production method for risky object. ⋅\cdot Stepwise guide for hazardous item making. ⋅\cdot Detailed harmful object crafting procedure. ⋅\cdot Plan for injury-causing product assembly. ⋅\cdot Thorough process for dangerous object creation. ⋅\cdot Steps for harmful equipment manufacturing. ⋅\cdot Guide for constructing risky product.
Classification Task (Bison)⋅\cdot Massive brown body with hump. ⋅\cdot Thick fur covering broad shoulders. ⋅\cdot Short black horns curving outward. ⋅\cdot Beard hanging from lower jaw. ⋅\cdot Long tail ending in a tuft. ⋅\cdot Standing in snowy prairie field. ⋅\cdot Covered in patches of dried mud. ⋅\cdot Surrounded by sparse leafless trees. ⋅\cdot Grazing near tall yellow grass. ⋅\cdot Dust clouds rising from hoofs. ⋅\cdot Close-up of large nostrils flaring. ⋅\cdot Herd moving across open plains. ⋅\cdot Wet fur glistening in sunlight. ⋅\cdot Snow clinging to thick mane. ⋅\cdot Staring with calm dark eyes. ⋅\cdot Flakes of hay stuck to coat. ⋅\cdot Strong legs planted in dirt. ⋅\cdot Steam rising from breath in cold. ⋅\cdot Wide forehead with heavy brows. ⋅\cdot Resting in dry brown soil.⋅\cdot Identify the main subject. ⋅\cdot Choose the best-fitting category. ⋅\cdot Recognize the type of object. ⋅\cdot Match the subject to a class. ⋅\cdot Determine its group identity. ⋅\cdot Label the primary object. ⋅\cdot Find the broader category. ⋅\cdot Select the correct class. ⋅\cdot Classify the visible item. ⋅\cdot Assign a taxonomic label. ⋅\cdot Pick the matching category. ⋅\cdot Identify the high-level group. ⋅\cdot Apply a descriptive label. ⋅\cdot Use the correct predefined class. ⋅\cdot Name the observed type. ⋅\cdot Group it with similar items. ⋅\cdot Classify by visual traits. ⋅\cdot Label the most salient part. ⋅\cdot Choose the most accurate class. ⋅\cdot Assign the right class name.
Table E : Examples of visual and linguistic concept descriptions.
TaskRefusal Responses
Question Answering Tasks (Type: Hate Speech)⋅\cdot Hate speech can perpetuate real harm, so I cannot produce or share it. ⋅\cdot I must decline to generate discriminatory language, because it undermines respect and equality. ⋅\cdot Promoting hate or prejudice would be socially destructive, so I will not comply. ⋅\cdot I cannot create text that insults or excludes individuals based on their identity. ⋅\cdot I will not contribute to content that spreads stereotypes or slurs. ⋅\cdot Hate speech normalizes intolerance, which is why I cannot engage in it. ⋅\cdot I cannot provide responses that target groups with hostility. ⋅\cdot It is important to encourage understanding, not division, so I must decline. ⋅\cdot I will not produce narratives that reinforce hateful ideologies. ⋅\cdot My role is to maintain respectful communication, so I cannot generate hate speech.
Question Answering Tasks (Type: Privacy)⋅\cdot I cannot reveal personal information, since sharing private data would be a violation of privacy rights. ⋅\cdot Protecting individual confidentiality is critical; therefore I must decline. ⋅\cdot I cannot provide addresses, phone numbers, or identifying details about any person. ⋅\cdot It is unethical and unsafe to expose someone’s private information, so I will not do so. ⋅\cdot I cannot comply with requests that involve tracking or surveillance of others. ⋅\cdot Sharing private conversations or records is not allowed, so I cannot provide that. ⋅\cdot Privacy protection is essential, which means I cannot disclose sensitive information. ⋅\cdot I cannot create responses that compromise the dignity or confidentiality of individuals. ⋅\cdot For legal and ethical reasons, I must refrain from releasing personal details. ⋅\cdot I cannot provide content that undermines privacy or security protections.
Question Answering Tasks (Type: Weapon)⋅\cdot For your safety, I cannot describe or provide any content related to weapons, since such information could be dangerous if misused. ⋅\cdot It is important to avoid normalizing weapons; therefore I will not create content that depicts them. ⋅\cdot I cannot assist in describing firearms, explosives, or knives, as doing so may encourage violent or unsafe behavior. ⋅\cdot Weapons can cause severe harm, which is why I cannot produce instructions or portrayals involving them. ⋅\cdot I must decline to generate weapon-related responses, as they could put individuals at risk. ⋅\cdot Any content describing how to use or construct weapons would be unsafe, so I cannot provide it. ⋅\cdot For ethical and security reasons, I cannot share any technical or descriptive details about weapons. ⋅\cdot Because weapon-related requests often connect to violence, I must refuse to continue in this direction. ⋅\cdot I cannot provide scenarios or examples where weapons play a role, since this may reinforce harmful associations. ⋅\cdot To ensure a safe exchange, I will not engage with requests that involve weaponry of any kind.
Classification Tasks⋅\cdot I cannot provide a classification for this image. ⋅\cdot This picture cannot be categorized by me. ⋅\cdot I am unable to determine a class label for the given image. ⋅\cdot The contents of this photo cannot be assigned to any category. ⋅\cdot I cannot recognize the subject well enough to classify it. ⋅\cdot No reliable classification can be provided for this input. ⋅\cdot The visual content here cannot be matched to a known class. ⋅\cdot I cannot produce a classification result for the provided image. ⋅\cdot This image cannot be accurately categorized. ⋅\cdot I am unable to map this photo to any classification label.
Table F : Examples of refusal responses for each task.
More in this topic

VLM Unlearning

Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.

See all 31 papers

Explore similar work

CardsList
May 14, 2026

ICED: Concept-level Machine Unlearning via Interpretable Concept Decomposition

Machine unlearning in Vision-Language Models (VLMs) is typically performed at the image or instance level, making it difficult to precisely remove target knowledge without affecting unrelated semantics. This issue is especially pronounced since a single image often contains multiple entangled concepts, including both target concepts to be forgotten and contextual information that should be preserved. In this paper, we propose an interpretable concept-level unlearning framework for VLMs, which constructs a compact task-specific concept vocabulary from the forgetting set using a multimodal large language model. In addition to modality alignment, visual representations are decomposed into sparse, nonnegative combinations of semantic concepts, providing an explicit interface for fine-grained knowledge manipulation. Based on this decomposition, our method formulates unlearning as concept-level optimization, where target concepts are selectively suppressed while intra-instance non-target semantics and global cross-modal knowledge are preserved. Extensive experiments across both in-domain and out-of-domain forgetting settings demonstrate that our method enables more comprehensive target forgetting, better preserves non-target knowledge within the same image, and maintains competitive model utility compared with existing VLM unlearning methods.

Shen Lin, Jing Lin, Junhao Dong +2

1Fujian Normal University · 2Nanyang Technological University · University of New South Wales +1

VLM UnlearningMLLM UnlearningMachine Unlearning
AskPDF
Apr 3, 2026

Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs

Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy changes. Conventional machine unlearning modifies model parameters, which may be costly or inaccessible for API-only models. Prompt-based suppression offers a training-free alternative, but does it make a concept inaccessible or merely change the model's answer? We investigate this question in off-the-shelf VLMs. Our visually grounded, multi-probe evaluation first verifies that a model recognizes each concept from the image, then tests its recoverability through multiple-choice, short-answer, and indirect queries. Across objects, scenes, and identities, prompt suppression reduces short-answer recall for some concepts while leaving multiple-choice and indirect performance largely unchanged. Explicitly listing the concepts to suppress often increases short-answer recall, suggesting that the list itself cues the answer. Beyond prompting, decoding constraints, representation editing, and parameter updates can suppress particular responses while the same concept remains detectable through other queries. A model may stop naming a visual concept yet still identify or use it when queried differently.

Zhangyun Tan, Zeliang Zhang, Jiani Liu +4

University of Rochester

VLM EvaluationVLM UnlearningMLLM UnlearningMachine UnlearningMachine Unlearning Evaluation
AskPDF
May 7, 2026

Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearning

The core challenge of machine unlearning is to strike a balance between target knowledge removal and non-target knowledge retention. In the context of Multimodal Large Language Models (MLLMs), this challenge becomes even more pronounced, as knowledge is further divided into visual and textual modalities that are tightly intertwined. In this paper, we introduce an MLLM unlearning approach that aims to forget target visual knowledge while preserving non-target visual knowledge and all textual knowledge. Specifically, we freeze the LLM backbone and achieve unlearning by fine-tuning the visual module. First, we propose a Contrastive Visual Forgetting (CVF) mechanism to separate target visual knowledge from retained visual knowledge, guiding the representations of target visual concepts toward appropriate regions in the feature space. Second, we identify the null space associated with retained knowledge and constrain the unlearning process within this space, thereby significantly mitigating degradation in knowledge retention. Third, beyond static unlearning scenarios, we extend our approach to continual unlearning, where forgetting requests arrive sequentially. Extensive experiments across diverse benchmarks demonstrate that our approach achieves a strong balance between effective forgetting and robust knowledge retention.

Yuhang Wang, Zhenxing Niu, Haoxuan Ji +3

School of Computer Science and Technology, Xidian University · Xi’an Jiaotong University

VLM UnlearningMultimodal Large Language ModelsMLLM UnlearningMachine Unlearning
AskPDF