cs.CVOct 8, 2025

Cross-Modal Attention Guided Unlearning in Vision-Language Models

Authors: Karuna Bhaila, Aneesh Komanduri, Minh-Hao Van, Xintao Wu

Organizations: Department of Electrical Engineering and Computer Science, University of Arkansas

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in multi-modal understanding for tasks such as Visual Question Answering (VQA), which requires models to infer outputs based on visual and textual context simultaneously. However, models may memorize some private and/or sensitive information during training and regurgitate it in inference. Recently, machine unlearning has been leveraged to address such leakage of private data in LLMs. VLMs add a layer of complexity to this process, as the visual context in the query may additionally contain sensitive information. To address this issue, we explore unlearning for VLMs, specifically for the VQA task. We explore the role of visual tokens for output generation in VLMs using cross-modal attention and utilize it to formulate Cross-Modal Attention Guided Unlearning (CAGUL), a lightweight and efficient VLM unlearning framework. In contrast to computationally expensive finetuning methods, CAGUL utilizes external modules to encode unlearning information in visual tokens of low importance for relevant queries. Empirical results show that the transformed visual tokens prevent leakage and retain reference model behavior to achieve desirable trade-offs in forget quality and model utility without altering pretrained model parameters.

Figures & tables

Explore similar work

CardsList
  1. One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models

    Jul 17, 2026Sudharshan Balaji, Yili Ren, Guangjing Wang +2VLM UnlearningVLM Robustness

  2. Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning

    Oct 31, 2026Laura Yao, Haochen Zhang, Jinhao Duan +2VLM Unlearning

  3. On the Robustness of Machine Unlearning for Vision-Language Models

    May 26, 2026Yujie Lin, Kaidi Jia, Jiayao Ma +2VLM EvaluationVLM Unlearning