Machine Unlearning Evaluation

Follow this topic Ask about this topic

Machine unlearning evaluation asks whether a model has genuinely lost the influence of specified training examples, facts, or users—not merely stopped producing them on familiar prompts. Common approaches compare behavior and utility with retrained models, measure membership or extraction risk, probe hidden representations, and test recovery through adversarial queries, fine-tuning, or sequential updates. Benchmarks span classifiers, language and vision-language models, generative systems, and agents, with checks for collateral damage to related knowledge and retained capabilities. A central tension remains: behavioral tests are practical but can miss recoverable information, while stronger audits may require costly reference models or risk exposing private data themselves.

Latest papers

105 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Secure Forgetting: A Framework for Privacy-Driven Unlearning in Large Language Model (LLM)-Based Agents
Apr 1, 2026Dayong Ye, Tainqing Zhu, Congcong Zhu +5LLM Unlearning Privacy-Preserving Language Models
SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning
Mar 27, 2026Cai Selvas-Sala, Lei Kang, Lluis GomezVLM Unlearning Machine Unlearning Evaluation
Computer Vision Center · Universitat Polit`ecnica de Catalunya · Universitat Aut`onoma de Barcelona
Computation and Communication Efficient Federated Unlearning via On-server Gradient Conflict Mitigation and Expression
Mar 14, 2026Minh-Duong Nguyen, Senura Hansaja, Le-Tuan Nguyen +4Federated Unlearning Federated Learning
VinUniversity · University of Sydney · Trinity College Dublin
The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning
Mar 11, 2026Raj Sanjay Shah, Jing Huang, Keerthiram Murugesan +2LLM Unlearning Machine Unlearning Evaluation
Georgia Institute of Technology · Stanford University · IBM Research
A Comprehensive Evaluation of LLM Unlearning Robustness under Multi-Turn Interaction
Feb 28, 2026Ruihao Pan, Suhang WangLLM Unlearning Machine Unlearning Evaluation
Pennsylvania State University
Unlearning Evaluation through Subset Statistical Independence
Feb 28, 2026Chenhao Zhang, Muxing Li, Feng Liu +2Machine Unlearning Evaluation Machine Unlearning
University of Queensland · University of Melbourne · RIKEN Center for Advanced Intelligence Project +1
ROKA: Robust Knowledge Unlearning against Adversaries
Feb 28, 2026Jinmyeong Shin, Joshua Tapia, Nicholas Ferreira +3Machine Unlearning Machine Unlearning Evaluation
University of California, Merced, USA · California State University, East Bay, USA
Benchmarking Unlearning for Vision Transformers
Feb 23, 2026Kairan Zhao, Iurie Luca, Peter TriantafillouMachine Unlearning Machine Unlearning Evaluation
University of Warwick
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning
Feb 23, 2026Anna Borisiuk, Andrey Savchenko, Alexander Panchenko +1Machine Unlearning LLM Unlearning
AIRI · Skoltech · Sber AI Lab +2
Understanding Empirical Unlearning with Combinatorial Interpretability
Feb 22, 2026Shingo Kodama, Niv Cohen, Micah Adler +1Machine Unlearning Machine Unlearning Evaluation
Middlebury College · New York University · MIT +1
Easy Data Unlearning Bench
Feb 18, 2026Roy Rinberg, Pol Puigdemont, Martin Pawelczyk +1Machine Unlearning Machine Unlearning Evaluation
Harvard University · LIONS, EPFL, Lausanne, Switzerland.
Suppression or Deletion: A Restoration-Based Representation-Level Analysis of Machine Unlearning
Feb 18, 2026Yurim Jang, Jaeung Lee, Dohyun Kim +2Machine Unlearning Machine Unlearning Evaluation
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea
RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories
Dec 3, 2025Roy Rinberg, Usha Bhalla, Igor Shilov +2LLM Evaluation LLM Unlearning
Harvard University · MATS · Imperial College London +1
CAFÉ: Causal Black-Box Testing of Machine Unlearning
Sep 20, 2025Anna Mazhar, Sainyam GalhotraMachine Unlearning Evaluation Black-Box Model Auditing
Cornell University, USA
Instance-Level Difficulty: A Missing Perspective in Machine Unlearning
Oct 3, 2024Hammad Rizwan, Mahtab Sarvmaili, Hassan Sajjad +1Machine Unlearning Machine Unlearning Evaluation
Department of Computer Science, Dalhousie University, Canada.

Apr 1, 2026J/K move · Enter open · S save

Secure Forgetting: A Framework for Privacy-Driven Unlearning in Large Language Model (LLM)-Based Agents

Dayong Ye, Tainqing Zhu, Congcong Zhu, Feng He+4

LLM Unlearning Privacy-Preserving Language Models Machine Unlearning Evaluation

Open paper AskPDF arXiv

Large language model (LLM)-based agents have recently gained considerable attention due to the powerful reasoning capabilities of LLMs. Existing research predominantly focuses on enhancing the task performance of these agents in diverse scenarios. However, as LLM-based agents become increasingly integrated into real-world applications, significant concerns emerge regarding their accumulation of sensitive or outdated knowledge. Addressing these concerns requires the development of mechanisms that allow agents to selectively forget previously learned knowledge, giving rise to a new term LLM-based agent unlearning. This paper initiates research on unlearning in LLM-based agents. Specifically, we propose a novel and comprehensive framework that categorizes unlearning scenarios into three contexts: state unlearning (forgetting specific states or items), trajectory unlearning (forgetting sequences of actions) and environment unlearning (forgetting entire environments or categories of tasks). Within this framework, we introduce a natural language-based unlearning method that trains a conversion model to transform high-level unlearning requests into actionable unlearning prompts, guiding agents through a controlled forgetting process. Moreover, to evaluate the robustness of the proposed framework, we introduce an unlearning inference adversary capable of crafting prompts, querying agents, and observing their behaviors in an attempt to infer the forgotten knowledge. Experimental results show that our approach effectively enables agents to forget targeted knowledge while preserving performance on untargeted tasks, and prevents the adversary from inferring the forgotten knowledge.