Machine Unlearning Evaluation

Follow this topic Ask about this topic

Machine unlearning evaluation asks whether a model has genuinely lost the influence of specified training examples, facts, or users—not merely stopped producing them on familiar prompts. Common approaches compare behavior and utility with retrained models, measure membership or extraction risk, probe hidden representations, and test recovery through adversarial queries, fine-tuning, or sequential updates. Benchmarks span classifiers, language and vision-language models, generative systems, and agents, with checks for collateral damage to related knowledge and retained capabilities. A central tension remains: behavioral tests are practical but can miss recoverable information, while stronger audits may require costly reference models or risk exposing private data themselves.

Latest papers

105 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Behavioral Audit of Machine Unlearning Has a Privacy Cost
Jun 12, 2026Liou Tang, James Joshi, Ashish KunduPrivacy Auditing Machine Unlearning
University of Pittsburgh, Pittsburgh, PA, USA · Cisco Research, San Jose, CA, USA
MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs
Jun 11, 2026He Li, Haoang Chi, Qizhou Wang +6VLM Unlearning Multimodal Large Language Models
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · TMLR Group, Hong Kong Baptist University, Hong Kong, China · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China +2
REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference
Jun 5, 2026Anurag Sharma, Sai Teja Chunchu, Prasenjit Mitra +2Multi-Label Classification Machine Unlearning
IIT Kharagpur · India · Carnegie Mellon University +4
SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation
May 29, 2026Petros Andreou, Jamie Lanyon, Axel Finke +1Machine Unlearning Machine Unlearning Evaluation
Department of Computer Science, School of Science, Loughborough University, Loughborough, UK · School of Mathematics, Statistics and Physics, Newcastle University, Newcastle upon Tyne, NE1 7RU, UK
Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning
May 29, 2026Polina Dolgova, Sebastian U. StichSelf-Supervised Knowledge Distillation Machine Unlearning
CISPA Helmholtz Center for Information Security · Universität des Saarlandes · Saarbrücken, Germany
AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis
May 28, 2026Saeedeh Davoudi, Reihaneh Iranmanesh, Ophir Frieder +1Healthcare Privacy Leakage in Language Models
1IR Lab, Computer Science Department, Georgetown University, Washington D.C.
MAAT: Multi-phase Adapter-Aware Targeted Unlearning
May 28, 2026Suryash Yagnik, Shubham Gaur, Saksham Thakur +3Machine Unlearning LLM Unlearning
1Indian Institute of Information Technology, Bhopal, India · University of California, Santa Cruz, USA · 3Independent Researcher +2
RULER: Representation-Level Verification of Machine Unlearning
May 26, 2026Georgina Cosma, Axel FinkeMachine Unlearning Machine Unlearning Evaluation
Department of Computer Science, Loughborough University, UK · School of Mathematics, Statistics and Physics, Newcastle University, UK
On the Robustness of Machine Unlearning for Vision-Language Models
May 26, 2026Yujie Lin, Kaidi Jia, Jiayao Ma +2VLM Evaluation VLM Unlearning
1Xiamen University
Measuring the Depth of LLM Unlearning via Activation Patching
May 23, 2026Jaeung Lee, Dohyun Kim, Jaemin JoLLM Auditing LLM Unlearning
Sungkyunkwan University Republic of Korea
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models
May 20, 2026Divyaksh Shukla, Ashutosh ModiShortcut Learning in Language Models Confidence Estimation in Language Models
Indian Institute of Technology Kanpur (IIT Kanpur)
Form and Function: Machine Unlearning as a Problem of Misaligned States
May 17, 2026Kennon StewartQuasi-Newton Methods Machine Unlearning
Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA
Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries
May 17, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangMemorization in Language Models LLM Auditing
Northeastern University, USA · University of Illinois Urbana-Champaign, USA · Southern Methodist University, USA
Forgetting That Sticks: Quantization-Permanent Unlearning via Circuit Attribution
May 14, 2026Saisab Sadhu, Pratinav Seth, Vinay Kumar SankarapuMachine Unlearning Post-Training Quantization
Lexsi Labs
Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation
May 14, 2026Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho +1Multilingual Language Model Evaluation Privacy Leakage in Language Models
GSCST, Seoul National University · Department of Artificial Intelligence, Chung-Ang University · Korean Surgical Researcher Foundation, Republic of Korea +1
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
May 9, 2026Jiahui Guang, Zexun Zhan, Zhenlin Xu +5VLM Evaluation VLM Unlearning
Harbin Institute of Technology, Shenzhen, China · Sichuan University, Chengdu, China · Harbin Institute of Technology, Weihai, China +3
Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning
May 9, 2026Renjie Gu, Jiazhen Du, Yihua Zhang +1LLM Unlearning LLM Honesty
Fudan University · Central South University · Michigan State University
Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation
May 9, 2026Weidong Zheng, Kongyang Chen, Yuanwei Guo +1Classification Machine Unlearning
School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou 510335, China · School of Artificial Intelligence, Guangzhou University, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510006, China +1
ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models
May 7, 2026Yuhang Wang, Wenjie Mei, Junkai Zhang +3Privacy-Preserving Language Models Multimodal Large Language Models
School of Computer Science and Technology Xidian University
Not Every Subject Should Stay: Machine Unlearning for Noisy Engagement Recognition
May 6, 2026Alexander VedernikovMachine Unlearning Machine Unlearning Evaluation
Helsinki, Finland
Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks
May 5, 2026JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3VLM Evaluation VLM Unlearning
Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score
May 4, 2026Abdullah Ahmad Khan, Hamid Laga, Ferdous SohelVLM Evaluation VLM Unlearning
Murdoch University
Towards Reliable Testing of Machine Unlearning
Apr 16, 2026Anna Mazhar, Sainyam GalhotraData Leakage Automated Software Testing
Cornell University Ithaca, NY, USA
Graph Propagated Projection Unlearning: A Unified Framework for Vision and Audio Discriminative Models
Apr 13, 2026Shreyansh Pathak, Jyotishman DasMachine Unlearning Machine Unlearning Evaluation
Indian Institute of Technology Jodhpur
Efficient Unlearning through Maximizing Relearning Convergence Delay
Apr 10, 2026Khoa Tran, Simon S. WooMachine Unlearning Machine Unlearning Evaluation
CSE Department, Sungkyunkwan University, South Korea · Secure Machines Lab
An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations
Apr 9, 2026Yichen Gao, Altay Unal, Akshay Rangamani +1Machine Unlearning Machine Unlearning Evaluation
Department of Computer Science & Engineering, The Ohio State University · Department of Data Science, New Jersey Institute of Technology
Is your algorithm unlearning or untraining?
Apr 9, 2026Eleni Triantafillou, Ahmed Imtiaz Humayun, Monica Ribero +3Machine Unlearning Machine Unlearning Evaluation
Google · Hasso Plattner Institute
Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation
Apr 6, 2026Houzhe Wang, Xiaojie Zhu, Chi ChenFederated Unlearning Machine Unlearning Evaluation
School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Kingdom of Saudi Arabia
Erasure or Erosion? Evaluating Compositional Degradation in Unlearned Text-To-Image Diffusion Models
Apr 6, 2026Arian Komaei Koma, Seyed Amir Kasaei, Ali Aghayari +2Diffusion Model Unlearning Machine Unlearning
Department of Computer Engineering, Sharif University of Technology
Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs
Apr 3, 2026Zhangyun Tan, Zeliang Zhang, Jiani Liu +4VLM Evaluation VLM Unlearning
University of Rochester

Jun 12, 2026J/K move · Enter open · S save

Behavioral Audit of Machine Unlearning Has a Privacy Cost

Liou Tang, James Joshi, Ashish Kundu

University of Pittsburgh, Pittsburgh, PA, USA · Cisco Research, San Jose, CA, USA

Privacy Auditing Machine Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

The removal of learned data from Machine Learning models through Machine Unlearning (MU) has been widely studied; however, there has yet to be an agreed-upon scheme for auditing MU. Existing work has shown that a dishonest model owner can falsify evidence to avoid executing MU, while curious auditors (and adversaries) can infer the privacy-sensitive properties of the model and its training data even with limited access. Yet auditing of MU under mutual distrust between the model owner and the auditor remains unexplored. We provide an information-theoretic proof for this scenario: for convex ML models, a generic audit scheme that relies solely on querying the model for behavioral signals cannot identify insufficiently unlearned models without revealing membership information of the retained set. Therefore, auditing MU under the assumption of a dishonest model owner and an honest-but-curious auditor faces an inherent privacy-audit tradeoff. Our empirical results on convex models strongly supports this result, while further experiments demonstrate that this privacy-audit tension persists in non-convex models. Our results call for a more careful consideration of the privacy-audit tension under a realistic auditor threat model, and serve as a foundation for more scrutiny of designs of privacy-preserving audit schemes for the MU pipeline. We also release our code implementation at https://github.com/LiouTang/Behavioral-Unlearn-Audit.