Authors: Piyush Tiwary, Anushka Srivastava, Prathosh A. P
Organizations: Department of Electrical Communication Engineering, Indian Institute of Science, Bengaluru, KA 560012, India · Independent Researcher · Department of Electrical Communication Engineering, Bengaluru, KA 560012, India
Machine unlearning aims to remove a user's data from a trained model after a deletion request. Existing methods are majorly evaluated on three axes - how well they forget, how much capability they retain, and how robust they are to membership inference attacks, but not on whether the provider can offer the user any evidence that the deletion occurred. To fill this gap, we introduce auditability as a fourth axis, defined as the separation between requested (forget set) and retained (retain set) data under a statistic the provider computes. We show that across eleven unlearning methods on TOFU, MUSE and RWKU no method satisfies all four axes, with different methods falling short on different axes. To address this, we propose a two stage method that can be used on top of any unlearning method, in the the first stage we propose GapWiden which uses a hinge loss to increase the difference between audit statistics of forget set and retain set. In the second stage we propose GapVeil, an inference time rejection-sampler which uses the base model as the drafter and re-samples based on the audit statistics of the drafted response. We observe that using our method, auditability and leakage reach oracle parity for every unlearning method, and forget performance converges to the oracle; retain performance improves for nine of eleven methods but remains short of parity. Our results show that evidence of deletion or auditability can be constructed by design, and that treating it as an explicit requirement changes the viability of existing unlearning methods.
Figures & tables
Figure 1: Overview of proposed method. Left: The Unlearning Dilemma across the four evaluation axes. Right: (a) GapWiden post-unlearning method and (b) GapVeil inference-time rejection sampling.
Figure 2: Four-axis unlearning evaluation. (a) Spider plot comparing representative baselines ( SatImp , RMU , NPO ) against our method and the retrained Oracle. (b) Per-method performance improvement from initial unlearned models ( θu ) to our proposed method across Auditability, Leakage Robustness, Forget, and Retain performance.
Method
Auditability
Leakage
Forget
Retain
CEU ( Yang, 2025 )
×
✓
×
×
DPO ( Zhang et al., 2024b )
✓
×
✓
✓
GradAscent ( Maini et al., 2024 )
×
∼
×
×
GradDiff ( Maini et al., 2024 )
✓
×
✓
×
NPO ( Zhang et al., 2024b )
✓
✓
✓
×
PDU ( Entesari et al., 2026 )
✓
✓
✓
×
Table 1: Comparison of several unlearning methods on TOFU-forget10B (✓: satisfies, ∼ : partial, × : fails). Detailed metric definitions are provided in Appendix C .
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_8B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Table 2: Per-split evaluation on TOFU using Llama-8B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
MUSE-News
MUSE-Books
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_muse_paired.csv \tblmname
\tblNewsFrgBase
\tblNewsRetBase
\tblNewsLeakBase
\tblNewsAudBase
\tblBooksFrgBase
\tblBooksRetBase
\tblBooksLeakBase
\tblBooksAudBase
\tblBooksAudOurs
\tblNewsFrgOurs
\tblNewsRetOurs
\tblNewsLeakOurs
\tblNewsAudOurs
\tblBooksFrgOurs
\tblBooksRetOurs
\tblBooksLeakOurs
Table 3: Per-split evaluation on MUSE using Llama-7B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
Figure 3: GapWiden injects audit signal on un-auditable baselines. Score distributions for forget and retain sets across the three un-auditable methods, before and after GapWiden across four audit statistics. GapWiden collapses the retain distribution near zero while shifting forget mass beyond the decision margin. Numbers report separation AUROC.
Figure 4: GapVeil suppresses behavioral leakage against black-box attackers. Distribution of black-box abnormality scores on forget versus comparison sets for unlearning baselines and under GapVeil . Values report attacker advantage ∣AUROC−0.5∣ in bold and signed AUROC in parentheses.
Figure 8
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
lr
Epochs
Objective coefficients
GradAscent
10−5
2.5
– (unconstrained ascent on Df )
GradDiff
10−5
7
γ=1.0,α=1.0
NPO
5×10−5
10
β=0.1,γ=1.0,α=1.0
DPO
10−5
10
β=0.1,γ=1.0,α=1.0 ; IDK preference pairs
SimNPO
5×10−5
10
β=2.5,γ=1.0,α=0.5
RMU
10−5
10
steering coeff. 2 , α=1 , layer-7 activations
Appendix
Table 4: Method specific hyperparameters used for obtaining the unlearned model.
Method
lr
Epochs
Objective coefficients
NPO
3×10−5
10
β=0.1,γ=1.0,α=1.0 (recipe default)
CEU
4×10−6
3
suppress true token from position 2 on
SimNPO
10−5
10
β=0.7,γ=1.0,α=0.1
RMU
10−3
10
steering coeff. 6.5 ; layers 5 – 7 down_proj only
SatImp
10−5
10
β1=4.0,β2=0.1,γ=1.0,α=0.1
WGA
10−5
10
β=5.0,α=0.1,γ=1.0
Appendix
Table 5: Method specific hyperaparameters used for obtaining unlearned model on MUSE.
Sampling steps, K
3
Energy, e⋆
best of {log-prob, KL, JS, entropy} gap, chosen per method
Threshold, τ
argmax0.85⋅TPR+0.15⋅TNR on forget vs. retain energy scores
Draft sampling
temperature 1.0 , do_sample=True
Appendix
Table 6: GapVeil sampling configuration, fixed across all reported results.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_1B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 7: Per-split evaluation on TOFU using Llama-1B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_3B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 8: Per-split evaluation on TOFU using Llama-3B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_8B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 9: Per-split evaluation on TOFU using Llama-8B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_rwku_3B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 10: Per-split evaluation on RWKU using Llama-3B model. Splits forget01, forget05 and forget10 contain RWKU targets 1–2, 1–10 and 1–20 (20 question-answer pairs each). Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_rwku_8B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 11: Per-split evaluation on RWKU using Llama-8B model. Splits forget01, forget05 and forget10 contain RWKU targets 1–2, 1–10 and 1–20 (20 question-answer pairs each). Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
Figure 9: Abnormality score distribution after each stage in proposed pipeline. Distribution of black-box abnormality scores on forget versus comparison sets for unlearning baselines and under GapWiden and GapVeil . Values report attacker advantage ∣AUROC−0.5∣ in bold and signed AUROC in parentheses.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Machine unlearning has been extensively studied in response to growing privacy concerns and regulatory requirements. However, auditing whether unlearning algorithms have truly erased the influence of specific data remains an open challenge. The lack of reliable and practical auditing mechanisms can lead to critical privacy risks, such as residual information leakage. This paper initiates a systematic investigation into whether existing unlearning algorithms can truly forget the designated data. We propose the first practical and general-purpose auditing framework for machine unlearning, inspired by the concept of proof of ignorance. Our framework addresses the key practicality limitations of existing methods by eliminating the need for retraining-from-scratch baselines, avoiding the training of large numbers of shadow models, and requiring no intrusive intervention in the original training process. To evaluate the effectiveness of our framework, we first conduct validation experiments to verify its soundness and completeness. We then perform comprehensive experiments across six datasets and ten representative unlearning methods. The results demonstrate that our framework reliably distinguishes between successful and failed unlearning. In particular, we observe that retraining-based and fine-tuning-based methods can achieve effective unlearning, even when the target data remain in the original dataset. In contrast, de-optimization-based methods fail to achieve true unlearning and instead degrade the model's performance. Fisher/Hessian-based methods also fail to unlearn requested data, even formal certification is provided. Moreover, we show that our framework is robust against fake unlearning attempts and generalizes well to large language models.
Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes data-dependent lower bounds on the unlearning parameter ε using membership inference attacks. Evaluating multiple unlearning algorithms, we find a sharp separation: algorithms with rigorous guarantees, such as model clipping and rewind-to-delete, achieve very small ε bounds that do not falsify their unlearning guarantees, whereas empirical methods such as Hessian-based unlearning, interleaved ascent-descent, ascent on the forget set, and fine-tuning on the retain set exhibit large bounds, indicating poor unlearning. Our auditor provides a practical tool for empirically falsifying unlearning claims through a hypothesis-testing framework, and we validate it on CIFAR-100 and Shakespeare text.
Prior white-box studies show that large language models can retain latent traces of target knowledge after unlearning, even when the knowledge is no longer expressed in their outputs. However, existing audits remain limited to one-off diagnostics: it is unclear whether these residual signals can predict future recovery under continued training or serve as reliable optimization targets. Resolving this gap is essential to determine whether internal auditing can move beyond post-hoc evaluation toward proactive risk monitoring and safer unlearning. We propose J-Access, an inference-time audit that uses the Jacobian lens to map intermediate representations into vocabulary space and measures how often target concepts remain accessible along the model's output pathway. We hypothesize that residual accessibility reflects recovery susceptibility: knowledge that remains closer to the output pathway requires less fine-tuning to restore, leading to faster recovery. We audit 398 public unlearned models spanning eight unlearning methods. We find that: (1) most unlearned models retain access above the retain-only gold level; (2) pre-attack accessibility predicts recovery speed and extent at the model level, but cannot identify which specific facts will be recovered; and (3) directly minimizing J-Access does not promote genuine deletion. Instead, the model learns to hide knowledge from the audit, producing lower audit scores but greater post-attack recovery. These findings position J-Access as a model-level diagnostic for assessing residual susceptibility in unlearned models. We argue internal audits should serve as an independent diagnostic dimension in unlearning evaluation, and should not be converted into optimization targets without validation.
Zirui Song, Huaxing Liu, Xiang Wang +8
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · AMAP, Alibaba Group · Peking University