Authors: Piyush Tiwary, Anushka Srivastava, Prathosh A. P
Organizations: Department of Electrical Communication Engineering, Indian Institute of Science, Bengaluru, KA 560012, India · Independent Researcher · Department of Electrical Communication Engineering, Bengaluru, KA 560012, India
Machine unlearning aims to remove a user's data from a trained model after a deletion request. Existing methods are majorly evaluated on three axes - how well they forget, how much capability they retain, and how robust they are to membership inference attacks, but not on whether the provider can offer the user any evidence that the deletion occurred. To fill this gap, we introduce auditability as a fourth axis, defined as the separation between requested (forget set) and retained (retain set) data under a statistic the provider computes. We show that across eleven unlearning methods on TOFU, MUSE and RWKU no method satisfies all four axes, with different methods falling short on different axes. To address this, we propose a two stage method that can be used on top of any unlearning method, in the the first stage we propose GapWiden which uses a hinge loss to increase the difference between audit statistics of forget set and retain set. In the second stage we propose GapVeil, an inference time rejection-sampler which uses the base model as the drafter and re-samples based on the audit statistics of the drafted response. We observe that using our method, auditability and leakage reach oracle parity for every unlearning method, and forget performance converges to the oracle; retain performance improves for nine of eleven methods but remains short of parity. Our results show that evidence of deletion or auditability can be constructed by design, and that treating it as an explicit requirement changes the viability of existing unlearning methods.
Figures & tables
Figure 1: Overview of proposed method. Left: The Unlearning Dilemma across the four evaluation axes. Right: (a) GapWiden post-unlearning method and (b) GapVeil inference-time rejection sampling.
Figure 2: Four-axis unlearning evaluation. (a) Spider plot comparing representative baselines ( SatImp , RMU , NPO ) against our method and the retrained Oracle. (b) Per-method performance improvement from initial unlearned models ( θu ) to our proposed method across Auditability, Leakage Robustness, Forget, and Retain performance.
Method
Auditability
Leakage
Forget
Retain
CEU ( Yang, 2025 )
×
✓
×
×
DPO ( Zhang et al., 2024b )
✓
×
✓
✓
GradAscent ( Maini et al., 2024 )
×
∼
×
×
GradDiff ( Maini et al., 2024 )
✓
×
✓
×
NPO ( Zhang et al., 2024b )
✓
✓
✓
×
PDU ( Entesari et al., 2026 )
✓
✓
✓
×
Table 1: Comparison of several unlearning methods on TOFU-forget10B (✓: satisfies, ∼ : partial, × : fails). Detailed metric definitions are provided in Appendix C .
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_8B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Table 2: Per-split evaluation on TOFU using Llama-8B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
MUSE-News
MUSE-Books
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_muse_paired.csv \tblmname
\tblNewsFrgBase
\tblNewsRetBase
\tblNewsLeakBase
\tblNewsAudBase
\tblBooksFrgBase
\tblBooksRetBase
\tblBooksLeakBase
\tblBooksAudBase
\tblBooksAudOurs
\tblNewsFrgOurs
\tblNewsRetOurs
\tblNewsLeakOurs
\tblNewsAudOurs
\tblBooksFrgOurs
\tblBooksRetOurs
\tblBooksLeakOurs
Table 3: Per-split evaluation on MUSE using Llama-7B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
Figure 3: GapWiden injects audit signal on un-auditable baselines. Score distributions for forget and retain sets across the three un-auditable methods, before and after GapWiden across four audit statistics. GapWiden collapses the retain distribution near zero while shifting forget mass beyond the decision margin. Numbers report separation AUROC.
Figure 4: GapVeil suppresses behavioral leakage against black-box attackers. Distribution of black-box abnormality scores on forget versus comparison sets for unlearning baselines and under GapVeil . Values report attacker advantage ∣AUROC−0.5∣ in bold and signed AUROC in parentheses.
Figure 8
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
lr
Epochs
Objective coefficients
GradAscent
10−5
2.5
– (unconstrained ascent on Df )
GradDiff
10−5
7
γ=1.0,α=1.0
NPO
5×10−5
10
β=0.1,γ=1.0,α=1.0
DPO
10−5
10
β=0.1,γ=1.0,α=1.0 ; IDK preference pairs
SimNPO
5×10−5
10
β=2.5,γ=1.0,α=0.5
RMU
10−5
10
steering coeff. 2 , α=1 , layer-7 activations
Appendix
Table 4: Method specific hyperparameters used for obtaining the unlearned model.
Method
lr
Epochs
Objective coefficients
NPO
3×10−5
10
β=0.1,γ=1.0,α=1.0 (recipe default)
CEU
4×10−6
3
suppress true token from position 2 on
SimNPO
10−5
10
β=0.7,γ=1.0,α=0.1
RMU
10−3
10
steering coeff. 6.5 ; layers 5 – 7 down_proj only
SatImp
10−5
10
β1=4.0,β2=0.1,γ=1.0,α=0.1
WGA
10−5
10
β=5.0,α=0.1,γ=1.0
Appendix
Table 5: Method specific hyperaparameters used for obtaining unlearned model on MUSE.
Sampling steps, K
3
Energy, e⋆
best of {log-prob, KL, JS, entropy} gap, chosen per method
Threshold, τ
argmax0.85⋅TPR+0.15⋅TNR on forget vs. retain energy scores
Draft sampling
temperature 1.0 , do_sample=True
Appendix
Table 6: GapVeil sampling configuration, fixed across all reported results.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_1B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 7: Per-split evaluation on TOFU using Llama-1B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_3B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 8: Per-split evaluation on TOFU using Llama-3B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_8B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 9: Per-split evaluation on TOFU using Llama-8B model. Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_rwku_3B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 10: Per-split evaluation on RWKU using Llama-3B model. Splits forget01, forget05 and forget10 contain RWKU targets 1–2, 1–10 and 1–20 (20 question-answer pairs each). Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
forget01
forget05
forget10
Method
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
Frg
Ret
Leak
Aud
\csvreader [ head to column names, head to column names prefix=tbl, late after line= , late after last line= , ]tables/results_rwku_8B_paired.csv \tblmname
\tblfOneFrgBase
\tblfOneRetBase
\tblfOneLeakBase
\tblfOneAudBase
\tblfFiveFrgBase
\tblfFiveRetBase
\tblfFiveLeakBase
\tblfFiveAudBase
\tblfTenFrgBase
\tblfTenRetBase
\tblfTenLeakBase
\tblfTenAudBase
\tblfTenAudOurs
\tblfOneFrgOurs
\tblfOneRetOurs
\tblfOneLeakOurs
\tblfOneAudOurs
\tblfFiveFrgOurs
\tblfFiveRetOurs
\tblfFiveLeakOurs
\tblfFiveAudOurs
\tblfTenFrgOurs
\tblfTenRetOurs
\tblfTenLeakOurs
Appendix
Table 11: Per-split evaluation on RWKU using Llama-8B model. Splits forget01, forget05 and forget10 contain RWKU targets 1–2, 1–10 and 1–20 (20 question-answer pairs each). Performance comparison across forget ( Frg ), retain ( Ret ), leakage ( Leak ), and auditability ( Aud ) metrics, evaluated with and without our proposed framework. For each baseline pair, the larger value is highlighted in bold ; ties remain unbolded. The last row averages each column over all the methods.
Figure 9: Abnormality score distribution after each stage in proposed pipeline. Distribution of black-box abnormality scores on forget versus comparison sets for unlearning baselines and under GapWiden and GapVeil . Values report attacker advantage ∣AUROC−0.5∣ in bold and signed AUROC in parentheses.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.