cs.CLOct 15, 2026

Behavioral Forgetting Is Not Privacy in Machine Unlearning

Authors: Tyler Lizzo, Larry Heck

Organizations: AI Virtual Assistant (AVA) Lab, Georgia Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine unlearning aims to remove targeted information from large language models, but existing evaluations often measure forgetting only on a designated benchmark distribution. We argue that such behavioral forgetting is an insufficient standard for privacy-oriented unlearning, and that evaluation should instead test whether forgetting persists across the access paths through which the same information may be recovered. We demonstrate this with a controlled case study of cross-lingual querying: across multiple multilingual model families, forgetting varies substantially when semantically equivalent information is queried in different languages and scripts. A single access path is therefore enough to expose a real gap between behavioral forgetting and information inaccessibility, motivating broader evaluation of access-path generalization and adversarial recoverability alongside in-distribution forgetting.

Figures & tables

Figure 1: Cross-lingual access-path evaluation for Llama 3 using UNLEARN ( Lizzo and Heck, 2025 ) Each panel fixes the unlearning language, while points evaluate the same targeted facts through different linguistic access paths. The dashed line denotes the TOFU forgetting criterion ( p=0.1p=0.1 ); model utility is normalized relative to the base model. Apparent unlearning success varies with the language used for evaluation. See Appendix A.5 for additional results across model families.
Figure 2: Representational basis of access-path-dependent forgetting. Removing the shared multilingual subspace induces forgetting across linguistic access paths, whereas removing a language-specific residual primarily affects the corresponding language.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

 Llama 3MixtralAya 23
MethodUtility ↑\uparrowForget ↑\uparrowUtility ↑\uparrowForget ↑\uparrowUtility ↑\uparrowForget ↑\uparrow
Base Model0.62272.37×10−152.37{\times}10^{-15}0.59123.0×10−163.0{\times}10^{-16}0.55036.4×10−246.4{\times}10^{-24}
DPO0.53571.06×10−161.06{\times}10^{-16}0.51002.5×10−142.5{\times}10^{-14}0.41824.3×10−214.3{\times}10^{-21}
Gradient Ascent0.00001.43×10−221.43{\times}10^{-22}0.00001.35×10−191.35{\times}10^{-19}0.00006.28×10−186.28{\times}10^{-18}
KL Minimization0.00004.07×10−324.07{\times}10^{-32}0.00007.23×10−197.23{\times}10^{-19}0.00002.38×10−262.38{\times}10^{-26}
Gradient Difference0.58722.83×10−252.83{\times}10^{-25}0.54835.83×10−255.83{\times}10^{-25}0.49786.19×10−296.19{\times}10^{-29}
Appendix
Table 1: Comparison of unlearning performance on the English-English TOFU benchmark [ Maini et al., 2024 ] for three models: Llama 3, Mixtral, and Aya 23. Among unlearning methods, the best value in ecah column is bolded and the second-best is underlined. ✓marks forget quality above α=0.1\alpha=0.1 .
LanguageHuman Pass RateCOMET-Kiwi
Spanish98.75%0.8818
Hindi96.5%0.8769
Hindi (Romanized)94.25%–
Italian99.25%0.8858
Chinese95.5%0.8610
Chinese (Pinyin)93.25%–
Appendix
Table 2: Translation validation results for cTOFU. COMET-Kiwi scores measure semantic consistency between English source text and translated outputs using a reference-free quality estimation model. Human validation pass rates report the percentage of manually inspected examples preserving factual consistency and author identity across translations. Human evaluation used 400 randomly sampled examples per language.
ModelActive ParametersTotal ParametersHuggingFace Checkpoint
Mistral 7B7B7Bmistralai/Mistral-7B-v0.1
Mixtral 8 ×\times 7B12.9B46.7Bmistralai/Mixtral-8x7B-v0.1
Magistral Small24B24Bmistralai/Magistral-Small-2506
LLaMA 2 7B7B7Bmeta-llama/Llama-2-7b-hf
LLaMA 3 8B8B8Bmeta-llama/Meta-Llama-3-8B
LLaMA 4 Scout17B109Bmeta-llama/Llama-4-Scout-17B-16E
Appendix
Table 3: Pretrained language models used in our evaluation. Active parameters denote the number of parameters used during a forward pass. Total parameters denote the full parameter count of the model. All checkpoints are obtained from HuggingFace.
  Llama 3MixtralAya 23
UnlearnEvalUtil.FQUtil.FQUtil.FQ
BaseEnglish0.62272.37×10−152.37{\times}10^{-15}0.59122.74×10−162.74{\times}10^{-16}0.55036.47×10−246.47{\times}10^{-24}
Spanish0.59194.31×10−134.31{\times}10^{-13}0.53934.68×10−134.68{\times}10^{-13}0.51932.63×10−212.63{\times}10^{-21}
Italian0.61028.84×10−178.84{\times}10^{-17}0.56699.31×10−199.31{\times}10^{-19}0.51173.21×10−193.21{\times}10^{-19}
Hindi0.41047.67×10−177.67{\times}10^{-17}––0.39295.73×10−215.73{\times}10^{-21}
Hindi (R)0.52937.88×10−127.88{\times}10^{-12}––0.46233.51×10−183.51{\times}10^{-18}
Appendix
Table 4: Full one-to-one cross-lingual unlearning results across Llama 3 , Mixtral , and Aya 23 . Rows indicate the evaluation language within each unlearning-language block. A ✓\checkmark indicates satisfaction of the TOFU forgetting criterion ( p>0.1p>0.1 ).
More in this topic

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 349 papers

Explore similar work

CardsList

  1. LLM Unlearning Evaluation with TRIAGE

    Sep 26, 2026Danial Ataee, Peter TriantafillouMachine Unlearning EvaluationMachine Unlearning

    Department of Computer Science University of Warwick

  2. Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning

    Sep 1, 2026Miso Kim, Georu Lee, Seungwon Jeong +1Privacy Leakage in Language ModelsMachine Unlearning

    Dongguk University-Seoul

  3. Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

    Jul 10, 2026Amit Peleg, Naman Deep Singh, Naama Pearl +2Factual Knowledge in Language ModelsMachine Unlearning

    Tübingen AI Center, University of Tübingen