Machine unlearning aims to remove targeted information from large language models, but existing evaluations often measure forgetting only on a designated benchmark distribution. We argue that such behavioral forgetting is an insufficient standard for privacy-oriented unlearning, and that evaluation should instead test whether forgetting persists across the access paths through which the same information may be recovered. We demonstrate this with a controlled case study of cross-lingual querying: across multiple multilingual model families, forgetting varies substantially when semantically equivalent information is queried in different languages and scripts. A single access path is therefore enough to expose a real gap between behavioral forgetting and information inaccessibility, motivating broader evaluation of access-path generalization and adversarial recoverability alongside in-distribution forgetting.
Figures & tables
Figure 1: Cross-lingual access-path evaluation for Llama 3 using UNLEARN ( Lizzo and Heck, 2025 ) Each panel fixes the unlearning language, while points evaluate the same targeted facts through different linguistic access paths. The dashed line denotes the TOFU forgetting criterion ( p=0.1 ); model utility is normalized relative to the base model. Apparent unlearning success varies with the language used for evaluation. See Appendix A.5 for additional results across model families.
Figure 2: Representational basis of access-path-dependent forgetting. Removing the shared multilingual subspace induces forgetting across linguistic access paths, whereas removing a language-specific residual primarily affects the corresponding language.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Llama 3
Mixtral
Aya 23
Method
Utility ↑
Forget ↑
Utility ↑
Forget ↑
Utility ↑
Forget ↑
Base Model
0.6227
2.37×10−15
0.5912
3.0×10−16
0.5503
6.4×10−24
DPO
0.5357
1.06×10−16
0.5100
2.5×10−14
0.4182
4.3×10−21
Gradient Ascent
0.0000
1.43×10−22
0.0000
1.35×10−19
0.0000
6.28×10−18
KL Minimization
0.0000
4.07×10−32
0.0000
7.23×10−19
0.0000
2.38×10−26
Gradient Difference
0.5872
2.83×10−25
0.5483
5.83×10−25
0.4978
6.19×10−29
Appendix
Table 1: Comparison of unlearning performance on the English-English TOFU benchmark [ Maini et al., 2024 ] for three models: Llama 3, Mixtral, and Aya 23. Among unlearning methods, the best value in ecah column is bolded and the second-best is underlined. ✓marks forget quality above α=0.1 .
Language
Human Pass Rate
COMET-Kiwi
Spanish
98.75%
0.8818
Hindi
96.5%
0.8769
Hindi (Romanized)
94.25%
–
Italian
99.25%
0.8858
Chinese
95.5%
0.8610
Chinese (Pinyin)
93.25%
–
Appendix
Table 2: Translation validation results for cTOFU. COMET-Kiwi scores measure semantic consistency between English source text and translated outputs using a reference-free quality estimation model. Human validation pass rates report the percentage of manually inspected examples preserving factual consistency and author identity across translations. Human evaluation used 400 randomly sampled examples per language.
Model
Active Parameters
Total Parameters
HuggingFace Checkpoint
Mistral 7B
7B
7B
mistralai/Mistral-7B-v0.1
Mixtral 8 × 7B
12.9B
46.7B
mistralai/Mixtral-8x7B-v0.1
Magistral Small
24B
24B
mistralai/Magistral-Small-2506
LLaMA 2 7B
7B
7B
meta-llama/Llama-2-7b-hf
LLaMA 3 8B
8B
8B
meta-llama/Meta-Llama-3-8B
LLaMA 4 Scout
17B
109B
meta-llama/Llama-4-Scout-17B-16E
Appendix
Table 3: Pretrained language models used in our evaluation. Active parameters denote the number of parameters used during a forward pass. Total parameters denote the full parameter count of the model. All checkpoints are obtained from HuggingFace.
Llama 3
Mixtral
Aya 23
Unlearn
Eval
Util.
FQ
Util.
FQ
Util.
FQ
Base
English
0.6227
2.37×10−15
0.5912
2.74×10−16
0.5503
6.47×10−24
Spanish
0.5919
4.31×10−13
0.5393
4.68×10−13
0.5193
2.63×10−21
Italian
0.6102
8.84×10−17
0.5669
9.31×10−19
0.5117
3.21×10−19
Hindi
0.4104
7.67×10−17
–
–
0.3929
5.73×10−21
Hindi (R)
0.5293
7.88×10−12
–
–
0.4623
3.51×10−18
Appendix
Table 4: Full one-to-one cross-lingual unlearning results across Llama 3 , Mixtral , and Aya 23 . Rows indicate the evaluation language within each unlearning-language block. A ✓ indicates satisfaction of the TOFU forgetting criterion ( p>0.1 ).
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving other capabilities. However, existing evaluations rely primarily on behavioral benchmarks, which assess whether a model appears to forget but provide limited insight into how unlearning changes the model or affects related knowledge. We introduce TRIAGE (Tripartite Representation-internal Introspection for Adjacency Gap Evaluation), a benchmark-agnostic evaluation framework for characterizing these changes. TRIAGE uses diagonal approximations of the Fisher information and Hessian to measure changes in parameter sensitivity and local curvature, and utilizes a Forget / Adjacent-Retain / Generic-Retain partition to quantify an adjacency gap in semantically related knowledge. Based on the magnitude and distribution of these changes, TRIAGE further classifies each algorithm's update as no-op, partially localized, collateral dominant, or globally destructive. Across 12 unlearning methods, four language models, and the WMDP, TOFU, and MUSE benchmarks, we find that methods with similar behavioral forgetting can produce substantially different internal changes and patterns of collateral damage. These signatures also vary across models and benchmarks, indicating that the effects of unlearning are not determined solely by the unlearning algorithm. TRIAGE can be applied alongside existing unlearning benchmarks to complement behavioral evaluation with a model-internal view of how unlearning reshapes the model's parameter space and affects retained knowledge.
Danial Ataee, Peter Triantafillou
Department of Computer Science University of Warwick
Machine unlearning for large language models (LLMs) often assumes that a pre-defined forget set matches what the model has memorized, but this frequently breaks in realistic privacy settings where the original training data is inaccessible. We term this gap forget-set misalignment and identify two cases. In Under Unlearning, the forget set omits memorized information and leakage persists. In Out-of-Knowledge Unlearning, the algorithm is driven to "forget" knowledge the model never learned, perturbing parameters and degrading utility. Using gradient-level analysis, we show these behaviors arise from misaligned unlearning targets rather than specific optimization choices. We then propose CONfession-to-Forget-Set (CONFS), a data-blind framework that constructs model-aligned forget sets by eliciting and formalizing the model's memorized knowledge. Across synthetic, multimodal, and real-world benchmarks, CONFS approaches Gold-standard performance on several metrics and achieves a competitive forgetting-utility balance, while preserving utility better than other data-blind forget-set constructions.
Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call under-forgetting, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call over-forgetting. Both failures reflect an asymmetric generalization problem. Forget evaluation must cover diverse query formulations of the same target facts, testing whether forgetting holds beyond exact training prompts. Retain evaluation must probe a far larger and implicitly defined set, namely every fact disjoint from the forget target. The retain set thus defines the effective forget set, yet current datasets provide no fine-grained annotation of this forget-retain boundary. We address this with SUITE, an evaluation protocol and training corpus that captures forget-retain structure for real-world factual domains. Methods trained on SUITE improve substantially, showing that training data is as important as algorithmic design. Building on the obtained insights, we introduce JensUn++, an unlearning algorithm that achieves the best forget-retain utility trade-off across three LLMs, in both sequential and joint unlearning settings. Code and datasets are available at https://amitpeleg.github.io/forget-narrowly-retain-broadly