Memorization in Language Models

Overview

Researchers study when models reproduce training text or personal data, retain factual knowledge, and generalize beyond examples—questions that matter for privacy, benchmark validity, and reliable use. Common approaches include canary and text-extraction attacks, membership inference, matched training and non-training comparisons, and mechanistic analysis of activations and training dynamics; evaluations include privacy audits and unlearning benchmarks such as TOFU and MUSE. Mitigations range from data filtering and training-objective changes to fine-tuning, unlearning, and targeted edits to model activations or parameters, while agent-memory benchmarks test long-horizon recall. A central tension is that storing information does not guarantee it will be revealed, and reducing leakage must preserve useful recall and generalization.

Latest papers

CardsList

125 papers12 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. Constructing and Validating Memorisation Audits for EHR Foundation Models

    Nov 11, 2026Inigo Stibbe, Joshua Placidi, Padmanabhan Ramnarayan +1Memorization in Language ModelsPrivacy Leakage in Language Models

  2. An Information Bottleneck for Post-hoc Memorization Mitigation in LLMs

    Nov 5, 2026Dina EL Zein, Shashi Kumar, James HendersonMemorization in Language ModelsPrivacy-Preserving Language Models

  3. Forget What Comes Next: LLM Unlearning via Statistical Independence

    Nov 2, 2026Andrii Shkabrii, Claudia Plant, Martin Pawelczykhilbert-schmidt independence criterionLLM Unlearning

  4. From Representation to Readout: Mechanistic Analysis of Membership Signals in Language Models

    Oct 30, 2026Wenxi Dai, Jingyu Hu, Yanting Yang +1Privacy Leakage in Language ModelsMemorization in Language Models

  5. Characterizing Recoverable Expert-Utilization Signatures in Fine-Tuned Mixture-of-Experts LLMs

    Oct 29, 2026Fei Wang, Baochun Li, Bo LiMixture-of-Experts Language ModelsMemorization in Language Models

  6. Self-Keyed In-Parameter Memorization for External Memory-Free Dialogue Recall in Frozen Language Models

    Oct 27, 2026XinYu Piao, Myunghoon Ryu, Joongheon Kim +1Conversational MemoryMemorization in Language Models

  7. How Much Information Can We Fit inside a Transformer?

    Oct 25, 2026Amirhossein Basareh, Rishabh Sanjay, Daniel de Mello +1Memorization in Language ModelsTransformer

  8. ReproBAIT: Can coding agents build their own biological AI tools?

    Oct 14, 2026Shiv Muthupandiyan, Tolga H. Dur, Mac Walker +8AI Coding AgentsAI Agent Benchmarks

  9. When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

    Oct 6, 2026Vedant Palit, Florent Draye, Nicolas Zucchet +2Memorization in Language ModelsLLM Fine-Tuning

  10. How Learning Governs Unlearning across the Memorization-Generalization Spectrum

    Oct 6, 2026Hwiyeong Lee, Hyelim Lim, Ingyu Bang +2Memorization in Language ModelsMachine Unlearning

  11. Learning a Fact Is Not Learning How to Retrieve It

    Oct 2, 2026Chaemin Jang, Jihee Kim, Dongman LeeKnowledge Retrieval in Language ModelsFactual Knowledge in Language Models

  12. Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context

    Sep 28, 2026Muyu He, Yuchen Liu, Ran Tao +1Memorization in Language ModelsLLM Interpretability

  13. MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

    Sep 28, 2026Zineddine Tighidet, Andrea Mogini, Jiali Mei +2LLM EvaluationMemorization in Language Models

  14. Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models

    Sep 28, 2026Florian Eichin, Philipp Mondorf, Andrei Mircea +3Memorization in Language ModelsData Repetition in Language Model Training

  15. How Linear Attention Remembers

    Sep 27, 2026Kichang Lee, JaeYeon Park, Songkuk Kim +1Linear AttentionMemorization in Language Models

  16. TRAP: Understanding and Mitigating Privacy Memorization in Language Models

    Sep 26, 2026Muhammed Ustaomeroglu, Ziyue Xu, Hanshen Xiao +3Safety-Preserving Fine-Tuning of Language ModelsMemorization in Language Models

  17. Memory vs. Context? Influential Factors of Factual Recall in Language Models

    Sep 21, 2026Guilhem Fouilhé, Nicholas Asher, Philippe MullerPrompt SensitivityMemorization in Language Models

  18. Specification Oracles

    Sep 11, 2026Atticus Cull, Justin McCarthyMemory-Augmented Language ModelsFactual Knowledge in Language Models

  19. The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination

    Sep 10, 2026Xi Wang, Shijia Xu, Rongfeng GuoMemorization in Language ModelsHallucination in Language Models

  20. Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss

    Sep 10, 2026Zhijian Li, Stefan Larson, Kevin LeachMemorization in Language ModelsLLM Fine-Tuning

  21. Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

    Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouMemorization in Language ModelsLanguage Model Robustness