Memorization in Language Models

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 75

All topics

Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes
Oct 8, 2026Ali Satvaty, Narjes Sharafi, Jirui Qi +2Memorization in Language Models Privacy Leakage in Language Models
University of Groningen · Leiden University
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Oct 6, 2026Vedant Palit, Florent Draye, Nicolas Zucchet +2Memorization in Language Models LLM Fine-Tuning
MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI +3
How Learning Governs Unlearning across the Memorization-Generalization Spectrum
Oct 6, 2026Hwiyeong Lee, Hyelim Lim, Ingyu Bang +2Memorization in Language Models Machine Unlearning
Seoul National University · Hanyang University · Chung-Ang University
Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context
Sep 28, 2026Muyu He, Yuchen Liu, Ran Tao +1Memorization in Language Models LLM Interpretability
Independent · University of Pennsylvania · Drexel University
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
Sep 28, 2026Zineddine Tighidet, Andrea Mogini, Jiali Mei +2LLM Evaluation Memorization in Language Models
BNP Paribas Paris, France · Sorbonne Université Paris, France · Criteo AI Lab
Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
Sep 28, 2026Florian Eichin, Philipp Mondorf, Andrei Mircea +3Memorization in Language Models Data Repetition in Language Model Training
MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany
Memory vs. Context? Influential Factors of Factual Recall in Language Models
Sep 21, 2026Guilhem Fouilhé, Nicholas Asher, Philippe MullerPrompt Sensitivity Memorization in Language Models
IRIT ANITI · IRIT CNRS · IRIT Université de Toulouse
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
Sep 15, 2026William L. Tong, Aryo Lotfi, Emmanuel Abbe +6Memorization in Language Models Long-Context Modeling
Harvard University · Apple
The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination
Sep 14, 2026Xi Wang, Shijia Xu, Rongfeng GuoMemorization in Language Models Hallucination in Language Models
Hefei Institutes of Physical Science, Chinese Academy of Sciences, China · University of Science and Technology of China, China · Chongqing University, China +1
Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss
Sep 10, 2026Zhijian Li, Stefan Larson, Kevin LeachMemorization in Language Models LLM Fine-Tuning
University of Southern California, Los Angeles · Vanderbilt University, Nashville
Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
Sep 9, 2026Arman Nik KhahMemorization in Language Models LLM Auditing
The University of Texas at Dallas
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouMemorization in Language Models Language Model Robustness
Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA
From Retrieval to Weights: Parametric Individualization of Small Language Models with Individual Text Corpora
Sep 9, 2026Christoph Wigbels, Ali Abusaleh, Markus T. Jansen +2Memory-Augmented Language Models Memorization in Language Models
Bergische Universität Wuppertal · Goethe-Universität Frankfurt
Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry
Sep 1, 2026Shengfang Zhai, Leo Marchyok, Yuling Shi +4Memorization in Language Models Membership Inference Attacks
National University of Singapore · Oregon State University · Shanghai Jiao Tong University +1
Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization
Aug 31, 2026Dishu Yang, Jingjing Liu, Jize LiLLM Alignment Memorization in Language Models
Khoury College of Computer Sciences Northeastern University San Jose, USA · Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, USA · Department of Economics Boston University Boston, USA
Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training
Aug 19, 2026Zachary Speck, Asa ShepardLanguage Model Pretraining Memorization in Language Models
Arizona State University · Williams College
Memorization Diagnostics for Code LLMs Should be Scale-Aware
Aug 13, 2026Prateek Kumar Rajput, Abdoul Aziz Bonkoungou, Alberick Euraste Djiré +6Memorization in Language Models Code Language Models
University of Luxembourg, Esch-sur-Alzette, Luxembourg.
Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning
Aug 1, 2026Zachary Nicholas Houghton, Vsevolod KapatsinskiMemorization in Language Models Language Modeling
University of Oregon · Vail Systems, Inc.
MemSFT: Mitigating Alignment Tax with an External Parametric Memory
Jul 28, 2026Jiarui Wang, Xiang Shi, Jiaqi Cao +8LLM Alignment Memory-Augmented Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · School of Computer Science, Shanghai Jiao Tong University +1
Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization
Jul 28, 2026Akshay SasiLLM Quantization Memorization in Language Models
Independent Researcher
Reasoning or Memorization: Can LLMs Understand and Generate Chinese Xiehouyu Riddles?
Jul 26, 2026Hai Hu, Siyuan Song, Chongtian Shao +3LLM Evaluation Memorization in Language Models
1The Hong Kong Polytechnic University · 2The University of Texas at Austin · 3Shanghai Jiao Tong University
How Much Can a LoRA Adapter Memorize? Measuring Adapter Capacity in Bits
Jul 23, 2026Kaizhen Tan, Heqing Du, Yang FengMemorization in Language Models Privacy Leakage in Language Models
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Language Model Pretraining Rotary Positional Embeddings
Tsinghua University
Extractable Memorization From First Principles
Jul 14, 2026A. Feder Cooper, Marika Swanberg, Jamie Hayes +5LLM Evaluation Memorization in Language Models
Yale, AVERI, Stanford · Google Research · Google DeepMind +2
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Jul 10, 2026Roberto Garcia, Jerry Liu, Ronny Junkins +3Transformer Memorization in Language Models
Institute for Computational & Mathematical Engineering, Stanford University · Department of Computer Science, Stanford University · Department of Computer Science and Engineering, University at Buffalo
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
Jul 9, 2026Lu Dai, Ziyang Rao, Yili Wang +3Memorization in Language Models LLM Fine-Tuning
HKUST · HKUST(GZ)
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jul 6, 2026Jingwei Zuo, Cong Zeng, Ilyas Chahed +6Memorization in Language Models Neural Network Memorization
Technology Innovation Institute Abu Dhabi, UAE
Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbed
Jun 30, 2026Zhichao Fan, Zexin Zhuang, Yanhang LiMemorization in Language Models LLM Auditing
University of Illinois Urbana-Champaign · Southern Methodist University · Northeastern University
Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTs
Jun 26, 2026Cristhian Kapelinski, Diego KreutzPrivacy-Preserving Language Models Memorization in Language Models
Federal University of Pampa (UNIPAMPA), Alegrete, Brazil
Internal Data Repetition Destroys Language Models
Jun 23, 2026Jessica Chudnovsky, Joshua Kazdan, Noam Levi +6Memorization in Language Models Language Model Scaling Laws
1Stanford Computer Science. · 2Stanford Statistics. · 3Tel Aviv University. +1

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

Ali Satvaty, Narjes Sharafi, Jirui Qi, Suzan Verberne+1

University of Groningen · Leiden University

Memorization in Language Models Privacy Leakage in Language Models Retrieval-Augmented Generation

Open paper AskPDF arXiv

Large language models (LLMs) can expose memorized training sequences under prefix-based extraction: given a prefix from a training example, the model may assign high probability to the original continuation. In deployed systems, however, prefixes are rarely evaluated in isolation. They often appear together with instructions, retrieved documents, or other task-specific context, as in retrieval-augmented generation (RAG). This motivates examining whether contextual conditioning mitigates memorization or merely changes the set of memorized samples that become extractable. We investigate this issue through paired item-level measurements of probabilistic suffix extraction. For each prefix-suffix pair, we score the target suffix under an empty prompt and under retrieved contexts of varying relevance, across three open-weight instruction-tuned models. We find that context does not simply erase memorization. Instead, extractable memorization consists of a context-robust core and a context-sensitive boundary. Many samples that are extractable without context remain extractable under the retrieved context, especially as the prefix length increases. At the same time, context mainly affects marginal samples near the extraction threshold: it suppresses some exposures, but also enables new ones that are missed by prefix-only evaluation. These findings qualify the view that RAG reduces memorization risk. Context can lower aggregate extraction by suppressing boundary cases, yet robustly extractable samples persist, and context-enabled extractability remains security-relevant.

Momentum

Related topics

Latest papers 75