Memorization Dynamics of Fill-in-the-Middle Pretraining

Tobias von Arx; Tanguy Dieudonné

cs.CLMay 21, 2026

Memorization Dynamics of Fill-in-the-Middle Pretraining

Authors: Tobias von Arx, Tanguy Dieudonné

Abstract

Fill-in-the-middle (FIM) is a pretraining objective widely used to equip causal language models with infilling ability, yet its effect on verbatim memorization remains underexplored. We study the memorization dynamics of FIM in a controlled setting by pretraining matched Llama 3.2 models with FIM and standard left-to-right (LTR) objectives on a FineWeb-Gutenberg corpus containing repeated Gutenberg excerpts. With prefix-based probes, FIM more often recovers short or partially matching spans, while LTR more often assigns high confidence to long exact continuations. We observe that verbatim extraction under FIM-training grows approximately linearly with repetitions over the tested range. Evaluating native FIM-format probes reveals that suffix context is not sufficient: verbatim recall under FIM-training remains strongly anchored in prefix context. Our results also show that evaluating only one span length or probing format can miss important nuances in memorization behavior.

Explore similar work

Extractable Memorization From First Principles
Jul 14, 2026A. Feder Cooper, Marika Swanberg, Jamie Hayes +5Memorization Principles
Continual Memorization of Factoids in Language Models
Nov 11, 2024Howard Chen, Jiayi Geng, Adithya Bhaskar +2Memorization
Extracting Training Data from Diffusion Language Models via Infilling
May 22, 2026Yihan Wang, N. AsokanDiffusion Language Models Memorization

Jul 14, 2026 · cs.LGJ/K move · Enter open · S save

Extractable Memorization From First Principles

A. Feder Cooper, Marika Swanberg, Jamie Hayes, Lea Duesterwald+4

Yale, AVERI, Stanford · Google Research · Google DeepMind · Cornell · Stanford

Recent work on extractable memorization in LLMs suffers from two contrasting validity problems. Some studies overstate extraction, e.g., relying on sequences too short to distinguish memorization from predictability. Others imply that extraction is unreliable evidence of memorization, since models can also reproduce real-world text they weren't explicitly trained on. In different ways, both overlook what makes a valid extraction claim: the model must generate a training sequence with high enough probability to indicate memorization. To determine what's high enough, one has to perform a matched comparison: measuring the generation probabilities of both the training sequences of interest and comparable non-training sequences. Because non-training sequences cannot have been memorized, their probabilities provide a baseline for predictability; a training sequence exceeding this baseline provides evidence of memorization. We formalize matched comparisons in two ways: (1) a conformal test that calibrates a threshold to a chosen FPR when training and non-training sequences are sampled from populations, and (2) a census that calibrates against a matched non-training document when the object is a single document (e.g., a book). We show that matched comparisons enable rigorous, calibrated memorization claims, and reveal where prior setups have validity issues. For instance, on Wikipedia OLMo 2 32B reproduces non-training 10-token suffixes roughly 24% as often as training ones: that share of the training generation rate reflects false positives, not memorization. For Llama 3.1 70B on books, the thresholds we calibrate are as low as 1e-27, supporting memorization claims for sequences that no feasible sampling budget would extract. Based on these results, we refine "extractable memorization" to require a valid memorization claim and near-certain generation within a realistic budget.

Memorization Principles

Open paper Ask about it PDF

Abstract

Explore similar work

Extractable Memorization From First Principles

Continual Memorization of Factoids in Language Models

Extracting Training Data from Diffusion Language Models via Infilling