We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We study memorization and the extension of symbolic rules in neural sequence models by using sequences whose complexity is regulated by Lempel--Ziv--Welch (LZW) compression. Under ROTE, each architecture is trained as the same finite-context conditional predictor and is evaluated using teacher-forced one-step prediction as well as closed-loop rollout on the withheld symbols. Following a shared prediction-and-rollout evaluation routine, the benchmark evaluates gated recurrent, minimal recurrent, attention-based, and hybrid recurrent-attention models with their native computational characteristics preserved. Beyond standard predictive metrics, the benchmark reports normalized string distances, training time, memory usage, and parameter count across an LZW-complexity sweep. The study establishes a connection between the complexity of algorithmic sequences and the memorization capacity of neural architectures, revealing the trade-offs involving memorization quality, rollout stability, and computational expense. Our software and reproducible experimental code can be obtained from https://github.com/nla-group/rote.
Figures & tables
c=κ(u)
∣u∣ median (range)
r⋆(u) median (range)
α100max
10
11.0 (10–18)
3.0 (2–9)
0.000
30
42.0 (33–82)
5.0 (4–12)
0.000
50
83.5 (65–171)
6.0 (4–14)
0.000
70
127.5 (97–263)
6.5 (4–16)
0.000
90
174.0 (127–365)
7.0 (4–18)
0.000
Table 1: Seed diagnostics grouped by target LZW complexity. Statistics summarize the eight generated seeds at each c value, corresponding to four alphabet sizes and two stochastic seeds. Length and r⋆ are reported as median with range; α100max is the maximum ambiguity rate at the benchmark context length w=100 .
Figure 1: Teacher-forced loss and closed-loop rollout error as functions of LZW complexity. Both panels use the same model identities and summarize the 640 training runs. Lower values are better for both metrics.
Figure 2: Closed-loop degradation over the rollout horizon. Errors introduced early in the rollout become part of the subsequent context, so this plot measures stability of the learned symbolic dynamics rather than one-step classification alone. Markers are shown every ten rollout steps to improve readability; the curves are computed over all horizons.
Model
Exact
DL@ c=90
Params (M)
Time (s)
Time/epoch (s)
Memory (MB)
GRU
0.925±0.265
0.015±0.034
0.152
47.59
1.11
187.30
LinearAttention
0.912±0.284
0.013±0.040
1.581
8.88
1.27
2407.24
LSTM
0.887±0.318
0.041±0.092
0.202
74.56
1.14
213.13
minGRU
0.787±0.412
0.079±0.141
0.067
6.30
0.11
196.04
minLSTM
0.762±0.428
0.082±0.146
0.101
114.09
2.78
130.48
RWKV
0.613±0.490
0.113±0.161
1.585
10.99
1.68
656.22
Table 2: Model-level summary over all alphabet sizes and LZW complexities. Exact denotes the fraction of runs for which the 100-symbol rollout exactly matches the withheld suffix. Exact and DL@ c=90 are reported as mean ± standard deviation over successful evaluations (runs that completed training); cost statistics are medians.
Figure 3: Efficiency views of the same benchmark results. The Pareto plot compares rollout quality against measured compute, while the parameter plots show that increasing Pθ alone does not explain the observed ordering.
Model
Params (M)
Exact
DL
Test loss
Time (s)
Memory (MB)
GRU
0.236±0.001
1.000
0.000±0.000
0.047±0.003
58.10±9.93
230.59±3.12
LSTM
0.203±0.001
1.000
0.000±0.000
0.056±0.014
101.04±11.98
214.64±3.04
LinearAttention
0.224±0.000
1.000
0.000±0.000
0.031±0.010
6.93±1.35
445.91±3.03
Performer
0.188±0.000
1.000
0.000±0.000
0.036±0.011
105.68±29.92
953.26±2.53
Transformer
0.235±0.000
0.875
0.005±0.014
0.040±0.010
12.08±3.23
335.90±2.44
RWKV
0.226±0.000
0.625
0.134±0.197
0.039±0.006
26.05±15.14
253.61±3.43
Table 3: High-complexity matched-size robustness check at c=90 . Each row summarizes eight evaluations: two alphabet sizes n∈{4,8} , two generated seeds, and two training runs. Widths are chosen to approximate a target size P0=0.2M parameters. Exact is the fraction of 100-step rollouts with zero DL distance. DL, test loss, train time, and memory are reported as mean ± standard deviation.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Fixed-budget and matched-size results on the common high-complexity subset c=90 , n∈{4,8} . Points show means and vertical bars show one standard deviation over two seeds and two runs for each alphabet size. The left panel measures closed-loop symbolic stability through normalized DL distance; the right panel measures local teacher-forced prediction through test loss.
Neural network memorization studies how models store and retrieve training examples, how this differs from generalization, and how memory can be measured or controlled.
Memorization in language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce Memory Dial, a training framework that makes memorization pressure an explicit, controllable variable. Memory Dial interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter α, producing a family of models identical in architecture and training setup (within each sweep), differing only in memorization pressure. Experiments across six architectures and five benchmarks demonstrate that: (1) α reliably controls memorization pressure, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Additional analyses show that the effect is robust across a range of sharpening temperatures, differs qualitatively from single-temperature cross-entropy, transfers to multilingual settings, and is detectable even on naturally occurring single-occurrence sequences. Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.
Xiangbo Zhang, Ali Emami
Georgia Institute of Technology · Emory University
Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.
Florian Eichin, Philipp Mondorf, Andrei Mircea +3
MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany
We introduce a straightforward yet effective method to empirically study memorization in deep neural networks for classification tasks. Our approach augments each training sample with auxiliary random labels, which are then predicted by a random label prediction head (RLP-head). RLP-heads can be attached at arbitrary depths of a network, predicting random labels from the corresponding intermediate representation and thereby enabling analysis of how memorization capacity evolves across layers. By interpreting the RLP-head performance as an empirical estimate of Rademacher complexity, we obtain a direct measure of both sample-level memorization and model capacity. We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets. Building on this approach, we further propose a novel regularization technique based on the output of the RLP-head, which demonstrably reduces memorization. Interestingly, our experiments reveal that reducing memorization can either improve or impair generalization, depending on the dataset and training setup. These findings challenge the traditional assumption that overfitting is equivalent to memorization and suggest new hypotheses to reconcile these seemingly contradictory results. The source code is available at https://github.com/MarlonBecker/RandomLabelHeads
Marlon Becker, Jonas Konrad, Luis Garcia Rodriguez +1