ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences

Xinye Chen; Stefan Güttel; Mohammad Mozaffari

cs.LGSep 25, 2026

ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences

Authors: Xinye Chen, Stefan Güttel, Mohammad Mozaffari

Organizations: Sorbonne Université, CNRS, LIP6 Paris, France · University of Toronto Toronto, Canada

Abstract

We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We study memorization and the extension of symbolic rules in neural sequence models by using sequences whose complexity is regulated by Lempel--Ziv--Welch (LZW) compression. Under ROTE, each architecture is trained as the same finite-context conditional predictor and is evaluated using teacher-forced one-step prediction as well as closed-loop rollout on the withheld symbols. Following a shared prediction-and-rollout evaluation routine, the benchmark evaluates gated recurrent, minimal recurrent, attention-based, and hybrid recurrent-attention models with their native computational characteristics preserved. Beyond standard predictive metrics, the benchmark reports normalized string distances, training time, memory usage, and parameter count across an LZW-complexity sweep. The study establishes a connection between the complexity of algorithmic sequences and the memorization capacity of neural architectures, revealing the trade-offs involving memorization quality, rollout stability, and computational expense. Our software and reproducible experimental code can be obtained from https://github.com/nla-group/rote.

Figures & tables

$c=\kappa(u)$	$\|u\|$ median (range)	$r^{\star}(u)$ median (range)
10	11.0 (10–18)	3.0 (2–9)
30	42.0 (33–82)	5.0 (4–12)
50	83.5 (65–171)	6.0 (4–14)
70	127.5 (97–263)	6.5 (4–16)
90	174.0 (127–365)	7.0 (4–18)

Table 1: Seed diagnostics grouped by target LZW complexity. Statistics summarize the eight generated seeds at each

c

value, corresponding to four alphabet sizes and two stochastic seeds. Length and

r^{\star}

are reported as median with range;

\alpha_{100}^{\max}

is the maximum ambiguity rate at the benchmark context length

w=100

Figure 1: Teacher-forced loss and closed-loop rollout error as functions of LZW complexity. Both panels use the same model identities and summarize the 640 training runs. Lower values are better for both metrics.

Figure 2: Closed-loop degradation over the rollout horizon. Errors introduced early in the rollout become part of the subsequent context, so this plot measures stability of the learned symbolic dynamics rather than one-step classification alone. Markers are shown every ten rollout steps to improve readability; the curves are computed over all horizons.

Model	Exact	DL@ $c=90$	Params (M)	Time (s)	Time/epoch (s)	Memory (MB)
GRU	$0.925\pm 0.265$	$0.015\pm 0.034$	0.152	47.59	1.11	187.30
LinearAttention	$0.912\pm 0.284$	$0.013\pm 0.040$	1.581	8.88	1.27	2407.24
LSTM	$0.887\pm 0.318$	$0.041\pm 0.092$	0.202	74.56	1.14	213.13
minGRU	$0.787\pm 0.412$	$0.079\pm 0.141$	0.067	6.30	0.11	196.04
minLSTM	$0.762\pm 0.428$	$0.082\pm 0.146$	0.101	114.09	2.78	130.48
RWKV	$0.613\pm 0.490$	$0.113\pm 0.161$	1.585	10.99	1.68	656.22

Table 2: Model-level summary over all alphabet sizes and LZW complexities. Exact denotes the fraction of runs for which the 100-symbol rollout exactly matches the withheld suffix. Exact and DL@

c=90

are reported as mean

\pm

standard deviation over successful evaluations (runs that completed training); cost statistics are medians.

Panel of Figure 3 — Figure 3: Efficiency views of the same benchmark results. The Pareto plot compares rollout quality against measured compute, while the parameter plots show that increasing $P_{\theta}$ alone does not explain the observed ordering.

Model	Params (M)	Exact	DL	Test loss	Time (s)	Memory (MB)
GRU	$0.236\pm 0.001$	1.000	$0.000\pm 0.000$	$0.047\pm 0.003$	$58.10\pm 9.93$	$230.59\pm 3.12$
LSTM	$0.203\pm 0.001$	1.000	$0.000\pm 0.000$	$0.056\pm 0.014$	$101.04\pm 11.98$	$214.64\pm 3.04$
LinearAttention	$0.224\pm 0.000$	1.000	$0.000\pm 0.000$	$0.031\pm 0.010$	$6.93\pm 1.35$	$445.91\pm 3.03$
Performer	$0.188\pm 0.000$	1.000	$0.000\pm 0.000$	$0.036\pm 0.011$	$105.68\pm 29.92$	$953.26\pm 2.53$
Transformer	$0.235\pm 0.000$	0.875	$0.005\pm 0.014$	$0.040\pm 0.010$	$12.08\pm 3.23$	$335.90\pm 2.44$
RWKV	$0.226\pm 0.000$	0.625	$0.134\pm 0.197$	$0.039\pm 0.006$	$26.05\pm 15.14$	$253.61\pm 3.43$

Table 3: High-complexity matched-size robustness check at

c=90

. Each row summarizes eight evaluations: two alphabet sizes

n\in\{4,8\}

, two generated seeds, and two training runs. Widths are chosen to approximate a target size

P_{0}=0.2\,\mathrm{M}

parameters. Exact is the fraction of 100-step rollouts with zero DL distance. DL, test loss, train time, and memory are reported as mean

\pm

standard deviation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Fixed-budget and matched-size results on the common high-complexity subset

c=90

n\in\{4,8\}

. Points show means and vertical bars show one standard deviation over two seeds and two runs for each alphabet size. The left panel measures closed-loop symbolic stability through normalized DL distance; the right panel measures local teacher-forced prediction through test loss.

Neural Network Memorization

Neural network memorization studies how models store and retrieve training examples, how this differs from generalization, and how memory can be measured or controlled.

See all 64 papers

Memory Dial: A Training Framework for Controllable Memorization in Language Models
Apr 6, 2026Xiangbo Zhang, Ali EmamiMemorization in Language Models
Georgia Institute of Technology · Emory University
Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
Sep 28, 2026Florian Eichin, Philipp Mondorf, Andrei Mircea +3Memorization in Language Models Data Repetition in Language Model Training
MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany
Random Label Prediction Heads for Studying Memorization in Deep Neural Networks
Jul 13, 2026Marlon Becker, Jonas Konrad, Luis Garcia Rodriguez +1Neural Network Generalization Neural Network Memorization
University of Münster, Germany

Apr 6, 2026J/K move · Enter open · S save

Memory Dial: A Training Framework for Controllable Memorization in Language Models

Xiangbo Zhang, Ali Emami

Georgia Institute of Technology · Emory University

Memorization in Language Models

Open paper AskPDF arXiv

Memorization in language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce Memory Dial, a training framework that makes memorization pressure an explicit, controllable variable. Memory Dial interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter

α

, producing a family of models identical in architecture and training setup (within each sweep), differing only in memorization pressure. Experiments across six architectures and five benchmarks demonstrate that: (1)

α

reliably controls memorization pressure, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Additional analyses show that the effect is robust across a range of sharpening temperatures, differs qualitatively from single-temperature cross-entropy, transfers to multilingual settings, and is detectable even on naturally occurring single-occurrence sequences. Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.

Abstract

Neural Network Memorization

Explore similar work

Memory Dial: A Training Framework for Controllable Memorization in Language Models

Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models

Random Label Prediction Heads for Studying Memorization in Deep Neural Networks