Going Deeper With Memories

Afshin Khadangi

cs.LGNov 3, 2026

Going Deeper With Memories

Authors: Afshin Khadangi

Organizations: SnT, University of Luxembourg

Abstract

Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive. We introduce DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation. A learned invertible subband transform constructs a hierarchy of shorter sequences, while a lossless carrier and explicit detail sidebands preserve byte identity. Fast weight memory blocks operate along the encoder, with iterative shared state computation at the compressed bottleneck, while an attention based super resolution decoder restores byte resolution representations. DeepMem achieves stronger language modeling performance than a matched byte level Transformer on FineWeb-Edu and PG19, and obtains the best macro average performance across the evaluated FLORES languages. It also improves PIQA accuracy while remaining competitive on code modeling, multiple choice reasoning, and byte level corruption robustness. Bottleneck refinement requires approximately $7{,}600\times$ fewer counted FLOPs than equivalent full resolution reasoning, while maintaining competitive evaluation throughput.

Figures & tables

Model	FineWeb-Edu	PG19	Stack-smol	FLORES
Transformer	1.0457 $\pm$ 0.019	1.4635 $\pm$ 0.023	2.4474 $\pm$ 0.077	2.0082 $\pm$ 0.028
FastWeight	1.0187 $\pm$ 0.008	1.4956 $\pm$ 0.019	2.9558 $\pm$ 0.047	2.0599 $\pm$ 0.022
BLT	1.0336 $\pm$ 0.009	1.5920 $\pm$ 0.050	2.6138 $\pm$ 0.065	2.0233 $\pm$ 0.021
AU-Net	1.0319 $\pm$ 0.006	1.4885 $\pm$ 0.022	2.4894 $\pm$ 0.058	2.0077 $\pm$ 0.020
DeepMem (NR)	1.0145 $\pm$ 0.010	1.4462 $\pm$ 0.022	2.5021 $\pm$ 0.046	1.9758 $\pm$ 0.017
DeepMem	1.0134 $\pm$ 0.008	1.4402 $\pm$ 0.017	2.4854 $\pm$ 0.034	1.9694 $\pm$ 0.020

Table 1: Language-modeling performance in bits per byte (BPB). Lower is better. DeepMem (NR) represents DeepMem with no shared state refinement.

Model	English	German	Arabic	Hindi	Bengali	Thai	Vietnamese	Swahili	Macro
T	1.2013 $\pm$ 0.020	2.4014 $\pm$ 0.036	2.1194 $\pm$ 0.028	1.4265 $\pm$ 0.013	1.6157 $\pm$ 0.020	1.7914 $\pm$ 0.021	2.3884 $\pm$ 0.032	3.1216 $\pm$ 0.053	2.0082 $\pm$ 0.028
FW	1.2484 $\pm$ 0.013	2.3674 $\pm$ 0.019	2.1350 $\pm$ 0.026	1.4196 $\pm$ 0.011	1.6205 $\pm$ 0.023	1.7722 $\pm$ 0.016	2.4547 $\pm$ 0.022	3.4613 $\pm$ 0.044	2.0599 $\pm$ 0.022
BLT	1.2543 $\pm$ 0.013	2.3916 $\pm$ 0.027	2.1316 $\pm$ 0.027	1.4430 $\pm$ 0.013	1.6065 $\pm$ 0.022	1.7627 $\pm$ 0.016	2.3588 $\pm$ 0.016	3.2376 $\pm$ 0.032	2.0233 $\pm$ 0.021
AU	1.2208 $\pm$ 0.019	2.3831 $\pm$ 0.022	2.1172 $\pm$ 0.028	1.4202 $\pm$ 0.011	1.6175 $\pm$ 0.023	1.7526 $\pm$ 0.010	2.3635 $\pm$ 0.019	3.1865 $\pm$ 0.027	2.0077 $\pm$ 0.020
DM (NR)	1.2134 $\pm$ 0.013	2.3473 $\pm$ 0.014	2.0406 $\pm$ 0.019	1.4518 $\pm$ 0.015	1.5996 $\pm$ 0.012	1.6569 $\pm$ 0.016	2.2988 $\pm$ 0.018	3.1978 $\pm$ 0.028	1.9758 $\pm$ 0.017
DM	1.2007 $\pm$ 0.015	2.3408 $\pm$ 0.019	2.0383 $\pm$ 0.019	1.4403 $\pm$ 0.013	1.6132 $\pm$ 0.016	1.6535 $\pm$ 0.016	2.3059 $\pm$ 0.027	3.1624 $\pm$ 0.032	1.9694 $\pm$ 0.020

Table 2: Language-level BPB on FLORES. Lower is better. Macro is the unweighted mean across the eight languages. T, FW, AU, DM (NR), and DM represent Transformer, FastWeight, AU-Net, DeepMem with no shared state refinement, and DeepMem , respectively.

Model	FineWeb-Edu	PG19	Stack-smol	Average
Transformer	147,788	146,590	146,690	147,023
FastWeight	87,577	86,663	87,095	87,112
BLT	109,302	109,792	109,932	109,675
AU-Net	114,565	115,658	118,965	116,396
DeepMem (NR)	137,963	136,905	139,287	138,051
DeepMem	118,963	120,905	119,287	119,718

Table 3: Mean BPB evaluation throughput in processed bytes per second. Higher is better.

Model	HellaSwag	PIQA	ARC-E	ARC-C	Mean
Transformer	28.81 $\pm$ 0.4	60.55 $\pm$ 0.5	35.96 $\pm$ 0.6	22.74 $\pm$ 0.5	37.02 $\pm$ 0.5
FastWeight	29.95 $\pm$ 0.4	60.61 $\pm$ 0.5	37.72 $\pm$ 0.6	20.07 $\pm$ 0.3	37.09 $\pm$ 0.5
BLT	28.80 $\pm$ 0.4	60.55 $\pm$ 0.5	33.16 $\pm$ 0.7	21.07 $\pm$ 0.4	35.90 $\pm$ 0.5
AU-Net	28.64 $\pm$ 0.4	60.66 $\pm$ 0.5	34.67 $\pm$ 0.6	20.86 $\pm$ 0.4	36.21 $\pm$ 0.5
DeepMem (NR)	29.87 $\pm$ 0.4	61.81 $\pm$ 0.7	36.32 $\pm$ 0.6	21.74 $\pm$ 0.3	37.44 $\pm$ 0.5
DeepMem	30.30 $\pm$ 0.4	60.83 $\pm$ 0.7	36.49 $\pm$ 0.6	23.41 $\pm$ 0.3	37.76 $\pm$ 0.5

Table 4: Zero-shot multiple-choice accuracy (%). Higher is better. Mean is the unweighted average across the four tasks. DeepMem (NR) represents DeepMem with no shared state refinement.

Model	Clean	Drop	Repeat	Antspeak	Uppercase	Randomcase
T	28.81 $\pm$ 0.4	27.97 $\pm$ 0.5	27.50 $\pm$ 0.4	25.93 $\pm$ 0.4	28.79 $\pm$ 0.5	27.38 $\pm$ 0.4
FW	29.95 $\pm$ 0.4	29.18 $\pm$ 0.4	28.54 $\pm$ 0.4	27.06 $\pm$ 0.4	29.89 $\pm$ 0.4	28.58 $\pm$ 0.4
BLT	28.80 $\pm$ 0.4	28.98 $\pm$ 0.4	27.92 $\pm$ 0.4	26.02 $\pm$ 0.4	28.68 $\pm$ 0.4	27.42 $\pm$ 0.4
AU	28.64 $\pm$ 0.4	28.86 $\pm$ 0.4	27.54 $\pm$ 0.4	26.10 $\pm$ 0.4	28.42 $\pm$ 0.4	27.56 $\pm$ 0.4
DM	30.30 $\pm$ 0.4	29.16 $\pm$ 0.4	28.64 $\pm$ 0.4	26.89 $\pm$ 0.4	29.82 $\pm$ 0.3	29.16 $\pm$ 0.4

Table 5: HellaSwag accuracy (%) under byte-level perturbations. Higher is better. T, FW, AU and DM represent Transformer, FastWeight, AU-Net and DeepMem , respectively.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Mean BPB				Throughput (B/s)
Variant	FineWeb	PG19	Stack	FLORES	FineWeb	PG19	Stack	Average
DeepMem	1.0134	1.4402	2.4854	1.9694	118,963	120,905	119,287	119,718
No shared-state refinement	1.0145	1.4462	2.5021	1.9758	137,963	136,905	139,287	138,051
No carrier $\rightarrow$ sem.	1.0141	1.4482	2.4845	1.9830	121,953	122,663	122,754	122,457
All-scale fusion	1.0142	1.4413	2.4082	1.9851	117,374	118,468	119,369	118,404
All attention	1.0613	1.4827	2.4527	2.0498	140,812	134,798	139,944	138,518

Appendix

Table A.1 : Architecture results for language modeling and BPB-evaluation throughput. BPB (mean) is lower better; processed bytes per second (B/s) is higher better.

Variant	English	German	Arabic	Hindi	Bengali	Thai	Vietnamese	Swahili	Macro
DeepMem	1.2007	2.3408	2.0383	1.4403	1.6132	1.6535	2.3059	3.1624	1.9694
No shared-state refinement	1.2134	2.3473	2.0406	1.4518	1.5996	1.6569	2.2988	3.1978	1.9758
No carrier $\rightarrow$ sem.	1.2206	2.3625	2.0522	1.4513	1.6225	1.6542	2.3284	3.1726	1.9830
All-scale fusion	1.2187	2.3512	2.0530	1.4553	1.6135	1.6887	2.3256	3.1748	1.9851
All attention	1.2246	2.4511	2.1191	1.5030	1.6574	1.7709	2.4596	3.2131	2.0498

Appendix

Table A.2 : Mean FLORES BPB for the architecture configurations. Lower is better.

Variant	Hella	PIQA	ARC-E	ARC-C	Mean
DeepMem	30.30	60.83	36.49	23.41	37.76
No shared-state refinement	29.87	61.81	36.32	21.74	37.44
No carrier $\rightarrow$ sem.	29.76	60.99	35.61	23.75	37.53
All attention	28.70	59.68	33.68	21.74	35.95

Appendix

Table A.3 : Mean zero-shot multiple-choice accuracy (%) for the architecture configurations. Higher is better.

Variant	Clean	Drop	Repeat	Antspeak	Uppercase	Randomcase	Mean
DeepMem	30.30	29.16	28.64	26.89	29.82	29.16	29.00
No carrier $\rightarrow$ sem.	29.76	29.33	28.66	26.90	29.44	28.52	28.77
No shared-state refinement	29.87	29.13	28.56	27.23	29.50	28.78	28.85
All attention	28.70	27.56	26.95	25.92	28.07	27.32	27.42

Appendix

Table A.4 : Mean HellaSwag accuracy (%) under prompt-level byte perturbations. Higher is better.

Test	Cutoffs	$\boldsymbol{\delta_{\mathrm{rep}}}$	Max prefix diff.	Max suffix diff.	Result
Midpoint	1	$0$	$0$	$4.3369$	Pass
Exhaustive	2,048	$0$	$5.0664{\times}10^{-6}$	$5.4749$	Pass

Appendix

Table B.5 : Future-perturbation causality tests for the 2,048-byte all-scale fusion configuration under controlled FP32 execution. Prefix differences must be at most

10^{-5}

. Lower prefix difference is better.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 526 papers

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
Depth Memory
Oct 13, 2026David González-Martínez, Philipp Nazari, T. Konstantin Rusch +1Memory-Augmented Language Models Associative Memory
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center +2
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language Modeling Byte-Level Language Model
Rice University · Amazon Science

Oct 5, 2026J/K move · Enter open · S save

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

Jie Wang, Shiwei Luo, Qi Zhang, Yuanbin Wu

School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China

Language Model Scaling Byte-Level Language Model Language Modeling Tokenization in Language Models

Open paper AskPDF arXiv

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to

25\%

of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields

3.4\times

more accepted tokens than in subword Transformers.

Abstract

Language Modeling

Explore similar work

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

Depth Memory

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer