Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.
Figures & tables
Figure 1: Scaling curves with respect to table size. ClimbMix, nine-layer model (20.7M backbone), ten table sizes. Validation bits per byte (lower is better) against table size. Curves show Chorus, Engram and FactorEngram; isolated points show OE, TN-gram and two X-gram configurations. One seed per point.
Figure 2: Existing n-gram embedding tables and Chorus. (a) Engram, TN-gram and FactorEngram keep a separate n-gram table for every place that reads one, including the input. (b) Chorus looks up one set of tables once and reuses the result at the input and in every other layer. Each layer rescales the refined embedding using its own hidden state (dashed arrows) and maps it through its own projection.
Method
ClimbMix 9-layer model 20.7M backbone 76M tables
FineWeb 9-layer model 17.1M backbone 26M tables
FineWeb 18-layer model 133M backbone 60M tables
Plain model
0.9969 ± 0.0007
1.2422 ± 0.0008
1.0823 ± 0.0006
OE
0.9885 ± 0.0003
—
—
X-gram
0.9688 ± 0.0007
—
—
Engram
0.9616 ± 0.0006
1.2051 ± 0.0002
1.0665 ± 0.0001
TN-gram
0.9602 ± 0.0024
1.1983 ± 0.0002
1.0648 ± 0.0006
FactorEngram
0.9565 ± 0.0005
1.2177 ± 0.0001
1.0768 ± 0.0005
Table 1: Validation loss across all three settings. Validation bits per byte (lower is better), mean ± standard error over three seeds. The plain model is the original backbone trained with no added n-gram tables.
Figure 3: Training curves at two model sizes. FineWeb. (a) Nine-layer model (17.1M backbone), 26M tables. (b) 18-layer model (133M backbone), 60M tables. Plain model means no added n-gram tables. Both panels show validation bits per byte on a fixed 2,048-sequence subset during training, mean ± standard error over three seeds. Final full-validation results are in Table 1 .
Figure 4: Downstream accuracy. FineWeb, 18-layer model (133M backbone), 60M tables. Plain model means no added n-gram tables. Accuracy of Chorus minus that of each baseline on 22 tasks and on their average, in percentage points; mean ± standard error of paired differences over three seeds.
Table size
AdamW state BF16 (GiB)
Ember state FP32 (MiB)
Saved (GiB)
Saved (%)
76M
0.27
2.11
0.27
99.25
300M
1.12
8.85
1.11
99.23
1.2B
4.47
35.67
4.44
99.22
2.4B
8.94
71.44
8.87
99.22
Table 2: GPU optimizer memory saved by Ember. ClimbMix, nine-layer model (20.7M backbone). The state memory is computed using the conventional storage precision for each optimizer.
Memory
Table update
BPB ↓ (mean ± s.e.)
Engram
AdamW
0.9616 ± 0.0006
Engram
Ember
0.9575 ± 0.0004
FactorEngram
AdamW
0.9565 ± 0.0005
FactorEngram
Ember
0.9860 ± 0.0017
Shared bigram table
AdamW
0.9645 ± 0.0004
Shared bigram table
Ember
0.9622 ± 0.0006
Table 3: The design against the table optimizer, and a simpler shared table. ClimbMix, nine-layer model (20.7M backbone), 76M tables. Validation bits per byte, mean ± standard error over three seeds.
Figure 5: Ablation studies of Chorus components. ClimbMix, nine-layer model (20.7M backbone), 76M tables. Bars show increases in validation loss; bars and reference gaps use one seed. Dashed vertical lines mark Chorus’s gaps to FactorEngram and Engram. Removing the input or layer reads removes the addition of the refined embedding to the token embedding or attention values, respectively. Table size is controlled: in “w/o token table U ,” its parameters are reallocated to the n-gram tables; in “Split input and layer tables,” the same total table size is divided equally between the two pools.
Figure 6: Effect of the longest n-gram order. ClimbMix, nine-layer model (20.7M backbone). Validation bits per byte for each longest n-gram order N , using all orders from 2 through N . Solid lines show fixed total table sizes of 76M and 1.2B. The dashed line shows the growing-size condition: each table keeps the same row count as orders are added, for a total of 177.8M to 1.2B parameters.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
ClimbMix
FineWeb, small
FineWeb, large
Layers / width
9 / 512
9 / 512
18 / 1,024
Backbone parameters
20.71M
17.06M
133.25M
Vocabulary size
8,192
1,024
1,024
Training tokens
3.15B
3.15B
5.24B
Table size
20M–9.6B
26M
60M
Appendix
Table 4: Model and training settings. Backbone and table sizes are in millions or billions of parameters. Method comparisons use three seeds; the scaling curve uses one seed at each size.
Table size (B parameters)
Method
GPUs
Validation BPB ↓
Update GPU-hours
Peak memory (GiB/GPU)
0.020
Chorus
1
0.964671
1.58
8.4
0.020
FactorEngram
1
0.974093
1.87
10.0
0.020
Engram
1
0.973753
1.56
9.0
0.039
Chorus
1
0.955231
1.59
8.5
0.039
FactorEngram
1
0.963001
1.87
10.2
0.039
Engram
1
0.969229
1.56
9.1
Appendix
Table 5: Full scaling results. ClimbMix, nine-layer model (20.7M backbone), one seed per setting. Validation BPB, training-update GPU-hours summed across devices, and peak training memory per GPU.
Figure 7: Validation-loss margins during training. ClimbMix, nine-layer model (20.7M backbone), one seed per size. Each line is one checkpoint; positive values mean that Chorus has lower validation loss than the baseline. Margins are in thousandths of BPB.
Task
Examples
Plain model mean ± s.e.
FactorEngram mean ± s.e.
Engram mean ± s.e.
TN-gram mean ± s.e.
Chorus (ours) mean ± s.e.
HellaSwag (0-shot)
10,042
34.72 ± 0.12
34.84 ± 0.06
35.46 ± 0.07
35.51 ± 0.19
36.28 ± 0.17
Jeopardy
2,117
0.05 ± 0.00
0.03 ± 0.02
0.06 ± 0.02
0.06 ± 0.02
0.06 ± 0.02
Wikidata QA
20,321
12.88 ± 0.68
12.51 ± 0.94
17.38 ± 1.20
18.57 ± 1.28
25.84 ± 2.74
ARC-Easy
2,376
35.30 ± 0.33
35.34 ± 0.14
36.17 ± 0.04
35.79 ± 0.21
35.93 ± 0.54
ARC-Challenge
1,172
21.96 ± 0.31
22.13 ± 0.42
21.99 ± 0.54
22.55 ± 0.44
22.33 ± 0.30
COPA
100
59.67 ± 1.76
60.33 ± 1.86
61.00 ± 1.00
61.67 ± 0.88
59.67 ± 0.88
Appendix
Table 6: Eighteen-layer downstream results: tasks 1–11. FineWeb, 133M backbone, 60M tables. Accuracy (%), mean ± standard error over three seeds; bold is best for each task. Plain model means no added n-gram tables.
Task
Examples
Plain model mean ± s.e.
FactorEngram mean ± s.e.
Engram mean ± s.e.
TN-gram mean ± s.e.
Chorus (ours) mean ± s.e.
Winograd
273
56.41 ± 0.63
56.17 ± 0.74
57.02 ± 0.98
57.39 ± 1.27
56.90 ± 0.44
Winogrande
1,267
52.46 ± 0.32
52.12 ± 0.48
51.75 ± 1.08
51.35 ± 0.09
50.83 ± 1.18
Dyck languages
1,000
15.33 ± 1.03
12.93 ± 4.92
13.93 ± 0.79
16.37 ± 1.27
12.43 ± 1.13
LSAT-AR
230
21.88 ± 0.29
24.06 ± 0.72
24.49 ± 0.95
23.33 ± 1.47
22.90 ± 0.63
CS algorithms
1,320
40.98 ± 0.88
39.04 ± 0.96
38.11 ± 0.20
40.56 ± 1.60
39.12 ± 0.86
Operators
210
14.44 ± 1.14
14.92 ± 1.24
16.35 ± 2.64
15.56 ± 1.30
17.46 ± 0.57
Appendix
Table 7: Eighteen-layer downstream results: tasks 12–22. Same models as Table 6 . Accuracy (%) and the overall mean score, mean ± standard error over three seeds; bold is best for each task and for the overall mean.
Setting
Method
WikiText-2 BPB ↓ mean ± s.e.
LAMBADA (%) ↑ mean ± s.e.
HellaSwag (%) ↑ mean ± s.e.
FineWeb
Plain model
1.2813 ± 0.0019
21.50 ± 0.59
29.40 ± 0.03
FactorEngram
1.2502 ± 0.0008
22.64 ± 0.26
29.65 ± 0.07
Engram
1.2339 ± 0.0003
22.95 ± 0.27
30.02 ± 0.18
TN-gram
1.2288 ± 0.0016
24.69 ± 0.23
30.22 ± 0.11
Chorus (ours)
1.2238 ± 0.0015
24.50 ± 0.47
29.94 ± 0.15
ClimbMix
Plain model
1.2966 ± 0.0031
19.82 ± 0.29
29.58 ± 0.01
Appendix
Table 8: Nine-layer external evaluations. FineWeb (17.1M backbone, 26M tables) and ClimbMix (20.7M backbone, 76M tables). WikiText-2 BPB and LAMBADA and HellaSwag accuracy, mean ± standard error over three seeds.
Model
Method
Read sites
Added FLOPs
Backbone FLOPs
Relative (%)
9 layers, width 512
Engram
2
2.10M
21.8M
+9.7%
9 layers, width 512
TN-gram
2
11.56M
21.8M
+53.1%
9 layers, width 512
FactorEngram
2
12.59M
21.8M
+57.9%
9 layers, width 512
Chorus
5
2.76M
21.8M
+12.7%
18 layers, width 1,024
Plain
0
0.00M
152.0M
+0.0%
18 layers, width 1,024
Engram
2
4.20M
152.0M
+2.8%
Appendix
Table 9: Computation added per token. FineWeb models. Counts use multiply-accumulates in matrix products and convolutions; the backbone includes attention over the 1,024-token context and the output projection.
Method
Added FLOPs (% of backbone)
Training time (h) mean ± s.e.
Peak memory (GB) mean ± s.e.
Plain model
0.0
17.0 ± 0.1
5.3 ± 0.0
FactorEngram
33.1
20.5 ± 0.0
6.5 ± 0.0
Engram
2.8
20.1 ± 0.4
6.0 ± 0.0
TN-gram
24.3
19.3 ± 0.0
6.5 ± 0.0
Chorus
11.4
19.3 ± 0.3
6.2 ± 0.0
Appendix
Table 10: Training cost. FineWeb, 18-layer model (133M backbone), 60M tables, on RTX 5090 GPUs. Time spent on training updates and peak training memory, mean ± standard error over three seeds.
Configuration
Table size (M)
Other parameters (M)
BPB ↓
Δ BPB vs reference
Chorus (reference)
76.44
2.75
0.946917
–
Token table only (no n-gram tables)
6.95
2.75
0.973654
+0.026737
w/o token table U and rescaling by gate stℓ
76.44
1.44
0.963851
+0.016934
w/o token table U
76.44
2.75
0.963423
+0.016506
w/o reading at layers
74.48
0.79
0.960298
+0.013381
w/o reading at input
75.92
2.23
0.958322
+0.011405
Appendix
Table 11: Full component ablations. ClimbMix, nine-layer model, one seed per configuration. Parameter counts are in millions. Rows with a total size of 76.44M reallocate parameters to table rows; other rows also remove the omitted component’s parameters. The AdamW row uses learning rate 0.001; the main figure uses the best of three learning rates.
Table size (B parameters)
Group
Chorus
Engram
FactorEngram
0.020
Q1
0.840620
0.849840
0.850147
Q2
0.941097
0.950044
0.950421
Q3
0.996503
1.005630
1.005846
Q4
1.086553
1.095585
1.096049
0.039
Q1
0.831665
0.845513
0.840246
Q2
0.931786
0.945722
0.939592
Appendix
Table 12: Validation loss by phrase frequency. ClimbMix, nine-layer model (20.7M backbone), one seed per setting. Validation BPB for four equally sized groups of passages, ordered from most frequent three-token contexts (Q1) to least frequent (Q4) in training. Bold is best within each row.
Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.
Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token (n-gram) memories. However, they rely on separate hash tables for each n-gram order, which introduces hash collisions and prevents nested n-grams from sharing the underlying latent structures. To address these issues, we propose Tensorized Engram (TN-gram), a compact memory module that represents tensorized n-gram embeddings through shared factors in the Canonical Polyadic (CP) form. TN-gram learns shared token-position factors together with order-absorption vectors to encode the embeddings of different n-gram order. Comprehensive experiments demonstrate that TN-gram matches or even outperforms Engram-style n-gram modules while requiring much fewer parameters.
Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides +3
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · AIP, RIKEN, Tokyo, Japan.
Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit. Each embedding is stored in its own hashed slot and modulated by a single scalar gate. As a result, polysemous patterns cannot selectively read out the components of their memory that are relevant to the context. Moreover, parameters are shared only through hash collisions, which are largely unrelated to semantics. We propose FactorEngram, a factorized n-gram memory with basis-level contextual gating. FactorEngram retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components. The same dictionary is also used for gating. The backbone hidden state is scored against each basis vector to gate the corresponding coefficient before reconstruction, which lets the context modulate each memory component individually. FactorEngram also covers both individual tokens and multi-token n-grams, and we systematically study where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones, FactorEngram improves language modeling and downstream task performance. Ablation studies confirm the contribution of each component and identify insertion before the attention sublayer in the middle layers as an effective configuration.
Bowen Yang, Jingbo Zhou, Qinghong Miao +1
Large Model Frontier Research Department, Baidu Inc., China · Nanyang Technological University