Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.
Figures & tables
Figure 1: Scaling curves with respect to table size. ClimbMix, nine-layer model (20.7M backbone), ten table sizes. Validation bits per byte (lower is better) against table size. Curves show Chorus, Engram and FactorEngram; isolated points show OE, TN-gram and two X-gram configurations. One seed per point.
Figure 2: Existing n-gram embedding tables and Chorus. (a) Engram, TN-gram and FactorEngram keep a separate n-gram table for every place that reads one, including the input. (b) Chorus looks up one set of tables once and reuses the result at the input and in every other layer. Each layer rescales the refined embedding using its own hidden state (dashed arrows) and maps it through its own projection.
Method
ClimbMix 9-layer model 20.7M backbone 76M tables
FineWeb 9-layer model 17.1M backbone 26M tables
FineWeb 18-layer model 133M backbone 60M tables
Plain model
0.9969 ± 0.0007
1.2422 ± 0.0008
1.0823 ± 0.0006
OE
0.9885 ± 0.0003
—
—
X-gram
0.9688 ± 0.0007
—
—
Engram
0.9616 ± 0.0006
1.2051 ± 0.0002
1.0665 ± 0.0001
TN-gram
0.9602 ± 0.0024
1.1983 ± 0.0002
1.0648 ± 0.0006
FactorEngram
0.9565 ± 0.0005
1.2177 ± 0.0001
1.0768 ± 0.0005
Table 1: Validation loss across all three settings. Validation bits per byte (lower is better), mean ± standard error over three seeds. The plain model is the original backbone trained with no added n-gram tables.
Figure 3: Training curves at two model sizes. FineWeb. (a) Nine-layer model (17.1M backbone), 26M tables. (b) 18-layer model (133M backbone), 60M tables. Plain model means no added n-gram tables. Both panels show validation bits per byte on a fixed 2,048-sequence subset during training, mean ± standard error over three seeds. Final full-validation results are in Table 1 .
Figure 4: Downstream accuracy. FineWeb, 18-layer model (133M backbone), 60M tables. Plain model means no added n-gram tables. Accuracy of Chorus minus that of each baseline on 22 tasks and on their average, in percentage points; mean ± standard error of paired differences over three seeds.
Table size
AdamW state BF16 (GiB)
Ember state FP32 (MiB)
Saved (GiB)
Saved (%)
76M
0.27
2.11
0.27
99.25
300M
1.12
8.85
1.11
99.23
1.2B
4.47
35.67
4.44
99.22
2.4B
8.94
71.44
8.87
99.22
Table 2: GPU optimizer memory saved by Ember. ClimbMix, nine-layer model (20.7M backbone). The state memory is computed using the conventional storage precision for each optimizer.
Memory
Table update
BPB ↓ (mean ± s.e.)
Engram
AdamW
0.9616 ± 0.0006
Engram
Ember
0.9575 ± 0.0004
FactorEngram
AdamW
0.9565 ± 0.0005
FactorEngram
Ember
0.9860 ± 0.0017
Shared bigram table
AdamW
0.9645 ± 0.0004
Shared bigram table
Ember
0.9622 ± 0.0006
Table 3: The design against the table optimizer, and a simpler shared table. ClimbMix, nine-layer model (20.7M backbone), 76M tables. Validation bits per byte, mean ± standard error over three seeds.
Figure 5: Ablation studies of Chorus components. ClimbMix, nine-layer model (20.7M backbone), 76M tables. Bars show increases in validation loss; bars and reference gaps use one seed. Dashed vertical lines mark Chorus’s gaps to FactorEngram and Engram. Removing the input or layer reads removes the addition of the refined embedding to the token embedding or attention values, respectively. Table size is controlled: in “w/o token table U ,” its parameters are reallocated to the n-gram tables; in “Split input and layer tables,” the same total table size is divided equally between the two pools.
Figure 6: Effect of the longest n-gram order. ClimbMix, nine-layer model (20.7M backbone). Validation bits per byte for each longest n-gram order N , using all orders from 2 through N . Solid lines show fixed total table sizes of 76M and 1.2B. The dashed line shows the growing-size condition: each table keeps the same row count as orders are added, for a total of 177.8M to 1.2B parameters.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
ClimbMix
FineWeb, small
FineWeb, large
Layers / width
9 / 512
9 / 512
18 / 1,024
Backbone parameters
20.71M
17.06M
133.25M
Vocabulary size
8,192
1,024
1,024
Training tokens
3.15B
3.15B
5.24B
Table size
20M–9.6B
26M
60M
Appendix
Table 4: Model and training settings. Backbone and table sizes are in millions or billions of parameters. Method comparisons use three seeds; the scaling curve uses one seed at each size.
Table size (B parameters)
Method
GPUs
Validation BPB ↓
Update GPU-hours
Peak memory (GiB/GPU)
0.020
Chorus
1
0.964671
1.58
8.4
0.020
FactorEngram
1
0.974093
1.87
10.0
0.020
Engram
1
0.973753
1.56
9.0
0.039
Chorus
1
0.955231
1.59
8.5
0.039
FactorEngram
1
0.963001
1.87
10.2
0.039
Engram
1
0.969229
1.56
9.1
Appendix
Table 5: Full scaling results. ClimbMix, nine-layer model (20.7M backbone), one seed per setting. Validation BPB, training-update GPU-hours summed across devices, and peak training memory per GPU.
Figure 7: Validation-loss margins during training. ClimbMix, nine-layer model (20.7M backbone), one seed per size. Each line is one checkpoint; positive values mean that Chorus has lower validation loss than the baseline. Margins are in thousandths of BPB.
Task
Examples
Plain model mean ± s.e.
FactorEngram mean ± s.e.
Engram mean ± s.e.
TN-gram mean ± s.e.
Chorus (ours) mean ± s.e.
HellaSwag (0-shot)
10,042
34.72 ± 0.12
34.84 ± 0.06
35.46 ± 0.07
35.51 ± 0.19
36.28 ± 0.17
Jeopardy
2,117
0.05 ± 0.00
0.03 ± 0.02
0.06 ± 0.02
0.06 ± 0.02
0.06 ± 0.02
Wikidata QA
20,321
12.88 ± 0.68
12.51 ± 0.94
17.38 ± 1.20
18.57 ± 1.28
25.84 ± 2.74
ARC-Easy
2,376
35.30 ± 0.33
35.34 ± 0.14
36.17 ± 0.04
35.79 ± 0.21
35.93 ± 0.54
ARC-Challenge
1,172
21.96 ± 0.31
22.13 ± 0.42
21.99 ± 0.54
22.55 ± 0.44
22.33 ± 0.30
COPA
100
59.67 ± 1.76
60.33 ± 1.86
61.00 ± 1.00
61.67 ± 0.88
59.67 ± 0.88
Appendix
Table 6: Eighteen-layer downstream results: tasks 1–11. FineWeb, 133M backbone, 60M tables. Accuracy (%), mean ± standard error over three seeds; bold is best for each task. Plain model means no added n-gram tables.
Task
Examples
Plain model mean ± s.e.
FactorEngram mean ± s.e.
Engram mean ± s.e.
TN-gram mean ± s.e.
Chorus (ours) mean ± s.e.
Winograd
273
56.41 ± 0.63
56.17 ± 0.74
57.02 ± 0.98
57.39 ± 1.27
56.90 ± 0.44
Winogrande
1,267
52.46 ± 0.32
52.12 ± 0.48
51.75 ± 1.08
51.35 ± 0.09
50.83 ± 1.18
Dyck languages
1,000
15.33 ± 1.03
12.93 ± 4.92
13.93 ± 0.79
16.37 ± 1.27
12.43 ± 1.13
LSAT-AR
230
21.88 ± 0.29
24.06 ± 0.72
24.49 ± 0.95
23.33 ± 1.47
22.90 ± 0.63
CS algorithms
1,320
40.98 ± 0.88
39.04 ± 0.96
38.11 ± 0.20
40.56 ± 1.60
39.12 ± 0.86
Operators
210
14.44 ± 1.14
14.92 ± 1.24
16.35 ± 2.64
15.56 ± 1.30
17.46 ± 0.57
Appendix
Table 7: Eighteen-layer downstream results: tasks 12–22. Same models as Table 6 . Accuracy (%) and the overall mean score, mean ± standard error over three seeds; bold is best for each task and for the overall mean.
Setting
Method
WikiText-2 BPB ↓ mean ± s.e.
LAMBADA (%) ↑ mean ± s.e.
HellaSwag (%) ↑ mean ± s.e.
FineWeb
Plain model
1.2813 ± 0.0019
21.50 ± 0.59
29.40 ± 0.03
FactorEngram
1.2502 ± 0.0008
22.64 ± 0.26
29.65 ± 0.07
Engram
1.2339 ± 0.0003
22.95 ± 0.27
30.02 ± 0.18
TN-gram
1.2288 ± 0.0016
24.69 ± 0.23
30.22 ± 0.11
Chorus (ours)
1.2238 ± 0.0015
24.50 ± 0.47
29.94 ± 0.15
ClimbMix
Plain model
1.2966 ± 0.0031
19.82 ± 0.29
29.58 ± 0.01
Appendix
Table 8: Nine-layer external evaluations. FineWeb (17.1M backbone, 26M tables) and ClimbMix (20.7M backbone, 76M tables). WikiText-2 BPB and LAMBADA and HellaSwag accuracy, mean ± standard error over three seeds.
Model
Method
Read sites
Added FLOPs
Backbone FLOPs
Relative (%)
9 layers, width 512
Engram
2
2.10M
21.8M
+9.7%
9 layers, width 512
TN-gram
2
11.56M
21.8M
+53.1%
9 layers, width 512
FactorEngram
2
12.59M
21.8M
+57.9%
9 layers, width 512
Chorus
5
2.76M
21.8M
+12.7%
18 layers, width 1,024
Plain
0
0.00M
152.0M
+0.0%
18 layers, width 1,024
Engram
2
4.20M
152.0M
+2.8%
Appendix
Table 9: Computation added per token. FineWeb models. Counts use multiply-accumulates in matrix products and convolutions; the backbone includes attention over the 1,024-token context and the output projection.
Method
Added FLOPs (% of backbone)
Training time (h) mean ± s.e.
Peak memory (GB) mean ± s.e.
Plain model
0.0
17.0 ± 0.1
5.3 ± 0.0
FactorEngram
33.1
20.5 ± 0.0
6.5 ± 0.0
Engram
2.8
20.1 ± 0.4
6.0 ± 0.0
TN-gram
24.3
19.3 ± 0.0
6.5 ± 0.0
Chorus
11.4
19.3 ± 0.3
6.2 ± 0.0
Appendix
Table 10: Training cost. FineWeb, 18-layer model (133M backbone), 60M tables, on RTX 5090 GPUs. Time spent on training updates and peak training memory, mean ± standard error over three seeds.
Configuration
Table size (M)
Other parameters (M)
BPB ↓
Δ BPB vs reference
Chorus (reference)
76.44
2.75
0.946917
–
Token table only (no n-gram tables)
6.95
2.75
0.973654
+0.026737
w/o token table U and rescaling by gate stℓ
76.44
1.44
0.963851
+0.016934
w/o token table U
76.44
2.75
0.963423
+0.016506
w/o reading at layers
74.48
0.79
0.960298
+0.013381
w/o reading at input
75.92
2.23
0.958322
+0.011405
Appendix
Table 11: Full component ablations. ClimbMix, nine-layer model, one seed per configuration. Parameter counts are in millions. Rows with a total size of 76.44M reallocate parameters to table rows; other rows also remove the omitted component’s parameters. The AdamW row uses learning rate 0.001; the main figure uses the best of three learning rates.
Table size (B parameters)
Group
Chorus
Engram
FactorEngram
0.020
Q1
0.840620
0.849840
0.850147
Q2
0.941097
0.950044
0.950421
Q3
0.996503
1.005630
1.005846
Q4
1.086553
1.095585
1.096049
0.039
Q1
0.831665
0.845513
0.840246
Q2
0.931786
0.945722
0.939592
Appendix
Table 12: Validation loss by phrase frequency. ClimbMix, nine-layer model (20.7M backbone), one seed per setting. Validation BPB for four equally sized groups of passages, ordered from most frequent three-token contexts (Q1) to least frequent (Q4) in training. Bold is best within each row.