Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China · ICT AI Competence Center, Huawei Technologies Co., Ltd, Shanghai, China
As a new scaling technique for large language models, the n-gram embedding improves the base ability of a model while adding almost no inference cost. Mainstream research realizes it through joint training of the n-gram embedding with the backbone, which severely restricts the settings in which an n-gram embedding can be used. We therefore propose training an n-gram embedding onto an existing model through incremental training, to extend where the technique can deliver value. Specifically, we explore several settings of the n-gram embedding under incremental training---the learning rate of the embedding, the vocabulary size, and the injection position---and train an n-gram embedding onto the existing model under the best configuration. Experiments show that the best configuration under incremental training does not fully coincide with that under pretraining; moreover, the incrementally trained knowledge is written predominantly into the n-gram embedding, while the general ability of the backbone is retained throughout this training process.
Figures & tables
Figure 1: (a) Existing works introduce the n-gram embedding at the pretraining stage and jointly optimize it with the backbone from scratch; (b) this paper considers incrementally attaching an n-gram embedding to a pretrained model that originally lacks the module, to learn domain knowledge. The two scenarios differ in the scale of the corpus and the initial state of the parameters.
Figure 2: The side-branch structure of Engram and the three key configurations during augmentation and incremental training (yellow boxes). The n-gram embedding branch leaves the backbone’s computation graph untouched: et is retrieved with the n tokens ending at position t as the key and, weighted by the gate αl,t , additively merged into the residual stream at the insertion layers. The insertion layers are chosen from [0,…,L] and multiple layers may be used; the vocabulary size determines the number of slots in the n-gram embedding table; both are fixed once the model is augmented. The learning rate is set separately for the backbone and the n-gram embedding, forming the dual-learning-rate scheme.
Figure 3: Training loss of incrementally training the n-gram embedding under different learning rates, shown over the first 30,000 training steps. (a) Under the single learning rate the curve coincides throughout with fine-tuning without the n-gram embedding; after the learning rate of the n-gram embedding is increased, the training loss drops markedly. (b) Difference between the loss under the single learning rate and that of fine-tuning without the n-gram embedding, which remains below 2.6×10−3 throughout.
Learning rate
1×10−6
5×10−6
5×10−5
Injection position
layer 0
layer 14
layer 0
layer 14
layer 0
layer 14
∥hl,t∥∥yl,t∥
0.050
0.009
0.075
0.012
0.177
0.123
Table 1: Contribution of the n-gram embedding at each injection layer, measured at step 30,000 under different learning rates of the n-gram embedding. yl,t and hl,t are the n-gram embedding output and the hidden state at layer l , position t , respectively.
Vocabulary size
Telecom (en)
Telecom (zh)
SFT w/o n-gram embedding
36.59
18.81
M=60k
37.86
19.09
M=152k
38.70
21.66
M=1520k
41.99
23.18
Table 2: Sweep over the vocabulary size of the n-gram embedding. Injection positions fixed; the SFT w/o n-gram embedding row serves as reference.
Figure 4: Performance on the two telecom test sets versus the vocabulary size M of the n-gram embedding (log scale on the horizontal axis). The score rises monotonically with M on both test sets, and every configuration surpasses the SFT baseline without the n-gram embedding (dashed line).
Injection position
Telecom (en)
Telecom (zh)
SFT w/o n-gram embedding
36.59
18.81
Layer 0 (embedding layer)
38.95
21.62
Layer 1
37.24
21.23
Layer 14 (middle layer)
41.48
18.92
Layers {0,14}
41.99
23.18
Layers {1,14}
41.65
20.29
Table 3: Sweep over the injection position of the n-gram embedding. Vocabulary size fixed at 1520k ; the SFT w/o n-gram embedding row serves as reference.
Telecom (en)
Telecom (zh)
MMLU
BIG-bench
Qwen3-0.6B
31.45
12.73
48.33
41.76
SFT
36.59
18.81
48.87
45.10
SFT w/ n-gram embedding
41.99
23.18
42.23
39.80
Table 4: Incremental training with the n-gram embedding versus the baselines.
Figure 5: Heatmap of the gate values, averaged over the tokens of each word, after incremental training with the n-gram embedding. The injection strength of the n-gram embedding is pronounced on domain-relevant words.
Telecom (en)
Telecom (zh)
MMLU
BIG-bench
Qwen3-0.6B
31.45
12.73
48.33
41.76
SFT w/ n-gram embedding
41.99
23.18
42.23
39.80
n-gram embedding unloaded
27.91
13.37
49.70
44.37
Table 5: Unloading experiment: locating where domain knowledge is stored. The two n-gram embedding rows share the same set of weights and differ only in whether the n-gram embedding branch is enabled at inference.
The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each pass only processes individual embedding rows, which are much smaller than the storage used for the whole embedding table. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers), which builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU memory is limited, using COUNT to let backbone parameters fill GPU memory gives better performance than splitting memory with additional learned n-gram embeddings. Beyond efficiency, COUNT allows switching in a different table of n-gram counts from a domain-specific corpus at inference time to adapt to a new task without any training.
Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.
Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.