Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China · ICT AI Competence Center, Huawei Technologies Co., Ltd, Shanghai, China
As a new scaling technique for large language models, the n-gram embedding improves the base ability of a model while adding almost no inference cost. Mainstream research realizes it through joint training of the n-gram embedding with the backbone, which severely restricts the settings in which an n-gram embedding can be used. We therefore propose training an n-gram embedding onto an existing model through incremental training, to extend where the technique can deliver value. Specifically, we explore several settings of the n-gram embedding under incremental training---the learning rate of the embedding, the vocabulary size, and the injection position---and train an n-gram embedding onto the existing model under the best configuration. Experiments show that the best configuration under incremental training does not fully coincide with that under pretraining; moreover, the incrementally trained knowledge is written predominantly into the n-gram embedding, while the general ability of the backbone is retained throughout this training process.
Figures & tables
Figure 1: (a) Existing works introduce the n-gram embedding at the pretraining stage and jointly optimize it with the backbone from scratch; (b) this paper considers incrementally attaching an n-gram embedding to a pretrained model that originally lacks the module, to learn domain knowledge. The two scenarios differ in the scale of the corpus and the initial state of the parameters.
Figure 2: The side-branch structure of Engram and the three key configurations during augmentation and incremental training (yellow boxes). The n-gram embedding branch leaves the backbone’s computation graph untouched: et is retrieved with the n tokens ending at position t as the key and, weighted by the gate αl,t , additively merged into the residual stream at the insertion layers. The insertion layers are chosen from [0,…,L] and multiple layers may be used; the vocabulary size determines the number of slots in the n-gram embedding table; both are fixed once the model is augmented. The learning rate is set separately for the backbone and the n-gram embedding, forming the dual-learning-rate scheme.
Figure 3: Training loss of incrementally training the n-gram embedding under different learning rates, shown over the first 30,000 training steps. (a) Under the single learning rate the curve coincides throughout with fine-tuning without the n-gram embedding; after the learning rate of the n-gram embedding is increased, the training loss drops markedly. (b) Difference between the loss under the single learning rate and that of fine-tuning without the n-gram embedding, which remains below 2.6×10−3 throughout.
Learning rate
1×10−6
5×10−6
5×10−5
Injection position
layer 0
layer 14
layer 0
layer 14
layer 0
layer 14
∥hl,t∥∥yl,t∥
0.050
0.009
0.075
0.012
0.177
0.123
Table 1: Contribution of the n-gram embedding at each injection layer, measured at step 30,000 under different learning rates of the n-gram embedding. yl,t and hl,t are the n-gram embedding output and the hidden state at layer l , position t , respectively.
Vocabulary size
Telecom (en)
Telecom (zh)
SFT w/o n-gram embedding
36.59
18.81
M=60k
37.86
19.09
M=152k
38.70
21.66
M=1520k
41.99
23.18
Table 2: Sweep over the vocabulary size of the n-gram embedding. Injection positions fixed; the SFT w/o n-gram embedding row serves as reference.
Figure 4: Performance on the two telecom test sets versus the vocabulary size M of the n-gram embedding (log scale on the horizontal axis). The score rises monotonically with M on both test sets, and every configuration surpasses the SFT baseline without the n-gram embedding (dashed line).
Injection position
Telecom (en)
Telecom (zh)
SFT w/o n-gram embedding
36.59
18.81
Layer 0 (embedding layer)
38.95
21.62
Layer 1
37.24
21.23
Layer 14 (middle layer)
41.48
18.92
Layers {0,14}
41.99
23.18
Layers {1,14}
41.65
20.29
Table 3: Sweep over the injection position of the n-gram embedding. Vocabulary size fixed at 1520k ; the SFT w/o n-gram embedding row serves as reference.
Telecom (en)
Telecom (zh)
MMLU
BIG-bench
Qwen3-0.6B
31.45
12.73
48.33
41.76
SFT
36.59
18.81
48.87
45.10
SFT w/ n-gram embedding
41.99
23.18
42.23
39.80
Table 4: Incremental training with the n-gram embedding versus the baselines.
Figure 5: Heatmap of the gate values, averaged over the tokens of each word, after incremental training with the n-gram embedding. The injection strength of the n-gram embedding is pronounced on domain-relevant words.
Telecom (en)
Telecom (zh)
MMLU
BIG-bench
Qwen3-0.6B
31.45
12.73
48.33
41.76
SFT w/ n-gram embedding
41.99
23.18
42.23
39.80
n-gram embedding unloaded
27.91
13.37
49.70
44.37
Table 5: Unloading experiment: locating where domain knowledge is stored. The two n-gram embedding rows share the same set of weights and differ only in whether the n-gram embedding branch is enabled at inference.