cs.CLNov 16, 2026

Augmenting Pretrained Language Models with N-gram Embeddings through Incremental Training

Authors: Shaohua Hu, Xusheng Gu, Haofei Yang, Minghao Shao, Nanhui Zhao, Caoyun Fan, Fangming Li

Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China · ICT AI Competence Center, Huawei Technologies Co., Ltd, Shanghai, China

Abstract

As a new scaling technique for large language models, the n-gram embedding improves the base ability of a model while adding almost no inference cost. Mainstream research realizes it through joint training of the n-gram embedding with the backbone, which severely restricts the settings in which an n-gram embedding can be used. We therefore propose training an n-gram embedding onto an existing model through incremental training, to extend where the technique can deliver value. Specifically, we explore several settings of the n-gram embedding under incremental training---the learning rate of the embedding, the vocabulary size, and the injection position---and train an n-gram embedding onto the existing model under the best configuration. Experiments show that the best configuration under incremental training does not fully coincide with that under pretraining; moreover, the incrementally trained knowledge is written predominantly into the n-gram embedding, while the general ability of the backbone is retained throughout this training process.

Figures & tables

Explore similar work

CardsList
  1. The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

    Oct 14, 2026Ian Magnusson, Noah A. Smith, Pang Wei KohLanguage ModelingLanguage Model Scaling

  2. Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

    Nov 17, 2026Stephen Chung, Wenyu Du