Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling
Organizations: Dalle Molle Institute for Artificial Intelligence (IDSIA) · National University of Science and Technology POLITEHNICA Bucharest
Abstract
Subword tokenization introduces a computational layer in language models, where many distinct token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-token loss across evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input.
Figures & tables
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Lang. | Tokenizer | Canonical Seq. Len. | Aux Seq. Len. | Aux/Can Ratio | % Unsplit Tokens | Tok/Word | Tok/Byte |
|---|---|---|---|---|---|---|---|---|
| FineWeb-Edu | aya-23 | 145.89 | 303.64 | 2.08 | 17.13 | 1.27 | 0.21 | |
| GPT-2 | 148.28 | 320.81 | 2.16 | 15.05 | 1.29 | 0.21 | ||
| ChatDoctor | aya-23 | 86.58 | 178.73 | 2.06 | 18.00 | 1.34 | 0.22 | |
| GPT-2 | 89.15 | 187.52 | 2.10 | 18.19 | 1.38 | 0.23 | ||
| C++ | aya-23 | 88.23 | 158.69 | 1.80 | 34.19 | 1.98 | 0.33 | |
| GPT-2 | 93.07 | 167.09 | 1.80 | 35.68 | 2.09 | 0.35 |
Autoregressive Language Modeling
Autoregressive language modeling predicts sequences one step at a time, using prior tokens to generate text, speech, code, and other data.
See all 139 papers