cs.CLJan 6, 2026

Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling

Authors: Adrian Cosma, Stefan Ruseti, Emilian Radoi, Mihai Dascalu

Organizations: Dalle Molle Institute for Artificial Intelligence (IDSIA) · National University of Science and Technology POLITEHNICA Bucharest

Abstract

Subword tokenization introduces a computational layer in language models, where many distinct token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-token loss across evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

    Jun 15, 2026Michael K. Chen, Xikun Zhang, Fan Bai +2Language Model PretrainingData Augmentation