Subword tokenization introduces a computational layer in language models, where many distinct token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-token loss across evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input.
Figures & tables
Figure 1: Diagram of our architecture. The main causal decoder operates on canonical BPE tokens, while a lightweight single-block causal encoder consumes sampled subword variants (e.g., dinosaur → dino + saur | d + inosaur ) and injects them into the main trunk via a block-causal cross-attention operation.
Figure 2: Evaluations on various chat datasets across pretraining duration for two model sizes: μ=1 - top and μ=2 - bottom . We show training runs with the number of data repetitions R∈{16,24,32,64} for clarity. Training with subword variants consistently delays overfitting in these scenarios.
Figure 3: Performance across multilingual fine-tuning for μ=1 across 32 epochs. The model pretrained with the GPT-2 tokenizer ( top ) does not improve with auxiliary subword variants, whereas the variant trained with aya-23 tokenizer ( bottom ) and subword variants consistently outperforms the standard Transformer decoder when repeating data.
Figure 4: Pretraining curves for various numbers of data repetitions for an 88M parameter model ( μ=1 , left) and a 244M parameter model ( μ=2 , right) . Training with subword variants delays overfitting for R>16 .
Figure 5: Loss during pretraining for runs using attention dropout and token-level Gaussian noise as augmentations for R=32 and μ=1 .
Figure 6: Loss across two benchmarks for runs using attention dropout ( top) and token-level Gaussian noise ( bottom ) as augmentations for R=32 and μ=1 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Dataset
Lang.
Tokenizer
Canonical Seq. Len.
Aux Seq. Len.
Aux/Can Ratio
% Unsplit Tokens
Tok/Word
Tok/Byte
FineWeb-Edu
aya-23
145.89
303.64
2.08
17.13
1.27
0.21
GPT-2
148.28
320.81
2.16
15.05
1.29
0.21
ChatDoctor
aya-23
86.58
178.73
2.06
18.00
1.34
0.22
GPT-2
89.15
187.52
2.10
18.19
1.38
0.23
C++
aya-23
88.23
158.69
1.80
34.19
1.98
0.33
GPT-2
93.07
167.09
1.80
35.68
2.09
0.35
Appendix
Table 1: Descriptive statistics for canonical (Can.) and auxiliary (Aux.) tokenizations across datasets, languages, and tokenizers. These statistics quantify both the additional sequence-level overhead introduced by the auxiliary branch and the amount of tokenization variation available to the method in each setting.
As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction (xt+i for i>1). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.
In the standard LM training pipeline, subword tokenisation is applied as a preprocessing step. Subword segmental language modelling is an alternative paradigm in which tokenisation is learned during training, allowing the model to discover subword units that optimise its training objective. In this paper, we present our submission to the 2026 BabyLM Challenge, for which we develop two new subword segmental LMs: SubSegGPT and SubSegDeBERTa. SubSegGPT is a decoder-only model that learns tokenisation during autoregressive pretraining. SubSegDeBERTa is an encoder-based model that jointly learns to generate and tokenise masked words. We train both for the Strict and Strict-small tracks. Our top submission to Strict is SubSegDeBERTa, which achieves notable gains in zero-shot evaluation. Our top submission to Strict-small is SubSegGPT, which outperforms tokenisation-based baselines. Our results show that learnable subword tokenisation can improve sample-efficiency for BabyLM pretraining. We analyse the subword learning dynamics of our models and find that tokenisation gradually converges on subword units that balance morphological alignment and fine-grained segmentation.
Francois Meyer
Department of Computer Science University of Cape Town
Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.