Data Repetition in Language Model Training

This field studies what happens when language models see the same documents or examples multiple times during pretraining, fine-tuning, or iterative training. Controlled multi-epoch experiments, repetition-aware scaling laws, and data-mixture studies measure effects on validation loss, downstream benchmarks, memorization, and output diversity. Reuse can make scarce, high-quality data more useful, and replay can help preserve or improve capabilities, but excessive or concentrated repetition can cause overfitting, fragile reasoning, and repetitive outputs. A central open tension is how to choose repetition rates across model sizes, datasets, and training budgets, since the value of another pass depends on data quality, ordering, regularization, and the amount of unique data.