Authors: Joshua Kazdan, Noam Levi, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo, David Donoho
Organizations: Department of Statistics, Stanford University · AI4Science, EPFL · Department of Computer Science, Stanford University · ServiceNow Research · IMC Trading
Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what constitutes a ``duplicate'': beyond surface-form matches, semantically equivalent documents (e.g. translations) may induce redundant training signals once models become sufficiently capable. Practically, this means that semantic duplicates operate increasingly like exact duplicates during training. We present evidence that duplication is scale-dependent in two ways. First, as model capability increases, cross-entropy loss gradients for semantically equivalent documents become more aligned. Smaller models, by contrast, produce gradients that reflect surface similarity (e.g., shared tokens) rather than semantic similarity. Second, we embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m. For moderate corpus sizes, the cosine similarity between nearest-neighbors follows an isotropic power law baseline. However, as corpus size grows to hundreds of billions of tokens, the nearest-neighbor similarities deviate sharply, indicating accelerated semantic collisions. Finally, controlled pretraining on data sampled with replacement from pools of finite unique documents shows that limited uniqueness yields mild degradation for small models, but rapidly increasing loss penalties for larger models, breaking naive scaling extrapolation. We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus. Our results identify and resolve an unstudied source of scale-dependence, allowing for more accurate prediction at scale.
Figures & tables
Figure 1: Semantic-preserving transformations yield more aligned gradients for larger/stronger models. We sample N=1000 FineWeb-Edu-Dedup documents and compute per-document gradients of normalized next-token cross-entropy (Eq. 2 ) for each model. We report mean cosine similarity between (i) unrelated document pairs (negative baseline) and (ii) each document and its transformed counterpart (positives), including translations and light surface perturbations. Smaller/weaker models exhibit gradient similarity dominated by surface cues (language/casing), often failing to separate positives from negatives. As capability increases, positives become consistently more aligned than the negative baseline. Error bars show per-document standard deviation. Per-model-family results are in Figure 8 .
Figure 2: Semantic sensitivity emerges over training and is accelerated by scale. For a fixed model family, we compute AUC to detect whether a candidate gradient corresponds to a semantic-preserving transformation of the same document versus an unrelated document, with cosine similarity to the original document gradient as the score. Early in training, AUC remains near 0.5 because gradients are dominated by surface-form features (language/casing). With additional optimizer steps, AUC increases, indicating that gradients increasingly reflect semantic content. Larger models reach a given AUC with fewer steps.
Figure 3: NN cosine similarity scaling deviates sharply at large corpus sizes. We embed 190M FineWeb-Edu-Dedup documents with EmbeddingGemma-300m and sample subsets of size ranging from 104 - 108 without replacement. For each N , we estimate the mean nearest-neighbor cosine similarity using FAISS. Dashed lines show best-fit power laws over the small- N regime where the uniform/vMF null predicts E[Δi]∝N−2/d . Beyond a scale threshold, the empirical curve steepens (smaller gaps than predicted), indicating substantially more near neighbors than expected under isotropic baselines.
Figure 4: Tail collision rates accelerate with dataset size. For fixed thresholds T , we estimate the fraction of points with nearest-neighbor similarity Mi≥T . These increase exponentially, as predicted under an isotropic baseline.
Figure 5: Nearest-neighbor cosine similarity scaling laws collapse an order of magnitude earlier for synthetic datasets: We embed the fully-synthetic pretraining dataset Recycling-the-Web ( Nguyen et al., 2025 ) and find that the scaling law discovered in Figure 3 occurs an order of magnitude earlier for synthetic data, suggesting that the diversity of synthetic pretraining datasets should be improved.
Figure 6: Finite unique data pools induce scale-dependent degradation and break naive scaling extrapolation. We train model ladders at matched compute while sampling training documents with replacement from pools of size K (exact repeats allowed). We compare against an approximately-infinite baseline with negligible repeats. Left: train and validation loss versus compute/scale for each K . Right: fractional loss change relative to the baseline (Eq. 9 ). Small models scale normally under small K , while larger models exhibit rapidly increasing penalties, implying that scaling ladders can underestimate main-run loss when effective uniqueness is limited.
Figure 7: Predictable scaling can be restored by accounting for limited semantic diversity: We use dataset size and mean cosine similarity to estimate K via Equation 34 (left). We then plug our estimate of Keff into Equation 28 to estimate the loss (Center). This produces scaling curves that align closely with the empirical eval losses (right).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Semantic-preserving transformations yield more aligned gradients for larger/stronger models: We display the same data as in Figure 1 .
Figure 9: Predictions of eval loss using cosine intensity-based estimation of Keff .
Figure 10: Predictions of eval loss using true K .