Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
Authors: Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli, Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh
Organizations: Institute of Computer Science, University of St. Gallen, St. Gallen, Switzerland · Institute of Computer Science in Vorarlberg, University of St. Gallen, Dornbirn, Austria
Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.
Figures & tables
Figure 1Figure 2
Subset Level
% Full Dataset
Size (seqs)
Tokens (in M)
Tokens/Param
Train Time (h:mm)
∙
27
0.1
128
0.13
0.056
0:08
∙
28
0.2
256
0.26
0.111
0:15
∙
29
0.3
512
0.52
0.222
0:31
∙
210
0.7
1,024
1.05
0.444
1:01
∙
211
1.4
2,048
2.10
0.888
2:02
∙
212
2.7
4,096
4.19
1.776
3:44
Table 2: Dataset statistics and training cost based on 2.4M trainable parameters.
Fraction of full-data accuracy
Subset
Data (%)
Train Time (h)
Cost (%)
80%
213
7
8
7
90%
215
29
32
26
95%
216
56
60
49
100%
217
100
123
100
Table 3: Cost–performance trade-offs under fixed training schedule
Figure 5: Validation accuracy on AllTheNews2.0 dataset vs. subset tokens for fixed training budget
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Training Dataset
Validation Dataset
Number of Sequences
131,072
20,000
Sequence Length
1024 tokens
1024 tokens
Total Token Count
76,800,000
10,240,000
Padding Tokens
21,504,000 (28%)
2,867,200 (28%)
Appendix
Table 4: Training and validation dataset statistics of articles after cleaning ( >500 tokens)
Subset Level
Val. Loss
Val. Perplexity
Val. Accuracy (%)
27
11.24±0 .07)
76320±4973
0.40±0 .12)
28
9.71±0 .10)
16453±1588
2.12±0 .41)
29
8.68±0 .05)
5911±268
3.13±0 .21)
210
8.21±0 .02)
3683±80
4.00±0 .50)
211
7.70±0 .08)
2203±178
5.74±1 .20)
212
7.36±0 .04)
1574±63
7.13±0 .27)
Appendix
Table 5: Robustness to random subset sampling across seeds. Mean ± standard deviation over five seeds at epoch 1.
Figure 6: Empirical best validation CE loss per subset size compared to the Kaplan scaling-law prediction on AllTheNews2.0
Figure 7: Cross-entropy loss on AllTheNews2.0 dataset under a fixed training budget.
Figure 8: Validation accuracy (top) and cross-entropy loss (bottom) on WikiText-103 under a fixed training budget.
Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.
Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling.
Boqian Wu, Qiao Xiao, Patrik Okanovic +6
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.
Jingwen Liu, Ezra Edelman, Surbhi Goel +1
Columbia University · University of Pennsylvania · Kempner Institute, Harvard University