Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

Götz-Henrik Wiegand; Lorena Raichle; Rico Städeli; Tomas Hrycej; Bernhard Bermeitinger; Siegfried Handschuh

cs.LGApr 10, 2026

Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

Authors: Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli, Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh

Organizations: Institute of Computer Science, University of St. Gallen, St. Gallen, Switzerland · Institute of Computer Science in Vorarlberg, University of St. Gallen, Dornbirn, Austria

Abstract

Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.

Figures & tables

Subset Level		% Full Dataset	Size (seqs)	Tokens (in M)	Tokens/Param	Train Time (h:mm)
$\bullet$	$2^{7}$	0.1	128	0.13	0.056	0:08
$\bullet$	$2^{8}$	0.2	256	0.26	0.111	0:15
$\bullet$	$2^{9}$	0.3	512	0.52	0.222	0:31
$\bullet$	$2^{10}$	0.7	1,024	1.05	0.444	1:01
$\bullet$	$2^{11}$	1.4	2,048	2.10	0.888	2:02
$\bullet$	$2^{12}$	2.7	4,096	4.19	1.776	3:44

Table 2: Dataset statistics and training cost based on 2.4M trainable parameters.

Fraction of full-data accuracy	Subset	Data (%)	Train Time (h)	Cost (%)
80%	$2^{13}$	7	8	7
90%	$2^{15}$	29	32	26
95%	$2^{16}$	56	60	49
100%	$2^{17}$	100	123	100

Table 3: Cost–performance trade-offs under fixed training schedule

Figure 5: Validation accuracy on AllTheNews2.0 dataset vs. subset tokens for fixed training budget

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Training Dataset	Validation Dataset
Number of Sequences	131,072	20,000
Sequence Length	1024 tokens	1024 tokens
Total Token Count	76,800,000	10,240,000
Padding Tokens	21,504,000 (28%)	2,867,200 (28%)

Appendix

Table 4: Training and validation dataset statistics of articles after cleaning (

>500

tokens)

Subset Level	Val. Loss	Val. Perplexity	Val. Accuracy (%)
$2^{7}$	$11.24\pm 0$ .07)	$76\,320\pm 4973$	$0.40\pm 0$ .12)
$2^{8}$	$9.71\pm 0$ .10)	$16\,453\pm 1588$	$2.12\pm 0$ .41)
$2^{9}$	$8.68\pm 0$ .05)	$5911\pm 268$	$3.13\pm 0$ .21)
$2^{10}$	$8.21\pm 0$ .02)	$3683\pm 80$	$4.00\pm 0$ .50)
$2^{11}$	$7.70\pm 0$ .08)	$2203\pm 178$	$5.74\pm 1$ .20)
$2^{12}$	$7.36\pm 0$ .04)	$1574\pm 63$	$7.13\pm 0$ .27)

Appendix

Table 5: Robustness to random subset sampling across seeds. Mean

\pm

standard deviation over five seeds at epoch 1.

Figure 6: Empirical best validation CE loss per subset size compared to the Kaplan scaling-law prediction on AllTheNews2.0

Figure 7: Cross-entropy loss on AllTheNews2.0 dataset under a fixed training budget.

Panel of Figure 8 — Figure 8: Validation accuracy (top) and cross-entropy loss (bottom) on WikiText-103 under a fixed training budget.

Language Model Scaling Laws

Language model scaling laws describe how model size, training data, compute, and inference effort jointly shape loss, capabilities, and efficiency.

See all 141 papers

Squeezing More from Limited Data with Recursive Transformers
Aug 27, 2026Serdar Gülbahar, Lukas Edman, Alexander FraserLanguage Model Pretraining Language Model Scaling Laws
Technical University of Munich
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
May 19, 2026Jingwen Liu, Ezra Edelman, Surbhi Goel +1Deep Learning Optimization Statistical Learning Theory
Columbia University · University of Pennsylvania · Kempner Institute, Harvard University

Aug 27, 2026J/K move · Enter open · S save

Squeezing More from Limited Data with Recursive Transformers

Serdar Gülbahar, Lukas Edman, Alexander Fraser

Technical University of Munich

Language Model Pretraining Language Model Scaling Laws Representation Learning

Open paper AskPDF arXiv

Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.

Abstract

Language Model Scaling Laws

Explore similar work

Squeezing More from Limited Data with Recursive Transformers

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases