Squeezing More from Limited Data with Recursive Transformers

Serdar Gülbahar; Lukas Edman; Alexander Fraser

cs.CLAug 27, 2026

Squeezing More from Limited Data with Recursive Transformers

Authors: Serdar Gülbahar, Lukas Edman, Alexander Fraser

Organizations: Technical University of Munich

Abstract

Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.

Figures & tables

Figure 1: Standard embeddings use a single

V\times H

matrix, while factorized embeddings introduce an embedding-size bottleneck by mapping tokens through an

E

-dimensional space before projecting to the

H

-dimensional hidden space. This reduces embedding parameter count when

E\ll H

and can act as a useful regularizer.

Figure 2: Optimal scale with standard Transformers on the BabyLM baseline corpus across three representative evaluations. The preferred parameter count depends on the downstream target: BLiMP saturates relatively early, LAMBADA becomes more scale-favoring as dataset size grows, and COMPS continues to benefit from larger models at moderate and high data budgets.

Figure 3: Final training loss for the standard-model optimal-scale sweep on the BabyLM baseline corpus. The x-axis uses the same parameter counts as Figure 2 . All data subsets use the same tokenizer and vocabulary, so losses are directly comparable. Training loss keeps falling even when downstream performance saturates or declines.

Figure 4: RecursiveGPT architecture. Tokens are processed by applying a shared Transformer block

R

times. Each recurrent step

r

uses separate normalization and bias parameters for depth conditioning.

Figure 5: Average accuracy for recursive models with $(H,E){=}(640,192)$ trained with recurrent depth $R$ swept from 4 to 32 on the 10% subset of the 100M BabyLM-baseline corpus from the optimal scale experiment in Section 3 . The horizontal reference lines mark the three-seed average scores from Table 1 for the best standard model and an unfactorized standard model with the same hidden size.

Word Budget	Model	Params	BLiMP	COMPS	LAMBADA p@5	Average
10M	Standard	41.1M	68.97	52.88	12.93	44.93
10M	Standard + FE	28.7M	70.09	52.62	12.48	45.07
10M	RecursiveGPT + FE ( $R{=}16$ )	27.6M	70.95	52.96	13.48	45.80
100M	Standard	1.22B	79.71	60.65	50.04	63.47
100M	RecursiveGPT + FE ( $R{=}24$ )	124.0M	80.06	58.70	47.01	61.92
100M	RecursiveGPT-Large ( $R{=}24$ )	404.1M	80.70	60.31	50.77	63.93

Table 1: Comparison between standard and recursive models at 10M and 100M words. FE denotes factorized embeddings and

R

denotes recurrent depth. The 10M rows include the best standard model from Section 3 with

H{=}384

, an unfactorized

H{=}384

E{=}384

model, and a depth-16 recursive model with factorized embeddings, all trained on the same 10% subset of the 100M BabyLM-baseline corpus used in the standard-model sweep. All rows are averaged over seeds 0, 1, and 2. The 100M standard row reports the best standard model by average score. Evaluation scores are reported as percentages; Average is the arithmetic mean across the three evaluations.

Word Budget	Model	Params	BLiMP	BLiMP S.	EWoK	Entity T.	COMPS	Average
10M	GPT-BERT (Causal)	31.0M	71.70	63.20	49.50	34.60	52.80	54.36
10M	AMLM Hard Decay	34.7M	71.40	59.20	51.00	44.20	54.20	56.00
10M	RecursiveGPT + FE	27.6M	72.21	58.28	52.01	24.51	53.33	52.07
100M	GPT-BERT (Causal)	120.0M	79.30	70.40	52.30	30.90	58.30	58.24
100M	RecursiveGPT + FE	124.0M	80.06	70.82	54.74	34.42	58.70	59.75
100M	RecursiveGPT-Large	404.1M	80.70	70.75	54.97	32.67	60.31	59.88

Table 2: Comparison to the BabyLM 2025 GPT-BERT causal baselines ( Charpentier and Samuel, 2024 ) , the highest-scoring causal models on both tracks ( Charpentier et al., 2025 ) , and AMLM hard decay, the winning masked language model in the 10M-word NLP track ( Edman and Fraser, 2025 ) . RecursiveGPT scores are averaged over seeds 0, 1, and 2. Entity T. denotes Entity Tracking, and BLiMP S. denotes BLiMP Supplement. Scores are reported as percentages; Average is the arithmetic mean across the five evaluations.

Variant	Params	BLiMP	COMPS	LAMBADA p@5	Average
RecursiveGPT + FE $(H,E){=}(640,192)$	27.6M	70.85	53.16	14.46	46.16
Shared norm parameters	27.6M	71.38	52.83	13.97	46.06
No Factorized Embeddings ( $H{=}640$ )	56.7M	68.13	52.75	13.81	44.90
No Factorized Embeddings ( $H{=}384$ )	30.5M	68.70	52.41	11.17	44.09

Table 3: Ablations of the 10M-word depth-16 RecursiveGPT model trained on the 10% subset of the 100M BabyLM-baseline corpus with seed 0. FE denotes factorized embeddings. Shared norm parameters shares the normalization parameters across recurrent steps instead of using step-specific normalization and bias. No FE removes factorized input and output vocabulary maps. Scores are reported as percentages; Average is the arithmetic mean across the three evaluations.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$E$	Embed	Total	BLiMP	COMPS	LAMBADA	Average
64	4.2M	20.2M	70.51	51.96	9.79	44.09
128	8.5M	24.5M	69.79	52.33	12.49	44.87
192	12.7M	28.7M	70.09	52.62	12.48	45.07
256	17.0M	32.9M	69.02	52.55	12.43	44.67
320	21.2M	37.2M	69.25	52.31	13.01	44.86
384	25.2M	41.1M	68.97	52.88	12.93	44.93

Appendix

Table 5: Factorized-embedding sweep on the 10M-word setting with hidden size fixed at

H{=}384

. Embed and Total report vocabulary-map (input embedding plus output head) and total parameter counts; scores are percentages averaged over seeds 0, 1, and 2, LAMBADA is evaluated as pass@5, and Average is the arithmetic mean of the scores.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
Apr 10, 2026Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli +3Language Model Scaling Laws Scaling Laws
Institute of Computer Science, University of St. Gallen, St. Gallen, Switzerland · Institute of Computer Science in Vorarlberg, University of St. Gallen, Dornbirn, Austria
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language Modeling Self-Attention
Harvard University
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1

Apr 10, 2026J/K move · Enter open · S save

Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli, Tomas Hrycej+2

Institute of Computer Science, University of St. Gallen, St. Gallen, Switzerland · Institute of Computer Science in Vorarlberg, University of St. Gallen, Dornbirn, Austria

Language Model Scaling Laws Scaling Laws

Open paper AskPDF arXiv

Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.

Abstract

Language Model Pretraining

Explore similar work

Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training