Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts

Eren Unlu

cs.AIApr 5, 2026

Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts

Authors: Eren Unlu

Organizations: Globeholder, Paris, France

Abstract

Width expansion offers a practical route to reuse smaller causal-language-model checkpoints, but selecting a widened warm start is not solved by zero-step preservation alone. We study dense width growth as a candidate-selection problem over full training states, including copied weights, optimizer moments, and scheduler state. In a small-scale TinyStories proxy, we compare exact-copy, perturbative, asymmetric-reset, and structured non-clone warm starts under matched continuation budgets. We evaluate zero-step preservation, short-lag probe metrics, and downstream continuation utility in deterministic and stochastic regimes. The picture is mixed and partially replicated through a reduced-pool seed-1 check. Exact-copy symmetric warm starts rank first in every completed 16-step probe and in the completed stochastic 128-step continuations at seed-0 steps 1000 and 2000 plus reduced seed-1 step 2000. By contrast, the structured non-clone challenger wins deterministic 128-step continuation. Early escape from the inherited cloned subspace is therefore not a universal selector: it helps in long deterministic continuation, but it misleads at short lag and under stochastic continuation. The result is narrow but useful: for dense width growth at this scale, preservation is not a universal ranking criterion, and the best replacement signal depends on both regime and lag budget.

Figures & tables

Component	Value
Dataset	roneneldan/TinyStories / default
Tokenizer	Byte-level BPE, NFKC normalization, vocab 8192 , min frequency 2 , trained on 50000 examples
Packing	prepend <bos> , append <eos> , flatten, then pack into 257 -token rows for next-token prediction
Parent model	decoder-only LM with learned token and position embeddings, 6 layers, d_model=256 , n_heads=8 , mlp_multiplier=4.0 , RMSNorm, SwiGLU
Parent dropout	0.0
Parent optimization	AdamW, lr=0.0003 , min_lr=3e-05 , betas=(0.9, 0.95) , weight_decay=0.1 , grad_clip=1.0

Table 1

Recipe	Init	Opt state	Sched	Perturb	Warmup	Role
scratch-large	scratch	fresh	constant	0.0000	0	Fresh large child with no inherited weights or optimizer state.
exact-copy	exact-copy	symmetric	constant	0.0000	0	Exact function-preserving width copy with symmetric AdamW state inheritance.
exact-copy + perturb	exact-copy	symmetric	constant	0.0010	0	Exact-copy widened child with small isotropic post-copy perturbation.
exact-copy + asym-reset	exact-copy	asym-reset	constant	0.0000	0	Exact-copy widened child with new-clone AdamW moments reset out of the cloned subspace.
exact-copy + asym-reset + rewarm	exact-copy	asym-reset	fresh cosine	0.0000	2	Asymmetric-reset widened child probed under a fresh global cosine rewarmup.
ref-subspace	ref-subspace	ref-slice copy	constant	0.0000	0	Embed the parent exactly into one active width slice, keep the added slice dormant at step 0, and copy optimizer state only into the preserved reference slice.

Table 2

Setting	Regime	Horizon	Winner	Delta AUC	Best selector
S0 / 1000 / 16D	deterministic	16	exact-copy	0.0791	zero loss, zero KL, probe KL, probe RMS
S0 / 1000 / 16S	stochastic	16	exact-copy	0.1093	zero loss, zero KL, probe RMS
S0 / 1000 / 128D	deterministic	128	ref-subspace	-3.9633	probe KL, probe escape
S0 / 1000 / 128S	stochastic	128	exact-copy	0.1047	zero loss, zero KL, probe RMS
S0 / 2000 / 16D	deterministic	16	exact-copy	0.0324	zero loss, zero KL, probe KL, probe RMS
S0 / 2000 / 16S	stochastic	16	exact-copy	0.0892	zero loss, zero KL, probe KL, probe RMS

Table 3

Figure 1: Lag-budget and regime reconciliation for the main structured non-clone challenger. Each point shows the validation-loss AUC gap between refsubspace_statecopy and exactcopy_symmetric, computed as refsubspace minus exactcopy, so negative values favor refsubspace_statecopy and positive values favor exactcopy_symmetric. In the deterministic panel, every available series moves downward from the 16-step probe to the 128-step continuation, and the seed-0 step-1000 series plus both step-2000 series cross below zero. In the stochastic panel, all available series remain positive: the seed-0 step-1000 series stays close to zero at 128 steps, while the seed-0 and seed-1 step-2000 series move farther positive.

Figure 2: Selector top-1 regret across completed settings. Each cell shows the regret of a selector relative to the best candidate in the same report, with lower values better. The left panel covers the full seed-0 study; the right panel covers the reduced seed-1 replication. Probe KL is the strongest overall low-cost selector, staying at zero regret across all reduced seed-1 settings and all seed-0 deterministic long-horizon settings, but the seed-0 step-1000 stochastic 128-step run shows a small Probe RMS win over Probe KL. Probe escape reaches zero regret only in deterministic 128-step continuation, where the structured non-clone challenger actually wins. Probe RMS remains competitive in short and stochastic settings but misses both deterministic long-horizon reversals. Zero-step KL is often acceptable at short lag, but it fails when deterministic long-horizon continuation favors refsubspace_statecopy.

Setting	Winner	Exact-copy AUC	Ref-subspace AUC	Delta AUC ( ref - exact )	Mean delta / step
S0 / 1000 / 16D	exact-copy	44.5945	44.6737	0.0791	0.0049
S0 / 1000 / 16S	exact-copy	44.1903	44.2996	0.1093	0.0068
S0 / 1000 / 128D	ref-subspace	360.9172	356.9538	-3.9633	-0.0310
S0 / 1000 / 128S	exact-copy	351.1863	351.2910	0.1047	0.0008
S0 / 2000 / 16D	exact-copy	39.9548	39.9872	0.0324	0.0020
S0 / 2000 / 16S	exact-copy	39.9700	40.0591	0.0892	0.0056

Table 6

Setting	Zero-step loss	Zero-step KL	Probe KL	Probe RMS	Probe Escape
S0 / 1000 / 16D	0.0000	0.0000	0.0000	0.0000	0.0791
S0 / 1000 / 16S	0.0000	0.0000	0.1093	0.0000	0.1093
S0 / 1000 / 128D	3.9633	3.9633	0.0000	3.9633	0.0000
S0 / 1000 / 128S	0.0000	0.0000	0.1047	0.0000	0.1047
S0 / 2000 / 16D	0.0000	0.0000	0.0000	0.0000	0.0324
S0 / 2000 / 16S	0.0000	0.0000	0.0000	0.0000	0.0892

Table 7

Target	16 -step AUC pick	Regret @ 128	16 -step final pick	Regret @ 128	Spearman
S0 / 1000 / 128D	exact-copy	3.9633	ref-subspace	0.0000	0.8286
S0 / 1000 / 128S	exact-copy	0.0000	exact-copy + perturb	0.1539	0.9429
S0 / 2000 / 128D	exact-copy	0.6090	exact-copy	0.6090	0.9429
S0 / 2000 / 128S	exact-copy	0.0000	exact-copy	0.0000	1.0000
S1 / 2000 / 128D	exact-copy	0.5574	exact-copy	0.5574	0.8000
S1 / 2000 / 128S	exact-copy	0.0000	exact-copy	0.0000	1.0000

Table 8

Selector	Mean regret	Max regret	Zero-regret settings	Mean Spearman to utility
Zero-step loss	0.4745	3.9633	7	0.4191
Zero-step KL	0.4745	3.9633	7	0.4191
Probe KL	0.0178	0.1093	10	0.9619
Probe RMS	0.4275	3.9633	9	0.8571
Probe Escape	0.2015	0.9876	3	-0.5000

Table 9

Claim ID	Audited statement	Status	Quantitative support
C1	Exact-copy symmetric wins every completed 16-step setting.	pass	6/6 short settings
C2	Reference-subspace wins every completed deterministic 128-step setting.	pass	3/3 deterministic long settings
C3	Exact-copy symmetric wins every completed stochastic 128-step setting.	pass	3/3 stochastic long settings
C4	Probe KL is the strongest overall low-cost selector by mean top-1 regret.	pass	Zero-step loss=0.4745, Zero-step KL=0.4745, Probe KL=0.0178, Probe RMS=0.4275, Probe Escape=0.2015
C5	Probe escape is exact only in deterministic 128-step settings.	pass	zero-regret escape: 3/3 deterministic long, 0/9 all other settings

Table 10

Language Model Scaling

Language model scaling studies how model size, training data, compute, and inference-time resources affect capability and efficiency.

See all 60 papers

When is Warmstarting Effective for Scaling Language Models?
May 13, 2026Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4Language Model Scaling Laws Language Model Scaling
University of Freiburg · Zuse School ELIZA · University of Technology Nuremberg +3
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Feb 2, 2026Qifan Yu, Xinyu Ma, Zhijian Zhuo +7Deep Learning Optimization
State Key Laboratory of General Artificial Intelligence, Peking University · ByteDance Seed · Work done at ByteDance Seed.
Length Generalization Needs Proper Regularization
Oct 3, 2026Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1Length Generalization Language Model Generalization
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon +1

May 13, 2026J/K move · Enter open · S save

When is Warmstarting Effective for Scaling Language Models?

Neeratyoy Mallik, Maciej Janowski, Johannes Hog, Herilalaina Rakotoarison+3

University of Freiburg · Zuse School ELIZA · University of Technology Nuremberg+3

Language Model Scaling Laws Language Model Scaling

Open paper AskPDF arXiv

Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited practical adoption in large-scale training. We attribute this to two underexplored factors: (1) an overemphasis on preserving the smaller model's performance at initialization, which constrains operator design for new architectures, and (2) insufficient analysis of how growth interacts with hyperparameters and scaling behavior, compounded by inconsistent growth factors across the literature. We show that preserving the base model's initial post-growth performance is not necessary for strong final performance, and that simple, architecture-agnostic growth strategies can outperform more complex warmstarting operators. Crucially, we empirically identify an upper bound on the growth factor

g

beyond which training from scratch is more efficient. We observe this across multiple ablation setups. Notably, this limit is also present, but unreported, in prior published results. Across our experiments on dense MLPs and dense language models, we find that a

2\times

growth factor is the most reliable in yielding convergence speedups, with gains most pronounced under 20 tokens/parameter budgets and diminishing as budget increases. We fit scaling laws over these observations to provide predictive guidance for practitioners deciding when and how much to grow. Together, our analysis provides practical guidelines and empirical limits for model growth.

Abstract

Language Model Scaling

Explore similar work

When is Warmstarting Effective for Scaling Language Models?

SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning

Length Generalization Needs Proper Regularization