Small Models Scout Bottleneck Order for Large-Model Data Control

Seungmin Choi; Jiwon Sung; Muhammad Umer; Abhiram Rao Gorle; Guijin Son; Youngjae Yu; John M. Cioffi

cs.AIAug 14, 2026

Small Models Scout Bottleneck Order for Large-Model Data Control

Authors: Seungmin Choi, Jiwon Sung, Muhammad Umer, Abhiram Rao Gorle, Guijin Son, Youngjae Yu, John M. Cioffi

Organizations: Stanford University · Seoul National University

Abstract

Small proxy models are commonly used to identify data mixtures for larger-scale training. We ask whether their training trajectories reveal another transferable structure: the order in which larger models should resolve skill bottlenecks. We formulate first-passage skill training, where each monitored skill has a target floor and the objective is to minimize the tokens required to reach all floors. We introduce LogFloor, a closed-loop controller that directs each round toward current bottlenecks, producing phase-ordered resolution trajectories. Across five bAbI skill slices on Qwen2.5-1.5B, LogFloor reduces token cost by 56.2% on average. In 70M-to-12B transfer, three-round replay of a 70M scout path reaches every floor in all eight target runs, saving 30.9% by pair mean, 39.4% in pooled training tokens, and 37.6% under source-cost accounting. On MMLU-control, a frozen scout path succeeds across all eight 12B runs. Collapsing a path to its static marginal mixture or reversing its phase order removes most benefits, while bottleneck labels alone remain partially useful. These results identify phase-ordered bottleneck resolution as a transferable curriculum structure for monitored skill-targeted training.

Figures & tables

Figure 1: A simplified visualization of the proposed LogFloor controller for $K=5$ monitored skills. The bottleneck skill at each round $r$ is depicted in orange. At each round, LogFloor shifts the largest share of the next allocation $w_{r}$ to this bottleneck, and the resulting ordered sequence $\{w_{r}\}$ forms the bottleneck-resolving trajectory that we later replay on larger target models.

Term	Definition
Pooled saving	Savings after summing token costs across runs.
Pair mean saving	Mean of the savings of each matched run.
Hit	All monitored floors are reached within the horizon.
Route first-hit round	First round where all monitored floors are reached.
Heldout audit	Heldout evaluation at the route first-hit round.
W/T/L	Wins, ties, and losses against matched uniform training.

Table 1: Evaluation terminology.

Figure 2: A representative bottleneck-resolving trajectory. Uniform training clears all target floors at zero-indexed round 42, while LogFloor clears them at zero-indexed round 11 by repeatedly prioritizing this floor-pressure bottleneck. Red boxes in the lower heatmap identify the bottleneck skill selected at each round.

Floor	LogFloor pair mean	LogFloor pooled	Greedy pooled	Skill-It pooled	ODM pooled
2.8	36.3%	59.2%	54.4%	-12.1%	7.2%
2.9	28.0%	53.2%	52.0%	-22.9%	4.8%
3.0	27.1%	49.1%	49.1%	-13.3%	12.2%
3.1	48.7%	55.6%	55.7%	3.6%	20.9%
3.2	57.8%	62.2%	52.4%	-5.3%	-19.6%
3.3	45.0%	55.4%	44.9%	-3.8%	-17.9%

Table 2: First-passage target sweep on Qwen2.5-1.5B. Each target floor has five seeds. LogFloor reaches the route target in

30/30

runs, and

28/30

route first-hits also clear the heldout target on all floors.

Figure 3: Small-to-large trajectory transfer. Replayable target-probe replay transfers source LogFloor trajectories across six Pythia settings and outperforms static summaries on average.

Method	Hit	Win	Target tokens	Pair mean	Pooled	Incl. scout
12B uniform	8/8	–	825,094	0.0%	0.0%	0.0%
12B online LogFloor	8/8	8/8	499,711	34.4%	39.4%	39.4%
70M target-probe replay	8/8	7/8	500,375	30.9%	39.4%	37.6%
Probe + frozen target mix	7/8	–	–	–	–	–
70M phase replay (oracle)	8/8	7/8	535,150	30.1%	35.1%	33.4%

Table 3: 70M-to-12B transfer accounting. Replayability, probe, oracle, and source-cost controls over eight target runs. Hit and Win count route hits and wins against matched uniform runs; rows with a missed target omit aggregate savings.

Figure 4: 70M-to-12B transfer diagnostics. The left plot juxtaposes target-only savings for 12B online and 70M target-probe replay with cost-inclusive savings for 70M phase replay and RegMix. The right plot shows that static proxy search improves the 70M proxy score but transfers non-monotonically to 12B.

Comparison	Pairs	Deficit cos	Alloc. cos	Top- $2$
Diagonal seed $i\!\to\!i$	$8$	$0.807$	$0.844$	$0.930$
Off-diagonal $i\!\to\!j$	$56$	$0.814$	$0.839$	$0.935$

Table 4: 70M-to-12B phase geometry audit. Before-hit trajectories are aligned by normalized phase, and entries are means over diagonal and off-diagonal source–target seed pairs.

Schedule	Hit	W/T/L	Pair mean	Pooled rounds
12B online LogFloor	$8/8$	$4/3/1$	$25.6\%$	$33.6\%$
70M matched replay	$8/8$	$5/1/2$	$23.5\%$	$38.8\%$
Frozen scout seed $2$	$8/8$	$6/2/0$	$46.8\%$	$60.3\%$
$3$ -round target probe	$8/8$	$5/2/1$	$17.2\%$	$31.0\%$
$3$ -round probe + frozen target mix	$7/8$	$4/1/3$	–	–

Table 5: MMLU control transfer diagnostics. Rows aggregate eight Pythia-12B target seeds unless noted. W/T/L denotes wins, ties, and losses against matched uniform runs. Rows with a missed target omit aggregate savings.

Schedule	Retained signal	Hit	Pair mean	Pooled rounds	Pooled tokens
Correct seed $2$ scout	full phase indexed allocation path	$8/8$	$46.8\%$	$60.3\%$	$59.8\%$
Static collapse	marginal allocation only	$8/8$	$-2.9\%$	$10.3\%$	$9.1\%$
Reverse	same allocations in reverse order	$8/8$	$-13.6\%$	$6.0\%$	$5.3\%$
Random permutation	same allocations in shuffled order	$6/8$	–	–	–
Label replay	bottleneck identities in correct order	$8/8$	$27.9\%$	$25.0\%$	$22.4\%$
Label reverse	same identities in reverse order	$8/8$	$-66.7\%$	$-11.2\%$	$-10.8\%$

Table 6: Decomposition of the frozen source seed 2 trajectory. All rows use the same eight Pythia-12B targets and the same floors for each target. Rows with a missed target omit aggregate savings.

Scout floor	$2.8$	$3.0$	$3.3$
	Target floor
$2.8$	–	$4/5$	$5/5$
$3.0$	$5/5$	–	$4/5$
$3.3$	$5/5$	$5/5$	–

Table 7: Replay across target floors on Qwen2.5-1.5B. Each cell gives route hits out of five target seeds after the same three-round target probe; darker shading marks full

5/5

hits. Gray cells mark identical scout and target floors.

Axis	Result	Role
Target sweep	Qwen2.5-1.5B saves $56.2\%$ pooled tokens across six floors	primary validation
Skill count	$K\in\{5,7,10,15\}$ : $41.8\%$ aggregate and $40.1\%$ paired savings	floor-count robustness
Model size	Pythia 70M–2.8B wins on all evaluated seeds, $42.1\%$ mean saving	scale robustness
Pretraining stage	positive from step 16K onward, $53.9\%$ at the final checkpoint	stage robustness
SuperGLUE control	$51.2\%$ (base) and $53.7\%$ (CB-substituted) route savings; $7/10$ heldout-confirmed	beyond bAbI
Architecture and frontier	Mamba self-control positive; single-seed 72B pilot hits at round 7 (uniform), 2 (online), 3 (replay)	scope evidence

Table 8: Breadth and scope summary.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 344 papers

What do Language Models Learn and When? The Implicit Curriculum Hypothesis
Apr 9, 2026Emmy Liu, Kaiser Sun, Millicent Li +4Emergent Abilities in Language Models Language Model Pretraining
Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1
Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle
Oct 15, 2026Yuanjian Xu, Jianing Hao, Tianze Sun +7LLM Training Data Mixture Optimization
The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology · OpenCSG +4
What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use
Sep 26, 2026Yixiao Chen, Ke Cheng, Jiangtao Guan +5Language Model Pretraining Factual Knowledge in Language Models
AI Data, Tencent · Harvard University · AMS, Tencent

Apr 9, 2026J/K move · Enter open · S save

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Emmy Liu, Kaiser Sun, Millicent Li, Isabelle Lee+3

Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University+1

Emergent Abilities in Language Models Language Model Pretraining

Open paper AskPDF arXiv

Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (

ρ= .81

across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (

R^2 = .68

.84

across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.

Abstract

LLM Training

Explore similar work

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use