What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use

Yixiao Chen; Ke Cheng; Jiangtao Guan; Shuo Huang; Yue Liu; Jun Zhang; Yuhong Liu; Jie Jiang

cs.LGSep 26, 2026

What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use

Authors: Yixiao Chen, Ke Cheng, Jiangtao Guan, Shuo Huang, Yue Liu, Jun Zhang, Yuhong Liu, Jie Jiang

Organizations: AI Data, Tencent · Harvard University · AMS, Tencent

Abstract

What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve its causal relation, vary shortcut-bearing context, and re-audit the residual. Formation-sensitive selection and prerequisite ordering accelerate a binding-matching-transport path; a brief early prefix from the same training multiset retains a validation advantage through 100B tokens. Availability counterfactuals then distinguish writing content from invoking available memory, while paired supervision and context-opportunity ranking improve matched route decisions and long-context answer likelihood. A continuous 350M-model experiment connects the three interventions on the same facts: early circuit training improves subsequent learning, and the complete sequence outperforms stage-replacement controls on facts withheld from Use teaching. Independent query surfaces and opposed-source decisions expose conditional arbitration as the remaining frontier. Together, these results show why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples.

Figures & tables

Figure 1: Three failures, three teaching targets. (a) Repeated dax should retrieve 7; the read fails. (b) “Ada wrote the notes” has not been acquired for unaided recall. (c) Query

q

asks for the latest fictional story’s capital, Marseille, but memory supplies Paris. Dashed paths denote candidate operations; crosses mark failed read/write steps; the solid red arrow marks wrong routing. Bottom entries compare model predictions (red) with target answers (panel colors). These are illustrative cases, not measured outputs.

Stage	Missing operation	Diagnostic	Intervention	Residual bottleneck $\rightarrow$
Circuit	path not formed	component gradients / formation-time MDA	mechanism-matched ordering	content unavailable $\rightarrow$ Store
Store	required content unavailable	M-only / evidence counterfactuals	Write; test invocation at the boundary	selection conflict $\rightarrow$ Use
Use	which circuit/source wins	query-conditioned routing probes	paired routing / CCUR	multi-arm arbitration $\rightarrow$ open

Table 1: The residual-bottleneck chain in our investigation. Each intervention addresses an identified missing operation; the remaining bottleneck motivates the next stage.

Figure 2: Circuit residual contracts; available-source conflicts persist. (a) Circuit excess-NLL mass from checkpoint-local diagnosis on 272 fixed natural identities. (b) Composition of a separate 942-target conflict residual; lines normalize total mass to each arm’s 0.1B value. Pale/left bars: Ordinary; solid/right: Curriculum. Appendix B gives roster construction, rules, and sensitivity.

Figure 3: Circuit training advances formation with persistent gains. (a) Binding forms in both arms; reordering advances matching. (b) Endpoint-minus-formation NLL is positive for Overall and ambiguous recall at 1.2B and over normalized 0.1–1.2B AUC; 1.2B PPL reductions are 2.57% and 1.30%. (c) The same strict matching-only teacher comparison favors formation-time selection in all six domains at 1.2B. (d) Curriculum crosses Ordinary’s final PPL at 82B and stays lower through 100B. Appendix C gives teacher trajectories, order controls, and the measured-exit sweep.

Figure 4: Domain-selective gains motivate an availability audit. (a) At 2.5B in the 160M order study, fixed-multiset Joint-to-matching improves Overall, Math, and Code PPL over interleaving, but not Web. (b) Relevant-source, masked-source, and shuffled-context QA on 435 fixed Web facts at 2.5B shows both arms use explicit evidence. Appendix Tables 11 and 15 give full presentation/NLL comparisons and reverse-order controls, respectively.

Arm	Active operation	M-only gain	Natural-use gain	B accuracy (%)
C0	matched controls	0.636	0.894	82.03
C1	Write	0.740	0.971	83.98
C2	Use	0.723	1.201	93.36
C3	Write + Use	0.806	1.294	95.31

Table 2: Available facts benefit most from invocation training. Parent-minus-continuation NLL gains (

\uparrow

); B accuracy is first-divergent-token candidate choice, not free generation. All arms share 512 facts, two matched slots, parent, and 25M budget. Use trains carrier A; natural gain averages A/B on those facts. Appendix D gives carrier and factorial contrasts.

Figure 5: Matched supervision improves engagement and long-context ranking. (a) Paired supervision outperforms Ordinary and balanced unpaired on familiar queries; independent phrasings test transfer. (b) On a document-disjoint training pool, CCUR beats LongCE and score-shuffle; whiskers are paired 95% endpoint intervals. (c) CCUR minus LongCE accuracy on opposed queries shows a source-prior shift, not joint arbitration. Positive values in (b,c) favor CCUR .

Figure 6: The repairs compose on common facts. Both routes receive target Store and paired Use after pretraining. On 640 Store-taught, Use-untaught facts, memory improves mainly during Store; the joint-use task improves further during Use. Each point evaluates the same fixed queries; lower answer-token NLL is better.

Training route	Joint-use NLL $\downarrow$	Pair accuracy (%) $\uparrow$
Complete sequence	2.970	5.26
No early curriculum	3.050	3.39
Control facts in Store	3.245	1.41
Unpaired Use	3.019	3.18
Ordinary continuation	3.870	0.99

Table 3: Each stage contributes to final use. Same 640 facts and 1,920 familiar-query pairs. Pair accuracy requires both authorized candidates to win at their first divergent token. “Control facts” replaces target Store; “unpaired” replaces the complete Use construction.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Block	Item	Specification	Pair status
Model	Architecture	GPT decoder; 28 layers; $d=2{,}048$ ; SwiGLU 5,120	matched
Model	Parameters	,484,900,352	matched
Model	Attention	-head MHA; RoPE; SDPA	matched
Model	Norm / vocab	RMSNorm; 65,536 tokens	matched
Input	Sequence / batch	,048 tokens; 512 sequences (1,048,576 targets/update)	matched
Train	Optimizer	AdamW; $\beta=(0.9,0.95)$ ; weight decay 0.1	matched

Appendix

Table 4: Configuration of the matched 1.485B-parameter/100B-token experiment.

Evidence	Model / start	Matched object	Support
Moving residual	.485B pair	dynamic identities; 32 fixed-stratum cases; 942 conflicts	separate frozen panels
MDA teacher time	M proxy; 1.485B target	formation-time vs endpoint-time MDA; matching-only selection	run per arm
Circuit confirmation	Pythia-style 160M @100M tokens	clean/swap, binding ablation, forced transport	runs $\times$ 3 domains
Order	Pythia-style 160M	same selected multiset; 2.5B endpoint	run per arm
Exit	.485B target	same multiset; 3B endpoint	run per arm
Long horizon	.485B target	same 100B multiset, positions only	dense 75–100B trajectory

Appendix

Table 5: Evidence map. Separate rows use separate populations; shared terminology does not merge them.

Model	Configuration
Pythia 160M auxiliary	GPT-NeoX; 12 layers; width 768; 12 heads; FFN 3,072; GELU; LayerNorm; official 50,304-token vocabulary and tokenizer
M selector teacher	,746,176 parameters; 22 layers; width 1,024; 16-head MHA; SwiGLU 2,816; RMSNorm; RoPE; tied embeddings; primary 65,536-token tokenizer
Teacher training	,048-token sequences; effective batch 512; AdamW, $\beta=(0.9,0.95)$ , weight decay 0.1; LR $3\times 10^{-4}$ to $3\times 10^{-5}$ ; 95 warm-up updates; cosine schedule to 3B

Appendix

Table 6: Auxiliary architectures and the selector proxy. The Pythia suite and the primary pretraining pipeline have separate tokenizations.

Arm	Tokens	C share	S share	U share	L share	Mass
Ordinary	0.3B	25.88	25.35	24.68	24.09	194.42
Ordinary	1.2B	9.80	49.63	2.64	37.92	103.40
Ordinary	3B	4.67	51.95	1.42	41.96	82.27
Ordinary	100B	2.56	44.84	0.00	52.60	60.00
Curriculum	0.3B	25.80	25.00	25.67	23.53	199.02
Curriculum	1.2B	2.96	54.92	2.17	39.95	90.56

Appendix

Table 7: Fixed retrospective strata on the same 32 natural cases. C/S/U/L are Circuit/Store/Use/Local excess-mass shares (%); Mass is absolute excess NLL. Each group contains eight fixed identities and retains its label.

Arm	Tokens	C	S	U	Mass	Arm	Tokens	C	S	U	Mass
O	0.100	35.74	11.34	52.91	2430.94	C	0.100	33.53	11.87	54.60	2277.94
O	0.481	12.27	10.60	77.12	2132.58	C	0.481	7.78	8.14	84.08	2058.97
O	0.800	5.75	9.39	84.86	1420.16	C	0.800	3.40	10.67	85.93	1209.27
O	1.200	4.74	15.19	80.08	1136.97	C	1.200	1.25	17.21	81.53	926.17
O	3.000	1.99	23.23	74.77	740.32	C	3.000	5.74	23.93	70.33	771.53
O	30.00	2.93	13.39	83.68	536.51	C	30.00	2.35	13.85	83.80	585.74

Appendix

Table 8: Selected 942-target conflict cells. Arms: O = Ordinary, C = Curriculum. The C/S/U columns report Circuit/Store/Use shares of actionable excess-NLL mass; Mass is unnormalized.

Arm	Tokens	Circuit cases	Circuit mass	Circuit share	Total mass / solved
Ordinary	0.3B	37	227.10	13.7%	1658.47 / 0
Curriculum	0.3B	50	318.19	19.0%	1673.17 / 0
Ordinary	0.6B	9	41.80	3.1%	1364.17 / 7
Curriculum	0.6B	1	8.14	0.7%	1243.58 / 20
Ordinary	1.2B	0	0.00	0.0%	906.40 / 66
Curriculum	1.2B	0	0.00	0.0%	834.39 / 94

Appendix

Table 9: Adjudication-free first-checkpoint audit on 272 fixed identities. Mass is absolute excess NLL above 2; solved means full-view target rank

=1

Audit or quantity	Setting or observed range
Positive strong-rule threshold multipliers	$0.5,\ 0.75,\ 1.25,\ 1.5$
Changed assignments	.8–3.1%
Actionable Circuit / Use shares	.0–19.2% / 80.8–82.0%
Natural-case severity cutoffs	$0,\ 1,\ 2,\ 3,\ 4$
Ordinary Circuit share at 0.3B	.7–26.3% across the cutoff grid
Ordinary Circuit share at 100B	.1, 4.6, 2.6, 0, 0%, in cutoff order

Appendix

Table 10: Residual-audit sensitivity. The threshold-rescaling audit, natural-case severity sweep, and adjudication-free trajectory use their own frozen populations. Percentage shares refer to excess-loss mass.

View	Cases	Ordinary NLL	Curriculum NLL	Gap	Target top-1
Original Web continuation	435	4.821	5.719	0.898	31.3 / 16.6
Context QA	435	5.288	5.575	0.288	45.1 / 42.1
Paraphrased context QA	435	5.115	5.444	0.329	46.9 / 43.4
Minimal-source QA	435	4.556	4.877	0.321	54.0 / 52.4
Source-masked QA	435	7.300	7.587	0.287	19.3 / 15.6
Shuffled-context QA	435	7.893	8.041	0.148	11.7 / 8.5

Appendix

Table 11: Same-fact presentation counterfactual. NLL is target NLL; gap is Curriculum-minus-Ordinary. Top-1 reports Ordinary/Curriculum accuracy (%).

Figure 7: Circuit data construction. Preserve key–match–successor roles while varying content and source realization. Formation-time sensitivity selects natural rows; joint binding–matching followed by matching-only selection orders prerequisites; a pilot calibrates the exit.

Domain	Binding		Matching			Transport
	Clean–corrupt	Patch	Clean	Swapped	Q/K $\Delta$ NLL	Eligible	V/O $\Delta$ NLL
Web	.00703	.00575	.00958	$-.11722$	.09406	86	.35515
Math	.00748	.00326	.04658	$-.15676$	.72399	139	.68819
Code	.00690	.00222	.06866	$-.15794$	1.18290	124	1.14056

Appendix

Table 12: Auxiliary 160M functional confirmation. Top: domain-probe binding, matching, and transport controls, where

\Delta

NLL is ablated minus clean. Bottom: natural-carrier confirmation at 100M training tokens in the main and independent runs. Every registered gate passes.

Tokens	Formation	Endpoint	Formation	Endpoint
	Overall NLL		Ambiguous recall NLL
0.1B	7.58435	7.57876	6.48897	6.46127
0.2B	6.28988	6.30441	4.97248	4.97939
0.3B	5.70661	5.74712	4.37319	4.42689
0.45B	5.10654	5.11048	3.71070	3.66996
0.6B	4.58137	4.59874	2.75410	2.74828

Appendix

Table 13: Strict matching-only comparison of formation-time and endpoint-time MDA teachers. Lower is better.

Teacher	Selection order	Binding $\uparrow$	Matching $\uparrow$	Induction $\uparrow$	Overall NLL $\downarrow$	Ambiguous recall NLL $\downarrow$
Formation	Joint-to-matching	.7128	.3071	2.4977	4.8303	3.0933
Formation	Matching-only	.7084	.2964	2.4410	4.8223	3.0960
Endpoint	Joint-to-matching	.6996	.2903	2.4098	4.8392	3.1095
Endpoint	Matching-only	.7141	.2709	2.2298	4.8513	3.1117

Appendix

Table 14: Normalized AUC over 0.1–1.2B for the static teacher-by-organization factorial. Teacher denotes the matching-score checkpoint; the binding-score checkpoint is fixed.

Order	Overall	Math	Code	Ambiguous recall NLL	Web
Joint-to-matching	7.6686	18.3830	4.0426	0.6923	41.4245
Interleaved	7.6872	18.4269	4.0561	0.6955	41.3716
Matching-to-joint	7.7018	18.4238	4.0703	0.7026	41.2755

Appendix

Table 15: Same selected warm-up multiset, different order, evaluated at 2.5B. All columns report PPL except ambiguous recall NLL; lower is better.

Exit	Overall	Code	STEM	Books	Papers	English Web	Chinese Web
Ordinary	25.9247	3.7900	17.2590	29.6611	21.5048	35.3020	70.5480
0.3B	25.8479	3.7790	17.2387	29.4727	21.5418	35.2636	70.0989
0.6B	25.7401	3.7798	17.2283	29.2783	21.5368	35.1533	69.3255
1.2B	25.6834	3.7703	17.1703	29.2310	21.4604	35.1711	68.9548
2.0B	25.7771	3.7780	17.2160	29.4766	21.4597	35.2412	69.2234

Appendix

Table 16: Warm-up exit sweep at the common 3B endpoint. Values are PPL; lower is better.

Tokens	Ordinary	Curriculum
75B	12.6560	12.6019
80B	12.6177	12.5621
82B	12.6043	12.5497
90B	12.5714	12.5164
95B	12.5625	12.5078
100B	12.5580	12.5034

Appendix

Table 17: Selected validation PPL from the paired 100B run.

Run	Overall PPL gain (%)	Ambiguous recall NLL direction
1	1.26	lower
2	1.58	lower by 0.00740
3	1.00	higher by 0.00321

Appendix

Table 18: Curriculum versus Ordinary across independent 3B initializations. Gains use

100(P_{\rm Ord}-P_{\rm Curr})/P_{\rm Ord}

Domain	Ordinary	Curriculum
Books	15.4227	15.3798
Chinese Web	21.6994	21.3875
Code	2.4922	2.4882
English Web	16.9887	16.9355
Papers	11.2392	11.2190
STEM	9.2482	9.2225

Appendix

Table 19: All six domain PPLs at the matched 100B endpoint.

Figure 8: Store data construction. Preserve target content and the cue–value relation; vary cue surface, carrier, and optimizer position. Evidence rescue after M-only failure calls for writing; M-only success followed by carrier suppression calls for invocation training.

Selector	Frozen cases	Evidence-present accuracy
H/L/M M-causal gate	2,048	100.00%
Blind/hard	2,048	31.84%
Random hard	2,048	5.08%

Appendix

Table 20: Availability-gated and generic hard-example populations, each with 2,048 selected cases. The gated set is frozen from the 2,220 qualifying M-missing candidates. Evidence-present success is part of that selection rule.

View	Input construction
M-only	Short factual question followed by Answer: ; no source passage.
Evidence	Original answer-containing passage followed by the same question.
Surface	The source sentence truncated immediately before its answer; scored as a completion.
Carrier A / B	The same question preceded by background with answer-containing sentences removed; A uses a Background instruction, B a Notes instruction.

Appendix

Table 21: Store input views. The question and candidate answers stay fixed. Evidence availability and answer-free context change separately.

Write steps $k$	Exact margin	Unseen paraphrase	Broad NLL cost
12	+0.01296	+0.02679	+0.227%
14	+0.03729	+0.02959	+0.298%
15	+0.05009	+0.02910	+0.328%
16	+0.05744	+0.03181	+0.358%
20	+0.07162	+0.03394	+0.674%

Appendix

Table 22: Controlled relation-writing dose. Margin gains are relative to the parent; broad NLL cost is relative to ordinary continuation.

Cue view	Parent	Correct pairing	Wrong pairing
Exact	$-0.07528$	$-0.03799$	$-0.12884$
Unseen paraphrase	$-0.03448$	$-0.00490$	$-0.04868$

Appendix

Table 23: Relation controls at the low-dose operating point (

k=14

). Entries are no-source correct-minus-competing margins; wrong-pair training changes the main-relation margin by

-0.0536

while favoring the swapped assignment.

Arm	Active operation	M-only	Carrier A	Carrier B	A/B mean	Drag change
C0	matched controls	0.636	0.901	0.887	0.894	$-0.927$
C1	Write	0.740	0.975	0.967	0.971	$-0.741$
C2	Use	0.723	1.213	1.189	1.201	$-1.814$
C3	Write + Use	0.806	1.307	1.282	1.294	$-1.751$

Appendix

Table 24: Natural Write

\times

Use factorial. Gains are parent-minus-continuation NLL; larger is better. A/B mean is the natural-use gain in Table 2 . Use trains carrier A; B changes the prompt around the same answer-free background and frozen facts. All arms share two intervention slots and a 25M-token continuation, with matched non-drag facts in disabled slots. Parent, ordinary rows, loss, optimizer positions, updates, and budget are fixed.

Population	Contrast	Value
Dragged	Write main effect	0.085
Dragged	Use main effect	0.315
Dragged	Write $\times$ Use interaction	0.016
Non-drag control	Write-only drag change	$-0.358$
Non-drag control	Use-only drag change	$+0.393$
Frozen Store (Store41)	Target-NLL decrease	1.903

Appendix

Table 25: Store audit summaries for separate populations. The first block reports natural-use NLL effects on dragged facts; the second reports drag change on matched non-drag controls (negative means less suppression). The final block reports target-NLL decreases from 3B to 100B on frozen class rosters.

Figure 9: Use data construction. In the illustrative conflict, the passage says Rome while established knowledge says Paris. Changing task authority changes the prescribed answer. Preserve that conditional relation across facts and query forms; long-window selection instead concentrates full–local opportunity. Table 26 gives paired, unpaired, and reversed constructions.

Fact	Question	Edited passage	Knowledge
A	What is the capital of France?	Capital: Rome	Paris
B	What is the capital of Germany?	Capital: Vienna	Berlin

Appendix

Table 26: Illustrative paired, unpaired, and reversed training records.

C

authorizes the passage and

M

established knowledge. Each listed arrow gives the supervised target; the other answer is its competitor. For readability, both illustrative records are assigned

t=T_{0}

. The construction follows the implemented allocation; the facts illustrate it rather than report model outputs.

Checkpoint	Arm	Override	Reject	Strict pair	Strict family
+100M	Ordinary	73.57	23.23	4.27	0.10
+100M	Balanced unpaired	73.47	34.70	9.43	1.00
+100M	True paired	73.53	39.67	13.83	1.90
+150M	Ordinary	74.77	22.63	4.47	0.00
+150M	Balanced unpaired	74.10	35.37	10.60	0.80
+150M	True paired	77.13	37.03	14.87	2.90

Appendix

Table 27: Familiar-query positive control over 1,000 unseen fact families. Accuracy is reported as a percentage; strict family requires every registered surface for a family.

Round	Ordinary@100M	Paired@100M	Ordinary@150M	Paired@150M
3 templates	7.00	5.45	6.55	4.55
32 templates	2.35	2.95	2.60	2.70
10 $\times$ families	6.70	2.95	6.60	2.95
Pair-worst	6.70	3.75	6.60	3.85

Appendix

Table 28: Strict-pair accuracy (%) under independently phrased queries. Each repair is evaluated at two saved points.

Checkpoint	Arm	4K	8K	12K
Midpoint	LongCE	4.2271	4.5152	4.4247
Midpoint	Score-shuffled	4.1894	4.4917	4.4060
Midpoint	CCUR	4.1479	4.4368	4.3580
Endpoint	LongCE	3.5081	3.7562	3.8365
Endpoint	Score-shuffled	3.4721	3.7069	3.7906
Endpoint	CCUR	3.4103	3.6264	3.7179

Appendix

Table 29: Document-disjoint data-pool replication. All arms share the parent, LongCE objective, matched strata, 92 updates, and 96.47M tokens. Lower answer-token NLL is better.

Length	Control	NLL gain	95% interval
4K	LongCE	0.0979	[0.0928, 0.1031]
4K	Score-shuffled	0.0618	[0.0574, 0.0663]
8K	LongCE	0.1298	[0.1223, 0.1374]
8K	Score-shuffled	0.0806	[0.0741, 0.0872]
12K	LongCE	0.1186	[0.1109, 0.1264]
12K	Score-shuffled	0.0727	[0.0665, 0.0790]

Appendix

Table 30: Paired endpoint uncertainty for the document-disjoint training pool. Positive control-minus- CCUR NLL favors CCUR . Intervals resample complete evaluation cases within task, preserving answer-token aggregation (50,000 draws).

View/test	Full	Local only	Full $-$ local	Source removal	Swap winner	Swap value
Gain	.00789	$-.00634$	.01424	$-.00080$	$-.00490$	$-.00761$

Appendix

Table 31: Use-localization controls on the original pool. Values are CCUR -minus-LongCE truth-log-probability gains.

Checkpoint	Arm	Context override	Memory reject	Both
Midpoint	LongCE	38.55	59.60	1.70
Midpoint	CCUR	38.90	58.65	1.15
Endpoint	LongCE	40.55	55.85	1.30
Endpoint	CCUR	41.10	55.00	1.20

Appendix

Table 32: Fixed 1,000-family context-versus-memory panel. Accuracy is reported as a percentage; Both requires both query modes for one family.

Audit	Arm or class	Value
Route offset	Ordinary	0.0282
	True paired	0.1009
	Reversed relation	$-0.0884$
Overall PPL	LongCE	19.3240
	CCUR	19.1569
BridgeSet NLL gain	Circuit	.00281

Appendix

Table 33: Additional Use diagnostics. Blocks report separate populations and metrics: an independent matched relation panel at +100M, Overall PPL at the document-disjoint endpoint, and a domain-matched 96-case BridgeSet. BridgeSet entries are NLL gains; specificity subtracts the mean gain of the other classes.

Stage	Condition	Result
Gate	Empty target, replacement character, or no alphanumeric character	Invalid
	$a\geq 0.25$ or source-only target rank $=1$	Available source
	$r_{F}\geq 2$ or full NLL $\geq 2$ or $m_{F}<0$	Bad output
	Bad output and $[w\geq 0.05$ or $c\geq 0.05$ or $(l\geq 0.10\land r_{L}<r_{F})]$	Actionable
	Valid but nonactionable	Nonactionable
Strong rules, in precedence order

Appendix

Table 34: Frozen conflict-classifier rules. Quantities

a,c,w,l

are defined in Appendix B ;

r_{F}

and

r_{L}

are full and best-local target ranks, and

m_{F}

is the full-view margin against the strongest wrong candidate. Within each ordered block, apply the first matching rule. Frozen adjudication uses 360 representative records.

Item	Setting
Teacher parameters	,746,176
Teacher checkpoints	Binding: 200.278M tokens; matching formation: 1.501B; endpoint: 3.000B
Matching components	Teacher Q/K heads L9H5 and L9H13
Prefix allocation	Joint selection 40%; matching-only 60%
Primary prefix	,701 joint + 352,051 matching-selected sequences; 1.2017B tokens
Domain slots	Code 10%, STEM 10%, Books 25%, Papers 15%, English Web 25%, Chinese Web 15%

Appendix

Table 35: Circuit data and protocol settings. These are fixed experimental instances of the construction described in the text; teacher checkpoints and target-model checkpoints are distinct.

Item	Setting
M-missing gate	,000 candidates; 11,997 valid; 2,220 M-missing
Natural-use filtering	,468 complete-view candidates; 7,247 M-available; 2,212 strict-drag cases
Natural factorial	dragged facts + 512 matched non-drag controls
Slot schedule	Write and Use slots, each repeated 3 times among 24 updates
Packed row	Four independently attended 512-token segments; one supervised target/segment
Parent / learning rate	Same 100B model and optimizer; $2\times 10^{-5}$

Appendix

Table 36: Store and paired-Use protocol settings. Gate candidates, natural invocation cases, and route-evaluation families are separate populations.

Item	Setting
Full window / scored positions	,384 tokens / 4,096–16,383 ( $12,288$ targets)
Target blocks / local history	Twelve 1,024-target blocks / 4,096 preceding tokens
Context extension	K $\rightarrow$ 16K; linear RoPE scaling by 8 ( Chen et al., 2023b )
Learning rate / batch	$2\times 10^{-5}$ / 64 complete windows per update
Original window pool	,170 windows across 8 source shards
Difficulty strata	High-full-NLL half of each shard; 20 equal-frequency bins

Appendix

Table 37: Long-context scoring, selection, and evaluation settings. The independent pool excludes all original training documents. All arms within a pool share the stated configuration.

Figure 10: Continuous branches with a shared initialization. A/B differ in early pretraining order. Target and control Store write matched but different facts. The paired/unpaired comparison shares a target-Store parent. Only arrows denote model-state inheritance.

Stage	Updates	Effective batch	Input positions	Learning rate
Pretraining (each A/B)	2,861	512	2,999,975,936	shared schedule
Store (each parent)	381	64	49,938,432	$3\times 10^{-5}$
Use (ordinary stream)	96	512	100,663,296	$2\times 10^{-5}$

Appendix

Table 38: Sequential training settings. Positions are model-input positions, not parameter counts or GPU costs.

Partition	Facts	Store-taught	Use-taught
Use teaching	384	yes	yes
Common tracking	640	yes	no
Development	256	yes	no

Appendix

Table 39: Fixed target-fact split. The main sequential results use only the 640 common tracking facts.

Route	Node	Reading NLL	Memory NLL	Joint-use NLL
A	Pretraining	3.156392	6.025372	3.863549
A	Store	2.179583	4.426465	3.192157
A	Use	2.170958	4.371545	3.050034
B	Pretraining	3.136392	5.995372	3.839549
B	Store	2.139583	4.346465	3.132157
B	Use	2.130958	4.291545	2.970034

Appendix

Table 40: All 18 stage-trajectory values on the same 640 common facts. A uses ordinary pretraining; B uses the early Circuit curriculum. Both then receive target Store and paired Use. Lower answer-token NLL is better.

Contrast	A	B	B minus A
Store: memory before minus after	1.599	1.649	0.050
Use: joint-use before minus after	0.142	0.162	0.020

Appendix

Table 41: Within-stage NLL reductions on the common facts. The final column is the difference of the two observed improvements, not a fitted interaction or significance estimate.

Route	Memory NLL	Joint-use NLL	Pair accuracy	Pair-NLL wins
Complete sequence	4.292	2.970	5.26%	—
No early curriculum	4.372	3.050	3.39%	58.28%
Control facts in Store	5.220	3.245	1.41%	79.74%
Unpaired Use	4.289	3.019	3.18%	55.10%
Ordinary continuation	6.026	3.870	0.99%	91.77%

Appendix

Table 42: All five final routes on the common task. Pair-NLL win rate sums the two full-answer NLLs within a query pair and counts when the complete route is lower than the listed control; ties are not wins. Its denominator is 1,920 pairs, and it is distinct from candidate accuracy.

Node or control	Reading margin	Memory margin	Pair NLL	Both correct
A pretraining	+1.375	$-0.469$	5.144	no
B pretraining	+1.450	$-0.375$	5.100	no
B + target Store	+6.250	+8.938	3.300	no
Complete sequence	+5.062	+7.031	2.850	yes
No early curriculum	+4.188	+5.312	3.011	no
Unpaired Use	+4.312	+6.469	2.975	no

Appendix

Table 43: One fact across the continuous chain and controls. Margins are authorized-minus-competing first-divergent-token scores in the reading and no-evidence views. Pair NLL sums the two authority decisions for the fixed query pair.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 441 papers

Capacity-Dependent Effects of Data Selection for Reasoning
Aug 13, 2026Cuong Dang, Hoang Anh Just, Ruoxi JiaInstruction-Tuning Data Selection Supervised Fine-Tuning
Department of Electrical and Computer Engineering Virginia Tech Blacksburg, VA, USA
Base Models Can Reason By Taking a Cue From Training Data
Oct 5, 2026Sophie L. Wang, Amil Dravid, Rulin Shao +3LLM Prompting RL for Language Model Reasoning
MIT · UC Berkeley · University of Washington +1
Beyond Correctness: Compute-Responsive Data Selection for Reasoning Model Post-Training
Oct 14, 2026Liang He, Xiangyu Li, Yuxin Wang +4Training Data Selection Language Model Post-Training

Aug 13, 2026J/K move · Enter open · S save

Capacity-Dependent Effects of Data Selection for Reasoning

Cuong Dang, Hoang Anh Just, Ruoxi Jia

Department of Electrical and Computer Engineering Virginia Tech Blacksburg, VA, USA

Instruction-Tuning Data Selection Supervised Fine-Tuning Selective Knowledge Distillation LLM Reasoning Knowledge Distillation

Open paper AskPDF arXiv

In reasoning supervised fine-tuning, candidate responses for the same instruction can differ substantially in how well they match the student's current distribution. Recent likelihood-based response selection methods suggest that responses closer to the student distribution provide more effective supervision, motivating the hypothesis that high-likelihood responses may generally be preferable for fine-tuning. In this paper, we revisit this intuition and show that the value of likelihood-based data selection depends critically on model capacity and training duration. Through controlled experiments on mathematical reasoning, using students ranging from 1.5B to 8B parameters and supervision generated by stronger teacher models, we observe a clear capacity-dependent ``{\color{SMALLCOLOR}Fast-Fit} / {\color{LARGECOLOR}Slow-Gain}'' pattern. High-likelihood data provides faster and more stable early improvements, especially for smaller models, but low-likelihood data becomes increasingly beneficial for larger models when training is allowed to continue longer. To explain this phenomenon, we analyze learning dynamics, showing that small models often fail to absorb low-likelihood supervision and instead fall into shallow or repetitive behaviors, while larger models are better able to move toward the teacher distribution under such data. We further provide a capacity-constrained theoretical view of distillation that clarifies how data difficulty, data span, and student capacity jointly govern transfer. Overall, our findings show that effective data selection for reasoning should be aware of model capacity and computing budget rather than based on a single universal preference for high-likelihood supervision.

Abstract

Language Model Pretraining

Explore similar work

Capacity-Dependent Effects of Data Selection for Reasoning

Base Models Can Reason By Taking a Cue From Training Data

Beyond Correctness: Compute-Responsive Data Selection for Reasoning Model Post-Training