Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

Zhuohan Wang; Haoran Ma; Tianyu Wu; Yuanlin Duan; Zichun Liao; Jieming Yu

cs.CLSep 26, 2026

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

Authors: Zhuohan Wang, Haoran Ma, Tianyu Wu, Yuanlin Duan, Zichun Liao, Jieming Yu

Organizations: Harvard University · Rutgers University · The Hong Kong University of Science and Technology

Abstract

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.

Figures & tables

Figure 1: OracleLadder workflow. A teacher compiles a fixed milestone roadmap from the parent problem and its reference solution

G(P)

. The student is tested on Track A (milestone-only) and on Track B (the parent under help levels

C_{1}

C_{2}

C_{3}

). Crossing the two tracks gives the reasoning-gap diagnosis.

Panel of Figure 2 — Figure 2: Worked example of the two diagnostic measurements (Qwen-base on family cabf0726 , numbers from the released 16K panel). The model solves all tested milestones in isolation, but the parent is solved only when milestone answers are supplied, yielding a milestone-execution gap .

Model	Direct accuracy	Solved directly	Solved by any probe	Added by milestones	MATH score
Qwen-base	5.4%	21.5%	42.9%	21.5%	45.4%
OutcomeRL-2K	10.7%	29.9%	48.0%	18.1%	65.4%
MilestoneRL-2K	10.6%	27.1%	46.6%	19.5%	63.4%
gpt-oss-20b	25.7%	45.5%	60.5%	15.0%	88.6%
Llama-70B	7.8%	14.7%	35.6%	20.9%	49.5%
DeepSeek-V3.1	28.4%	40.4%	51.7%	11.3%	85.5%

Table 1: Same-set summary on the 354 milestone families. The table separates direct solving, diagnostic recovery, and external MATH performance.

Figure 3: Problem-matched milestones drive recovery, and scalar accuracy does not determine diagnostic recovery. (a) MilestoneRL-2K specificity check: correct milestone information vs. random, generic, and mismatched controls. (b) Cross-model: per-rollout direct accuracy vs. family-level any-probe recovery on the same 354 families.

Condition	Main set ( $n{=}354$ )	MATH500 ( $n{=}35$ )	AIME 2024/25 ( $n{=}38$ )
$C_{1}$ direct	96	3	2
$C_{2}$ -random	78	2	5
$C_{2}$ -generic	96	2	5
$C_{2}$ -correct	123	15	17
$C_{3}$ -mismatched	119	13	14
$C_{3}$ -gold	140	15	15

Table 2: The roadmap effect replicates beyond the main set. Families solved under each condition. The main set uses MilestoneRL-2K. The new datasets use Qwen3-8B (thinking disabled) as screening anchor and student. AIME uses the default threshold of at least 1 correct attempt out of 8. MATH500 uses at least 3 of 8, because at the default threshold direct prompting already recovers 16 of its 35 families, which passed a screen of only 4 attempts (Appendix N gives every threshold).

Figure 4: Largest non-direct category across all six models: pass the milestone test, but no parent probe yields a correct rollout. Reasoning-gap taxonomy on the 354-family diagnostic set. See Section 4.5 for the audit reading.

Figure 5: Similar RL gains can move families through different parent-probe outcomes. (Left) Parent-probe outcomes for Qwen-base and the two RL runs. (Right) Change in each outcome relative to Qwen-base.

Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Anchor (subset)	Qwen- base	Outcome- RL-2K	Milestone- RL-2K	gpt-oss- 20b	Llama- 70B	DeepSeek- V3.1
Qwen-base ( $n{=}236$ )	—	140	152	102	142	99
OutcomeRL-2K ( $n{=}208$ )	144	—	150	100	132	101
MilestoneRL-2K ( $n{=}226$ )	152	145	—	103	139	104
gpt-oss-20b ( $n{=}165$ )	108	103	114	—	97	96
Llama-70B ( $n{=}230$ )	135	123	126	89	—	91
DeepSeek-V3.1 ( $n{=}156$ )	103	100	106	86	96	—

Appendix

Table 3: Per-anchor composition-gap counts with the 16K milestone test. Rows are anchor models, with the anchored subset size in parentheses. The anchored subset holds the families of the 354-family set where the anchor passes the milestone test and fails all 8

C_{1}

attempts. Each cell counts the families in that subset where the column model passes the milestone test and is Unrecovered under all three parent probes. A dash marks the anchor’s own column, which is left unscored. The milestone test and the parent probes both run at max_tokens=16384 with

K{=}8

Anchor	Qwen- base	Outcome- RL-2K	Milestone- RL-2K	gpt-oss- 20b	Llama- 70B	DeepSeek- V3.1
Qwen-base	1.00	+0.67	+0.33	+0.67	+0.67	+0.67
OutcomeRL-2K	+0.67	1.00	+1.00	+1.00	+0.67	+1.00
MilestoneRL-2K	+0.33	+1.00	1.00	+1.00	+1.00	+1.00
gpt-oss-20b	+0.67	+1.00	+1.00	1.00	+0.67	+1.00
Llama-70B	+0.67	+0.67	+1.00	+0.67	1.00	+0.67
DeepSeek-V3.1	+0.67	+1.00	+1.00	+1.00	+0.67	1.00

Appendix

Table 4: Kendall’s

\tau

between prober rankings under two anchor screens, with the 16K milestone test. Within each anchor screen, the five other models are ranked by their composition-gap counts in Table 3 . Each cell compares two anchors on the four models that are anchor in neither screen. The diagonal is 1 by definition. The mean over the 15 anchor pairs is

+0.80

, and all 15 are positive.

Anchor (subset)	Qwen-b.	Outc-2K	Mile-2K	gpt-oss	Llama-70B	DSeek
Qwen-b. ( $n{=}235$ )	—	143	150	103	144	104
Outc-2K ( $n{=}209$ )	147	—	146	102	126	101
Mile-2K ( $n{=}223$ )	154	144	—	106	135	104
gpt-oss ( $n{=}163$ )	110	103	115	—	91	99
Llama-70B ( $n{=}222$ )	137	119	123	84	—	86
DSeek ( $n{=}154$ )	103	97	104	89	88	—

Appendix

Table 5: Per-anchor composition-gap counts. Rows are anchor models, and columns are non-anchor probers. Anchored subset size in parentheses next to the anchor name. The cell is the count of families in the anchored subset for which the prober passes Stage 0 and is Unrecovered under all three parent probes. Anchor and prober Stage 0 readings are both at the 4K screening budget (the budget used during diagnostic-set construction) to keep anchor-vs-prober comparable.

Anchor	Qwen-b.	Outc-2K	Mile-2K	gpt-oss	Llama-70B	DSeek
Qwen-b.	1.00	+0.67	+0.33	+0.33	+1.00	+0.33
Outc-2K	+0.67	1.00	+1.00	+0.33	+0.67	+0.33
Mile-2K	+0.33	+1.00	1.00	+0.67	+0.67	+0.67
gpt-oss	+0.33	+0.33	+0.67	1.00	+0.67	+1.00
Llama-70B	+1.00	+0.67	+0.67	+0.67	1.00	+0.67
DSeek	+0.33	+0.33	+0.67	+1.00	+0.67	1.00

Appendix

Table 6: Kendall’s

\tau

between non-anchor prober ranks across anchor choices. Each cell compares the rank order of probers (by composition-gap count) under two different anchor screens, computed on the four probers the two anchor pairs share. Diagonal is

1

by definition.

Type	Role	%
Key_Move	Critical algebraic/geometric step	50.7
Model	Problem modeling or formulation	24.6
Compute	Direct computation	16.2
Normalize	Standardize representation	6.4
Other	Lemma, Warmup, Sanity	2.1

Appendix

Table 7: Milestone types and compiled-milestone distribution across the diagnostic set.

MilestoneRL-2K state	$n$	Key_Move	Model	Compute	Normalize	Other
Direct	96	86%	44%	27%	15%	6%
Roadmap-Needed	47	83%	57%	45%	6%	6%
Answers-Needed	22	82%	59%	45%	9%	5%
Unrecovered	189	86%	48%	32%	17%	3%

Appendix

Table 8: Milestone-type composition of families in each MilestoneRL-2K state. Each cell is the percentage of families in that state whose milestone set contains at least one milestone of the given type (rows are not mutually exclusive, and a family can contribute to multiple columns).

$\tau$	Direct	Roadmap-Needed	Answers-Needed	Unrecovered
1 (default)	96	47	22	189
2	65	44	19	226
3	46	40	17	251
4	36	27	21	270
5	26	21	21	286

Appendix

Table 9: Family state counts as the threshold

\tau

varies (MilestoneRL-2K,

n{=}354

). Row sum = 354 throughout. Unrecovered dominates for every

\tau

Condition	Solved	sr	$\Delta$ vs $C_{1}$	paired $p$
$C_{1}$ direct	96	.106	—	—
$C_{2}$ -random	78	.072	$-$ 18	$0.010$
$C_{2}$ -generic	96	.090	$0$	$1.00$
$C_{2}$ -correct	123	.145	+27	$\boldsymbol{0.001}$
$C_{3}$ -mismatched	119	.137	+23	$0.011$
$C_{3}$ -gold	140	.195	+44	$\boldsymbol{3\times 10^{-7}}$

Appendix

Table 10: MilestoneRL-2K corruption controls on the

n{=}354

main set. sr is per-rollout success rate.

C_{2}

-random and

C_{2}

-generic do not improve over direct prompting.

C_{3}

-mismatched lifts above

C_{1}

but stays below

C_{3}

-gold (the gap reflects the additional value of problem-matched milestone answers). Only

C_{2}

-correct and

C_{3}

-gold reach substantial lift. (Section 4.1 reports the headline numbers in prose.)

Model	$C_{2}$ -corr vs $C_{1}$	vs $C_{2}$ -rand	vs $C_{2}$ -gen	$C_{3}$ -gold vs $C_{3}$ -mism
Qwen-base	$+31$ , $2.4\text{e-}4$	$+51$ , $8.6\text{e-}9$	$+35$ , $8.2\text{e-}5$	$+18$ , $0.015$
OutcomeRL-2K step 180	$+29$ , $7.7\text{e-}4$	$+28$ , $7.6\text{e-}4$	$+27$ , $0.003$	$+11$ , $0.144$
MilestoneRL-2K step 180	$+27$ , $0.001$	$+45$ , $2.4\text{e-}7$	$+27$ , $0.002$	$+21$ , $0.005$
gpt-oss-20b	$+22$ , $0.003$	$+20$ , $0.010$	$+19$ , $0.011$	$+7$ , $0.324$
Llama-3.3-70B-Inst.	$+42$ , $4.5\text{e-}10$	$+46$ , $3.2\text{e-}10$	$+34$ , $2\text{e-}6$	$+27$ , $4.6\text{e-}4$
DeepSeek-V3.1	$+26$ , $4.2\text{e-}5$	$+29$ , $9\text{e-}6$	$+17$ , $0.006$	$+3$ , $0.701$

Appendix

Table 11: Per-model paired exact McNemar on the 354-family panel. Each cell:

\Delta

(families the correct condition solves but the control does not, minus the reverse), two-sided exact

p

-value. Roadmap-specific lift is significant for all six models.

C_{3}

-gold vs. mismatched saturates for OutcomeRL-2K, gpt-oss-20b, and DeepSeek-V3.1.

Figure 6: Specificity across all six models on the

n{=}354

diagnostic set. Solid bars are problem-matched conditions (

C_{1}

direct in gray,

C_{2}

-correct in blue,

C_{3}

-gold in green). Hatched red bars are control conditions (

C_{2}

-random,

C_{2}

-generic,

C_{3}

-mismatched). Numbers above bars: families solved with

\geq 1/8

correct rollouts. No control exceeds its matched correct counterpart. Per-model paired

p

-values are in Table 11 .

Model	direct	Any probe	Any-probe rank
	(mean [95% CI])	(mean [95% CI])	(95% CI range)
Qwen-base	76.0 [61, 92]	152.4 [134, 171]	5 [4, 5]
OutcomeRL-2K	106.0 [90, 123]	170.3 [153, 189]	3 [2, 4]
MilestoneRL-2K	95.8 [80, 112]	165.2 [147, 184]	4 [3, 4]
gpt-oss-20b	160.9 [143, 178]	213.9 [196, 232]	1 [1, 1]
Llama-70B	51.8 [39, 64]	126.0 [109, 145]	6 [6, 6]

Appendix

Table 12: Bootstrap 95% CIs on fingerprint counts (1000 resamples of 354 families with replacement). direct count and any-probe count (

=\textsc{Direct}+\textsc{Roadmap-Needed}+\textsc{Answers-Needed}

) with central tendency and

[2.5,97.5]

percentile interval.

	Qwen-base	OutcomeRL-2K	MilestoneRL-2K	gpt-oss-20b	Llama-70B	DeepSeek-V3.1
Qwen-base	—	0.01	0.02	0.00	1.00	0.00
OutcomeRL-2K	0.99	—	0.81	0.00	1.00	0.10
MilestoneRL-2K	0.97	0.15	—	0.00	1.00	0.03
gpt-oss-20b	1.00	1.00	1.00	—	1.00	1.00
Llama-70B	0.00	0.00	0.00	0.00	—	0.00
DeepSeek-V3.1	1.00	0.89	0.96	0.00	1.00	—

Appendix

Table 13: Pairwise ordering frequency on the any-probe count: fraction of 1000 bootstrap resamples (family-level) where the row model’s any-probe count exceeds the column model’s. Values near 0.5 indicate unstable order, and values near 1.0 indicate stable dominance.

Model	Milestone solve rate	Passing families	All families	Passing families only
Qwen-base	93%	306	76/50/26/202	70/42/23/171
OutcomeRL-2K step 180	92%	296	106/50/14/184	88/44/9/155
MilestoneRL-2K step 180	95%	314	96/47/22/189	88/40/19/167
gpt-oss-20b	93%	306	161/36/17/140	141/33/14/118
Llama-3.3-70B-Inst.	87%	268	52/46/28/228	38/38/21/171
DeepSeek-V3.1	87%	264	143/33/7/171	108/25/6/125

Appendix

Table 14: Stage 0 milestone solvability at max_tokens=16384 ,

K{=}8

. A family passes the milestone test iff the model solves every non- Integrate milestone with

\geq 1/8

correct rollouts. Fingerprint cells: Direct / Roadmap-Needed / Answers-Needed / Unrecovered .

Model	Pass (4K)	Pass (16K)	Pass flipped	Comp. gap (4K)	Comp. gap (16K)
Qwen-base	303	306	43	173	171
OutcomeRL-2K	300	296	38	155	155
MilestoneRL-2K	308	314	36	167	167
gpt-oss-20b	304	306	22	120	118
Llama-70B	261	268	39	164	171
DeepSeek-V3.1	256	264	60	127	125

Appendix

Table 15: Stage 0 replication at the 16K decoding budget. Pass counts are families that pass the milestone test (every non- Integrate milestone solves at least once at

K{=}8

). Composition-gap counts are families that pass Stage 0 and are Unrecovered under all three parent probes.

Model	Direct	Roadmap	MS-Exec	Composition	Missing-MS	Capability
Qwen-base	76	42	23	171	11	31
OutcomeRL-2K	106	44	9	155	11	29
MilestoneRL-2K	96	40	19	167	10	22
gpt-oss-20b	161	33	14	118	6	22
Llama-70B	52	38	21	171	15	57
DeepSeek-V3.1	143	25	6	125	9	46

Appendix

Table 16: Reasoning-gap taxonomy: Stage 0 milestone test (pass/fail)

\times

smallest help needed to unlock the parent, per model on the 354-family diagnostic set. Counts are families. Both Stage 0 and parent probing at max_tokens=16384 ,

K{=}8

	MATH hard-100		Diagnostic fingerprint, $n{=}354$
Model	Score	p@8	Direct	Roadmap	Answers	Unrec.	Any probe
Qwen-base	0.45	0.75	76	50	26	202	152
OutcomeRL-2K	0.65	0.88	106	50	14	184	170
MilestoneRL-2K	0.63	0.85	96	47	22	189	165
gpt-oss-20b	0.89	0.97	161	36	17	140	214
Llama-70B	0.49	0.68	52	46	28	228	126

Appendix

Table 17: External MATH score and diagnostic fingerprint. MATH hard-100 provides an external scalar comparator. The fingerprint columns summarize the 354-family diagnostic set.

	MATH score	Direct	Roadmap-Needed	Answers-Needed	Any probe
Qwen-base	0.454	76	50	26	152
OutcomeRL-2K step 180	0.654 ( $+20.0$ )	106 ( $+30$ )	50 ( $+0$ )	14 ( $-12$ )	170 ( $+18$ )
MilestoneRL-2K step 180	0.634 ( $+18.0$ )	96 ( $+20$ )	47 ( $-3$ )	22 ( $-4$ )	165 ( $+13$ )

Appendix

Table 18: Numerical values behind Figure 5 (Section 4.4 ): two-arm longitudinal readout on the 354-family diagnostic set at max_tokens=16384 ,

K{=}8

, starting from the same Qwen-base checkpoint. MATH hard-100 pass@1 shows both RL arms gaining comparably. The fingerprint shows them moving differently.

Figure 7: Training trajectories move through different diagnostic states. Stacked areas show parent-probe outcomes across checkpoints. Numbers above each bar are families solved by any probe. The MATH hard-100 pass@1 trajectory over the same window is in Appendix L .

Dimension	Shared / different?	Summary
Base model, optimizer, algorithm, horizon	shared	Qwen-base, GRPO, AdamW LR $1{\times}10^{-5}$ , 180 steps
Reward signal	different	parent correctness only vs. parent + milestone rollouts
Per-step rollout budget	different	$1{,}024$ vs. $512$ ( $\sim$ 184K vs. $\sim$ 92K total rollouts)
SFT warmup	different	none vs. 3-epoch R1-distilled CoT warmup
Training-data composition / curriculum	different	details in artifact (originals only vs. originals + milestones, distinct $\hat{p}$ filters)

Appendix

Table 19: Training-recipe dimensions of OutcomeRL-2K and MilestoneRL-2K. The two runs are not matched in compute.

Figure 8: Cross-model parent-probe outcome fingerprints on the

n{=}354

diagnostic set. Each bar partitions families by the smallest help level that unlocks the parent: Direct (

C_{1}

), Roadmap-Needed (

C_{2}

), Answers-Needed (

C_{3}

), or Unrecovered . The aggregate counts behind each bar are reported in Table 21 .

Model	Lineage	Parameters	Architecture	Training regime
Qwen-base	Qwen3	8B	dense	base
OutcomeRL-2K	Qwen3	8B	dense	RL-tuned (outcome)
MilestoneRL-2K	Qwen3	8B	dense	RL-tuned (milestone)
gpt-oss-20b	gpt-oss	20B (3.6B active)	MoE	reasoning-trained
Llama-3.3-70B-Instruct	Llama-3	70B	dense	instruction-tuned
DeepSeek-V3.1	DeepSeek-V3	671B (37B active)	MoE	reasoning-trained

Appendix

Table 20: Cross-model panel metadata. OutcomeRL-2K and MilestoneRL-2K are RL-trained from the Qwen-base checkpoint sharing base, LoRA rank 64, GRPO recipe, 180-step training horizon, and the 2,000-parent NuminaMath training pool. They differ in reward construction (outcome-only versus milestone-augmented) and in the other training details listed in Table 19 .

	$C_{1}$	$C_{2}$ -rand	$C_{2}$ -gen	$C_{2}$ -corr	$C_{3}$ -mism	$C_{3}$ -gold	Any probe
Qwen-base	76	56	72	107	103	121	152
OutcomeRL-2K step 180	106	107	108	135	122	133	170
MilestoneRL-2K step 180	96	78	96	123	119	140	165
gpt-oss-20b	161	163	164	183	187	194	214
Llama-3.3-70B-Instruct	52	48	60	94	73	100	126
DeepSeek-V3.1	143	140	152	169	160	163	183

Appendix

Table 21: Cross-model panel: families solved (

\geq 1

correct rollout out of

K{=}8

) on the 354-family diagnostic set. Any probe

=\textsc{Direct}+\textsc{Roadmap-Needed}+\textsc{Answers-Needed}

(out of 354) is the count of families solved by any probe.

	4K				16K
Model	Direct	R-Need	A-Need	Any	Direct	R-Need	A-Need	Any
Qwen-base	64	54	40	158	76	50	26	152
OutcomeRL-2K	79	50	27	156	106	50	14	170
MilestoneRL-2K	88	45	25	158	96	47	22	165
gpt-oss-20b	114	34	27	175	161	36	17	214
Llama-70B	54	48	26	128	52	46	28	126

Appendix

Table 22: Decoding-budget sensitivity: parent-probe outcome counts at max_tokens=4096 vs. 16384 on the same 354 families. OutcomeRL-2K, gpt-oss-20b, and DeepSeek-V3.1 gain 14 to 42 any-probe families at 16K. Qwen-base, MilestoneRL-2K, and Llama-3.3-70B-Instruct change by at most 7.

Step	0	15	30	45	60	75	90	105	120	135	150	165	180
MATH score	0.41	0.42	0.49	0.43	0.43	0.53	0.51	0.56	0.56	0.60	0.58	0.58	0.62

Appendix

Table 23: OutcomeRL-2K wandb single-rollout (

K{=}1

) MATH hard-100 pass@1 at 13 checkpoints, 15-step intervals, recorded at max_tokens=4096 (training-time validation budget, not the main parent-probing budget).

Model	Condition	$n$	no_boxed	no_boxed_trunc	unparsable	parsed_wrong	parsed_correct
Qwen-base	$C_{1}$	400	13.0%	2.5%	0.0%	80.0%	4.5%
Qwen-base	$C_{3}$ -gold	400	17.0%	3.8%	0.2%	62.3%	16.8%
MilestoneRL-2K	$C_{1}$	400	8.5%	32.5%	0.0%	51.0%	8.0%
MilestoneRL-2K	$C_{3}$ -gold	400	17.8%	12.5%	0.2%	51.0%	18.5%
Llama-3.3-70B-Inst.	$C_{1}$	400	7.2%	0.5%	0.0%	88.2%	4.0%
Llama-3.3-70B-Inst.	$C_{3}$ -gold	400	13.8%	0.0%	0.0%	71.0%	15.2%

Appendix

Table 24: Format / extraction-failure audit on a 50-family random subset. Percentages are of 400 rollouts per (model, condition) cell.

Condition	Solved	sr	$\Delta$ vs $C_{1}$	paired exact $p$
$C_{1}$ direct	13	.191	—	—
$C_{2}$ -random	12	.176	$-$ 1	$1.00$
$C_{2}$ -generic	13	.199	0	$1.00$
$C_{2}$ -correct	20	.438	+7	$\boldsymbol{0.016}$
$C_{3}$ -mismatched	19	.426	+6	$0.031$
$C_{3}$ -gold	20	.516	+7	$\boldsymbol{0.016}$

Appendix

Table 25: Held-out replication (

n{=}32

, MilestoneRL-2K step 180). sr is per-rollout success rate. Random and generic controls do not lift above

C_{1}

. Correct roadmap and roadmap-plus-answers do. The mismatched-answer control lifts above

C_{1}

but stays below

C_{3}

-gold.

	MATH500	AIME 2024/25	AIME 2025 slice
Problems screened	500	60	30
Kept by the screen (0 of 4 correct)	36 (7.2%)	38 (63.3%)	21 (70.0%)
Valid milestone families	36	38	21
Families after the structural filter	35	38	21
Mean tested milestones per family	4.7	5.2	6.1
Tested milestones solved alone ( $\geq$ 1 of 8)	47/163	75/199	43/128

Appendix

Table 26: Construction counts and five-way taxonomy on the second datasets. The student and screening anchor is Qwen3-8B (thinking disabled). Tested milestones are the non- Integrate milestones left after the leak rule. The six taxonomy rows count families and sum to the family count of each column.

Condition	Fam.	Attempts	Fam.	Attempts	Fam.	Attempts
	MATH500 ( $n{=}35$ )		AIME 2024/25 ( $n{=}38$ )		AIME 2025 slice ( $n{=}21$ )
$C_{1}$ direct	16	28 (10.0%)	2	3 (1.0%)	5	6 (3.6%)
$C_{2}$ -random	12	22 (7.9%)	5	7 (2.3%)	2	3 (1.8%)
$C_{2}$ -generic	16	27 (9.6%)	5	7 (2.3%)	4	8 (4.8%)
$C_{2}$ -correct	18	106 (37.9%)	17	74 (24.3%)	6	24 (14.3%)
$C_{3}$ -mismatched	17	91 (32.5%)	14	65 (21.4%)	5	16 (9.5%)

Appendix

Table 27: Recovery under the six conditions for Qwen3-8B (thinking disabled). Fam. counts families with at least 1 correct attempt out of 8 (pass@8). Attempts gives the correct attempts and their share, out of 280 (MATH500), 304 (AIME 2024/25), and 168 (AIME 2025 slice) attempts per condition.

	MATH500 ( $n{=}35$ )				AIME 2024/25 ( $n{=}38$ )
Condition	$\tau{=}1$	$\tau{=}2$	$\tau{=}3$	$\tau{=}4$	$\tau{=}1$	$\tau{=}2$	$\tau{=}3$	$\tau{=}4$
$C_{1}$ direct	16	8	3	1	2	1	0	0
$C_{2}$ -random	12	5	2	2	5	1	1	0
$C_{2}$ -generic	16	7	2	1	5	1	1	0
$C_{2}$ -correct	18	16	15	13	17	13	11	10
$C_{3}$ -mismatched	17	15	13	12	14	12	9	9

Appendix

Table 28: Family-level recovery as the success threshold

\tau

varies, for Qwen3-8B (thinking disabled). A family counts as recovered under a condition when at least

\tau

of its 8 attempts are correct. The column

\tau{=}1

is pass@8.

	Problems kept		Probed families kept
Anchor	MATH500	AIME 2024/25	MATH500	AIME 2024/25
gpt-oss-20b	5/500 (1.0%)	10/60 (16.7%)	4/35	9/38
Qwen3-8B (thinking disabled)	36/500 (7.2%)	38/60 (63.3%)	35/35	38/38
Llama-3.2-1B-Instruct	345/500 (69.0%)	58/60 (96.7%)	33/35	38/38

Appendix

Table 29: The same screen under three anchors. A problem is kept when the anchor fails all 4 direct attempts at max_tokens=16384 . Probed families kept counts the families of Table 27 that this anchor’s screen also keeps. Qwen3-8B keeps all of them by construction. Qwen3-8B (thinking disabled) and gpt-oss-20b run on Tinker. Llama-3.2-1B-Instruct runs on a hosted inference API with the provider’s chat template. For gpt-oss-20b on MATH500, a restarted run logged some attempts twice, and we keep the first record of each attempt.

Component	Math	Code
Parent problem	Competition problem with a final answer	LiveCodeBench problem that reads standard input, with test cases
Verifier	Symbolic check of the boxed answer against the gold answer	Run the program on up to 12 test cases and compare its output with the expected output
Milestone	Sub-goal with a checkable gold answer	Helper function with a signature, a docstring, at least 2 assert tests, and a gold implementation
Milestone test (Stage 0)	Solve each milestone alone	Implement each helper from its signature, docstring, and tests, graded by its asserts
$C_{1}$	Parent problem only	Problem statement only
$C_{2}$	Parent plus the roadmap of milestone descriptions	Statement plus helper signatures, docstrings, and an overall plan

Appendix

Table 30: OracleLadder components in math and their counterparts in the LiveCodeBench pilot.

Condition	Passing attempts	Rate	Problems recovered
$C_{1}$ direct	4/62	6.5%	1/10
$C_{2}$ helper plan	33/80	41.2%	6/10
$C_{3}$ plan $+$ verified gold helpers	71/80	88.8%	10/10

Appendix

Table 31: Code-pilot results for Qwen3-8B (thinking disabled) on 10 LiveCodeBench problems, graded by test execution. Problems recovered counts problems with at least 1 passing attempt out of 8.

C_{1}

completed 62 of 80 planned attempts, so its rate uses 62 attempts.

	Inkling	GPT-5.4
Milestones per family, all types	8.1	5.8
Tested milestones per family	6.8	2.1
Compute share	23.0%	21.1%
Lemma share	14.3%	12.7%
Sanity share	12.9%	16.6%
Model share	12.6%	17.2%

Appendix

Table 32: Roadmap structure under the two teachers on the 53 families with a valid Inkling roadmap. Type shares count all milestones, including the private Integrate step. Tested milestones are the non- Integrate milestones left after the leak rule, which the

C_{2}

-correct and

C_{3}

-gold prompts show.

	Released vs. Inkling	Coarse vs. fine
	(53 families)	(33 families)
Tested milestones per family	2.1 / 6.8	2.0 / 8.1
Families recovered under $C_{2}$ -correct	23 / 20	11 / 10
Families recovered under $C_{3}$ -gold	24 / 19	13 / 12
Same $C_{2}$ outcome	44 (83%)	28 (85%)
Same $C_{3}$ outcome	46 (87%)	28 (85%)

Appendix

Table 33: Per-family recovery by gpt-oss-20b under two roadmaps of the same families (

K{=}8

, max_tokens=16384 ). A family is recovered under a condition when at least 1 of its 8 attempts is correct. Left: released GPT-5.4 roadmaps against Inkling roadmaps. Right: coarse against fine GPT-5.4 roadmaps from the controlled ablation. Pairs of counts list the first roadmap, then the second.

Released label	Inkling label	Strict test	Fraction score
Direct	Direct	22	22
Composition-gap	Composition-gap	0	7
Composition-gap	Capability-gap	11	4
Composition-gap	Missing-milestone-gap	1	0
Composition-gap	Roadmap-gap	0	1
Composition-gap	Direct	0	1

Appendix

Table 34: Five-way labels of gpt-oss-20b on 39 families under the released GPT-5.4 roadmaps and under Inkling roadmaps with atomic gold answers. Rows list each transition from the released label to the Inkling label. The strict milestone test requires every tested milestone. The fraction score requires at least half of them.

C_{1}

is sampled afresh under the Inkling roadmaps.

	All gold answers			Statement-novel gold answers
Model	Own	Control	Ratio	Own	Control	Ratio
Qwen-base	13.6%	3.8%	3.6 $\times$	9.4%	0.8%	12.3 $\times$
MilestoneRL-2K	17.2%	4.4%	3.9 $\times$	12.5%	1.2%	10.7 $\times$
Llama-3.3-70B-Instruct	14.5%	4.3%	3.4 $\times$	9.9%	0.8%	12.6 $\times$
DeepSeek-V3.1	15.2%	5.2%	2.9 $\times$	11.3%	1.7%	6.6 $\times$

Appendix

Table 35: Share of milestone and attempt pairs whose unaided (

C_{1}

) attempt contains the milestone’s gold answer, on 50 families with 8 attempts each. The control checks the same gold answers against the attempts on the other 49 families. Statement-novel gold answers are the 104 of 110 that the problem statement does not contain.

Model	Solved alone	Not solved alone
Qwen-base	14.6% (120/824)	0.0% (0/56)
MilestoneRL-2K	18.6% (150/808)	1.4% (1/72)
Llama-3.3-70B-Instruct	14.3% (117/816)	17.2% (11/64)
DeepSeek-V3.1	17.7% (130/736)	2.8% (4/144)

Appendix

Table 36: Share of milestone and attempt pairs whose unaided attempt contains the milestone’s gold answer, split by whether the same model solves that milestone in its 16K milestone test (

\geq

1 of 8 attempts). Counts are attained pairs over all pairs.

Model	All lengths	$\leq$ 25 characters	$\leq$ 15 characters
Qwen-base	1.28	0.92	0.82
MilestoneRL-2K	1.37	0.97	0.83
Llama-3.3-70B-Instruct	1.51	1.10	0.93
DeepSeek-V3.1	1.02	0.76	0.80

Appendix

Table 37: Containment rate of the released gold answers divided by the containment rate of Inkling’s gold answers for the same 39 families, on the same unaided attempts. Each column restricts both sets of gold answers to the given length.

	MilestoneRL-2K state
OutcomeRL-2K state	Direct	Roadmap-Needed	Answers-Needed	Unrecovered	Row total
Direct	84	13	5	4	106
Roadmap-Needed	8	27	6	9	50
Answers-Needed	1	3	6	4	14
Unrecovered	3	4	5	172	184
Column total	96	47	22	189	354

Appendix

Table 38: OutcomeRL-2K

\to

MilestoneRL-2K state transitions on the

n{=}354

main set at max_tokens

{=}

16384,

K{=}8

Figure 9: The unrecovered subset is mixed. Triage of the

n{=}100

analyzed Unrecovered families under the leak-safe repair procedure. Bar segments show the mutually exclusive audit categories. The pessimistic upper bound on composition-related failures follows the bookkeeping in Appendix S .

Rater category	Count	%	Repair action	Leak-safe?
Decomposition incomplete	50	50%	Regenerate, leak-safe filter	Yes
Verifier artifact (bad canonical)	26	26%	Re-solve + replace canonical	Yes
Genuine composition failure	18	18%	Keep as-is	—
Beyond model capability	5	5%	Keep as-is	—
Milestone not standalone	1	1%	Keep as-is	—
Total	100	100%

Appendix

Table 39: Human audit of 100 unrecovered families from the

n{=}354

main set (inter-rater

\kappa{=}0.49

, validated at

\kappa{=}0.64

on a 50-family subset). Categories are mutually exclusive.

Final triage category	Count
Artifact (verifier-repaired or leak-safe decomposition-repaired)	37
verifier-repair fixes (19/25 attempted verifier-artifact families unlocked)	19
leak-safe decomposition repairs (17/45 attempted families unlocked)	17
verifier-artifact whose repair was not pipeline-applicable	1
Composition-related or unresolved after repair	57
rater-confirmed composition failures, kept as-is	18

Appendix

Table 40: Final triage of the

n{=}100

unrecovered audit sample after leak-safe repair attempts. Top-level categories are mutually exclusive and sum to 100. The pessimistic upper bound on composition-related failures treats all 17 leak-safe decomposition repairs as plausibly suspect (residual risk of partial Final_Synthesis -style leakage even after the symbolic-equivalence filter).

Source	$n$	Direct ( $C_{1}$ )	Roadmap-Needed	Answers-Needed	Unrecovered
Leak-safe decomposition repair (45/50)	45	7	3	7	28
Verifier-repair (25/26 attempted)	25	10	6	3	6
Rater-confirmed genuine, re-run fingerprint	18	6	3	2	7
Kept beyond capability (re-run)	5	0	1	1	3
Kept not standalone (re-run)	1	0	0	1	0

Appendix

Table 41: Oracle re-evaluation after repair on the audited subset. Rows compare leak-safe decomposition repair, verifier repair, and the rater-confirmed genuine subset on a fresh oracle pass.

Model	Families	Flagged	Flagged %	Before	After	Before	After
	4K families labeled			Share, 4K		Share, 16K
Qwen-base	173	43	24.9	48.9	36.7	48.3	36.7
OutcomeRL-2K	155	39	25.2	43.8	32.8	43.8	32.5
MilestoneRL-2K	167	37	22.2	47.2	36.7	47.2	36.4
gpt-oss-20b	120	33	27.5	33.9	24.6	33.3	24.3
Llama-3.3-70B-Instruct	164	38	23.2	46.3	35.6	48.3	37.0

Appendix

Table 42: Automated rubric review of each model’s composition-gap families. Flagged families are those labeled verifier or format errors. The last four columns give the category’s share of the 354 families before and after removing the flagged families, on the 4K snapshot the review labeled and on the 16K sets of the main text with each family’s label carried over.

Family	Reference answer or format issue	Qwen3-8B	gpt-oss-20b
ff548f26	none	0	6
0ad0ce27	none	1	8
a0759695	wrong reference and wrong milestone answers	0	7
ad098d0c	two-part question, the cascade reads the last box	8	8
713ff8c1	wrong reference answer	8	6
a3a636a8	malformed reference, a milestone answer gives the answer	8	8

Appendix

Table 43: Raw-output retest of the 18 families that both raters labeled Genuine_Composition . Each model receives the

C_{3}

-gold prompt 8 times (

K{=}8

, max_tokens=16384 , temperature 0.7). Cells count attempts whose boxed answers state the correct parent answer, graded by hand against a verified answer. On 29eb7827, all 16 attempts state the correct order in words and 2 box it.

Family	Reference answer in the set	Defect and verified answer
713ff8c1	48884	wrong, largest palindrome is $49894=101\cdot 494$
a0759695	\frac{56\pi}{15}	wrong, minimum volume is $16\pi/15$
a3a636a8	\leqslant-2	variable missing, answer is $a\leq-2$
330cbc5a	=2,b=\frac{1}{4}	variable missing, answer is $a{=}2$ , $b{=}1/4$
c52e7627	x_{1}=\frac{15}{2},y_{1}=...	second solution $(3/2,1/2)$ missing
b36a1161	\1	garbled, answer is $x=\pm 1$

Appendix

Table 44: Defective reference answers found in the 354-family set. The first five families belong to the 18 audited genuine-composition families.

Schema of expected answer	Disagreements	Share	LLM over-accept rate in schema
Expr (free-form formula)	34	58%	34/156 = 22%
Number	24	41%	24/224 = 11%
Interval / Tuple	1	2%	1/12 = 8%
Equation , Set , other	0	0%	0/8 = 0%

Appendix

Table 45: Per-schema breakdown of the 59 strict-NOT_ACCEPT / LLM-ACCEPT disagreements. Expr (formulas) and Number dominate. For the frontier judge, disagreements of this kind are mostly correct answers that the cascade cannot parse (Appendix V.1 ).

Judge and responses	$n$	accept	reject	cascade rejects	judge rejects
		Both	Both	Judge accepts,	Cascade accepts,
8B judge, 400-response audit	400	217	124	59 (14.75%)	0 (0.0%)
gpt-5.5, all responses	997	167	702	127 (12.7%)	1 (0.1%)
gpt-5.5, NuminaMath parents	698	112	460	125 (17.9%)	1 (0.1%)
gpt-5.5, AIME 2024/25	299	55	242	2 (0.7%)	0 (0.0%)

Appendix

Table 46: Paired grades of the strict cascade and two LLM judges against the reference answer. The first row is the 400-response audit above. The other rows use gpt-5.5 on 997 responses, in total and by source.

Cascade failure mode	Disagreements	Sampled	Adjudicated	Cascade side	Judge side
Parse error	75	24	22	21	1
No boxed answer	35	11	9	1	8
Not equivalent	17	5	3	2	1
All modes	127	40	34	24	10
Weighted to all 127				86.8 (68%)	40.2 (32%)

Appendix

Table 47: Grader-blind adjudication of judge-accept, cascade-reject disagreements, by the cascade’s failure mode. The cascade side counts correct answers the cascade could not match and cases graded against a defective reference answer. The last row weights each failure mode by its share of the 127 disagreements.

Schema	Example	%
EXPR	$\boxed{\frac{a^{2}}{8}}$	44.6
EQUATION	$\boxed{y=13-2x}$	19.2
NUMBER	$\boxed{42}$	15.7
INEQUALITY	$\boxed{n\geq 7}$	6.1
SET	$\boxed{\{1,2,3\}}$	5.7
TUPLE_LIST	$\boxed{(1,2),(3,4)}$	4.9

Appendix

Table 48: Answer schema distribution in the pipeline (all milestones,

n{=}4654

Explore similar work

A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning
Sep 15, 2026Zhongdi Qu, Carla P. GomesMathematical Reasoning Circuits in Language Models
Cornell University
Math Reasoning in LLMs is Organized by Approach, Not Topic
Sep 22, 2026Sajad Goudarzi, Samaneh Zamanifard, Moloud Nasiri +1Mathematical Reasoning Benchmarks LLM Interpretability
Clemson University
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
Jun 8, 2026Yiteng Mao, Kenan Xu, Yijia Lyu +3Mathematical Reasoning Benchmarks LLM Evaluation
University of Wisconsin–Madison · East China Normal University · New York University +2

Sep 15, 2026J/K move · Enter open · S save

A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning

Zhongdi Qu, Carla P. Gomes

Cornell University

Mathematical Reasoning Circuits in Language Models Language Model Robustness LLM Mathematical Reasoning

Open paper AskPDF arXiv

Large language models solve grade-school math word problems with high accuracy, yet a single irrelevant clause inserted into the problem can collapse it. We reconcile these observations with a mechanistic account. We show that the model's internal computation decomposes into a four-stage sequential pipeline, Schema Abstraction, Operation Planning, Operand Binding, and Computation, each stage producing a distinct intermediate representation in an identifiable band of layers. Using the same scaffold to diagnose distractor-induced failure, we localize the corruption to a single stage, Operation Planning, implemented by a set of attention heads whose causal role we validate bidirectionally. In short, we provide a mechanistic interpretation of math word problem reasoning in LLMs, and their failure when distracted.

Abstract

Explore similar work

A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning

Math Reasoning in LLMs is Organized by Approach, Not Topic

RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning