Model Discovery Generation Digestion D i g . − D i s c . \mathrm{Dig.}-\mathrm{Disc.} Dig. − Disc. E x e c . − G e n . \mathrm{Exec.}-\mathrm{Gen.} Exec. − Gen. 82.42 82.42 82.42 67.03 67.03 67.03 100.00 100.00 100.00 86.81 86.81 86.81 59.34 59.34 59.34 50.55 50.55 50.55 97.80 97.80 97.80 71.98 71.98 71.98 41.76 41.76 41.76 44.51 44.51 44.51 97.25 97.25 97.25 68.68 68.68 68.68 34.62 34.62 34.62 43.41 43.41 43.41 91.21 91.21 91.21 60.99 60.99 60.99 24.73 24.73 24.73 52.75 52.75 52.75 92.31 92.31 92.31 78.57 78.57 78.57 28.57 28.57 28.57 47.80 47.80 47.80 93.41 93.41 93.41 71.98 71.98 71.98 13.19 13.19 13.19 38.46 38.46 38.46 79.67 79.67 79.67 61.54 61.54 61.54 6.59 6.59 6.59 26.37 26.37 26.37 70.88 70.88 70.88 56.04 56.04 56.04 6.59 6.59 6.59 17.03 17.03 17.03 68.68 68.68 68.68 39.56 39.56 39.56 6.04 6.04 6.04 18.13 18.13 18.13 65.38 65.38 65.38 43.41 43.41 43.41 4.40 4.40 4.40 15.93 15.93 15.93 67.03 67.03 67.03 39.01 39.01 39.01 7.14 7.14 7.14 13.19 13.19 13.19 48.90 48.90 48.90 32.42 32.42 32.42 We evaluate 12 models spanning three lineages and a broad capability range. The OpenAI lineage includes the closed-source GPT-5.4 ( OpenAI, 2026b ) , GPT-5.4-Mini ( OpenAI, 2026a ) , and GPT-5.4-Nano ( OpenAI, 2026a ) , alongside the open-weight GPT-OSS-20B ( OpenAI, 2025 ) . The Qwen lineage comprises Qwen-3.6-27B ( Qwen Team, 2026b ) and Qwen-3.5-{27B, 9B, 4B} ( Qwen Team, 2026a ) . The DeepSeek lineage features DeepSeek-R1-0528-Qwen3-8B ( DeepSeek-AI, 2025 ) and DeepSeek-R1-Distill-Qwen-{32B, 14B, 7B} ( DeepSeek-AI, 2025 ) . We use high reasoning effort when available and a maximum generation budget of 120k tokens per problem.
3.2 Results Analysis
Table 3.1 presents the comprehensive evaluation results of all models on Prim . Performance in the Discovery and Digestion dimensions is assessed by the model’s capacity to isolate the underlying mathematical primitive, utilizing the scoring protocol detailed in § 2.3 . Meanwhile, Generation and Execution are evaluated based on standard final-answer accuracy, following the HLE benchmark ( Center for AI Safety et al., 2026 ) . Further details on the diagnostic evaluation and additional analyses are provided in Appendix B .
Figure 2: Joint composition of Generation and Discovery outcomes across models. Finding 1: Answer accuracy masks distinct capability profiles. Final-answer accuracy conflates structural discovery with downstream execution. As shown in Figure 2 , Discovery and Generation are positively associated overall, yet models with similar solving accuracy can exhibit sharply different structural capabilities. For example, Qwen3.6-27B and gpt-5.4-mini achieve comparable Generation performance, while differing by over 30 30 30 D + G − D^{+}G^{-} D + G − D − G + D^{-}G^{+} D − G + Figure 3: Discovery versus Generation across all evaluated models on Prim . Qualitative analysis further clarifies these asymmetries. For gpt-5.4, D + G − D^{+}G^{-} D + G − D − G + D^{-}G^{+} D − G + Finding 2: Correct primitives unlock latent execution capacity. To examine whether models possess downstream capabilities that are not realized during direct Generation, we evaluate Execution when the correct Mathematical Primitive is explicitly provided. As shown in Table 3.1 , primitive guidance improves performance by 17.58 17.58 17.58 29.67 29.67 29.67 Model Generation Execution Self-Generated Primitive Teacher Plan Teacher Primitive Gold Primitive gpt-5.4-mini 50.55 50.55 50.55 48.90 48.90 48.90 60.44 60.44 60.44 63.74 63.74 63.74 71.98 71.98 71.98 44.51 44.51 44.51 46.15 46.15 46.15 54.40 54.40 54.40 60.99 60.99 60.99 68.68 68.68 68.68 52.75 52.75 52.75 41.21 41.21 41.21 56.04 56.04 56.04 64.84 64.84 64.84 78.57 78.57 78.57 Table 2: Generation and Execution performance on Prim under different forms of reasoning support. Detailed results are presented in Table B.3.3 in Appendix B.3 . We further investigate whether this rescue can be attributed simply to task decomposition or generic procedural scaffolding. Table 2 compares four intervention settings. Conditioning the model on its own generated primitive provides little benefit: it yields only a marginal gain for gpt-5.4-nano and substantially degrades Qwen3.6-27B, indicating that a forced discover-then-execute decomposition alone is insufficient. We then compare two forms of external guidance generated by the same teacher model (gpt-5.4) from the problem alone: a step-by-step Teacher Plan , which serves as a procedural hint by specifying the major solution steps, and a Teacher Primitive , which is generated according to the definition of primitive. Although the Teacher Plan improves over direct Generation, the Teacher Primitive yields substantially larger gains across the evaluated models. The Gold Primitive further maximizes this effect, pushing accuracy to 71.98 % 71.98\% 71.98% 68.68 % 68.68\% 68.68% 78.57 % 78.57\% 78.57% Finding 3: Independent structural discovery is the dominant bottleneck. As shown in Table 3.1 , models perform substantially better on Digestion , recovering the Mathematical Primitive from a correct solution, than on Discovery , where the same primitive must be identified from the problem alone. This asymmetry is particularly pronounced outside the strongest OpenAI models. Even models with weak Discovery can often recover the organizing primitive once the relevant reasoning is exposed through a completed solution; for instance, Qwen3.6-27B rises from 24.73% Discovery to 92.31% Digestion and R1-Distill-32B from 6.04% to 65.38%.
Figure 4: Decomposition of Generation failures by Discovery ( D D D E E E These results reveal a pronounced gap between retrospective recognition and prospective discovery of mathematical structure: models can often recognize the key idea once it is instantiated in a valid solution, yet struggle to identify it independently before the derivation is known.
To further localize this bottleneck, we decompose Generation failures according to the joint outcomes of Discovery ( D D D E E E 83.6 % 83.6\% 83.6% D − D^{-} D − D − E + D^{-}E^{+} D − E + 3.3 Further Analysis: Which Failures Are Repairable?
While our preceding diagnosis reveals the failure modes of LLMs during inference, a natural follow-up question arises: how does post-training impact these mathematical reasoning failures? Of particular interest are discovery-limited failures, which exhibit substantial room for improvement (§ 3.2 ). We hypothesize that these failures lie just at the boundary of the models’ reasoning capabilities, making them highly amenable to repair through post-training. Therefore, we utilize Supervised Fine-Tuning (SFT) and On-Policy Self-Distillation (OPSD) ( Zhao et al., 2026 ) to investigate how different failure modes uniquely respond to these post-training paradigms.
Setup. Our training corpus contains 709 Mathematics Ph.D. qualifying-examination problems from 1991–2026, each paired with a human-authored proof (median length: 118 words); additional details are provided in Appendix C . We use qualifying-examination problems because their open-ended, proof-oriented nature closely matches the difficulty and structural reasoning demands of Prim . Since such problems generally lack algorithmically verifiable short answers, we focus on text-supervised post-training rather than outcome-verified methods such as RLVR. We evaluate SFT and OPSD on Qwen-3.5-27B, 9B, 4B ( Qwen Team, 2026a ) , using the default OPSD configuration, bfloat16 precision, a fixed seed of 42 42 42 On-Policy Self-Distillation. Given training examples { x , y } \{x,y\} { x , y } x x x y y y π 0 \pi_{0} π 0 t t t P t : = π S ( ⋅ ∣ y ^ < t , x ) P_{t}:=\pi_{S}(\cdot\mid\hat{y}_{<t},x) P t := π S ( ⋅ ∣ y ^ < t , x ) Q t : = π T ( ⋅ ∣ y ^ < t , x , y ) Q_{t}:=\pi_{T}(\cdot\mid\hat{y}_{<t},x,y) Q t := π T ( ⋅ ∣ y ^ < t , x , y ) y ^ < t \hat{y}_{<t} y ^ < t L O P S D = 1 T ∑ t = 1 T D ( P t , Q t ) = 1 T ∑ t = 1 T D ( π S ( ⋅ ∣ y ^ < t , x ) , π T ( ⋅ ∣ y ^ < t , x , y ) ) , \displaystyle\mathcal{L}_{\mathrm{OPSD}}=\frac{1}{T}\sum_{t=1}^{T}D(P_{t},Q_{t})=\frac{1}{T}\sum_{t=1}^{T}D\left(\pi_{S}(\cdot\mid\hat{y}_{<t},x),\pi_{T}(\cdot\mid\hat{y}_{<t},x,y)\right), L OPSD = T 1 t = 1 ∑ T D ( P t , Q t ) = T 1 t = 1 ∑ T D ( π S ( ⋅ ∣ y ^ < t , x ) , π T ( ⋅ ∣ y ^ < t , x , y ) ) , D D D D K L ( Q t ∥ P t ) D_{\mathrm{KL}}(Q_{t}\parallel P_{t}) D KL ( Q t ∥ P t ) D K L ( P t ∥ Q t ) D_{\mathrm{KL}}(P_{t}\parallel Q_{t}) D KL ( P t ∥ Q t ) P t P_{t} P t Q t Q_{t} Q t Results. Across the 27B, 9B, and 4B models, the base models leave 341 341 341 294 294 294 147 147 147 D − E + D^{-}E^{+} D − E + 147 147 147 D − E − D^{-}E^{-} D − E − 20.4 % 20.4\% 20.4% 21.1 % 21.1\% 21.1% D − E + D^{-}E^{+} D − E + Failure type SFT OPSD D − E + D^{-}E^{+} D − E + D − E − D^{-}E^{-} D − E − Table 3: Post-training repair rates of SFT and OPSD across initial Prim failure quadrants. times their respective rates of 6.8 % 6.8\% 6.8% 8.2 % 8.2\% 8.2% D − E − D^{-}E^{-} D − E − 4 Internalizing Primitive-Guided Reasoning
In this section, we introduce Absorb , a novel variant of on-policy self-distillation that improves the mathematical problem-solving capability of LLMs by internalizing primitive-guided reasoning, motivated by the observations in § 3 . Extensive experiments with Qwen models as backbones show that Absorb consistently outperforms strong baselines on a diverse set of mathematical reasoning benchmarks (§ 4.4 ). Further implementation details and ablation results are provided in Appendix E .