Mathematical Transfer in LLMs Follows Reasoning Approach More Than Topic

Sajad Goudarzi; Samaneh Zamanifard; Seyed Amin Seyed Haeri; Moloud Nasiri; Hamed Rahimian

cs.AISep 29, 2026

Mathematical Transfer in LLMs Follows Reasoning Approach More Than Topic

Authors: Sajad Goudarzi, Samaneh Zamanifard, Seyed Amin Seyed Haeri, Moloud Nasiri, Hamed Rahimian

Organizations: Clemson University · Coastal Carolina University

Abstract

When selecting mathematical training data for LLMs, a natural organizing principle is topic: probability examples for probability targets. An alternative is reasoning approach: worked solutions that share a solution method with the target, even when the mathematical domain differs. We ask which relation produces greater transfer after fine-tuning. We evaluate two counterbalanced $2\times2$ designs: probability and combinatorics crossed with invariant reasoning and double counting (2,000 problems), and number theory and geometry crossed with complement and pigeonhole reasoning (800 problems). In each design, every cell serves as the held-out target in turn: same-approach (SA) sources share the target's method but change the topic, while same-topic (ST) sources share the topic but change the method. Every source appears once in each role, so additive source-quality effects cancel from the equally weighted aggregate contrast. Across five base models and three training seeds per design, SA outperforms ST in all 40 seed-pooled model--target comparisons. Model-level advantages range from 8.2 to 16.2 percentage points in the primary design (mean: 10.8) and from 12.0 to 16.0 in the second design (mean: 14.3); all ten model-level 95% confidence intervals exclude zero. In both designs, ST sources are more similar to targets under embedding and lexical measures, so the SA advantage runs opposite to the measured ordering of statement-level resemblance. These findings identify reasoning approach as a more effective matching criterion than topic for mathematical transfer across the evaluated topic--approach combinations.

Figures & tables

Panel of Figure 1 — Figure 1: The $2\times 2$ rotational design (a) Each cell pairs a topic (column) with a solution approach (row). When one cell is held out as the target (here: probability/invariant), its SA source occupies the same row—shared approach, different topic—while its ST source occupies the same column—shared topic, different approach. (b) The target rotates through all four cells.

Panel of Figure 2 — Figure 2: Dataset construction and experimental protocol. Top: problems are generated with topic-and-approach conditioning, verified via unconstrained re-solution and multi-judge agreement, and corrected through final-answer auditing. Middle: the final dataset contains 500 problems per cell, split 450/50 for training and validation. Bottom: for each of four target rotations, a separate checkpoint is fine-tuned on the SA source and the ST source and evaluated on the held-out target; the primary outcome is their accuracy difference $\delta$ .

Figure 3: Approach-matched supervision transfers better in all five models. Each point averages four held-out targets and seeds 7, 17, and 27. Error bars are 95% CIs; right labels show point estimates. The mean across models is

+10.8

percentage points.

Figure 4: Positive and negative transfer coexist with the same ordering. Points are descriptive means of the four target-wise changes, pooled over three seeds. SA improves average performance in every model; ST improves it in four and lowers it in Qwen. Full target-level CIs appear in Appendix A .

Figure 5: Every target favors SA. All points pool three seeds. Error bars are 95% CIs. Invariant targets show larger contrasts, particularly in combinatorics, but the ordering is positive for both approaches in both topics. These are target-wise effects, not estimates of an intrinsic transferability ranking among approaches.

Figure 6: The ordering holds at every training budget and widens with data. Left: SA–ST for each model at 150, 300, and 450 source-training examples. Right: suite-mean change from base for each relation. All five models; single seed; epochs are fixed, so optimizer steps grow with the data.

Figure 7: Greater statement resemblance does not track greater transfer. ST exceeds SA on every reported statement measure. NN denotes nearest neighbor.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	SA–ST (pp)	95% CI
Qwen3-4B-Base	+16.2	$[+14.4,+17.9]$
Meta-Llama-3-8B	+10.5	$[+9.4,+11.6]$
DeepSeek-Math-7B	+10.2	$[+8.9,+11.5]$
Mistral-7B-v0.3	+9.1	$[+8.0,+10.2]$
Gemma-3-4B-pt	+8.2	$[+7.1,+9.3]$
Equal-weight model mean	+10.8	—

Appendix

Table 1: Model-level SA–ST contrasts, pooled over four targets and three seeds.

Model	Seed 7	Seed 17	Seed 27
Qwen3-4B-Base	+15.8	+16.5	+16.2
Meta-Llama-3-8B	+10.1	+11.4	+9.9
DeepSeek-Math-7B	+11.4	+10.4	+8.8
Mistral-7B-v0.3	+9.5	+9.1	+8.8
Gemma-3-4B-pt	+7.4	+9.6	+7.5

Appendix

Table 2: Per-seed SA–ST, averaged over the four targets.

Model	Target	Base	SA	ST
Qwen3-4B	P / I	28.4	37.3	21.9
	P / D	45.2	44.9	38.5
	C / I	28.8	50.8	22.9
	C / D	28.0	37.1	22.0
Llama-3-8B	P / I	2.2	17.9	4.8
	P / D	0.4	12.2	6.8

Appendix

Table 3: Absolute accuracy (%), pooled over the three seeds. P = probability; C = combinatorics; I = invariant; D = double counting.

Qwen3-4B-Base
Target	SA–base [95% CI]	ST–base [95% CI]	SA–ST [95% CI]
P / I	$+8.9\;[+4.9,+13.0]$	$-6.5\;[-10.5,-2.5]$	$+15.5\;[+11.7,+19.1]$
P / D	$-0.3\;[-4.5,+3.8]$	$-6.7\;[-10.5,-2.7]$	$+6.3\;[+3.1,+9.5]$
C / I	$+22.0\;[+17.1,+27.0]$	$-5.9\;[-10.0,-1.9]$	$+27.9\;[+24.2,+31.8]$
C / D	$+9.1\;[+4.7,+13.2]$	$-6.0\;[-9.8,-2.3]$	$+15.1\;[+11.6,+18.7]$
Meta-Llama-3-8B

Appendix

Table 4: Target-level accuracy differences (percentage points) with 95% CIs, pooled over seeds 7, 17, and 27.

Model	SA–ST	95% CI	SA–base	ST–base
Meta-Llama-3-8B	$+16.0$	$[+14.0,+17.8]$	$+18.1$	$+2.1$
Qwen3-4B-Base	$+15.7$	$[+13.0,+18.3]$	$+4.0$	$-11.7$
DeepSeek-Math-7B-Base	$+14.6$	$[+12.6,+16.7]$	$+17.7$	$+3.1$
Gemma-3-4B-pt	$+13.0$	$[+11.2,+14.8]$	$+14.9$	$+1.9$
Mistral-7B-v0.3	$+12.0$	$[+10.2,+13.7]$	$+13.1$	$+1.1$

Appendix

Table 5: Second-design model-level accuracy differences, in percentage points, averaged over four target cells and three seeds. Confidence intervals are for SA–ST.

Model	Seed 7	Seed 17	Seed 27
Meta-Llama-3-8B	$+17.1$	$+16.1$	$+14.6$
Qwen3-4B-Base	$+14.2$	$+16.2$	$+16.5$
DeepSeek-Math-7B-Base	$+15.1$	$+13.4$	$+15.2$
Gemma-3-4B-pt	$+15.1$	$+10.6$	$+13.1$
Mistral-7B-v0.3	$+10.0$	$+11.9$	$+14.1$

Appendix

Table 6: Second-design SA–ST contrasts by training seed, averaged over the four target cells. Values are in percentage points.

Model	Target	Base	SA	ST	SA–ST
Meta-Llama-3-8B	NT / Comp	1.0	5.7	0.7	$+5.0$
	NT / Pig	0.0	25.5	4.5	$+21.0$
	Geo / Comp	0.0	3.8	1.5	$+2.3$
	Geo / Pig	0.5	38.8	3.3	$+35.5$
Qwen3-4B-Base	NT / Comp	37.5	41.3	38.0	$+3.3$
	NT / Pig	49.0	53.2	29.2	$+24.0$

Appendix

Table 7: Second-design target-level accuracy (%) and SA–ST contrasts (percentage points), pooled over seeds 7, 17, and 27. NT = number theory; Geo = geometry; Comp = complement; Pig = pigeonhole. Differences are computed before rounding.

Model	SA–ST	SA–base	ST–base	LoRA SA–ST
Qwen3-4B-Base	+19.8	+11.0	$-8.8$	+16.5
Meta-Llama-3-8B	+6.9	+7.5	+0.6	+11.4
DeepSeek-Math-7B-Base	+12.2	+19.8	+7.6	+10.4
Gemma-3-4B-pt	+5.4	+8.0	+2.6	+9.6
Mean of the four above	+11.1	—	—	+12
Mistral-7B-v0.3 †	+0.5	+0.4	$-0.1$	+9.1

Appendix

Table 8: Full fine-tuning, averaged over the four target cells. The LoRA column is the matched low-rank run. † Mistral did not converge and is excluded from the mean.

Model	P / I	P / D	C / I	C / D
Qwen3-4B-Base	+28.0	+13.0	+25.0	+13.0
Meta-Llama-3-8B	+8.2	+1.0	+16.0	+2.4
DeepSeek-Math-7B-Base	+10.6	+8.8	+23.6	+5.6
Gemma-3-4B-pt	+10.4	+0.4	+10.4	+0.4
Mistral-7B-v0.3 †	+0.4	+0.8	+0.4	+0.4

Appendix

Table 9: Target-level SA–ST contrasts under full fine-tuning (percentage points).

	SA–ST			SA–base			ST–base
Model	150	300	450	150	300	450	150	300	450
Qwen3-4B-Base	+14.6	+16.2	+16.5	+11.1	+9.5	+10.2	$-3.5$	$-6.7$	$-6.3$
Meta-Llama-3-8B	+6.4	+9.5	+11.4	+10.5	+12.7	+14.9	+4.1	+3.1	+3.5
DeepSeek-Math-7B-Base	+6.7	+10.2	+10.4	+12.9	+15.7	+17.7	+6.2	+5.5	+7.3
Mistral-7B-v0.3	+5.0	+7.2	+9.1	+7.0	+8.9	+11.7	+2.0	+1.8	+2.6
Gemma-3-4B-pt	+4.8	+5.3	+9.6	+7.1	+8.0	+11.9	+2.2	+2.7	+2.3

Appendix

Table 10: Accuracy differences (percentage points) by number of source-training examples, averaged over the four target cells.

Model	Checkpoint
Qwen3-4B-Base	Qwen/Qwen3-4B-Base
Meta-Llama-3-8B	meta-llama/Meta-Llama-3-8B
DeepSeek-Math-7B-Base	deepseek-ai/deepseek-math-7b-base
Mistral-7B-v0.3	mistralai/Mistral-7B-v0.3
Gemma-3-4B-pt	google/gemma-3-4b-pt

Appendix

Table 11: Public checkpoints used in the primary suite.

Metric	Within	SA	ST
TF–IDF, all pairs	0.054	0.023	0.026
TF–IDF, NN	0.538	0.232	0.260
Embedding, all pairs	0.537	0.437	0.458
Embedding, NN	0.898	0.705	0.739
Centred emb., all pairs	0.153	$-0.031$	0.004
Centred emb., NN	0.811	0.445	0.518

Appendix

Table 12: Statement similarity by relation. Centred embeddings subtract the dataset-mean vector. Metrics are not interchangeable.

Cell	SA	ST
P / I	35.4%	63.0%
C / I	52.8%	47.0%
P / D	45.8%	54.2%
C / D	33.2%	64.4%

Appendix

Table 13: Share of problems whose nearest out-of-cell statement neighbor lies in the SA or ST cell (Qwen3 embeddings). The leftover share matches neither topic nor approach.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 420 papers

Sep 22, 2026

Math Reasoning in LLMs is Organized by Approach, Not Topic

Mathematical reasoning benchmarks are typically organized by topic, but language models may organize their internal computation by reusable reasoning approach instead. In this paper, we investigate whether open math-capable LLMs organize internally by topical sub-skill or by reasoning approach, and we present evidence that the approach is the key. We introduce a generation-replay protocol: a model first generates a solution, after which we replay the exact prompt-plus-generation trajectory and extract activation-importance signatures over the reasoning tokens. We cluster these signatures without supervision across eight models and five mathematical reasoning sources, then evaluate the recovered structure with structural, semantic, and intervention tests. Across all 40 model-source cells, the recovered clusters outperform matched-size random baselines. Two independent frontier-LLM judges find approach-level coherence in 77-82% of real clusters versus 6-11% in within-source controls, and topic-pure clusters usually receive labels finer than the topic itself. In approach-controlled prompting, changing the requested reasoning approach shifts cluster assignment in seven of eight model conditions, whereas paraphrases largely preserve it. These results indicate that math-capable LLMs organize internal mathematical computation by reasoning approach rather than benchmark topic. The implication is that topic-stratified benchmarks and topic-balanced training corpora can still miss the axis that matters: even deliberately topic-balanced corpora may remain imbalanced over reasoning approaches.

Sajad Goudarzi, Samaneh Zamanifard, Moloud Nasiri +1

Clemson University

Mathematical Reasoning Benchmarks LLM Interpretability LLM Mathematical Reasoning

Ask PDF

Oct 2, 2025

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

Recent advances in Reinforcement Post-Training (RPT) have substantially improved the mathematical reasoning capabilities of large language models (LLMs), but it remains unclear how well such gains transfer across languages. In this work, we study cross-lingual mathematical reasoning transfer in English-centric LLMs under controlled multilingual evaluation. We first evaluate a range of English-centric reasoning models on multilingual mathematical reasoning benchmarks and find that transfer is highly variable across initial model choice, target language, and training paradigm. Through controlled comparative experiments, we further find that stronger English-centric initialization does not necessarily lead to stronger relative transfer efficiency, even when it yields higher multilingual reasoning accuracy. Finally, we show that the largest improvement comes from moving beyond English-only supervision: adding a single parallel language yields a substantial gain, while further gains from additional languages are smaller but consistent. Overall, our results highlight the need to evaluate multilingual mathematical reasoning directly, using both absolute performance and relative transfer-efficiency metrics.

Wen Yang, Junhong Wu, Chong Li +2

School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research

Cross-Lingual Reasoning in Language Models Cross-Lingual Transfer LLM Reasoning Multilingual Language Model Evaluation

Ask PDF

Sep 29, 2026

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning

Post-training is central to mathematical reasoning in modern large language models (LLMs), but endpoint pass@1 alone underidentifies what has changed. Gains may reflect newly reachable solutions, cheaper sampling of latent solutions, surface robustness, or memorisation. We compare three post-training paths under a common diagnostic readout: our sufficiently trained off-policy distillation trajectories, released Qwen3 off-policy-plus-on-policy distillation endpoints, and a released DeepSeek-Math endpoint trained with Group Relative Policy Optimisation (GRPO). Our probe uses cross-surface pass@K over verbatim prompts, paraphrases, numerical isomorphisms, and translations, plus consistency, distribution-shape, and verified supervised-fine-tuning (SFT) membership analyses. We find two regimes. On easier AMC problems, large-K ceilings are near saturation, so post-training mainly compresses sample cost. On harder AIME problems, post-training expands the large-K ceiling over the base model: sufficient off-policy distillation already raises this ceiling, Qwen3 released endpoints raise it further, and DeepSeek-Math GRPO does not dominate sufficient off-policy distillation at large K. English-dominant distillation improves non-English reasoning but preserves language-tier gaps. A controlled-overfit audit finds limited sensitivity in current SFT-membership probes. Compression is one regime of post-training, not a universal explanation.

Hongyang Li, Yiming Zhu, Xiao Li +3

University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay

Language Model Distillation RL Post-Training LLM Post-Training LLM Mathematical Reasoning

Ask PDF

Abstract

Transfer Learning

Explore similar work

Math Reasoning in LLMs is Organized by Approach, Not Topic

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning