Mathematical Transfer in LLMs Follows Reasoning Approach More Than Topic

Sajad Goudarzi; Samaneh Zamanifard; Seyed Amin Seyed Haeri; Moloud Nasiri; Hamed Rahimian

cs.AISep 29, 2026

Mathematical Transfer in LLMs Follows Reasoning Approach More Than Topic

Authors: Sajad Goudarzi, Samaneh Zamanifard, Seyed Amin Seyed Haeri, Moloud Nasiri, Hamed Rahimian

Organizations: Clemson University · Coastal Carolina University

Abstract

When selecting mathematical training data for LLMs, a natural organizing principle is topic: probability examples for probability targets. An alternative is reasoning approach: worked solutions that share a solution method with the target, even when the mathematical domain differs. We ask which relation produces greater transfer after fine-tuning. We evaluate two counterbalanced $2\times2$ designs: probability and combinatorics crossed with invariant reasoning and double counting (2,000 problems), and number theory and geometry crossed with complement and pigeonhole reasoning (800 problems). In each design, every cell serves as the held-out target in turn: same-approach (SA) sources share the target's method but change the topic, while same-topic (ST) sources share the topic but change the method. Every source appears once in each role, so additive source-quality effects cancel from the equally weighted aggregate contrast. Across five base models and three training seeds per design, SA outperforms ST in all 40 seed-pooled model--target comparisons. Model-level advantages range from 8.2 to 16.2 percentage points in the primary design (mean: 10.8) and from 12.0 to 16.0 in the second design (mean: 14.3); all ten model-level 95% confidence intervals exclude zero. In both designs, ST sources are more similar to targets under embedding and lexical measures, so the SA advantage runs opposite to the measured ordering of statement-level resemblance. These findings identify reasoning approach as a more effective matching criterion than topic for mathematical transfer across the evaluated topic--approach combinations.

Figures & tables

Panel of Figure 1 — Figure 1: The $2\times 2$ rotational design (a) Each cell pairs a topic (column) with a solution approach (row). When one cell is held out as the target (here: probability/invariant), its SA source occupies the same row—shared approach, different topic—while its ST source occupies the same column—shared topic, different approach. (b) The target rotates through all four cells.

Panel of Figure 2 — Figure 2: Dataset construction and experimental protocol. Top: problems are generated with topic-and-approach conditioning, verified via unconstrained re-solution and multi-judge agreement, and corrected through final-answer auditing. Middle: the final dataset contains 500 problems per cell, split 450/50 for training and validation. Bottom: for each of four target rotations, a separate checkpoint is fine-tuned on the SA source and the ST source and evaluated on the held-out target; the primary outcome is their accuracy difference $\delta$ .

Figure 3: Approach-matched supervision transfers better in all five models. Each point averages four held-out targets and seeds 7, 17, and 27. Error bars are 95% CIs; right labels show point estimates. The mean across models is

+10.8

percentage points.

Figure 4: Positive and negative transfer coexist with the same ordering. Points are descriptive means of the four target-wise changes, pooled over three seeds. SA improves average performance in every model; ST improves it in four and lowers it in Qwen. Full target-level CIs appear in Appendix A .

Figure 5: Every target favors SA. All points pool three seeds. Error bars are 95% CIs. Invariant targets show larger contrasts, particularly in combinatorics, but the ordering is positive for both approaches in both topics. These are target-wise effects, not estimates of an intrinsic transferability ranking among approaches.

Figure 6: The ordering holds at every training budget and widens with data. Left: SA–ST for each model at 150, 300, and 450 source-training examples. Right: suite-mean change from base for each relation. All five models; single seed; epochs are fixed, so optimizer steps grow with the data.

Figure 7: Greater statement resemblance does not track greater transfer. ST exceeds SA on every reported statement measure. NN denotes nearest neighbor.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	SA–ST (pp)	95% CI
Qwen3-4B-Base	+16.2	$[+14.4,+17.9]$
Meta-Llama-3-8B	+10.5	$[+9.4,+11.6]$
DeepSeek-Math-7B	+10.2	$[+8.9,+11.5]$
Mistral-7B-v0.3	+9.1	$[+8.0,+10.2]$
Gemma-3-4B-pt	+8.2	$[+7.1,+9.3]$
Equal-weight model mean	+10.8	—

Appendix

Table 1: Model-level SA–ST contrasts, pooled over four targets and three seeds.

Model	Seed 7	Seed 17	Seed 27
Qwen3-4B-Base	+15.8	+16.5	+16.2
Meta-Llama-3-8B	+10.1	+11.4	+9.9
DeepSeek-Math-7B	+11.4	+10.4	+8.8
Mistral-7B-v0.3	+9.5	+9.1	+8.8
Gemma-3-4B-pt	+7.4	+9.6	+7.5

Appendix

Table 2: Per-seed SA–ST, averaged over the four targets.

Model	Target	Base	SA	ST
Qwen3-4B	P / I	28.4	37.3	21.9
	P / D	45.2	44.9	38.5
	C / I	28.8	50.8	22.9
	C / D	28.0	37.1	22.0
Llama-3-8B	P / I	2.2	17.9	4.8
	P / D	0.4	12.2	6.8

Appendix

Table 3: Absolute accuracy (%), pooled over the three seeds. P = probability; C = combinatorics; I = invariant; D = double counting.

Qwen3-4B-Base
Target	SA–base [95% CI]	ST–base [95% CI]	SA–ST [95% CI]
P / I	$+8.9\;[+4.9,+13.0]$	$-6.5\;[-10.5,-2.5]$	$+15.5\;[+11.7,+19.1]$
P / D	$-0.3\;[-4.5,+3.8]$	$-6.7\;[-10.5,-2.7]$	$+6.3\;[+3.1,+9.5]$
C / I	$+22.0\;[+17.1,+27.0]$	$-5.9\;[-10.0,-1.9]$	$+27.9\;[+24.2,+31.8]$
C / D	$+9.1\;[+4.7,+13.2]$	$-6.0\;[-9.8,-2.3]$	$+15.1\;[+11.6,+18.7]$
Meta-Llama-3-8B

Appendix

Table 4: Target-level accuracy differences (percentage points) with 95% CIs, pooled over seeds 7, 17, and 27.

Model	SA–ST	95% CI	SA–base	ST–base
Meta-Llama-3-8B	$+16.0$	$[+14.0,+17.8]$	$+18.1$	$+2.1$
Qwen3-4B-Base	$+15.7$	$[+13.0,+18.3]$	$+4.0$	$-11.7$
DeepSeek-Math-7B-Base	$+14.6$	$[+12.6,+16.7]$	$+17.7$	$+3.1$
Gemma-3-4B-pt	$+13.0$	$[+11.2,+14.8]$	$+14.9$	$+1.9$
Mistral-7B-v0.3	$+12.0$	$[+10.2,+13.7]$	$+13.1$	$+1.1$

Appendix

Table 5: Second-design model-level accuracy differences, in percentage points, averaged over four target cells and three seeds. Confidence intervals are for SA–ST.

Model	Seed 7	Seed 17	Seed 27
Meta-Llama-3-8B	$+17.1$	$+16.1$	$+14.6$
Qwen3-4B-Base	$+14.2$	$+16.2$	$+16.5$
DeepSeek-Math-7B-Base	$+15.1$	$+13.4$	$+15.2$
Gemma-3-4B-pt	$+15.1$	$+10.6$	$+13.1$
Mistral-7B-v0.3	$+10.0$	$+11.9$	$+14.1$

Appendix

Table 6: Second-design SA–ST contrasts by training seed, averaged over the four target cells. Values are in percentage points.

Model	Target	Base	SA	ST	SA–ST
Meta-Llama-3-8B	NT / Comp	1.0	5.7	0.7	$+5.0$
	NT / Pig	0.0	25.5	4.5	$+21.0$
	Geo / Comp	0.0	3.8	1.5	$+2.3$
	Geo / Pig	0.5	38.8	3.3	$+35.5$
Qwen3-4B-Base	NT / Comp	37.5	41.3	38.0	$+3.3$
	NT / Pig	49.0	53.2	29.2	$+24.0$

Appendix

Table 7: Second-design target-level accuracy (%) and SA–ST contrasts (percentage points), pooled over seeds 7, 17, and 27. NT = number theory; Geo = geometry; Comp = complement; Pig = pigeonhole. Differences are computed before rounding.

Model	SA–ST	SA–base	ST–base	LoRA SA–ST
Qwen3-4B-Base	+19.8	+11.0	$-8.8$	+16.5
Meta-Llama-3-8B	+6.9	+7.5	+0.6	+11.4
DeepSeek-Math-7B-Base	+12.2	+19.8	+7.6	+10.4
Gemma-3-4B-pt	+5.4	+8.0	+2.6	+9.6
Mean of the four above	+11.1	—	—	+12
Mistral-7B-v0.3 †	+0.5	+0.4	$-0.1$	+9.1

Appendix

Table 8: Full fine-tuning, averaged over the four target cells. The LoRA column is the matched low-rank run. † Mistral did not converge and is excluded from the mean.

Model	P / I	P / D	C / I	C / D
Qwen3-4B-Base	+28.0	+13.0	+25.0	+13.0
Meta-Llama-3-8B	+8.2	+1.0	+16.0	+2.4
DeepSeek-Math-7B-Base	+10.6	+8.8	+23.6	+5.6
Gemma-3-4B-pt	+10.4	+0.4	+10.4	+0.4
Mistral-7B-v0.3 †	+0.4	+0.8	+0.4	+0.4

Appendix

Table 9: Target-level SA–ST contrasts under full fine-tuning (percentage points).

	SA–ST			SA–base			ST–base
Model	150	300	450	150	300	450	150	300	450
Qwen3-4B-Base	+14.6	+16.2	+16.5	+11.1	+9.5	+10.2	$-3.5$	$-6.7$	$-6.3$
Meta-Llama-3-8B	+6.4	+9.5	+11.4	+10.5	+12.7	+14.9	+4.1	+3.1	+3.5
DeepSeek-Math-7B-Base	+6.7	+10.2	+10.4	+12.9	+15.7	+17.7	+6.2	+5.5	+7.3
Mistral-7B-v0.3	+5.0	+7.2	+9.1	+7.0	+8.9	+11.7	+2.0	+1.8	+2.6
Gemma-3-4B-pt	+4.8	+5.3	+9.6	+7.1	+8.0	+11.9	+2.2	+2.7	+2.3

Appendix

Table 10: Accuracy differences (percentage points) by number of source-training examples, averaged over the four target cells.

Model	Checkpoint
Qwen3-4B-Base	Qwen/Qwen3-4B-Base
Meta-Llama-3-8B	meta-llama/Meta-Llama-3-8B
DeepSeek-Math-7B-Base	deepseek-ai/deepseek-math-7b-base
Mistral-7B-v0.3	mistralai/Mistral-7B-v0.3
Gemma-3-4B-pt	google/gemma-3-4b-pt

Appendix

Table 11: Public checkpoints used in the primary suite.

Metric	Within	SA	ST
TF–IDF, all pairs	0.054	0.023	0.026
TF–IDF, NN	0.538	0.232	0.260
Embedding, all pairs	0.537	0.437	0.458
Embedding, NN	0.898	0.705	0.739
Centred emb., all pairs	0.153	$-0.031$	0.004
Centred emb., NN	0.811	0.445	0.518

Appendix

Table 12: Statement similarity by relation. Centred embeddings subtract the dataset-mean vector. Metrics are not interchangeable.

Cell	SA	ST
P / I	35.4%	63.0%
C / I	52.8%	47.0%
P / D	45.8%	54.2%
C / D	33.2%	64.4%

Appendix

Table 13: Share of problems whose nearest out-of-cell statement neighbor lies in the SA or ST cell (Qwen3 embeddings). The leftover share matches neither topic nor approach.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 420 papers

Math Reasoning in LLMs is Organized by Approach, Not Topic
Sep 22, 2026Sajad Goudarzi, Samaneh Zamanifard, Moloud Nasiri +1Mathematical Reasoning Benchmarks LLM Interpretability
Clemson University
How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?
Oct 2, 2025Wen Yang, Junhong Wu, Chong Li +2Cross-Lingual Reasoning in Language Models Cross-Lingual Transfer
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research
Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning
Sep 29, 2026Hongyang Li, Yiming Zhu, Xiao Li +3Language Model Distillation RL Post-Training
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay

Sep 22, 2026J/K move · Enter open · S save

Math Reasoning in LLMs is Organized by Approach, Not Topic

Sajad Goudarzi, Samaneh Zamanifard, Moloud Nasiri, Hamed Rahimian

Clemson University

Mathematical Reasoning Benchmarks LLM Interpretability LLM Mathematical Reasoning

Open paper AskPDF arXiv

Mathematical reasoning benchmarks are typically organized by topic, but language models may organize their internal computation by reusable reasoning approach instead. In this paper, we investigate whether open math-capable LLMs organize internally by topical sub-skill or by reasoning approach, and we present evidence that the approach is the key. We introduce a generation-replay protocol: a model first generates a solution, after which we replay the exact prompt-plus-generation trajectory and extract activation-importance signatures over the reasoning tokens. We cluster these signatures without supervision across eight models and five mathematical reasoning sources, then evaluate the recovered structure with structural, semantic, and intervention tests. Across all 40 model-source cells, the recovered clusters outperform matched-size random baselines. Two independent frontier-LLM judges find approach-level coherence in 77-82% of real clusters versus 6-11% in within-source controls, and topic-pure clusters usually receive labels finer than the topic itself. In approach-controlled prompting, changing the requested reasoning approach shifts cluster assignment in seven of eight model conditions, whereas paraphrases largely preserve it. These results indicate that math-capable LLMs organize internal mathematical computation by reasoning approach rather than benchmark topic. The implication is that topic-stratified benchmarks and topic-balanced training corpora can still miss the axis that matters: even deliberately topic-balanced corpora may remain imbalanced over reasoning approaches.

Abstract

Transfer Learning

Explore similar work

Math Reasoning in LLMs is Organized by Approach, Not Topic

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning