SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation

Haomin Zhuang; Xiangqi Wang; Yili Shen; Ying Cheng; Xiangliang Zhang

cs.AIApr 2, 2026

SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation

Authors: Haomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang

Organizations: University of Notre Dame

Abstract

Large language models often default to step-by-step computation even when efficient numerical shortcuts are available. This raises a basic question: do they exhibit number sense in a human-like behavioral sense, i.e., the ability to recognize numerical structure, apply shortcuts when appropriate, and avoid them when they are not? We introduce SenseMath, a controlled benchmark for evaluating structure-sensitive numerical reasoning in LLMs. SenseMath contains 4,800 items spanning eight shortcut categories and four digit scales, with matched strong-shortcut, weak-shortcut, and control variants. It supports three evaluation settings of increasing cognitive demand: Shortcut Use (whether models can apply shortcuts on shortcut-amenable problems); Applicability Judgment (whether they can recognize when a shortcut is appropriate or misleading); and Problem Generation (whether they can generate new problem items that correctly admit a given type of shortcut). Our evaluation across five LLMs, ranging from GPT-4o-mini to Llama-3.1-8B, shows a consistent pattern: when explicitly prompted, models readily adopt shortcut strategies and achieve substantial accuracy gains on shortcut-amenable items (up to 15%), yet under standard chain-of-thought prompting they spontaneously employ such strategies in fewer than 40% of cases, even when they demonstrably possess the requisite capability. Moreover, this competence is confined to the Use level; models systematically over-generalise shortcuts to problems where they do not apply, and fail to generate valid shortcut-bearing problems from scratch. Together, these results suggest that current LLMs exhibit procedural shortcut fluency without the structural understanding of when and why shortcuts work that underlies human number sense.

Figures & tables

Figure 1: SenseMath overview. Each item has matched strong-shortcut, weak-shortcut, and control variants administered under CoT and number-sense prompting. The matched design isolates selective shortcut exploitation from general prompt effects.

Figure 4: Average accuracy gain from NS prompting (vs. CoT) across all digit scales, by variant.

Figure 5: RQ2 analysis. (a) NS accuracy gain on strong items increases with digit scale for capable models; 8B models show no benefit. (b) Normalized NS improvement per category at

d{=}8

(\text{NS}-\text{CoT})/(1-\text{CoT})

; red dashed circle = zero baseline.

Task	Model	Overall (%)	Strong (%)	Control (%)
J1	GPT-4o-mini	66	100	0
	GPT-4.1-mini	65	89	16
	Qwen3-8B	65	98	0
	Qwen3-30B	64	97	4
	Llama-3.1-8B	61	83	21
		Overall (%)	SHORTCUT (%)	COMP (%)

Table 1: Applicability judgment task results. J1: shortcut appropriateness (251 items after filtering). J2: strategy identification (80 items, GPT models only).

Model	Fmt	S.Ans	C.Ans	SC.Ex	C.Blk	Var	Pass
Qwen3-30B	96	65	73	66	87	76	23/96
Qwen3-8B	100	69	64	55	83	71	19/96
GPT-4.1-mini	100	70	79	67	80	75	17/96
GPT-4o-mini	100	58	48	46	82	73	9/96
Llama-3.1-8B	100	51	46	38	83	52	2/96

Table 2: Problem generation: per-check pass rates (%) and overall pass-all-6-checks rate. All models evaluated on 96 prompts. Fmt = valid JSON format; S.Ans = strong answer correct; C.Ans = control answer correct; SC.Ex = shortcut exists in strong; C.Blk = control blocks shortcut; Var = variant matching (same template/scale).

Figure 6: Distribution of checks passed per generated item. Most items pass 4–5 of 6 checks, suggesting models are close to generating valid problems but consistently fail on the shortcut-existence constraint.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Strong-shortcut variant	Control variant
Structural (SS)	What is $98\times 34$ ? (A) 3322 (B) 3342 (C) 3332 (D) 3352 Shortcut: $98{=}(100{-}2)$ , so $100{\times}34-2{\times}34=3332$ .	What is $47\times 40$ ? (A) 1870 (B) 1890 (C) 1900 (D) 1880 No near-100 structure; direct multiplication required.
Cancellation (CI)	Evaluate: $71+28-27$ . (A) 82 (B) 92 (C) 72 (D) 62 Shortcut: $28{-}27{=}1$ , so $\approx 71{+}1=72$ .	Evaluate: $71+28-118$ . (A) $-19$ (B) $-1$ (C) $-11$ (D) $-31$ No near-cancellation; full arithmetic needed.
Equation (ER)	$23+22=\rule{14.22636pt}{0.4pt}+18$ . (A) 37 (B) 27 (C) 47 (D) 17 Shortcut: move 18 left, $23{+}22{-}18=27$ .	$20+93+\rule{14.22636pt}{0.4pt}=59+49$ . (A) 15 (B) $-15$ (C) $-5$ (D) 5 No direct identity; must compute both sides.
Option Elim (OE)	What is $70\times 16$ ? (A) 1123 (B) 1119 (C) 1121 (D) 1120 Shortcut: $70{\times}\text{even}\Rightarrow$ ends in 0; only 1120 qualifies.	What is $33\times 87$ ? (A) 3031 (B) 2973 (C) 3187 (D) 2871 All options plausible; no elimination possible.

Appendix

Table 3: Example SenseMath items (

d{=}2

). Each row shows one item with its strong and control variants; the strong variant admits a clean shortcut while the control requires direct computation. Correct answers are bolded .

Figure 7: Normalized NS improvement at

d{=}2

Figure 8: Normalized NS improvement at

d{=}4

Figure 9: Normalized NS improvement at

d{=}16

Panel of Figure 10 — Figure 10: G task: side-by-side model outputs for magnitude_estimation ( $d{=}4$ ). All 6 checks shown per model. Only Qwen3-30B passes all checks by choosing operands genuinely close to $10^{4}$ . The dominant failure across all other models is ✗ SC.Exist: operands that look round but do not satisfy the structural constraint for an effective power-of-10 shortcut.

Model	Prompt	Metric	$d{=}2$			$d{=}4$			$d{=}8$			$d{=}16$
			$S$	$W$	$C$	$S$	$W$	$C$	$S$	$W$	$C$	$S$	$W$	$C$
GPT-4o-mini	CoT	Acc $\uparrow$ (%)	88	88	85	78	70	65	75	62	55	67	53	46
		SU	.28	.18	.24	.27	.16	.18	.36	.21	.22	.37	.27	.22
	NS	Acc $\uparrow$ (%)	92 +4	92	87	84 +6	72	63	82 +7	73	62	70 +3	59	51
		SU	.46	.37	.44	.77	.65	.78	.84	.76	.80	.90	.80	.90
	Strict	Acc $\uparrow$ (%)	94	96	92	83	78	83	70	58	48	68	58	45

Appendix

Table 4: Full results across all prompting conditions and digit scales. Acc

\uparrow

(%) = accuracy; SU = shortcut usage rate (proportion) classified by GPT-4.1-mini judge.

S

= strong-shortcut,

W

= weak-shortcut,

C

= control. Green / red deltas on Acc NS show the change from Acc CoT on strong items. Open-weight models truncate under Strict at 512 tokens.

Model	Variant	Prompt	SU Rate	Accuracy	Avg Tok
Qwen3-8B	Base	CoT	10.0%	64.5%	376
(nothink)	Base	NC	36.5%	65.0%	340
	NS-DPO	CoT	11.5%	67.0%	372
	NS-DPO	NC	33.5%	64.0%	350
	Baseline-DPO	CoT	8.5%	66.0%	374
	Baseline-DPO	NC	34.5%	60.0%	350

Appendix

Table 5: In-domain results on the 200-problem eval set (500-example training). SU Rate = fraction of responses using shortcut strategy; Avg Tok = average response tokens. Best accuracy per model per prompt condition is bold .

Model	Variant	BBH	ARC	GPQA	MMLU-P	MedQA	LogiQA	CSQA
Qwen3-8B	Base	73.8	87.0	34.6	31.8	61.4	50.5	80.8
	NS-DPO	73.8	87.0	31.2	31.4	59.4	46.9	80.8
	Baseline-DPO	73.6	86.6	34.6	31.8	59.0	47.9	80.7
	NS-SFT	73.6	86.6	33.7	32.6	60.4	48.4	80.7
	CoT-SFT	73.4	86.3	32.8	30.6	60.6	48.7	80.9
	Comb-DPO	69.8	87.0	35.0	28.4	62.2	51.0	80.0

Appendix

Table 6: Out-of-distribution benchmark results after 500-example training. Most variants fall within

\pm

4pp of base, confirming no catastrophic forgetting.

Mathematical Problem Generation

See all 14 papers

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning
Sep 26, 2026Zhuohan Wang, Haoran Ma, Tianyu Wu +3LLM Mathematical Reasoning LLM Evaluation
Harvard University · Rutgers University · The Hong Kong University of Science and Technology
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
Jun 2, 2026Zetian Ouyang, Linlin Wang, Gerard de Melo +1Numerical Reasoning in Language Models Mathematical Reasoning Benchmarks
East China Normal University · Hasso Plattner Institute, University of Potsdam
Math Reasoning in LLMs is Organized by Approach, Not Topic
Sep 22, 2026Sajad Goudarzi, Samaneh Zamanifard, Moloud Nasiri +1Mathematical Reasoning Benchmarks LLM Interpretability
Clemson University

Sep 26, 2026J/K move · Enter open · S save

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

Zhuohan Wang, Haoran Ma, Tianyu Wu, Yuanlin Duan+2

Harvard University · Rutgers University · The Hong Kong University of Science and Technology

LLM Mathematical Reasoning LLM Evaluation Mathematical Reasoning Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.

Abstract

Mathematical Problem Generation

Explore similar work

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

Math Reasoning in LLMs is Organized by Approach, Not Topic