Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy while fewer quantify local runtime and energy, characterize failure modes, or apply paired statistical comparisons under controlled conditions. This paper presents a controlled, documented procedure for evaluating locally hosted LLMs on mathematical reasoning. It combines fixed inference settings, hierarchical answer extraction and verification, explicit failure-mode classification, and per-question resource measurement, and reports accuracy with paired significance tests and effect sizes. We demonstrate it in a preliminary study of three compact open-weight models under five billion parameters, Gemma3:4b (Google), Phi3:3.8b (Microsoft), and Qwen3:4b (Alibaba), across datasets spanning Grade 8 Math, Calculus I, and Advanced Probability and Statistics. All models ran through the same local inference server on one workstation, using a shared prompt template, controlled settings, and a matched question set per dataset. No single model dominates. Qwen3:4b is most accurate on two datasets and Gemma3:4b on Calculus I, yet Gemma3:4b returns roughly three times more correct answers per watt-hour than Qwen3:4b on every dataset while generating far fewer output tokens; Qwen3:4b requires substantially more generation time, energy, and output per question. Phi3:3.8b is substantially less accurate on all three datasets; its low extraction-failure rate indicates incorrect answers rather than unparsed output, though we caveat possible prompt-format effects. These preliminary findings indicate that accuracy alone is an insufficient basis for selecting a local model.
Figures & tables
Fig. 1: High-level overview of the evaluation procedure. Each stage is applied identically to every model: structured math problems are loaded from the dataset, converted into a standardized prompt and executed on the local inference server, parsed by a multi-stage extraction and verification module, classified into one of four outcome categories, and aggregated into accuracy and per-question resource metrics with paired statistical analysis.
Dataset
Matched N
Categories
Grade 8 Math
644
5
Calculus I
374
1
Advanced Probability and Statistics
884
1
TABLE I: Matched evaluation set sizes used for cross-model comparison. Grade 8 Math problems are distributed across five Common Core categories; Calculus I and Advanced Probability and Statistics are treated as single-category domains.
Model
Dataset
N
Accuracy
95% CI
Correct/Wh
Tokens/Q
Gemma3:4b
Adv. Prob. & Stat.
884
0.947
[0.930, 0.960]
6.53
440
Phi3:3.8b
Adv. Prob. & Stat.
884
0.104
[0.086, 0.126]
0.89
389
Qwen3:4b
Adv. Prob. & Stat.
884
0.974
[0.961, 0.983]
1.96
1,507
Gemma3:4b
Calculus I
374
0.864
[0.825, 0.895]
4.45
502
Phi3:3.8b
Calculus I
374
0.289
[0.245, 0.337]
0.96
1,189
Qwen3:4b
Calculus I
374
0.794
[0.750, 0.832]
1.62
1,485
TABLE II: Accuracy and efficiency on the matched evaluation set. N denotes the number of questions in the matched set. Gemma3:4b leads on correct answers per watt-hour for every dataset despite being outscored on raw accuracy for two of three. Qwen3:4b generates roughly three to four times more output tokens per question than Gemma3:4b.
Dataset
Pair (A vs. B)
Acc. Diff.
p (Holm)
h
Mag.
Adv. P&S
Gemma3 vs. Phi3
+0.843
<10−162
+2.02
large
Adv. P&S
Gemma3 vs. Qwen3
−0.027
0.002
−0.14
negl.
Adv. P&S
Phi3 vs. Qwen3
−0.870
<10−160
−2.16
large
Calc. I
Gemma3 vs. Phi3
+0.575
<10−44
+1.25
large
Calc. I
Gemma3 vs. Qwen3
+0.070
3.1×10−4
+0.19
negl.
Calc. I
Phi3 vs. Qwen3
−0.505
<10−40
−1.06
large
TABLE III: Pairwise accuracy comparisons via McNemar’s test with Holm correction on the matched evaluation set. All nine comparisons are statistically significant at α=0.05 . A positive accuracy difference (A − B) indicates Model A is more accurate; Cohen’s h reports effect-size magnitude.
Category
N
Gemma3:4b
Phi3:3.8b
Qwen3:4b
Expressions and Equations
186
0.505
0.005
0.591
Functions
88
0.864
0.000
0.977
Geometry
145
0.690
0.000
0.876
Statistics and Probability
60
0.933
0.000
0.950
The Number System
164
0.659
0.091
0.750
TABLE IV: Per-category accuracy on Grade 8 Math. Qwen3:4b leads every category; the two stronger models are closest on Statistics and Probability and both struggle on Expressions and Equations.
Metric
Gemma3:4b
Phi3:3.8b
Qwen3:4b
Advanced Probability and Statistics
Gen. Time (s)
6.79
4.36
18.25
Output Tokens
431
240
1,261
Energy/Q (Wh)
0.142
0.075
0.423
Calculus I
Gen. Time (s)
9.26
9.33
19.38
TABLE V: Per-question resource medians on the matched evaluation set: generation time, output tokens, and estimated GPU energy for each model. Qwen3:4b is the most resource-intensive model across all three datasets. Phi3:3.8b has the lowest resource medians on Advanced Probability and Statistics and Grade 8 Math, whereas Gemma3:4b has the lowest medians on Calculus I. Corresponding per-question mean output-token values appear in Table II , and per-question mean energy values are reported in the text.
Model
Dataset
Wrong Ans.
Extr. Fail.
Hard Fail.
Gemma3:4b
Adv. P&S
5.3%
0.0%
0.0%
Phi3:3.8b
Adv. P&S
89.6%
0.0%
0.0%
Qwen3:4b
Adv. P&S
2.6%
0.0%
0.0%
Gemma3:4b
Calculus I
13.6%
0.0%
0.0%
Phi3:3.8b
Calculus I
71.1%
0.0%
0.0%
Qwen3:4b
Calculus I
20.6%
0.0%
0.0%
TABLE VI: Failure-mode classification on the matched evaluation set. Wrong Answer Rate is the fraction of all evaluated responses for which an answer was parsed but determined to be incorrect. Extraction Failure Rate is the fraction of all evaluated responses from which no answer could be parsed. Extraction failures are negligible across all cells.