Choosing Quantized Code LLMs by File Size and Energy per Solved Task
Organizations: Independent Researcher
Abstract
A developer running a code assistant on a laptop downloads one model file, and local inference moves the energy of every request onto that machine. A common rule says a larger model at lower precision tends to beat a smaller one at higher precision. We ran 24 GGUF files of Qwen2.5-Coder at four sizes on a fanless MacBook Air M2 and priced each in energy per solved task on 542 EvalPlus tasks. The result is a selection guide indexed by Ollama's listed file size, with an evidence label on each pick. At 1 and 2 GB it picks the q4_K_M files of the 1.5B and 3B models, both on the correctness and energy frontier under every accounting. At 3 GB as listed, the larger-model rule picks 7B-q2_K, the only measured 7B file that fits, and there it does not pay. That file brings no significant gain in solves over the 2 GB pick, 3B-q4_K_M, and the 90% CI limits any 3B-q4_K_M deficit to 2.4 points. Yet 7B-q2_K needs 2.03 times the energy per solved task on CPU, GPU and memory rails, or 1.66 to 1.81 times on whole-system power. On HumanEval+, five alternating session pairs give a ratio of 1.925 against 1.956 in the main grid. At 8 GB, 7B-q6_K buys a significant gain over the 2 GB pick at a higher energy price. From 1.5B up, q4_K_M trails the highest precision measured at its size by less than 3 points (90% CI). We release per-task results, power logs and the analysis script.
Figures & tables
| 0.5B | 1.5B | 3B | 7B | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Recipe | BPW | GB | Pass | J | mJ | BPW | GB | Pass | J | mJ | BPW | GB | Pass | J | mJ | BPW | GB | Pass | J | mJ |
| q2_K | 3.16 | 3.00 | 67.0 | 141.2 | 585 | |||||||||||||||
| q3_K_M | 5.66 | 0.36 | 40.6 | 38.9 | 79 | 4.24 | 0.82 | 55.9 | 53.4 | 179 | 4.11 | 1.60 | 1.3 | 4,708 | 317 | 3.99 | 3.80 | 71.6 | 187.0 | 848 |
| q4_K_M | 6.35 | 0.40 | 42.3 | 42.3 | 93 | 5.08 | 0.99 | 58.3 | 39.3 | 157 | 4.99 | 1.90 | 67.5 | 69.4 | 294 | 4.91 | 4.70 | 73.1 | 145.7 | 663 |
| q5_K_M | 6.71 | 0.42 | 42.8 | 41.3 | 94 | 5.80 | 1.10 | 57.9 | 57.2 | 236 | 5.75 | 2.20 | 67.7 | 101.2 | 423 | 5.71 | 5.40 | 72.9 | 215.1 | 963 |
| q6_K | 8.09 | 0.51 | 44.5 | 49.1 | 115 | 6.56 | 1.30 | 58.1 | 44.9 | 185 | 6.56 | 2.50 | 66.1 | 86.4 | 352 | 6.56 | 6.30 | 74.4 | 183.1 | 837 |
| Budget | Pick (listed GB) | Solved (pass) | J/solve | In-budget comparison (solved, J/solve) | Evidence label |
|---|---|---|---|---|---|
| 0.5 GB | 0.5B- q5_K_M (0.42) | 232 (42.8%) | 41.3 | 0.5B- q4_K_M : 229, 42.3 J; 0.5B- q3_K_M : 220, 38.9 J | lead unresolved (raw 0.810) |
| 1 GB | 1.5B- q4_K_M (0.99) | 316 (58.3%) | 39.3 | 1.5B- q3_K_M : 303, 53.4 J (off the frontier by 26.3%) | lead unresolved (raw 0.203); frontier member, rails frontier margin 76.4% against 3B- q4_K_M |
| 2 GB | 3B- q4_K_M (1.90) | 366 (67.5%) | 69.4 | 1.5B- q4_K_M : 316, 39.3 J (9.2 points fewer). Trap: 3B- q3_K_M (1.60 GB) solves 7 [ 31 ] | significant lead (all-pairs Holm 0.001); frontier member, rails frontier margin 45.8% against 3B- q5_K_M |
| 4 GB | 7B- q3_K_M (3.80) | 388 (71.6%) | 187.0 | 3B- q5_K_M : 367, 101.2 J; 3B- q4_K_M : 366, 69.4 J; 7B- q2_K : 363, 141.2 J | lead unresolved (raw 0.024); off the frontier by 22.0%, unreplicated |
| 8 GB | 7B- q6_K (6.30) | 403 (74.4%) | 183.1 | 7B- q4_K_M (4.70): 396, 145.7 J, smallest file shown equivalent to q8_0 (8.10, over budget) | lead unresolved from 7B- q4_K_M to q6_K (raw 0.248) |
| Comparison | Diff. | Interval | Verdict | |
| A. Top precision minus q4_K_M (90% CI; post hoc margin) | ||||
| 0.5B, fp16 | +3.3 | [+0.9, +5.7] | 0.142 | non-inferiority not shown |
| 1.5B, fp16 | 0.2 | [ 2.6, +2.2] | 1.000 | equivalent |
| 3B, fp16 | 1.3 | [ 3.1, +0.6] | 0.972 | non-inferior |
| 7B, q8_0 | +0.7 | [ 1.1, +2.6] | 1.000 | equivalent |
| B. Top precision minus lowest rung (95% CI) | ||||
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers