When Self-Consistency Hurts: The Hidden Cost of Majority Voting in Small Language Models
Organizations: Department of Computer Science, University of Liverpool, Liverpool, United Kingdom · LIPEX, Shenzhen, China
Abstract
We study prompted chain-of-thought (CoT) with self-consistency (SC), i.e. sampling N traces and plurality-voting, on small language models, a regime where per-query compute budgets are tight yet SC is commonly applied with a fixed sample budget inherited from large-model practice, regardless of instance difficulty. Across two model families (Qwen-2.5-0.5B, Gemma-3-1B) at FP16 and MLX-Q4 (4-bit quantisation in Apple's MLX framework), on GSM8K and MATH-500, we run 8 per-task and 4 task-pooled experiments at n=200-400 with 1,000x paired-bootstrap confidence intervals (CIs). Our first result is an upper bound: an oracle gate deciding per prompt whether to invoke SC reaches 109-174% of always-SC's accuracy at 23-30% of its compute in all 12/12 cases, so at N=5 always-SC is dominated everywhere on our grid. That margin is exactly the rate at which SC flips correct greedy answers to wrong, here 3-10.5% of prompts, which on Qwen-Q4 x MATH-500 exceeds the wrong-to-right rate and leaves always-SC 3 pt worse than always-greedy. Our second result is a deployable approximation: a gate over greedy-pass signals skips 45-99% of SC calls, with bootstrap CI half-widths under 5 pt, removing 36-79% of always-SC compute. Accuracy retention at those operating points has point estimates of 98.1-98.8%, which we report as point estimates because their CIs are wide (lower bounds 76-86%) over an SC accuracy of only 15-36%. The evidence covers N=5 on two <=1B families and two math benchmarks, which is the scope we claim. Code is available at https://github.com/heywanrong/SiftSC.
Figures & tables
| Cell | Dataset | |||||
|---|---|---|---|---|---|---|
| Qwen-FP16 | GSM8K | |||||
| Qwen-Q4 | GSM8K | |||||
| Gemma-FP16 | GSM8K | |||||
| Gemma-Q4 | GSM8K | |||||
| Qwen-FP16 | MATH-500 | |||||
| Qwen-Q4 | MATH-500 |
| Cell | Method | skip CI | CI | CI | |
|---|---|---|---|---|---|
| Qwen-FP16 | logistic | ||||
| Qwen-Q4 | conf_only | ||||
| Gemma-FP16 | logistic | ||||
| Gemma-Q4 | logistic |
Self-Consistency in Language Models
Self-consistency in language models studies whether repeated outputs agree and how agreement can guide reasoning, confidence, and behavior.
See all 54 papers