Strategic Self-Consistency

Tori Qiu; Ander Artola Velasco; Manuel Gomez-Rodriguez

cs.LGSep 24, 2026

Strategic Self-Consistency

Authors: Tori Qiu, Ander Artola Velasco, Manuel Gomez-Rodriguez

Organizations: Carnegie Mellon University, Pittsburgh, USA · Max Planck Institute for Software Systems, Kaiserslautern, Germany

Abstract

Self-consistency has become a popular technique for enhancing the reasoning abilities of large language models by generating multiple reasoning paths and selecting the final answer through a majority vote. However, because model providers typically charge users in proportion to the number of reasoning paths generated, they have a financial incentive to artificially increase the path count. In this work, we show that an unfaithful provider can exploit this incentive using a simple, efficient algorithm while avoiding detection by an auditor: by generating and strategically reordering additional reasoning paths, the algorithm makes every path appear necessary to reach the majority. To validate our algorithm, we conduct experiments with multiple instruct models from the Llama and Qwen families, as well as reasoning models distilled from DeepSeek-R1, on benchmark datasets spanning mathematics, science, and question answering. Our results suggest that the distribution of additional reasoning paths generated by our algorithm is heavy-tailed and that substantial capacity to overcharge remains even under the best possible audit designed to keep the false-positive rate below $α= 0.1$ .

Figures & tables

	Llama-3.2-3B				Qwen2.5-7B				DeepSeek-R1-Distill-Qwen-7B
	Mean	Median	P75	P90	Mean	Median	P75	P90	Mean	Median	P75	P90
GSM8K	13.5	0.0	0.0	5.0	6.0	0.0	0.0	0.0	3.1	0.0	0.0	3.0
AIME	101.0	5.0	17.0	63.0	82.0	3.0	11.0	56.0	69.6	3.0	11.0	39.9
GPQA	22.8	0.0	3.0	17.0	18.2	0.0	2.0	7.0	1.0	0.0	0.0	0.0

Table 1: Summary statistics of the additional answers generated by an unfaithful provider. For each model and dataset, we report the mean, median, 75th percentile (P75), and 90th percentile (P90) of the additional answers

N^{\prime}-N

generated by Algorithm 1 on queries from GSM8K , AIME , and GPQA . In all experiments, we use the ASC stopping rule (Example 2 ) with

\gamma=0.95

and set

\alpha=0.1

Figure 1: Distribution of additional answers generated by an unfaithful provider. For queries from GSM8K , AIME , and GPQA , the figure shows the complementary cumulative distribution

\Pr(N^{\prime}-N\geq x)

of additional answers

N^{\prime}-N

generated by Algorithm 1 . In all experiments, we use the ASC stopping rule (Example 2 ) with

\gamma=0.95

and set

\alpha=0.1

Figure 2: Influence of the audit threshold on an unfaithful provider’s capacity to overcharge. For queries from GSM8K , AIME , and GPQA , the figure shows the

90

th percentile of the distribution of additional answers

N^{\prime}-N

generated by Algorithm 1 for different values of the audit threshold

\alpha

. In all experiments, we use the ASC stopping rule (Example 2 ) with

\gamma=0.95

. DS abbreviates DeepSeek-R1-Distill .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Additional answers generated by Algorithm 1 on synthetic distributions. For

K=2

, we report the empirical mean number of additional answers

N^{\prime}-N

generated by Algorithm 1 (dotted lines), together with the theoretical lower bound from Proposition 4 (solid lines), as a function of the top-two probability gap

p_{(1)}-p_{(2)}

. The shaded regions indicate

\pm 2

standard errors of the average.

Figure 4: Price increase from additional answers. For each query, we plot the additional price, measured in U.S. cents, charged for the $N^{\prime}-N$ additional reasoning paths generated by Algorithm 1 , as a function of the faithful stopping time $N$ . Color indicates the estimated top-two probability gap $\hat{p}_{(1)}-\hat{p}_{(2)}$ , with smaller gaps corresponding to more difficult queries. We set the audit threshold to $\alpha=0.1$ and use the ASC stopping rule with $\gamma=0.95$ .

Figure 5: Distributions of additional reasoning paths under ESC. For queries from GSM8K , AIME , and GPQA , each panel shows the complementary cumulative distribution

\Pr(N^{\prime}-N\geq x)

of the number of additional reasoning paths generated by Algorithm 1 beyond the faithful stopping time. Colors denote datasets, and line width distinguishes model variants within each family. We use ESC stopping rule with window size

5

as the stopping rule and set

\alpha=0.1

Figure 6: Influence of the audit threshold under ESC. For queries from GSM8K , AIME , and GPQA , we report the

90

th percentile of the number of additional answers

N^{\prime}-N

generated by Algorithm 1 using ESC for different audit thresholds

\alpha

. The setting

\alpha=0

corresponds to no audit. DS abbreviates DeepSeek-R1-Distill .

Figure 7: Distributions of additional reasoning paths under adaptive best-of-

\bm{N}

. For queries from GSM8K , AIME , and GPQA , each panel shows the complementary cumulative distribution

\Pr(N^{\prime}-N\geq x)

of additional reasoning paths generated beyond the faithful stopping time by adapting Algorithm 1 to a reward-threshold stopping rule. Colors denote datasets, and line width distinguishes model variants within each family. These experiments set

\alpha=0.1

Strategic Reasoning Risks in Language Models

Strategic reasoning risks in language models concern how planning and social inference can enable deception, manipulation, evasion, or unsafe choices.

See all 39 papers

Multidimensional Consistency Improves Reasoning in Language Models
Mar 4, 2025Huiyuan Lai, Xiao Zhang, Malvina NissimReasoning Consistency in Language Models LLM Mathematical Reasoning
CLCG, University of Groningen / The Netherlands
Reliable Chain-of-Thought via Prefix Consistency
May 8, 2026Naoto Iwase, Yuki Ichihara, Mohammad Atif Quamar +1Self-Consistency Decoding Reasoning Consistency in Language Models
Nagoya University · Nara Institute of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence +1
Optimal Self-Consistency for Efficient Reasoning with Large Language Models
Nov 15, 2025Austin Feng, Marius Alonso, Ambroise Odonnat +2Self-Consistency Decoding Inference-Time Scaling
1Yale University, New Haven, CT, USA · 2Noah’s Ark Lab, Paris, France · 3Inria, Rennes, France +1

Mar 4, 2025J/K move · Enter open · S save

Multidimensional Consistency Improves Reasoning in Language Models

Huiyuan Lai, Xiao Zhang, Malvina Nissim

CLCG, University of Groningen / The Netherlands

Reasoning Consistency in Language Models LLM Mathematical Reasoning Prompt Sensitivity

Open paper AskPDF arXiv

While Large language models (LLMs) have proved able to address some complex reasoning tasks, we also know that they are highly sensitive to input variation, which can lead to different solution paths and final answers. Answer consistency across input variations can thus be taken as a sign of stronger confidence. Leveraging this aspect, we introduce a framework where models are systematically pushed to diversify solution paths towards a final answer, thereby testing them for answer consistency across multiple input variations. Focusing on math problems, we induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used. Experiments on a wide range of open-source LLMs of various sizes show that reasoning consistency differs by variation dimension, and that by aggregating consistency across dimensions, our framework enhances mathematical reasoning performance on monolingual datasets GSM8K and MATH500, and the multilingual dataset MGSM. We also discuss the different efficacy of the types of variations induced and how they could be further leveraged in future work.

Abstract

Strategic Reasoning Risks in Language Models

Explore similar work

Multidimensional Consistency Improves Reasoning in Language Models

Reliable Chain-of-Thought via Prefix Consistency

Optimal Self-Consistency for Efficient Reasoning with Large Language Models