While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. Wang et al.'s self-consistency framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces semantic self-consistency, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks.
Figures & tables
Figure 1: Whereas baseline self-consistency comprises three steps: (1) Prompt a model with chain-of-thought, (2) generate n sampled sequences, and (3) choose results based on the most occurring final output, our proposed method, shown above, decides based on the semantic consistency of the employed reasoning path. Our assumption is that language models often apply the correct reasoning but lack the ability to conclude to the correct result.
Dataset
Method/Metric
Llama 2 7B
Mistral 7B
GPT 3.5
Llama 3 8B
GPT-4o mini
AQuA-RAT
Top prob sample
21.65
24.34
53.63
43.02
79.22
SC baseline
24.80
25.60
59.40
45.28
83.07
CPW
24.60 (-0.2)
29.00 (+3.4)
68.00 (+8.6)
46.06 (+0.78)
82.68 (-0.39)
SCW
25.00 (+0.2)
29.80 (+4.2)
65.40 (+6.0)
47.48 (+2.2)
86.18 (+3.11)
SVAMP
Top prob sample
31.90
65.18
77.42
70.55
85.62
SC baseline
46.50
68.50
79.80
73.33
89.80
Table 1: Accuracy comparison of CPW and SCW on different datasets and models, with SciBERT embeddings for AQuA-RAT and SVAMP and RoBERTa embeddings for StrategyQA.
Dataset
Method
Llama 2
Mistral
GPT 3.5
Llama 3
GPT4o mini
Best / Average
Best / Average
Best / Average
AQuA-RAT
SC baseline
24.8 / 24.8
25.6 / 25.6
59.4 / 59.4
45.28 / 45.28
83.07 / 83.07
Isolation Forest
28.45 / 26.04
26.61 / 25.97
65.27 / 63.73
72.25 / 68.59
70.86 / 69.78
K-nearest neighbors
25.40 / 25.37
25.91 / 25.66
62.81 / 60.04
68.10 / 66.74
71.65 / 70.81
One-class SVM
26.70 / 24.25
28.45 / 26.08
59.55 / 59.26
68.39 / 65.91
70.87 / 69.23
SVAMP
SC baseline
46.5 / 46.5
68.5 / 68.5
79.8 / 79.8
73.33 / 73.33
89.80 / 89.80
Table 2: Outlier detection performance on SVAMP, AQuA-RAT, and StrategyQA. Performance increase over baseline of n>1% featured in bold. Encoded based on SciBERT for mathematical reasoning and RoBERTa for commonsense.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Model
Avg. Seq. Length
Avg. Accuracy Increase (%)
Avg. BLEU Score
AQuA-RAT
GPT 3.5
102.40
7.30
0.342
AQuA-RAT
Mistral
53.24
3.80
0.031
AQuA-RAT
Llama 2
49.58
0.00
0.045
AQuA-RAT
Llama 3
56.21
1.49
0.185
AQuA-RAT
GPT-4o mini
83.65
1.36
0.358
SVAMP
GPT 3.5
49.71
0.85
0.440
Appendix
Table 3: Comparison of Sequence Length, Accuracy Increase, and BLEU Score across datasets and models
Model
SVAMP
AQuA-RAT
SQA
Mistral
1.01
2.05
1.50
Llama 2
2.13
0.90
1.25
GPT 3.5
0.33
0.57
0.70
Llama 3
1.20
1.80
1.40
GPT-4o mini
0.80
1.00
0.95
Appendix
Table 4: Accuracy deviation (%) across models and datasets.
Method
Avg. Accuracy (%)
baseline SC
46.50
Varied temp. SC (MV)
46.53
Varied temp. SC (weight)
48.54
Appendix
Table 5: Weighted self-consistency with varying levels of abstraction improves performance over baseline.
BERT-Model
avg distance ( ↓ )
RoBERTa
48.697
MathBERT
45.892 (-2.8)
SciBERT
45.281 (-3.4)
Appendix
Table 6: Featurizers finetuned on similar distributions tend to pack answers more tightly together
Model
AQuA-rat
SVAMP
SQA
Llama 2
24.16
42.47
47.60
Llama 3
46.06
72.33
17.6
Mistral
24.83
62.52
23.73
GPT 3.5
65.52
78.67
21.97
GPT-4o mini
83.46
89.62
36.68
Appendix
Table 7: Performance using k -means for outlier detection, with k=2
Figure 2: Average Rouge-N Scores across StrategyQA, AQuA-RAT, and SVAMP for Different Models
Model
AQuA-RAT
SVAMP
Llama 2
15.5
32.8
Mistral
16.7
47.1
GPT 3.5
25.3
63.9
Appendix
Table 9: Weighting results based on N-Gram overlap with n = 2
Set 1 ( t )
Set 2 ( t )
Set 3 ( t )
0.9
0.7
0.5
0.8
0.6
0.4
0.7
0.5
0.3
0.6
0.4
0.2
0.5
0.3
0.1
Appendix
Table 10: Each Temperature is tested on 1/5 of the samples per generation, to ensure an even distribution.
Table 12
Method
Model
Chosen cluster
Disregarded cluster
SVAMP
LLAMA 2
2.037
2.567
SVAMP
Mistral
2.981
3.800
SVAMP
GPT 3.5
4.428
4.513
SVAMP
GPT 4o mini
4.356
4.653
SVAMP
LLAMA 3
4.562
4.569
AQuA-RAT
LLAMA 2
0.838
0.670
Appendix
Table 11: Average Deviation for clusters
Figure 5: T-SNE reduced image based on a test on a subset of arithmetic reasoning examples, evaluated on 10, 15 and 20 generated outputs based on baseline self-consistency
Large language models (LLMs) can achieve strong reasoning performance through chain-of-thought (CoT) reasoning, yet they often generate unnecessarily long reasoning paths that incur high inference cost. Self-consistency-based approaches push accuracy higher still, but they require sampling and aggregating multiple reasoning trajectories, leading to substantial computational overhead. In this paper, we introduce a confidence-aware selective sampling framework that, at inference time, analyzes a single reasoning trajectory to adaptively determine whether to rely on that trajectory alone or trigger multi-path sampling. The framework uses trajectory-level numeric features and sentence-level linguistic features extracted from reasoning states to guide selective multi-path reasoning. We train it on MedQA and evaluate it in-domain on MedQA and under calibration-only transfer on MathQA, MedMCQA, and MMLU, without further fine-tuning. Experimental results show that the proposed framework maintains comparable performance to full and efficient multi-path reasoning baselines, with accuracy changes of −0.41±0.58 and −0.31±0.58 percentage points, respectively, while reducing token usage by 71.7±5.0 and 36.6±9.1. These findings demonstrate that reasoning trajectories contain rich signals for uncertainty estimation, enabling a simple, transferable mechanism to balance accuracy and efficiency in LLM reasoning.
Juming Xiong, Kevin Guo, Congning Ni +7
1Vanderbilt University · 2Vanderbilt University Medical Center · 3Intuit AI Research
Large Language Models (LLMs) reason in free-form chain-of-thought (CoT) without revealing which steps were well supported and which alternatives were rejected. In high-stakes domains, users then cannot judge if an answer rests on sound reasoning. Majority voting and multi-agent debate improve answers using model agreement, but do not expose agreement on individual steps. Reasoning Consensus (RC) extracts a typed reasoning graph from each CoT, merges equivalent steps across graphs from diverse models, and weights every step by how many chains attest to it. RC reveals this step-wise support and the varied justifications behind each answer. On 7 benchmarks across law, science, medicine, narrative reasoning and logic, RC matches majority voting and debate in accuracy (69.4% vs. 69.0-70.3%). An LLM judge prefers RC's best-supported graph to other graphs for the same answer. In a user study, participants read RC graphs faster than debate transcripts or vote tallies, and 14 of 18 preferred them for future use.
While Large language models (LLMs) have proved able to address some complex reasoning tasks, we also know that they are highly sensitive to input variation, which can lead to different solution paths and final answers. Answer consistency across input variations can thus be taken as a sign of stronger confidence. Leveraging this aspect, we introduce a framework where models are systematically pushed to diversify solution paths towards a final answer, thereby testing them for answer consistency across multiple input variations. Focusing on math problems, we induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used. Experiments on a wide range of open-source LLMs of various sizes show that reasoning consistency differs by variation dimension, and that by aggregating consistency across dimensions, our framework enhances mathematical reasoning performance on monolingual datasets GSM8K and MATH500, and the multilingual dataset MGSM. We also discuss the different efficacy of the types of variations induced and how they could be further leveraged in future work.