While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. Wang et al.'s self-consistency framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces semantic self-consistency, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks.
Figures & tables
Figure 1: Whereas baseline self-consistency comprises three steps: (1) Prompt a model with chain-of-thought, (2) generate n sampled sequences, and (3) choose results based on the most occurring final output, our proposed method, shown above, decides based on the semantic consistency of the employed reasoning path. Our assumption is that language models often apply the correct reasoning but lack the ability to conclude to the correct result.
Dataset
Method/Metric
Llama 2 7B
Mistral 7B
GPT 3.5
Llama 3 8B
GPT-4o mini
AQuA-RAT
Top prob sample
21.65
24.34
53.63
43.02
79.22
SC baseline
24.80
25.60
59.40
45.28
83.07
CPW
24.60 (-0.2)
29.00 (+3.4)
68.00 (+8.6)
46.06 (+0.78)
82.68 (-0.39)
SCW
25.00 (+0.2)
29.80 (+4.2)
65.40 (+6.0)
47.48 (+2.2)
86.18 (+3.11)
SVAMP
Top prob sample
31.90
65.18
77.42
70.55
85.62
SC baseline
46.50
68.50
79.80
73.33
89.80
Table 1: Accuracy comparison of CPW and SCW on different datasets and models, with SciBERT embeddings for AQuA-RAT and SVAMP and RoBERTa embeddings for StrategyQA.
Dataset
Method
Llama 2
Mistral
GPT 3.5
Llama 3
GPT4o mini
Best / Average
Best / Average
Best / Average
AQuA-RAT
SC baseline
24.8 / 24.8
25.6 / 25.6
59.4 / 59.4
45.28 / 45.28
83.07 / 83.07
Isolation Forest
28.45 / 26.04
26.61 / 25.97
65.27 / 63.73
72.25 / 68.59
70.86 / 69.78
K-nearest neighbors
25.40 / 25.37
25.91 / 25.66
62.81 / 60.04
68.10 / 66.74
71.65 / 70.81
One-class SVM
26.70 / 24.25
28.45 / 26.08
59.55 / 59.26
68.39 / 65.91
70.87 / 69.23
SVAMP
SC baseline
46.5 / 46.5
68.5 / 68.5
79.8 / 79.8
73.33 / 73.33
89.80 / 89.80
Table 2: Outlier detection performance on SVAMP, AQuA-RAT, and StrategyQA. Performance increase over baseline of n>1% featured in bold. Encoded based on SciBERT for mathematical reasoning and RoBERTa for commonsense.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Model
Avg. Seq. Length
Avg. Accuracy Increase (%)
Avg. BLEU Score
AQuA-RAT
GPT 3.5
102.40
7.30
0.342
AQuA-RAT
Mistral
53.24
3.80
0.031
AQuA-RAT
Llama 2
49.58
0.00
0.045
AQuA-RAT
Llama 3
56.21
1.49
0.185
AQuA-RAT
GPT-4o mini
83.65
1.36
0.358
SVAMP
GPT 3.5
49.71
0.85
0.440
Appendix
Table 3: Comparison of Sequence Length, Accuracy Increase, and BLEU Score across datasets and models
Model
SVAMP
AQuA-RAT
SQA
Mistral
1.01
2.05
1.50
Llama 2
2.13
0.90
1.25
GPT 3.5
0.33
0.57
0.70
Llama 3
1.20
1.80
1.40
GPT-4o mini
0.80
1.00
0.95
Appendix
Table 4: Accuracy deviation (%) across models and datasets.
Method
Avg. Accuracy (%)
baseline SC
46.50
Varied temp. SC (MV)
46.53
Varied temp. SC (weight)
48.54
Appendix
Table 5: Weighted self-consistency with varying levels of abstraction improves performance over baseline.
BERT-Model
avg distance ( ↓ )
RoBERTa
48.697
MathBERT
45.892 (-2.8)
SciBERT
45.281 (-3.4)
Appendix
Table 6: Featurizers finetuned on similar distributions tend to pack answers more tightly together
Model
AQuA-rat
SVAMP
SQA
Llama 2
24.16
42.47
47.60
Llama 3
46.06
72.33
17.6
Mistral
24.83
62.52
23.73
GPT 3.5
65.52
78.67
21.97
GPT-4o mini
83.46
89.62
36.68
Appendix
Table 7: Performance using k -means for outlier detection, with k=2
Figure 2: Average Rouge-N Scores across StrategyQA, AQuA-RAT, and SVAMP for Different Models
Model
AQuA-RAT
SVAMP
Llama 2
15.5
32.8
Mistral
16.7
47.1
GPT 3.5
25.3
63.9
Appendix
Table 9: Weighting results based on N-Gram overlap with n = 2
Set 1 ( t )
Set 2 ( t )
Set 3 ( t )
0.9
0.7
0.5
0.8
0.6
0.4
0.7
0.5
0.3
0.6
0.4
0.2
0.5
0.3
0.1
Appendix
Table 10: Each Temperature is tested on 1/5 of the samples per generation, to ensure an even distribution.
Table 12
Method
Model
Chosen cluster
Disregarded cluster
SVAMP
LLAMA 2
2.037
2.567
SVAMP
Mistral
2.981
3.800
SVAMP
GPT 3.5
4.428
4.513
SVAMP
GPT 4o mini
4.356
4.653
SVAMP
LLAMA 3
4.562
4.569
AQuA-RAT
LLAMA 2
0.838
0.670
Appendix
Table 11: Average Deviation for clusters
Figure 5: T-SNE reduced image based on a test on a subset of arithmetic reasoning examples, evaluated on 10, 15 and 20 generated outputs based on baseline self-consistency