While Large language models (LLMs) have proved able to address some complex reasoning tasks, we also know that they are highly sensitive to input variation, which can lead to different solution paths and final answers. Answer consistency across input variations can thus be taken as a sign of stronger confidence. Leveraging this aspect, we introduce a framework where models are systematically pushed to diversify solution paths towards a final answer, thereby testing them for answer consistency across multiple input variations. Focusing on math problems, we induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used. Experiments on a wide range of open-source LLMs of various sizes show that reasoning consistency differs by variation dimension, and that by aggregating consistency across dimensions, our framework enhances mathematical reasoning performance on monolingual datasets GSM8K and MATH500, and the multilingual dataset MGSM. We also discuss the different efficacy of the types of variations induced and how they could be further leveraged in future work.
Figures & tables
Figure 1: Example of variations: A math problem is presented in different forms or languages, resulting in different reasoning paths and, in some cases, different final answers.
Figure 2: Overview of our Multidimensional Reasoning Consistency (MRC) framework: (i) COC changes the exemplars order; (ii) CPC rewrites the given questions in the same language; and (iii) CLC rewrites the given questions in different languages.
Figure 3: Reasoning accuracy of 4-shot for 8 different exemplar orders. Note that MIN, MEAN, and MAX denote the minimum, mean, and maximum scores, respectively.
Figure 4: Reasoning consistency on three dimensions of variation. Note that COC and CPC are evaluated on the monolingual benchmark GSM8K, while CLC is evaluated on the multilingual benchmark MGSM.
GSM8K
MATH500
Models
CoT
RwS
RwS+
RtS-
RtS
CPC
CoT
RwS
RwS+
RtS-
RtS
CPC
7-8B
Phi3-7B
88.2
84.5
87.0
84.8
88.1
90.0
40.4
36.2
41.8
39.6
43.4
46.6
Qwen2.5-7B
88.3
86.0
89.8
86.3
90.1
92.0
63.6
57.2
63.6
59.0
62.0
72.4
Qwen2.5-Math-7B
90.0
87.6
91.1
89.1
92.3
94.1
75.2
72.8
73.2
72.4
73.2
77.4
Llama3.1-8B
79.7
73.9
78.2
77.9
81.2
83.8
42.6
36.6
45.6
39.0
46.6
50.2
Table 1: Reasoning accuracy of CPC on the GSM8K and MATH500 benchmarks, obtained via aggregating vanilla CoT prompting and 4 different question rewriting settings. The best result for each model across the settings is shown in bold.
Models
BN
DE
EN
ES
FR
JA
RU
SW
TE
TH
ZH
CLC
7-8B
Phi3-7B
14.8
77.6
89.2
85.2
80.4
64.8
74.4
14.0
5.2
18.8
76.0
91.2
Qwen2.5-7B
67.2
72.4
91.6
82.8
72.0
64.8
70.8
16.4
29.2
75.6
74.0
92.8
Qwen2.5-Math-7B
16.8
76.8
92.8
82.0
76.8
61.6
78.8
4.0
5.6
51.2
85.6
93.6
Llama3.1-8B
57.6
64.4
80.8
73.6
63.6
52.4
68.0
55.6
49.6
58.8
63.6
78.8
Aya-expanse-8B
29.2
70.4
77.2
74.8
66.8
60.4
72.0
11.6
6.4
22.8
67.2
82.0
Table 2: Reasoning accuracy of CLC compared with vanilla CoT prompting on the MGSM benchmark. Bold numbers indicate the best result for each model across individual language settings and the CLC results aggregated across languages.
Models
BN
DE
EN
ES
FR
JA
RU
SW
TE
TH
ZH
CLC
7-8B
Phi3-7B
10.8
34.8
40.4
39.8
5.4
20.8
30.0
9.4
5.6
10.0
33.2
44.0
Qwen2.5-7B
33.4
51.6
63.6
50.0
33.0
50.2
55.8
17.6
17.0
42.2
49.0
67.4
Qwen2.5-Math-7B
29.2
61.2
75.2
62.0
53.4
50.8
65.8
3.8
16.2
23.6
70.6
79.4
Llama3.1-8B
22.2
31.0
42.6
35.4
7.2
29.4
38.6
24.8
9.4
26.8
32.8
47.2
Aya-expanse-8B
10.6
25.8
28.0
26.6
25.6
24.0
25.2
8.6
5.6
11.0
24.2
31.2
Table 3: Reasoning accuracy of CLC compared to vanilla CoT prompting on the machine-translated MATH500. Note that bold numbers indicate the best result for each model among different languages and CLC.
Datasets
MGSM
MATH500
Models
COC
CPC
CLC
O&P
O&L
P&L
MRC
COC
CPC
CLC
O&P
O&L
P&L
MRC
7-8B
Phi3-7B
92.4
92.0
91.2
93.2
93.2
93.2
94.8
43.2
46.6
44.0
49.0
47.6
51.2
49.6
Qwen2.5-7B
92.0
93.2
92.8
93.2
93.6
93.6
93.6
68.8
72.4
67.4
72.6
72.4
73.4
74.0
Qwen2.5-Math-7B
94.4
94.8
93.6
94.4
95.6
96.0
96.0
77.0
77.4
79.4
78.6
79.6
79.8
79.8
Llama3.1-8B
80.8
85.6
78.8
83.6
84.0
87.6
84.8
48.0
50.2
47.2
51.6
51.4
53.0
53.6
Table 4: Accuracy on MGSM and MATH500. Notes: (i) CPC uses 5 solution paths, COC and CLC use 8 each, and CLC uses 8 languages (excl. BN, SW, TE), so MRC contains a total of 19 paths (excluding the two identical English paths); (ii) best result for each model is bolded.
Figure 5: Reasoning accuracy of using varying numbers of reasoning paths on GSM8K.
Figure 6: Pearson correlation between models’ accuracy and different consistency scores.
Table 5: Case study from the MGSM test set, where solutions are generated by Qwen2.5-72B for the same question written in different languages. Note that, for brevity, we only show the English (EN) question here; the corresponding questions in the other languages are provided in Appendix A.4 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Examples of prompts for COC.
Figure 8: Examples of prompts for CPC.
Figure 9: Examples of prompts for CLC.
Orders (4-shot)
1
2
3
4
5
6
7
8
COC
7-8B
Phi3-7B
88.5
88.8
88.6
88.9
87.3
89.8
88.4
88.2
89.8
Qwen2.5-7B
88.3
87.0
87.6
88.0
88.0
87.8
88.5
88.2
90.5
Qwen2.5-Math-7B
90.0
89.7
91.3
92.3
91.5
90.5
90.8
91.0
92.6
Llama3.1-8B
79.7
78.8
79.0
79.0
80.1
79.1
77.8
78.7
80.1
Aya-expanse-8B
76.7
77.3
78.2
76.6
76.9
77.6
77.3
77.4
78.2
Appendix
Table 6: Reasoning accuracy of prompts in different orders on GSM8K compared to COC.
Orders (4-shot)
1
2
3
4
5
6
7
8
COC
7-8B
Phi3-7B
40.4
41.0
40.0
40.8
39.0
42.2
40.0
38.6
43.2
Qwen2.5-7B
63.6
64.6
64.8
64.4
63.6
65.2
61.8
62.4
68.8
Qwen2.5-Math-7B
75.2
74.4
75.2
75.6
73.2
75.4
74.6
74.2
77.0
Llama3.1-8B
42.6
44.0
41.8
41.4
42.4
41.6
45.0
42.6
48.0
Aya-expanse-8B
28.0
28.8
29.0
27.4
28.6
28.8
27.0
29.0
31.4
Appendix
Table 7: Reasoning accuracy of prompts in different orders on MATH500 compared to COC.
Figure 10: Examples of questions written in different languages.
Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.