A Large-Scale Evaluation of Role-Enhanced Zero-Shot Chain-of-Thought Prompting

Ibrahim Khan; Thai Van Nguyen; Yi Xia; Pratch Suntichaikul; Mustafa Can Gursesli; Daniel Fernandez Galeote; Juho Hamari; Ruck Thawonmas

cs.CLOct 21, 2026

A Large-Scale Evaluation of Role-Enhanced Zero-Shot Chain-of-Thought Prompting

Authors: Ibrahim Khan, Thai Van Nguyen, Yi Xia, Pratch Suntichaikul, Mustafa Can Gursesli, Daniel Fernandez Galeote, Juho Hamari, Ruck Thawonmas

Organizations: College of Information Science and Technology, Ritsumeikan University, Japan · Graduate School of Information Science and Technology, Ritsumeikan University, Japan · Independent Researcher, and an alumnus of the Graduate School of Information Science and Technology, Ritsumeikan University, Japan · Research Centre of Gameful Realities, Gamification Group, Tampere University, Finland

Abstract

This paper proposes Role-Enhanced Zero-Shot Chain-of-Thought (RE-ZS-CoT), a prompting framework that systematically integrates domain-specific expert roles to enhance reasoning in large language models (LLMs) without task-specific examples. We assess two versions of our approach. In the first version, static role selection, we evaluate eight predefined expert roles across arithmetic, commonsense, and reading comprehension benchmarks to determine their impact on reasoning performance. In the second version, dynamic role selection, each model autonomously selects the single most suitable role for each benchmark, enabling adaptive task-role alignment without manual specification. Across 20 LLMs and seven benchmarks, RE-ZS-CoT frequently surpasses zero-shot and zero-shot-CoT baselines, achieving performance improvements of up to 44 percentage points. The results reveal a strong correspondence between static and dynamic versions; the benchmarks where static roles yielded improvements were almost always those where dynamic roles also outperformed baselines, though static selection achieved peak performance. Ablation studies further show that unconventional roles degrade performance, underscoring the necessity of task-role alignment. The framework's scalability is validated across model sizes (2b-14b parameters), with smaller models like Granite3.3-8b achieving remarkable proficiency in arithmetic reasoning. RE-ZS-CoT advances zero-shot prompting by unifying role specialization with chain-of-thought reasoning, offering a practical, resource-efficient solution for improving LLM capabilities across diverse domains.

Figures & tables

Fig. 1: Examples of prompts and corresponding responses generated by Gemma3:12b model on the AQuA-RAT benchmark comparing zero-shot and zero-shot-CoT baselines to our RE-ZS-CoT and RE-ZS. The reasoning of the model is omitted for readability.

Fig. 2: Our approach’s pre and post questions, and split-phrase variants with example prompts. Figure 1 demonstrates the 2nd variant in action.

	AQuA-RAT				GSM8K				OpenbookQA
Model	0S	0CoT	Ours (S)	Ours (D)	0S	0CoT	Ours (S)	Ours (D)	0S	0CoT	Ours (S)	Ours (D)
GPT4.1	80.31	67.72	82.68	68.50	86.81	86.96	85.75	$\equiv$	79.40	87.20	94.4	93.00
GPT-4.1-mini	83.07	85.04	86.22	78.35	89.16	87.04	87.79	87.11	89.00	96.20	96.20	96.40
GPT-4.1-nano	75.98	71.65	77.95	70.87	87.87	86.73	86.66	85.37	87.40	86.20	91.40	90.00
GPT-4o	72.44	71.65	80.31	$\equiv$	88.63	89.08	88.48	$\equiv$	96.60	95.60	95.00	92.60
GPT-4o-mini	67.32	69.29	78.74	73.62	88.48	88.02	88.78	$\equiv$	92.60	80.20	84.80	83.60

TABLE I: Results of all models on the AQuA-RAT, GSM8K, and OpenbookQA datasets. The results are presented in zero-shot/zero-shot-CoT/Ours (static)/Ours (dynamic) format. Underlined scores indicate when our approach outperforms the baselines but is not the best result. Symbol “

\equiv

” indicates that the dynamic role was identical to the static role.

	StrategyQA				RACE-h				RACE-m				TriviaQA
Model	0S	0CoT	Ours (S)	Ours (D)	0S	0CoT	Ours (S)	Ours (D)	0S	0CoT	Ours (S)	Ours (D)	0S	0CoT	Ours (S)	Ours (D)
GPT-4.1	83.14	84.37	84.89	84.54	81.07	91.91	91.97	92.08	89.28	94.99	95.19	95.33	86.20	86.40	87.40	87.50
GPT-4.1-mini	79.30	81.09	82.27	81.66	84.19	89.37	90.37	89.68	89.69	93.18	94.08	93.87	79.90	82.80	84.90	84.50
GPT-4.1-nano	70.09	68.86	76.03	74.67	79.22	80.76	83.16	82.53	86.77	88.58	90.32	90.04	73.90	79.40	80.00	80.30
GPT-4o	80.57	81.09	82.45	82.27	92.31	89.25	89.11	89.57	94.71	93.38	94.29	94.22	85.60	87.00	87.10	87.00
GPT-4o-mini	76.90	79.04	79.30	79.34	87.25	86.59	87.34	86.54	92.27	92.76	93.11	92.62	77.70	79.50	80.60	79.90

TABLE II: Results of all models on the StrategyQA, RACE-h, RACE-m, and TriviaQA datasets. The results are presented in zero-shot/zero-shot-CoT/Ours (static)/Ours (dynamic) format. Underlined scores indicate when our approach outperforms the baselines but is not the best result.

	Poet
Model	AQuA-RAT	OpenBookQA	TriviaQA
GPT-4.1-nano	77.95 /37.01	91.40 /87.06	80.30 /78.50
GPT-4o-mini	78.74 /20.08	84.80 /75.00	80.60 /77.80
Gemma2-9b	55.51 /3.94	36.80/ 39.00	70.70 /63.60
Granite3.2-8b	60.63 /38.98	66.60 /53.00	70.60 /65.60
Llama3.2-3b	51.97 /35.04	62.00 /45.00	66.50 /61.00

TABLE III: Results for poet, comedian, and politician on AQuA-RAT, OpenBookQA, and TriviaQA benchmarks. Displayed as RE-ZS-CoT/auxiliary roles.

Panel of Figure 7 — Fig. 4: Role-phrase position results: comprehensive analysis of total counts across all benchmarks and all LLMs, respectively.

Panel of Figure 8 — Fig. 5: Effect of roles across all benchmarks.

Model	AQuA-RAT	OpenBookQA.	TriviaQA
GPT-4.1-nano	75.98/ 79.53	87.40 /84.40	73.90/ 76.00
GPT-4o-mini	67.32/ 79.92	92.60 /87.20	77.70/ 78.50
Gemma2-9b	54.72/54.72	88.60/ 88.80	69.70/ 70.70
Granite3.2-8b	53.54/ 56.69	66.80 /65.20	68.60/ 70.50
Llama3.2-3b	55.91 /54.72	30.20/ 49.80	61.60/ 66.10
Qwen2.5-3b	61.81 /60.63	68.60 /54.80	50.30/ 52.10

TABLE IV: Results for our Role-enhanced zero-shot on AQuA-RAT, OpenBookQA, and TriviaQA benchmarks. Displayed as zero-shot/RE-ZS.

	AQuA-RAT			OpenBookQA			TriviaQA
Model	w/o “Expert"	“Top"	“Amateur"	w/o “Expert"	“Top"	“Amateur"	w/o “Expert"	“Top"	“Amateur"
GPT-4.1-nano	0 %	0.40%	-0.39%	-1.84%	-5.34%	-6.24%	-7.24%	-0.10%	-1.00%
GPT-4o-mini	-1.18%	-0.51%	-5.79%	-2.03%	-2.33%	-3.93%	-1.10%	-0.80%	-2.90%
Gemma2-9b	-4.57%	-4.18%	-4.97%	-0.56%	-1.25%	-1.44%	-1.40%	-1.80%	-3.80%
Granite3.2-8b	-3.15%	-3.94%	-4.15%	-4.4%	-2.4%	-5.7%	0 %	-0.90%	-1.30%
Llama3.2-3b	-1.97%	3.54%	-2.39%	-12.00%	-18.2%	-20.6%	-1.80%	-2.30%	-3.20%

TABLE V: Results for lexical variations on AQuA-RAT, OpenBookQA, and TriviaQA benchmarks. Displayed as the relative performance increase or decrease from the RE-ZS-CoT.

Appendix figures & tables57 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Test split	Count
AQuA-RAT	AR	test	254
GSM8K	AR	test	1319
OepnbookQA	CR	dev	500
StrategyQA	CR	test	2290
RACE-m	RC	middle-test	1436
RACE-h	RC	high-test	3498

Appendix

TABLE VI: Details of each dataset. Test split shows the split used for evaluations in this research, while Count shows the number of included samples. For the Task , AR: Arithmetic Reasoning, CR: Commonsense Reasoning, RC: Reading Comprehension. * TriviaQA was downsampled to only 1000 records to save computational resources.

	Algebra			Count. & Prob			Geometry			Int. Algebra			Num. Th.			Prealgebra			Precalculus
Model	0S	0CoT	Ours(S)	0S	0CoT	Ours(S)	0S	0CoT	Ours(S)	0S	0CoT	Ours(S)	0S	0CoT	Ours(S)	0S	0CoT	Ours(S)	0S	0CoT	Ours(S)
GPT-4.1	88.12	87.87	88.54	79.75	81.01	80.59	70.98	69.73	73.31	71.87	72.20	73.09	91.30	90.19	91.11	83.01	83.81	83.70	63.74	65.75	64.10
GPT-4.1-mini	90.06	90.56	91.07	81.43	81.01	83.33	72.86	74.74	75.78	80.07	78.96	81.95	91.85	92.22	92.04	85.53	84.50	85.42	74.91	72.71	74.91
GPT-4.1-nano	85.93	86.86	87.45	70.68	70.46	72.78	61.17	60.54	62.63	65.01	66.67	66.11	85.19	84.07	87.04	78.30	78.87	81.29	56.78	62.09	61.90
GPT-4o	87.53	86.86	88.15	69.83	70.89	72.36	58.66	57.83	60.33	57.59	55.91	57.25	79.44	79.07	81.67	80.02	81.06	82.93	54.03	52.56	57.46
GPT-4o-mini	84.92	84.92	86.10	67.09	64.56	66.67	54.70	56.99	56.16	51.94	52.38	54.15	78.89	78.52	79.63	78.42	80.25	80.94	49.08	49.63	49.08

Appendix

TABLE VII: Results of all models on the MATH benchmark. The results are presented in zero-shot/zero-shot-CoT/RE-ZS-CoT(static) format.

Panel of Figure 13 — Fig. 6: Effect of roles and self-alignment across all benchmarks.

Model	Mathematician	Physicist	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
GPT-4.1-mini	100	9	0	1	2	88	0
GPT-4.1-nano	99	0	60	0	17	10	14
GPT-4o-mini	100	0	0	100	0	0	0

Appendix

TABLE VIII: Static Role Self-Alignment on AQuA (GPT Models)

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
GPT-4.1-mini	100	5	0	13	0	39	30	13
GPT-4.1-nano	100	0	1	82	0	16	1	0
GPT-4o-mini	100	0	0	0	63	37	0	0

Appendix

TABLE IX: Static Role Self-Alignment on GSM8K (GPT Models)

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
GPT-4.1-mini	7	8	0	92	4	0	0	89
GPT-4.1-nano	1	1	1	100	6	9	7	75
GPT-4o-mini	0	0	0	47	83	0	0	70

Appendix

TABLE X: Static Role Self-Alignment on OpenbookQA (GPT Models)

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
GPT-4.1-mini	3	2	0	97	0	0	0	98
GPT-4.1-nano	0	0	0	100	0	10	1	89
GPT-4o-mini	2	0	0	25	70	4	0	99

Appendix

TABLE XI: Static Role Self-Alignment on StrategyQA (GPT Models)

Model	Scientist	Literacy Specialist	Software Developer	Polymath
GPT-4.1-mini	0	100	85	15
GPT-4.1-nano	33	100	17	49
GPT-4o-mini	4	100	90	6

Appendix

TABLE XII: Static Role Self-Alignment on RACE(M&h) (GPT Models)

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
GPT-4.1-mini	5	5	0	16	76	0	36	62
GPT-4.1-nano	0	0	0	96	13	0	87	4
GPT-4o-mini	0	0	0	47	92	2	11	48

Appendix

TABLE XIII: Static Role Self-Alignment on TriviaQA (GPT Models)

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
Phi4-14b	100	12	0	0	17	0	66	5
Qwen2.5-14b	100	39	0	0	14	29	15	3
Gemma3-12b	4	0	0	0	0	0	99	97
Mistral-nemo-12b	95	0	0	7	82	2	13	1
Gemma2-9b	100	0	0	2	0	1	95	2
Granite3.2-8b	99	12	1	28	24	1	35	0

Appendix

TABLE XIV: Static Role Self-Alignment on AQuA

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
Phi4-14b	100	9	0	4	11	9	38	29
Qwen2.5-14b	100	15	0	0	0	78	5	2
Gemma3-12b	97	0	0	0	0	0	17	86
Mistral-nemo-12b	94	1	0	1	32	34	32	6
Gemma2-9b	100	0	0	7	76	1	14	2
Granite3.2-8b	98	9	1	9	12	12	59	0

Appendix

TABLE XV: Static Role Self-Alignment on GSM8K

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
Phi4-14b	27	20	1	22	51	0	2	77
Qwen2.5-14b	37	37	0	53	5	5	0	63
Gemma3-12b	0	0	0	100	0	0	0	100
Mistral-nemo-12b	0	0	1	88	25	3	0	83
Gemma2-9b	1	0	0	98	0	0	10	91
Granite3.2-8b	3	67	4	96	10	0	7	13

Appendix

TABLE XVI: Static Role Self-Alignment on OpenbookQA

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
Phi4-14b	51	12	1	56	0	4	17	59
Qwen2.5-14b	34	16	2	3	0	79	40	26
Gemma3-12b	0	0	0	94	0	0	6	100
Mistral-nemo-12b	0	5	0	82	3	9	14	87
Gemma2-9b	24	0	0	99	0	32	35	10
Granite3.2-8b	1	35	3	91	12	1	33	24

Appendix

TABLE XVII: Static Role Self-Alignment on StrategyQA

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
Phi4-14b	13	9	2	38	69	2	63	2
Qwen2.5-14b	2	2	0	0	98	0	95	3
Gemma3-12b	5	8	1	70	95	13	7	1
Mistral-nemo-12b	16	5	1	30	97	2	4	45
Gemma2-9b	9	0	0	46	100	1	44	0
Granite3.2-8b	65	60	13	25	11	10	3	10

Appendix

TABLE XVIII: Static Role Self-Alignment on RACE(M&h)

Model	Mathematician	Physicist	Physician	Scientist	Literacy Specialist	Engineer	Software Developer	Polymath
Phi4-14b	47	48	1	24	27	1	9	43
Qwen2.5-14b	47	50	0	8	43	2	1	49
Gemma3-12b	0	0	0	0	0	0	100	100
Mistral-nemo-12b	0	1	0	80	39	4	9	67
Gemma2-9b	1	0	0	80	17	0	7	94
Granite3.2-8b	4	27	2	92	54	2	9	10

Appendix

TABLE XIX: Static Role Self-Alignment on TriviaQA

Model	AQuA	GSM8K	OpenbookQA
GPT-4.1	Mathematics Educator (58)	Mathematician (87)	Cognitive Scientist (76)
GPT-4.1-mini	Mathematics Educator (95)	Mathematician (89)	Cognitive Scientist (81)
GPT-4.1-nano	Mathematics Educator (89)	Mathematics Educator (40)	Science Educator (37)
GPT-4o	Mathematician (61)	Mathematician (99)	Cognitive Scientist (96)
GPT-4o-mini	Mathematics Educator (38)	Mathematician (75)	Research Scientist (43)
Phi4-14b	Mathematics Educator (31)	Mathematics Educator (32)	Cognitive Scientist (14)

Appendix

TABLE XX: Dynamic Role Self-Alignment across three reasoning benchmarks (AQuA, GSM8K, and OpenbookQA). The numbers in parentheses indicate the total occurrences of the role among 100 model responses.

Model	StrategyQA	RACE(M&h)	TriviaQA
GPT-4.1	Research Scientist (39)	English Teacher (81)	Trivia Expert (62)
GPT-4.1-mini	Data Scientist (74)	Reading Comprehension Specialist (48)	Data Scientist (68)
GPT-4.1-nano	Logic Expert (21)	Educational Researcher (24)	Reading Comprehension Researcher (28)
GPT-4o	Cognitive Scientist (82)	English Teacher (26)	Trivia Expert (63)
GPT-4o-mini	Data Scientist (75)	Educational Assessment Specialist (43)	Data Scientist (98)
Phi4-14b	Ai Research Scientist (13)	Reading Specialist (13)	Knowledge Engineer (22)

Appendix

TABLE XXI: Dynamic Role Self-Alignment across three reasoning benchmarks (StrategyQA, RACE(M&h), and TriviaQA). The numbers in parentheses indicate the total occurrences of the role among 100 model responses.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	X	X	X	84.41	X	X	X
Mathematician (Post-question)	X	X	X	84.24	X	X	X
Mathematician (Split)	X	X	X	84.89	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XXII: Results for Static Role-Alignment on all prompt variants of GPT-4.1 on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	79.53	87.64	92.60	82.27	89.05	92.97	84.20
Mathematician (Post-question)	81.50	86.43	94.40	81.88	89.51	94.08	83.90
Mathematician (Split)	78.74	86.13	95.60	81.75	89.71	93.87	84.50
Physicist (Pre-question)	81.10	87.49	95.40	81.44	89.91	93.38	83.70
Physicist (Post-question)	77.95	85.75	93.80	81.35	89.19	93.87	83.90
Physicist (Split)	79.53	86.13	94.40	81.40	90.14	93.11	83.40

Appendix

TABLE XXIII: Results for Static Role-Alignment on all prompt variants of GPT-4.1-mini on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	65.75	86.66	87.20	76.03	79.82	87.19	76.60
Mathematician (Post-question)	65.75	84.53	86.20	75.11	82.33	89.42	79.50
Mathematician (Split)	62.99	84.84	86.20	75.28	81.59	89.97	78.70
Physicist (Pre-question)	67.32	85.60	88.40	72.84	79.65	87.60	76.40
Physicist (Post-question)	61.02	83.24	87.60	73.71	81.65	90.04	78.50
Physicist (Split)	62.99	83.24	86.00	73.80	81.22	89.35	79.30

Appendix

TABLE XXIV: Results for Static Role-Alignment on all prompt variants of GPT-4.1-nano on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	64.57	88.48	94.80	80.22	7.15	94.22	85.70
Mathematician (Post-question)	68.90	80.82	91.00	82.45	X	92.13	85.80
Mathematician (Split)	70.08	X	94.60	81.22	X	7.94	85.70
Physicist (Pre-question)	80.31	X	91.80	X	X	X	72.10
Physicist (Post-question)	72.44	X	86.40	X	X	X	X
Physicist (Split)	74.41	X	89.20	X	X	X	X

Appendix

TABLE XXV: Results for Static Role-Alignment on all prompt variants of GPT-4o on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	68.50	88.32	82.40	78.82	86.62	92.76	78.80
Mathematician (Post-question)	62.60	88.70	75.60	79.30	86.99	92.20	79.10
Mathematician (Split)	66.93	87.72	81.00	77.99	86.19	92.48	78.40
Physicist (Pre-question)	72.05	87.49	83.80	78.30	87.31	92.83	78.30
Physicist (Post-question)	69.69	88.70	74.40	78.34	86.54	92.20	79.00
Physicist (Split)	67.72	87.34	78.20	77.90	86.74	92.20	79.20

Appendix

TABLE XXVI: Results for Static Role-Alignment on all prompt variants of GPT-4o-mini on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	52.36	89.54	X	X	X	X	74.60
Mathematician (Post-question)	66.54	87.57	X	X	X	X	73.90
Mathematician (Split)	54.72	88.70	X	X	X	X	73.90
Physicist (Pre-question)	X	X	X	X	X	X	74.40
Physicist (Post-question)	X	X	X	X	X	X	73.90
Physicist (Split)	X	X	X	X	X	X	72.90

Appendix

TABLE XXVII: Results for Static Role-Alignment on all prompt variants of Phi4-14b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	59.06	87.34	X	X	X	X	X
Mathematician (Post-question)	59.06	87.26	X	X	X	X	X
Mathematician (Split)	52.76	85.97	X	X	X	X	X
Physicist (Pre-question)	64.57	0.61	X	X	X	X	72.90
Physicist (Post-question)	64.57	X	X	X	X	X	72.60
Physicist (Split)	61.81	X	X	X	X	X	72.30

Appendix

TABLE XXVIII: Results for Static Role-Alignment on all prompt variants of Qwen2.5-14b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	X	86.58	X	X	X	X	X
Mathematician (Post-question)	X	84.23	X	X	X	X	X
Mathematician (Split)	X	84.08	X	X	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XXIX: Results for Static Role-Alignment on all prompt variants of Gemma3-12b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	58.66	80.06	X	X	X	X	X
Mathematician (Post-question)	57.09	79.45	X	X	X	X	X
Mathematician (Split)	59.45	80.82	X	X	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XXX: Results for Static Role-Alignment on all prompt variants of Mistral-nemo-12b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	55.51	84.00	X	X	X	X	X
Mathematician (Post-question)	53.54	84.84	X	X	X	X	X
Mathematician (Split)	52.36	84.38	X	X	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XXXI: Results for Static Role-Alignment on all prompt variants of Gemma2-9b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	54.72	77.10	X	X	X	X	X
Mathematician (Post-question)	57.48	79.38	X	X	X	X	X
Mathematician (Split)	60.63	76.88	X	X	X	X	X
Physicist (Pre-question)	X	X	45.80	57.82	X	X	X
Physicist (Post-question)	X	X	63.60	59.04	X	X	X
Physicist (Split)	X	X	66.60	59.34	X	X	X

Appendix

TABLE XXXII: Results for Static Role-Alignment on all prompt variants of Granite3.2-8b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	62.60	77.94	70.60	61.92	X	X	X
Mathematician (Post-question)	57.09	78.77	75.40	67.12	X	X	X
Mathematician (Split)	59.06	79.30	71.40	64.98	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XXXIII: Results for Static Role-Alignment on all prompt variants of Granite3.3-8b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	60.24	79.45	X	X	X	X	X
Mathematician (Post-question)	54.72	79.38	X	X	X	X	X
Mathematician (Split)	59.06	79.53	X	X	X	X	X
Physicist (Pre-question)	X	X	51.80	67.99	X	X	71.10
Physicist (Post-question)	X	X	39.00	64.54	X	X	73.00
Physicist (Split)	X	X	51.80	69.04	X	X	72.90

Appendix

TABLE XXXIV: Results for Static Role-Alignment on all prompt variants of Llama3.1-8b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	14.96	42.84	X	35.07	X	X	45.50
Mathematician (Post-question)	15.35	41.02	X	34.10	X	X	46.90
Mathematician (Split)	20.08	41.77	X	36.11	X	X	46.00
Physicist (Pre-question)	17.72	42.53	35.20	33.36	24.70	39.69	42.20
Physicist (Post-question)	12.60	42.23	33.80	26.24	16.92	30.29	40.30
Physicist (Split)	16.14	40.49	34.20	29.21	19.18	33.29	42.70

Appendix

TABLE XXXV: Results for Static Role-Alignment on all prompt variants of Mistral-7b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	61.81	82.79	X	X	X	X	64.60
Mathematician (Post-question)	66.14	84.69	X	X	X	X	63.90
Mathematician (Split)	70.87	85.97	X	X	X	X	63.90
Physicist (Pre-question)	X	X	X	X	X	X	63.60
Physicist (Post-question)	X	X	X	X	X	X	64.50
Physicist (Split)	X	X	X	X	X	X	63.40

Appendix

TABLE XXXVI: Results for Static Role-Alignment on all prompt variants of Qwen2.5-7b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	51.57	X	X	X	X	X	X
Mathematician (Post-question)	51.18	X	X	X	X	X	X
Mathematician (Split)	51.97	X	X	X	X	X	X
Physicist (Pre-question)	X	X	57.80	66.90	X	X	X
Physicist (Post-question)	X	X	55.20	64.59	X	X	X
Physicist (Split)	X	X	57.80	64.06	X	X	X

Appendix

TABLE XXXVII: Results for Static Role-Alignment on all prompt variants of Llama3.2-3b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	58.27	72.40	52.60	57.86	X	X	55.30
Mathematician (Post-question)	58.66	76.72	54.00	60.92	X	X	55.20
Mathematician (Split)	59.45	78.17	55.60	60.39	X	X	54.90
Physicist (Pre-question)	X	X	48.60	57.51	X	X	X
Physicist (Post-question)	X	X	51.20	59.87	X	X	X
Physicist (Split)	X	X	49.40	59.91	X	X	X

Appendix

TABLE XXXVIII: Results for Static Role-Alignment on all prompt variants of Qwen2.5-3b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	28.35	62.47	X	57.12	X	X	X
Mathematician (Post-question)	26.38	61.56	X	54.98	X	X	X
Mathematician (Split)	25.59	61.41	X	54.45	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XXXIX: Results for Static Role-Alignment on all prompt variants of Gemma2-2b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	41.34	63.15	50.80	53.58	54.55	59.89	52.80
Mathematician (Post-question)	41.73	62.09	50.40	54.06	55.60	60.58	52.80
Mathematician (Split)	40.16	63.68	50.80	56.46	56.09	63.51	53.00
Physicist (Pre-question)	42.91	61.18	50.20	53.45	56.75	62.47	54.30
Physicist (Post-question)	39.37	64.14	49.80	53.97	57.72	62.88	54.70
Physicist (Split)	39.37	62.77	52.00	55.20	56.83	61.56	53.50

Appendix

TABLE XL: Results for Static Role-Alignment on all prompt variants of Granite3.2-2b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Prompt	AQuA-RAT	GSM8K	OpenbookQA	StrategyQA	RACE-h	RACE-m	TriviaQA
Mathematician (Pre-question)	53.15	70.43	X	55.24	X	X	X
Mathematician (Post-question)	51.57	73.62	X	58.08	X	X	X
Mathematician (Split)	53.15	69.14	X	56.81	X	X	X
Physicist (Pre-question)	X	X	X	X	X	X	X
Physicist (Post-question)	X	X	X	X	X	X	X
Physicist (Split)	X	X	X	X	X	X	X

Appendix

TABLE XLI: Results for Static Role-Alignment on all prompt variants of Granite3.3-2b on AQuA-RAT, GSM8K, OpenbookQA, StrategyQA, RACE-h, RACE-m, TriviaQA. Symbol ’X’ indicates that the role was not in the top two selected roles.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	68.50
Mathematics Educator (Post-question)	62.20
Mathematics Educator (Split)	61.81
GSM8K
(Same as Static)	$\equiv$

Appendix

TABLE XLII: Performance of dynamically-selected roles for GPT-4.1.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	74.80
Mathematics Educator (Post-question)	78.35
Mathematics Educator (Split)	77.17
GSM8K
Mathematician (Pre-question)	87.11

Appendix

TABLE XLIII: Performance of dynamically-selected roles for GPT-4.1-mini.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	69.69
Mathematics Educator (Post-question)	66.14
Mathematics Educator (Split)	70.87
GSM8K
Mathematics Educator (Pre-question)	85.37

Appendix

TABLE XLIV: Performance of dynamically-selected roles for GPT-4.1-nano.

AQuA-RAT
Dataset and Prompt Variant	Accuracy
(Same as Static)	$\equiv$
GSM8K
(Same as Static)	$\equiv$
OpenbookQA
Cognitive Scientist (Pre-question)	92.60

Appendix

TABLE XLV: Performance of dynamically-selected roles for GPT-4o.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	72.44
Mathematics Educator (Post-question)	73.62
Mathematics Educator (Split)	70.47
GSM8K
(Same as Static)	$\equiv$

Appendix

TABLE XLVI: Performance of dynamically-selected roles for GPT-4o-mini.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	53.15
Mathematics Educator (Post-question)	65.35
Mathematics Educator (Split)	54.33
GSM8K
Mathematics Educator (Pre-question)	89.31

Appendix

TABLE XLVII: Performance of dynamically-selected roles for Phi4-14b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	60.24
Mathematics Educator (Post-question)	59.84
Mathematics Educator (Split)	50.39
GSM8K
Math Teacher (Pre-question)	87.72

Appendix

TABLE XLVIII: Performance of dynamically-selected roles for Qwen2.5-14b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	50.79
Mathematics Educator (Post-question)	58.66
Mathematics Educator (Split)	23.23
GSM8K
Mathematics Educator (Pre-question)	83.09

Appendix

TABLE XLIX: Performance of dynamically-selected roles for Gemma3-12b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematical Linguist (Pre-question)	55.91
Mathematical Linguist (Post-question)	59.84
Mathematical Linguist (Split)	56.69
GSM8K
(Same as Static)	$\equiv$

Appendix

TABLE L: Performance of dynamically-selected roles for Mistral-nemo-12b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	51.57
Mathematics Educator (Post-question)	50.39
Mathematics Educator (Split)	49.61
GSM8K
Educational Psychologist (Pre-question)	81.96

Appendix

TABLE LI: Performance of dynamically-selected roles for Gemma2-9b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematics Educator (Pre-question)	58.27
Mathematics Educator (Post-question)	59.06
Mathematics Educator (Split)	59.45
GSM8K
Mathematics Educator (Pre-question)	77.10

Appendix

TABLE LII: Performance of dynamically-selected roles for Granite3.2-8b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Curriculum Developer (Pre-question)	59.84
Curriculum Developer (Post-question)	55.91
Curriculum Developer (Split)	60.63
GSM8K
Curriculum Developer (Pre-question)	77.33

Appendix

TABLE LIII: Performance of dynamically-selected roles for Granite3.3-8b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
(Same as Static)	$\equiv$
GSM8K
Math Educator (Pre-question)	80.36
Math Educator (Post-question)	79.61
Math Educator (Split)	79.76

Appendix

TABLE LIV: Performance of dynamically-selected roles for Llama3.1-8b.

AQuA-RAT
Dataset and Prompt Variant	Accuracy
(Same as Static)	$\equiv$
GSM8K
Data Scientist (Pre-question)	38.36
Data Scientist (Post-question)	41.47
Data Scientist (Split)	42.53

Appendix

TABLE LV: Performance of dynamically-selected roles for Mistral-7b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
(Same as Static)	$\equiv$
GSM8K
Math Teacher (Pre-question)	80.44
Math Teacher (Post-question)	84.91
Math Teacher (Split)	85.29

Appendix

TABLE LVI: Performance of dynamically-selected roles for Qwen2.5-7b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Mathematical Educator (Pre-question)	48.43
Mathematical Educator (Post-question)	46.06
Mathematical Educator (Split)	49.21
GSM8K
Math Educator (Pre-question)	68.61

Appendix

TABLE LVII: Performance of dynamically-selected roles for Llama3.2-3b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
(Same as Static)	$\equiv$
GSM8K
Math Educator (Pre-question)	73.77
Math Educator (Post-question)	76.35
Math Educator (Split)	73.24

Appendix

TABLE LVIII: Performance of dynamically-selected roles for Qwen2.5-3b.

AQuA-RAT
Dataset and Prompt Variant	Accuracy
Education Researcher (Pre-question)	24.80
Education Researcher (Post-question)	24.02
Education Researcher (Split)	16.93
GSM8K
(Same as Static)	$\equiv$

Appendix

TABLE LIX: Performance of dynamically-selected roles for Gemma2-2b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Educator (Pre-question)	43.31
Educator (Post-question)	40.55
Educator (Split)	42.91
GSM8K
Math Educator (Pre-question)	63.15

Appendix

TABLE LX: Performance of dynamically-selected roles for Granite3.2-2b.

Dataset and Prompt Variant	Accuracy
AQuA-RAT
Educational Data Scientist (Pre-question)	55.51
Educational Data Scientist (Post-question)	55.51
Educational Data Scientist (Split)	52.36
GSM8K
Mathematics Educator (Pre-question)	69.45

Appendix

TABLE LXI: Performance of dynamically-selected roles for Granite3.3-2b.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 429 papers

A Cross-Domain Evaluation of Prompting Techniques for Large Language Models
Oct 17, 2026Kevin Krazius, Tim Schlippe, Paul LibbrechtLLM Prompting LLM Evaluation
IU International University of Applied Sciences, Germany
Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
Aug 4, 2026Denys Pushkin, Albert Q. Jiang, Aryo Lotfi +2LLM Prompting CoT Reasoning
EPFL · Apple · Mistral AI
Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency
Mar 31, 2026Xingshuai Huang, Derek Li, Bahareh Nikpour +1CoT Reasoning LLM Reasoning
Huawei Technologies Canada, Canada · Huawei Noah’s Ark Lab, Canada

Oct 17, 2026J/K move · Enter open · S save

A Cross-Domain Evaluation of Prompting Techniques for Large Language Models

Kevin Krazius, Tim Schlippe, Paul Libbrecht

IU International University of Applied Sciences, Germany

LLM Prompting LLM Evaluation Few-Shot Prompting Automatic Prompt Optimization Retrieval-Augmented Generation

Open paper AskPDF arXiv

Prompt engineering has become a central mechanism for guiding the behavior of Large Language Models (LLMs) across a wide range of reasoning and language understanding tasks [1], [2]. However, existing studies frequently evaluate prompting techniques using different LLMs, datasets, or task settings, making direct comparisons across methods and domains difficult [3]. Consequently, in this paper we present a controlled cross-domain evaluation of Zero-Shot prompting, Few-Shot prompting, Retrieval-Augmented Generation (RAG), and automatic prompt optimization based on a PE2-inspired optimization pipeline [4]-[6]. All experiments were conducted with the same LLM and evaluated on six benchmark datasets covering arithmetic reasoning, logical reasoning, commonsense reasoning, and factual verification. The results show that no prompting strategy consistently outperforms the others across all domains, and pairwise statistical comparisons of the highest-performing configurations revealed no significant differences. RAG achieves the highest overall average accuracy of 78.33% when 300 examples are available for retrieval, while Few-Shot Prompting achieves comparatively high accuracies across several structured reasoning tasks. Zero-Shot Prompting remains competitive on tasks with self-contained inputs, whereas the PE2-inspired optimization approach produces stable but not systematically superior results compared to the other evaluated prompting techniques. Furthermore, increasing the number of task-specific examples does not consistently improve performance across prompting techniques, indicating strong domain and task dependency. These findings suggest that prompting techniques should be selected adaptively based on task characteristics rather than treated as universally applicable optimization strategies.

Abstract

LLM Prompting

Explore similar work

A Cross-Domain Evaluation of Prompting Techniques for Large Language Models

Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve

Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency