MoPrO-SD: Modular Prompt Optimization for Stance Detection

Meysam Sabbaghan; Nikesh Gyawali; Arman Zareian Jahromi; Sebastian Loera-Lozano; Doina Caragea; Cornelia Caragea

cs.CLOct 9, 2026

MoPrO-SD: Modular Prompt Optimization for Stance Detection

Authors: Meysam Sabbaghan, Nikesh Gyawali, Arman Zareian Jahromi, Sebastian Loera-Lozano, Doina Caragea, Cornelia Caragea

Organizations: Kansas State University · University of Illinois Chicago

Abstract

Stance detection remains challenging for large language models (LLMs) due to implicit opinions, sarcasm, and context-dependent expressions of stance. Although automatic prompt optimization offers an alternative to fine-tuning without updating model parameters, existing structured approaches are not specifically designed around stance-label semantics and class-specific ambiguities. We introduce MoPrO-SD, a stance-specific prompt optimization framework that decomposes prompts into functional semantic modules and refines them sequentially using LLM-based feedback. We study three refinement strategies: instruction-only optimization (I), instruction with reasoning (I+R), and instruction with per-example reasoning (I+R+E), using class-conditioned errors for stance-label refinement. We also introduce VaccineStance, a new human-annotated benchmark for vaccine stance detection. Experiments across four datasets and five open-weight LLMs show that MoPrO-SD improves upon the human-written Base prompt in many settings. On SemEval-2016, class-conditioned feedback achieves higher mean Macro-F1 than pooled feedback for four of five backbones. Both reasoning-based strategies, I+R and I+R+E, achieve strong performance, with their relative effectiveness varying across models and datasets, while remaining competitive with recent prompt optimization methods. The MoPrO-SD code and VaccineStance dataset are available on GitHub.

Figures & tables

Figure 1: Overview of MoPrO-SD. Starting from a six-module base prompt, MoPrO-SD sequentially refines the modules using instruction-only (I), instruction with reasoning (I+R), or instruction with per-example reasoning (I+R+E). After each iteration, the updated prompt is evaluated using validation Macro-F1, and the highest-scoring prompt is selected for final test evaluation.

Dataset	Method	Phi	GPT-OSS	Mistral	Gemma	Llama
SemEval	Base (zero-shot)	55.37 $\pm$ 0.12	60.02 $\pm$ 0.44	53.90 $\pm$ 0.28	65.82 $\pm$ 0.20	73.52 $\pm$ 0.14
	Base+CoT	66.60 $\pm$ 0.74	57.90 $\pm$ 0.87	67.06 $\pm$ 0.25	67.45 $\pm$ 0.43	71.77 $\pm$ 0.30
	Base+FS	57.90 $\pm$ 1.48	62.17 $\pm$ 0.79	64.69 $\pm$ 0.35	69.30 $\pm$ 0.05	72.28 $\pm$ 0.11
	Base+CoT+FS	64.85 $\pm$ 0.44	59.89 $\pm$ 0.28	67.32 $\pm$ 0.56	66.98 $\pm$ 0.11	70.41 $\pm$ 0.12
	Base+CoS	53.40 $\pm$ 0.81	60.18 $\pm$ 1.67	63.94 $\pm$ 0.37	59.92 $\pm$ 2.12	62.96 $\pm$ 1.45
	OPRO	67.49 $\pm$ 0.74	66.59 $\pm$ 0.94	56.61 $\pm$ 3.19	64.76 $\pm$ 1.39	73.45 $\pm$ 0.27

Table 1: Average Macro-F1 across targets for each dataset, model, and prompting strategy. I = instruction-only; I+R = instruction with reasoning; I+R+E = instruction with per-example reasoning. Base = human-written prompt; CoT = chain-of-thought prompting; FS = few-shot prompting; CoS = Chain-of-Stance prompting. Results are reported as mean

\pm

standard deviation across three independent runs. Shaded cells indicate differences relative to the Base score within each dataset–model column: gray denotes improvements, red denotes decreases, and darker shades indicate larger absolute differences. The best score within each dataset–model column is shown in bold .

Figure 2: Validation Macro-F1 trajectories across 10 optimization iterations using I+R+E. Curves show mean validation Macro-F1 across three independent runs, with shaded regions indicating the corresponding standard deviation.

Method	Mistral	Gemma	Llama
UniPrompt (Original Init.)	$71.06\pm 0.68$	$69.82\pm 0.62$	$73.99\pm 0.11$
UniPrompt (Shared Init.)	$\mathbf{71.42\pm 1.94}$	$\mathbf{69.90\pm 2.49}$	$73.33\pm 0.73$
SCULPT	$53.93\pm 0.40$	$65.41\pm 0.01$	$74.19\pm 0.48$
I+R+E (Ours)	$71.12\pm 0.85$	$69.68\pm 1.14$	$\mathbf{74.78\pm 0.38}$

Table 2: Comparison with structured prompt optimization methods on SemEval-2016. Results are Macro-F1 (%), reported as mean

\pm

standard deviation across runs. Exploratory two-sided Welch tests are unadjusted for multiple comparisons; Mistral–SCULPT gives

p<0.001

and Gemma–SCULPT

p=0.023

, with all others

p>0.05

	GPT-5 vs. Same LLM ( $\Delta$ Macro-F1)
Dataset	Phi	GPT-OSS	Mistral	Gemma	Llama
SemEval	+0.48	+0.25	$-$ 5.82	+2.23	+1.68
GunStance	+1.87	$-$ 1.56	$-$ 0.87	+2.78	+9.42
COVID-19	+0.10	+0.07	+1.04	+8.44	+9.23
VaccineStance	+2.27	+1.02	+3.66	+11.64	+8.55

Table 3: Effect of using GPT-5 as the external reasoner for the label-specific modules in I+R+E. Values report

\Delta

Macro-F1 = GPT-5

-

Same LLM. Full results are in Appendix G .

Method	Calls	In.	Out.	Total	Rel.
Base	1,249	768.9K	4.4K	773.4K	1.00 $\times$
Base+FS	1,249	1.14M	4.3K	1.14M	1.48 $\times$
Base+CoT	2,521	1.85M	423.4K	2.27M	2.94 $\times$
Base+CoT+FS	2,520	2.60M	420.8K	3.02M	3.91 $\times$
SCULPT	9,680	6.08M	306.5K	6.38M	8.26 $\times$
I+R+E (FP-5)	7,851	6.25M	133.8K	6.39M	8.26 $\times$

Table 4: LLM cost on SemEval-2016 using Mistral for one run across five targets. Costs include all algorithmic LLM calls under each evaluated protocol, including one final test pass. Relative cost is normalized by Base (773,391 tokens). FP-5 and FP-BM denote five-iteration and best-only-memory full-prompt I+R+E; Mod. denotes modular I+R+E.

Method	Phi	GPT-OSS	Mistral	Gemma	Llama
Base	55.37 $\pm$ 0.12	60.02 $\pm$ 0.44	53.90 $\pm$ 0.28	65.82 $\pm$ 0.20	73.52 $\pm$ 0.14
I+R+E (Full Prompt, 5 iter.)	65.69 $\pm$ 1.88	65.46 $\pm$ 0.62	67.02 $\pm$ 1.08	70.82 $\pm$ 0.39	74.03 $\pm$ 0.83
I+R+E (Full Prompt, Best-Only Memory, 10 iter.)	66.67 $\pm$ 0.76	65.10 $\pm$ 0.94	67.63 $\pm$ 0.61	70.53 $\pm$ 0.58	74.73 $\pm$ 0.77
I+R+E (Modular, 10 iter.)	68.63 $\pm$ 0.53	66.10 $\pm$ 1.14	71.12 $\pm$ 0.85	69.68 $\pm$ 1.14	74.78 $\pm$ 0.38

Table 5: Comparison of full-prompt and modular I+R+E optimization on SemEval-2016 (Macro-F1, mean

\pm

standard deviation over three runs). Standard full-prompt optimization uses five iterations due to context growth; the best-only-memory and modular variants use ten. Bold indicates the best optimized result for each backbone.

Routing	Phi	GPT	Mistral	Gemma	Llama
Pooled	$65.23\pm 3.87$	$65.05\pm 1.01$	$66.13\pm 1.94$	$67.47\pm 1.48$	$\mathbf{74.93\pm 0.89}$
Class-conditioned	$\mathbf{68.63\pm 0.53}$	$\mathbf{66.10\pm 1.14}$	$\mathbf{71.12\pm 0.85}$	$\mathbf{69.68\pm 1.14}$	$74.78\pm 0.38$

Table 7: Class-conditioned versus pooled error routing on SemEval-2016. Values are mean Macro-F1

\pm

standard deviation over three runs, averaged across five targets.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Target	Train	Valid	Test	Total
COVID-19-Stance	Anthony Fauci	1,464	200	200	1,864
	Face Masks	1,307	200	200	1,707
	School Closures	790	200	200	1,190
	Stay at Home Orders	972	200	200	1,372
	Total	4,533	800	800	6,133
GunStance	Guns Should Be Banned	900	900	900	2,700

Appendix

Table T1: Target-wise train, validation, and test distributions for the stance detection benchmarks.

Split	Favor	Against	Neutral	Total
Train	159	103	238	500
Validation	169	102	229	500
Test	396	259	345	1,000
Total	724	464	812	2,000

Appendix

Table T2: Class distribution of VaccineStance across the train, validation, and test splits.

Figure S1: Effect of optimization budget on SemEval-2016 under the I+R+E setting. Bars show mean test Macro-F1 over three independent runs using 1, 5, and 10 optimization iterations; error bars denote standard deviation.

Figure S2: Validation Macro-F1 trajectories over 10 optimization iterations across the four datasets using the I strategy. Curves show the mean across three independent runs, and shaded regions denote standard deviation. “Base” denotes the validation Macro-F1 of the original prompt before optimization.

Figure S3: Validation Macro-F1 trajectories over 10 optimization iterations across the four datasets using the I+R strategy. Curves show the mean across three independent runs, and shaded regions denote standard deviation. “Base” denotes the validation Macro-F1 of the original prompt before optimization.

Target	Method	Phi	GPT-OSS	Mistral	Gemma	Llama
		Model
Atheism	Base	37.88 $\pm$ 1.41	42.71 $\pm$ 0.96	20.15 $\pm$ 0.98	60.48 $\pm$ 0.22	63.74 $\pm$ 1.36
	Base+CoT	64.11 $\pm$ 0.56	33.65 $\pm$ 1.53	55.71 $\pm$ 0.50	63.34 $\pm$ 0.74	64.09 $\pm$ 0.34
	Base+FS	45.92 $\pm$ 2.87	52.47 $\pm$ 1.98	51.08 $\pm$ 0.97	65.84 $\pm$ 0.03	72.25 $\pm$ 0.23
	Base+CoT+FS	66.17 $\pm$ 0.57	44.94 $\pm$ 1.62	59.68 $\pm$ 1.61	61.99 $\pm$ 0.04	68.84 $\pm$ 0.56
	Base+CoS	51.23 $\pm$ 1.34	55.08 $\pm$ 3.22	56.77 $\pm$ 2.57	58.10 $\pm$ 0.93	64.56 $\pm$ 1.80

Appendix

Table T3: Per-target results on SemEval-2016 using Macro-F1. I = instruction-only; I+R = instruction + reasoning; I+R+E = instruction + per-example reasoning. Base = original prompt; CoT = chain-of-thought prompting; FS = few-shot prompting; CoS = Chain-of-Stance prompting. Values are mean

\pm

standard deviation across three runs. Shaded cells indicate differences from the corresponding Base score: gray denotes improvements, red denotes decreases, and darker shades indicate larger absolute differences. The best score for each target across all models and methods is shown in bold .

Target	Method	Phi	GPT-OSS	Mistral	Gemma	Llama
Guns should be banned	Base	39.84 $\pm$ 1.92	57.53 $\pm$ 0.81	53.75 $\pm$ 0.56	43.49 $\pm$ 0.23	52.17 $\pm$ 0.50
	Base+CoT	60.73 $\pm$ 1.24	57.26 $\pm$ 1.20	54.44 $\pm$ 0.43	49.46 $\pm$ 0.08	54.66 $\pm$ 0.81
	Base+FS	51.99 $\pm$ 2.07	58.69 $\pm$ 0.70	62.86 $\pm$ 0.13	57.18 $\pm$ 0.14	53.61 $\pm$ 0.29
	Base+CoT+FS	59.92 $\pm$ 0.53	56.41 $\pm$ 0.76	58.77 $\pm$ 0.58	62.44 $\pm$ 0.08	55.84 $\pm$ 1.10
	Base+CoS	42.52 $\pm$ 1.06	50.39 $\pm$ 1.50	62.19 $\pm$ 1.11	57.98 $\pm$ 0.49	46.87 $\pm$ 0.63
	OPRO	54.13 $\pm$ 3.03	56.74 $\pm$ 3.02	54.09 $\pm$ 0.62	42.60 $\pm$ 0.71	52.14 $\pm$ 0.84

Appendix

Table T4: Per-target results on GunStance using Macro-F1. I = instruction-only; I+R = instruction + reasoning; I+R+E = instruction + per-example reasoning. Base = original prompt; CoT = chain-of-thought prompting; FS = few-shot prompting; CoS = Chain-of-Stance prompting. Values are mean

\pm

standard deviation across three runs. Shaded cells indicate differences from the corresponding Base score within each model column: gray denotes improvements, red denotes decreases, and darker shades indicate larger absolute differences. The best score for each target across all models and methods is shown in bold .

Target	Method	Phi	GPT-OSS	Mistral	Gemma	Llama
Anthony Fauci	Base	57.47 $\pm$ 1.84	70.76 $\pm$ 3.03	72.65 $\pm$ 0.13	58.08 $\pm$ 0.43	70.91 $\pm$ 0.33
	Base+CoT	66.75 $\pm$ 0.46	73.35 $\pm$ 1.36	73.71 $\pm$ 0.83	61.95 $\pm$ 1.35	69.71 $\pm$ 0.42
	Base+FS	63.23 $\pm$ 2.49	73.28 $\pm$ 0.96	75.20 $\pm$ 0.77	64.53 $\pm$ 0.34	73.28 $\pm$ 0.50
	Base+CoT+FS	63.29 $\pm$ 0.86	74.83 $\pm$ 0.84	70.62 $\pm$ 0.63	67.22 $\pm$ 0.54	71.31 $\pm$ 1.10
	Base+CoS	53.62 $\pm$ 2.26	52.93 $\pm$ 0.69	62.66 $\pm$ 1.01	57.88 $\pm$ 3.59	60.67 $\pm$ 3.26
	OPRO	70.12 $\pm$ 0.32	69.48 $\pm$ 2.32	72.61 $\pm$ 2.42	59.12 $\pm$ 0.50	69.39 $\pm$ 0.49

Appendix

Table T5: Per-target results on COVID-19-Stance using Macro-F1. I = instruction-only; I+R = instruction + reasoning; I+R+E = instruction + per-example reasoning. Base = original prompt; CoT = chain-of-thought prompting; FS = few-shot prompting; CoS = Chain-of-Stance prompting. Values are mean

\pm

Figure S4: Macro-F1 comparison of zero-shot and few-shot prompting with the final I+R+E prompts across four benchmark datasets. Zero-shot uses no in-context demonstrations at inference; few-shot uses $k=1$ or $k=3$ retrieved demonstrations per stance class. Bars show means over three runs; error bars denote standard deviation.

Dataset	Reasoner	Phi	GPT-OSS	Mistral	Gemma	Llama
SemEval	I+R+E (Same)	68.63 $\pm$ 0.53	66.10 $\pm$ 1.14	71.12 $\pm$ 0.85	69.68 $\pm$ 1.14	74.78 $\pm$ 0.38
SemEval	I+R+E (GPT-5)	69.11 $\pm$ 0.26	66.35 $\pm$ 1.17	65.30 $\pm$ 2.46	71.91 $\pm$ 1.21	76.46 $\pm$ 0.27
GunStance	I+R+E (Same)	59.74 $\pm$ 0.26	62.12 $\pm$ 1.76	65.01 $\pm$ 2.18	60.21 $\pm$ 1.84	59.03 $\pm$ 3.58
GunStance	I+R+E (GPT-5)	61.61 $\pm$ 1.40	60.56 $\pm$ 0.04	64.14 $\pm$ 1.08	62.99 $\pm$ 1.34	68.45 $\pm$ 1.75
COVID-19-Stance	I+R+E (Same)	69.97 $\pm$ 2.43	71.14 $\pm$ 1.89	78.03 $\pm$ 2.24	69.55 $\pm$ 1.96	73.58 $\pm$ 3.29
COVID-19-Stance	I+R+E (GPT-5)	70.07 $\pm$ 1.07	71.21 $\pm$ 0.57	79.07 $\pm$ 0.26	77.99 $\pm$ 0.17	82.81 $\pm$ 0.10

Appendix

Table T6: Macro-F1 comparison of I+R+E using the classifier itself ( Same ) or GPT-5 as an external reasoner. Results are mean

\pm

standard deviation across three independent runs. Shaded GPT-5 cells indicate differences relative to the corresponding Same setting; gray denotes improvement and red denotes degradation, with darker shading indicating larger absolute differences.

Stance Detection

Stance detection identifies whether people support, oppose, or remain neutral toward a target, claim, or position in text and conversation.

See all 36 papers

A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection
Apr 29, 2026Genan Dai, Zini Chen, Yi Yang +1Multi-Agent LLM Systems LLM Evaluation
School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China
Multi-Agent Reasoning with Adaptive Worker Allocation for Stance Detection
Jun 10, 2026Meysam Sabbaghan, Arman Zareian Jahromi, Doina CarageaStance Detection Multi-Agent Reasoning
Department of Computer Science Kansas State University
Contextualized Prompting For Stance Detection On Social Media
Jun 4, 2026Tilman Beck, Shakib Yazdani, Simon Kruschinski +2Social Media Analysis Stance Detection
Institute of Intensive Care, University Hospital of Zurich and University of Zurich · Institute of Computer Science, University of Goettingen · GESIS Leibniz Institute for the Social Sciences +2

Apr 29, 2026J/K move · Enter open · S save

A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection

Genan Dai, Zini Chen, Yi Yang, Bowen Zhang

School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China

Multi-Agent LLM Systems LLM Evaluation Stance Detection LLM Prompting Multi-Agent Debate

Open paper AskPDF arXiv

Stance detection identifies the attitude of a text author toward a given target. Recent studies have explored various LLM-based strategies for this task, from zero-shot prompting to multi-agent debate. However, existing works differ in data splits, base models, and evaluation protocols, making fair comparison difficult. We conduct a systematic comparison that evaluates five methods across two categories -- prompt-based inference (Direct Prompting, Auto-CoT, StSQA) and agent-based debate (COLA, MPRF) -- on four datasets with 14 subtasks, using 15 LLMs from six model families with parameter sizes from 7B to 72B+. Our experiments yield several findings. First, on all models with complete results, the best prompt-based method outperforms the best agent-based method, while agent methods require 7 to 12 times more API calls per sample. Second, model scale has a larger impact on performance than method choice, with gains plateauing around 32B. Third, reasoning-enhanced models (DeepSeek-R1) do not consistently outperform general models of the same size on this task.

Abstract

Stance Detection

Explore similar work

A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection

Multi-Agent Reasoning with Adaptive Worker Allocation for Stance Detection

Contextualized Prompting For Stance Detection On Social Media