Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions

Ruiyang Qin; Qingzhuo Wang; Tian Wang; Zhihua Wei; Wen Shen

cs.LGAug 19, 2026

Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions

Authors: Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen

Organizations: School of Computer Science and Technology, Tongji University, Shanghai, China.

Abstract

The remarkable capabilities of large language models (LLMs) are often undermined by their instability. Even subtle and semantically irrelevant changes in prompts can cause dramatic fluctuations in performance, a phenomenon known as prompt sensitivity. Previous studies typically evaluate prompt sensitivity by comparing the LLM's final outputs when prompts change. However, such coarse-grained metrics fail to explain the internal reasons for prompt sensitivity. In this paper, we introduce interactions as a fine-grained tool to analyze prompt sensitivity of LLMs. Specifically, we decompose the output score of the LLM into a set of interactions. Each interaction represents a nonlinear relationship involving a set of input variables. We discover that subtle changes to prompts can trigger severe instability in interactions, even when the outputs of the LLM remain the same. To this end, we propose an Interaction-based Prompt Sensitivity (IPS) metric by quantifying changes in interactions when we introduce subtle changes to prompts. We apply the IPS metric to 50 open-source LLMs and uncover four factors that reduce the prompt sensitivity of LLMs, including supervised fine-tuning, increased model scales, dense architectures, and few-shot learning. More crucially, we discover a common mechanism by which these four factors reduce prompt sensitivity: all four factors tend to reduce the prompt sensitivity of low-order interactions (i.e., interactions involving few input variables).

Figures & tables

Figure 1: Using interactions for fine-grained analysis of prompt sensitivity. Given an input

\bm{x}

and a prompt template

T

, the LLM’s output score

v(``B"|\bm{x},T)

is equivalent to the output of an interaction-based logical model

\phi(\bm{x})

, i.e.,

v(``B"|\bm{x},T)=\phi(\bm{x})=\sum\nolimits_{S\subseteq N}I_{S}

. In this way, we can uncover the underlying reasons for the prompt sensitivity of LLMs by analyzing detailed interaction patterns. Specifically, we divide interactions into those that are stable to prompt changes ( i.e. stable interaction) and those that are not ( i.e. unstable interaction).

Figure 2: (a) Verifying the sparsity of interactions. We show absolute values of normalized interactions in a descending order. LLMs all encode a small number of salient interactions, while most of the interaction effects are negligible. (b) Verifying the quality of universal matching for any

2^{n}

masked inputs. The red line plots outputs of the LLM in an ascending order.

Figure 3: (a) Case studies of interaction-level analysis. The plots compare the interaction effects encoded by the LLM for the same input $\bm{x}$ under two semantically equivalent prompt templates, $T$ (blue) and $\hat{T}$ (red). The bottom row shows examples where the LLM’s output remains the same; the top row shows examples where it changes. Results show that interaction effects are highly unstable, even when the output of the LLM remains the same. (b) The distribution of salient interactions types of various LLMs.

Figure 4: Prompt sensitivity of all the 50 open-source LLMs in an ascending order.

Figure 10: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 11: A comparison of prompt sensitivity of different order types across different model scales.

Figure 12: (a) A comparison of the prompt sensitivity between instruct/chat models and base models. (b) A comparison of prompt sensitivity across different model scales. (c) A comparison of prompt sensitivity between MoE models and dense models. (d) A comparison of prompt sensitivity between 0-shot learning and few-shot learning.

Figure 13: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 14: A comparison of prompt sensitivity at the order-level across different model scales.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Model Family	Model Series	Model	Type	Architecture	Scale
Llama (10 models)	Llama 2 (6 models)	Llama-2-7b	Base	Dense	7B
		Llama-2-7b-chat	Chat	Dense	7B
		Llama-2-13b	Base	Dense	13B
		Llama-2-13b-chat	Chat	Dense	13B
		Llama-2-70b	Base	Dense	70B
		Llama-2-70b-chat	Chat	Dense	70B

Appendix

Table 1: A comprehensive list and characteristics of LLMs, grouped by model family and model series.

Figure 15: Different prompt templates. Red parts show the difference between the current prompt template with the first prompt template, i.e. , Prompt Template 1.

Figure 16: Prompt templates of few-shot learning.

Figure 17: Verifying the sparsity of interactions. We show absolute values of normalized interactions in a descending order. LLMs all encode a small number of salient interactions, while most of the interaction effects are negligible.

Figure 18: Verifying the quality of universal matching for any $2^{n}$ masked inputs. The red line plots outputs of the LLM in an ascending order.

Figure 19: A case study of interaction-level analysis revealing latent instability. The same input

\bm{x}

is formatted with two semantically identical templates,

T

and

\hat{T}

, differing only in letter case (e.g.,“Answer” vs. “ANSWER”). Although the LLM generates the same correct output (“D”) in both cases, the composition of the interaction-based logical model

\phi(\bm{x})

reveals significant internal divergence. Many interaction effects are highly unstable, changing in either sign or magnitude. This highlights a critical risk of prompt sensitivity that is invisible to output-level

Figure 21: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 22: A comparison of prompt sensitivity of different order types across different model scales.

Figure 23: A comparison of prompt sensitivity of low-, mid-, and high-order interactions between 0-shot learning and few-shot learning. Prompt sensitivity at all three order levels shows an clear drop when applying few-shot learning.

Metric	Value	Interpretation
Spearman’s $\rho$	0.9905	Rank Correlation: Measures the average similarity of the overall ranking trends. A value close to 1.0 indicates near-perfect monotonic consistency.
Pearson’s $r$	0.9957	Linearity: Measures the linear correlation of the raw IPS scores, indicating that the scale of sensitivity shifts linearly across thresholds.
Kendall’s $\tau$	0.9465	Pairwise Consistency: Indicates the probability that any pair of models maintains their relative order (better/worse) across different thresholds.
RMSE	1.72	Ranking Stability: On average, a model’s rank fluctuates by only $\pm 1.72$ positions across different threshold settings.
Top-10 Overlap	96.7%	SOTA Stability: The set of the top-10 most stable models remains 96.7% identical, ensuring reliable identification of the best-performing models.

Appendix

Table 2: Summary of consistency metrics across 16 different

\tau

thresholds (0.05–0.20). The high values in correlation metrics and low RMSE indicate that the relative ranking of model sensitivity remains stable regardless of the specific threshold used.

	IPS Score ( $\downarrow$ ) at Threshold $\tau$
Model Name	0.05	0.06	0.07	0.08	0.09	0.10	0.12	0.15	0.18	0.20
Qwen2.5-72B-Instruct	1.328	1.312	1.297	1.286	1.276	1.268	1.255	1.240	1.228	1.222
Qwen2-72B-Instruct	1.362	1.346	1.333	1.322	1.312	1.304	1.290	1.278	1.271	1.265
Qwen2-7B-Instruct	1.411	1.400	1.391	1.383	1.376	1.368	1.357	1.341	1.327	1.318
Qwen3-32B-Instruct	1.424	1.419	1.415	1.411	1.408	1.406	1.403	1.401	1.396	1.395
Qwen3-14B-Instruct	1.443	1.438	1.434	1.429	1.427	1.425	1.419	1.415	1.412	1.407

Appendix

Table 3: Detailed IPS scores for 50 LLMs across 10 different thresholds. The consistency in color gradients (implied by values) across rows confirms the robustness of the metric.

Figure 21

Figure 22

Figure 28: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 30: A comparison of prompt sensitivity at the order-level across different model scales.

Figure 25

Figure 26

Figure 35: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 37: A comparison of prompt sensitivity at the order-level across different model scales.

Figure 29

Figure 30

Figure 42: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 44: A comparison of prompt sensitivity at the order-level across different model scales.

Dense Model	Altering Tokens	Adding Tokens	Difference
internlm2-chat-7b	1.673	1.676	+0.003
llama-2-13b-chat	1.520	1.433	-0.087
llama-2-7b-chat	1.633	1.710	+0.077
llama-3-8b-instruct	1.450	1.467	+0.017
mistral-7b-v0.3-instruct	1.538	1.635	+0.097
olmo-2-7b-instruct	1.618	1.655	+0.037

Appendix

Table 4: IPS Scores of Dense Models on Different Perturbation Types. Higher scores indicate greater sensitivity. The more sensitive perturbation type for each model is highlighted in bold.

MoE Model	Altering Tokens	Adding Tokens	Difference
llama-moe-v1-3.5b-sft	1.736	1.724	-0.012
llama-moe-v2-3.8b-sft	1.698	1.655	-0.043
mixtral-8x7b-instruct	1.658	1.617	-0.041
olmoe-7b-instruct	1.725	1.699	-0.026
qwen1.5-moe-a2.7b-chat	1.669	1.634	-0.035
qwen3-30b-a3b-instruct	1.608	1.638	+0.030

Appendix

Table 5: IPS Scores of MoE Models on Different Perturbation Types. The more sensitive perturbation type for each model is highlighted in bold.

	Base Model		Instruct/Chat Model
Model Family	Cosine ( $\uparrow$ )	$L_{2}$ Dist. ( $\downarrow$ )	Cosine ( $\uparrow$ )	$L_{2}$ Dist. ( $\downarrow$ )
Llama-2-7B	$0.770$	75.12	0.781	$66.51$
Llama-2-13B	0.942	17.35	$0.927$	$19.69$
Llama-2-70B	0.911	33.65	$0.795$	$43.14$
Llama-3-8B	$0.856$	$79.77$	0.880	73.56
Mistral-7B-V0.3	$0.811$	$207.89$	0.843	154.43

Appendix

Table 6: Comparison of stability metrics (Cosine Similarity and

L_{2}

Distance) for Base vs. Instruct/Chat models. Unlike IPS, these metrics fail to show a consistent trend regarding the impact of Supervised Fine-Tuning.

Model Name	Architecture	Act. Params.	Type	IPS (ARC) $\downarrow$	IPS (MMLU) $\downarrow$
Qwen1.5-moe-a2.7b-chat	MoE	2.7B	Chat	1.665	1.640
Qwen2.5-3b-instruct	Dense	3B	Instruct	1.606	1.625
Olmoe-7b	MoE	1B	Base	1.714	1.717
Olmo-1b	Dense	1B	Base	1.632	1.658
Qwen3-30b-a3b	MoE	3B	Base	1.642	1.680
Qwen3-4b	Dense	4B	Base	1.568	1.599

Appendix

Table 7: Controlled variable analysis of prompt sensitivity between MoE and dense models.

Figure 37

Figure 38

Figure 50: Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors.

Figure 51: A comparison of prompt sensitivity at the order-level across different model scales.

Figure 41

Figure 42

Figure 58: A comparison of prompt sensitivity at the order-level across different model scales.

Prompt Sensitivity

Prompt sensitivity is the extent to which changes in wording, format, or context alter an AI model’s outputs and measured performance.

See all 115 papers

Understanding the Prompt Sensitivity
Apr 20, 2026Yang Liu, Chenhui ChuPrompt Sensitivity LLM Prompting
Kyoto University
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University
Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?
May 28, 2026Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek +2Prompt Sensitivity LLM Prompting
1Heinrich Heine University Düsseldorf · 2Jagiellonian University · 3IDEAS Research Institute

Apr 20, 2026J/K move · Enter open · S save

Understanding the Prompt Sensitivity

Yang Liu, Chenhui Chu

Kyoto University

Prompt Sensitivity LLM Prompting LLM Interpretability Language Model Robustness

Open paper AskPDF arXiv

Prompt sensitivity, which refers to how strongly the output of a large language model (LLM) depends on the exact wording of its input prompt, raises concerns among users about the LLM's stability and reliability. In this work, we consider LLMs as multivariate functions and perform a first-order Taylor expansion, thereby analyzing the relationship between meaning-preserving prompts, their gradients, and the log probabilities of the model's next token. We derive an upper bound on the difference between log probabilities using the Cauchy-Schwarz inequality. We show that LLMs do not internally cluster similar inputs like smaller neural networks do, but instead disperse them. This dispersing behavior leads to an excessively high upper bound on the difference of log probabilities between two meaning-preserving prompts, making it difficult to effectively reduce to 0. In our analysis, we also show which types of meaning-preserving prompt variants are more likely to introduce prompt sensitivity risks in LLMs. In addition, we demonstrate that the upper bound is strongly correlated with an existing prompt sensitivity metric, PromptSensiScore. Moreover, by analyzing the logit variance, we find that prompt templates typically exert a greater influence on logits than the questions themselves. Overall, our results provide a general interpretation for why current LLMs can be highly sensitive to prompts with the same meaning, offering crucial evidence for understanding the prompt sensitivity of LLMs. Code for experiments is available at https://github.com/ku-nlp/Understanding_the_Prompt_Sensitivity.

Abstract

Prompt Sensitivity

Explore similar work

Understanding the Prompt Sensitivity

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?