Brittlebench: Quantifying LLM robustness via prompt sensitivity

Angelika Romanou; Mark Ibrahim; Candace Ross; Chantal Shaib; Kerem Oktar; Samuel J. Bell; Anaelia Ovalle; Jesse Dodge

cs.LGFeb 27, 2026

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Authors: Angelika Romanou, Mark Ibrahim, Candace Ross, Chantal Shaib, Kerem Oktar, Samuel J. Bell, Anaelia Ovalle, Jesse Dodge,

Organizations: FAIR at Meta · EPFL · Northeastern University

Abstract

Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.

Figures & tables

	SCOPE COVERAGE					METHOD	ANALYSIS DEPTH		ARTIFACTS
Paper	Word-level	Prompt-level	Instructions-level	Paraphrasing	Task-specific	Variance decomposition	Reasoning analysis	Ranking stability	Software / Datasets
DOVE ( Habba et al., 2025 )	-			-	-	-	-	-	Dataset
PromptBench ( Zhu et al., 2024 )	-	-		-	-	-	-	-	Python lib
Mizrahi et al. (2024)	-	-		-	-	-	-		Code + Data
Alzahrani et al. (2024)	-			-	-	Token bias vs. Position bias	-		-
Salinas and Morstatter (2024)			-	-	-	-	-	-	-

Table 1: Comparison of Brittlebench with recent work on robustness evaluation. While prior work addresses complementary aspects of prompt robustness, Brittlebench brings these dimensions together in a unified framework, combining broad perturbation coverage with variance decomposition, reasoning and ranking-stability analyses, and an open-source meta-evaluation pipeline applicable to both open-weight and frontier API models.

Figure 1: The Brittlebench meta-evaluation framework. We select widely used benchmarks and apply semantics-preserving perturbations as described in Section 2 . We evaluate both the original and perturbed benchmarks using frontier and open-weight state-of-the-art models. Evaluating models on Brittlebench ’s perturbations provides a more comprehensive assessment by accounting for biases introduced by variations in input formats.

Commercial Frontier Models
Model	Word-level Perturbations		Prompt Padding		Prompt Augmentation		Paraphrasing		Average
	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$
Claude Opus 4.5	6.49	6.49	1.87	3.32	1.56	1.56	0.38	0.44	2.58	2.95
GPT-5	9.32	9.32	0.77	1.60	4.75	4.75	5.66	5.66	5.13	5.33
Open-Weight Models
Qwen-32b	7.54	7.54	5.41	5.41	0.83	0.83	0.49	0.49	3.57	3.57

Table 2: Mean

\Delta Drop

and

Range

(%) per model and perturbation category across the six MCQ benchmarks used for Brittlebench : MMLU, ARC, TruthfulQA, MathQA, GPQA, and LogiQA . Perturbations are grouped by the type of change they make to the input prompt. For each perturbation group,

\Delta Drop

denotes the largest drop from the clean baseline to the lowest-performing perturbation in the group, and

Range

denotes the difference between the best and worst perturbation performance within the group. Each pair of values reports these quantities averaged across the six benchmarks in the 0-shot setting. Disaggregated results for each perturbation group are provided in Appendix F . Confidence intervals for the reported aggregate values are provided in Appendix E .

Figure 2: Input perturbations systematically increase performance variance across models. Line plots show Accuracy (%) of each model on six benchmarks under zero-shot evaluation. The solid line shows baseline performance; dashed lines indicate the best- and worst-case accuracy across all perturbation types, with shaded bands highlighting the gap between the baseline and each extreme.

Figure 3: (a): Model sensitivity to perturbation intensity. Llama3.3-70B (top) and Qwen3-32B (bottom) accuracy (%) aggregated across benchmarks, under increasing numbers of surface-form perturbations, including typos and prompt-padding variations. Each line shows accuracy as a function of the number of perturbation occurrences, illustrating how perturbation intensity affects robustness. (b): Variance decomposition for models (top) and benchmarks (bottom) . From the model’s perspective, the brittleness score ( $\Pi_{m}$ ) quantifies whether a model’s performance variability across tasks is dominated by prompt sensitivity rather than intrinsic item difficulty. From the benchmark perspective, the brittleness score ( $\Pi_{b}$ ) indicates whether a benchmark primarily discriminates models based on task difficulty or on sensitivity to input perturbations.

Figure 4: Heatmap of Accuracy Drop (in %; $\downarrow$ ) from the baseline on MMLU for Qwen3-8B (0-shot) under single and paired input perturbations of Brittlebench . The first row reports the performance drop for individual perturbations, while each row shows the additional effect of applying a second perturbation on top of the first. Paraphrasing perturbations are always applied once. Higher values indicate larger degradation, revealing both compounding and counteracting effects between perturbations. Perturbations are sorted based on their average performance drop on all the benchmarks and models from highest (left/top) to lowest (right/bottom).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visual explanation of perturbations used in Brittlebench . Perturbations are grouped based on their type as described in Section 2 .

Commercial Frontier Models
Model	Word-level Perturbations		Prompt Padding		Prompt Augmentation		Average
	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$
Claude Opus 4.5	6.22	6.55	4.70	6.20	0.98	3.88	3.97	5.54
GPT-5	12.24	12.67	0.06	0.65	2.97	2.87	5.09	5.40
Open-Weight Models
Qwen-32b	7.52	7.30	6.05	1.71	1.64	1.05	5.07	3.35

Appendix

Table 4: Mean

\Delta Drop

and

Range

(%) per model and perturbation category across the six MCQ benchmarks used for Brittlebench : MMLU, TruthfulQA and GPQA .

Commercial Frontier Models
Model	Baseline	Typos		Drop Stop-words		Punctuation Spaces		Sequence Spaces		Word Merge		Word Split
		Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	$\Delta Drop$	$Range$
Claude Opus 4.5	78.64	74.36	4.28	74.08	4.56	78.45	0.19	72.15	6.49	76.72	1.92	76.74	1.90	6.49	6.49
GPT-5	73.97	64.65	9.32	65.63	8.34	70.29	3.68	69.47	4.50	70.19	3.78	70.42	3.55	9.32	9.32
Open-Weight Models
Qwen-32b

Appendix

Table 5: Average performance (%) across all used benchmarks on Brittlebench for Word Manipulation Perturbations . Each number is the average accuracy score on the 6 benchmarks used ( MMLU, ARC, TruthfulQA, MathQA, GPQA, LogiQA ) based on their harness-eval implementations for different few-shot settings.

Commercial Frontier Models
Model	Baseline	Persona		Emotion Prompt		Augmented Paraphrasing
		Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	$\Delta Drop$	$Range$
Claude Opus 4.5	78.64	77.68	0.96	78.05	0.59	77.08	1.56	1.56	1.56
GPT-5	73.97	69.96	4.01	69.90	4.07	69.22	4.75	4.75	4.75
Open-Weight Models
Qwen-32b

Appendix

Table 6: Average performance (%) across all used benchmarks on Brittlebench for Prompt Augmentation Perturbations . Each number is the average accuracy score on the 6 benchmarks used ( MMLU, ARC, TruthfulQA, MathQA, GPQA, LogiQA ) based on their harness-eval implementations for different few-shot settings.

Model	Baseline	Quotes		Spaces		New lines
		Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	$\Delta Drop$	$Range$
Commercial Frontier Models
Claude Opus 4.5	78.64	76.77	1.87	80.10	-1.46	79.64	-1.00	1.87	3.32
GPT-5	73.97	73.20	0.77	74.80	-0.83	73.86	0.11	0.77	1.60
Open-Weight Models
Qwen-32b

Appendix

Table 7: Average performance (%) across all used benchmarks on Brittlebench for Prompt Padding Perturbations . Each number is the average accuracy score on the 6 benchmarks used ( MMLU, ARC, TruthfulQA, MathQA, GPQA, LogiQA ) based on their harness-eval implementations for different few-shot settings.

Commercial Frontier Models
Model	Baseline	Lexical Paraphrasing		Syntactical Paraphrasing		Rule-free Paraphrasing
		Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	Acc ( $\uparrow$ )	Drop ( $\downarrow$ )	$\Delta Drop$	$Range$
Claude Opus 4.5	78.64	78.56	0.08	78.26	0.38	78.71	-0.07	0.38	0.44
GPT-5	73.97	69.14	4.83	68.31	5.66	69.42	4.55	5.66	5.66
Open-Weight Models
Qwen-32b

Appendix

Table 8: Average performance (%) across all used benchmarks on Brittlebench for Paraphrasing Perturbations . Each number is the average accuracy score on the 6 benchmarks used ( MMLU, ARC, TruthfulQA, MathQA, GPQA, LogiQA ) based on their harness-eval implementations for different few-shot settings.

Zero-shot Perturbation	Rank Corr.	Few-shot Perturbation	Rank Corr.
punct_spaces	0.9102	word_split	0.9673
rule_free_par	0.8857	punct_spaces	0.9592
word_split	0.8776	typos	0.9510
lexical_par	0.8694	word_merge	0.9429
spaces	0.8694	seq_spaces	0.9347
syntactic_par	0.8449	drop_stopwords	0.9347

Appendix

Table 9: Mean Spearman rank correlation by perturbation for zero-shot and few-shot settings. Averaged across tasks.

Commercial Frontier Models
Model	Word-level Perturbations		Prompt Padding		Prompt Augmentation		Paraphrasing
	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$	$\Delta Drop$ ( $\downarrow$ )	$Range$
Claude Opus 4.5	6.49 [4.22, 10.12]	6.49 [4.22, 10.15]	1.87 [0.15, 4.77]	3.32 [1.04, 7.10]	1.56 [0.32, 4.50]	1.56 [0.50, 4.69]	0.38 [0.00, 1.49]	0.44 [0.28, 2.00]
GPT-5	9.32 [4.08, 17.22]	9.32 [4.08, 17.22]	0.77 [0.00, 10.91]	1.60 [0.49, 12.56]	4.75 [0.58, 11.56]	4.75 [0.64, 11.56]	5.66 [0.69, 12.35]	5.66 [1.24, 12.35]
Open-Weight Models
Qwen-32b	7.54 [3.19, 15.60]	7.54 [3.50, 15.80]	5.41 [4.28, 7.11]	5.41 [4.28, 7.11]	0.83 [0.54, 1.46]	0.83 [0.58, 1.74]	0.49 [0.17, 1.18]	0.49 [0.19, 1.18]

Appendix

Table 10: Mean

\Delta Drop

and

Range

(%) per model and perturbation category across the six MCQ benchmarks used for Brittlebench : MMLU, ARC, TruthfulQA, MathQA, GPQA, and LogiQA , along with 95% bootstrap confidence intervals computed by resampling benchmarks.

Model	Benchmark	Perturbation	Performance	Drop
Qwen-32B	HumanEval	baseline	37.20	–
		word-level	0.92	36.28
		padding	30.69	6.51
		prompt-augm	17.75	19.45
		all perturbations	16.45	20.75
Qwen-32B	GSM8K	baseline	65.05	–

Appendix

Table 11: Impact of semantics-preserving prompt perturbations on generation-based benchmarks (HumanEval and GSM8K) for Qwen-32B and Llama-3.3-70B in the zero-shot setting. We report baseline performance and signed performance changes relative to the baseline under different perturbation categories. Negative drops indicate improvements over the baseline.

Figure 6: CoT improves robustness across most, but not all benchmarks, for Claude Opus 4.5.

Fixed Effects
	Estimate	SE	$z$	$p$
Intercept (Non-CoT, Baseline)	0.880	0.038	23.01	$<.001^{***}$
CoT (vs Non-CoT)	0.012	0.002	5.12	$<.001^{***}$
Perturbed (vs Baseline)	-0.039	0.044	-0.89	.373
CoT $\times$ Perturbed	0.008	0.003	2.81	.005 ∗∗
Random Effects (Variance / SD)

Appendix

Table 12: CoT weakly mitigates against brittleness in perturbations. The regression uses accuracy (0/1) as the dependent variable. The key predictor is the CoT

\times

Perturbed interaction: a positive coefficient indicates that perturbation hurts CoT less than Non-CoT, meaning CoT is more robust. Model includes random intercepts for benchmark and condition. Note that the significance of direct effects is difficult to interpret in the presence of interactions.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Figure 7: (a): Perturbations cause drops to reasoning quality, though with variation across benchmarks in the size of the drop. (b): Correct answers are systematically linked with high reasoning scores across all benchmarks.

	Estimate	SE	$z$	$p$
Fixed Effects
Intercept (Baseline)	89.21	2.14	41.69	$<.001^{***}$
Perturbed (vs Baseline)	-2.15	0.07	-30.99	$<.001^{***}$
Random Effects (Variance)
Benchmark	27.43
Residual	121.21

Appendix

Table 13: Reasoning quality is significantly lower for perturbations than baseline. The regression uses overall reasoning quality (0–100 scale) as the DV. Model includes random intercepts for benchmark.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Prompt Sensitivity

Prompt sensitivity is the extent to which changes in wording, format, or context alter an AI model’s outputs and measured performance.

See all 108 papers

perturber: A Stateless Service for Semantics-Preserving Text Perturbations
Oct 8, 2026Khoi-Nguyen TranAdversarial Perturbations
IBM Research
Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Feb 19, 2026Bogdan Kostić, Conor Fallon, Julian Risch +1Prompt Sensitivity LLM Evaluation
Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin
LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances
Sep 7, 2026Vamsi Krishna Kodavali, Rituraj SinghLLM Evaluation Instruction Following
Samsung

Oct 8, 2026J/K move · Enter open · S save

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Khoi-Nguyen Tran

IBM Research

Adversarial Perturbations

Open paper AskPDF arXiv

Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer. We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families. It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction. We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model's robustness, measuring meaning-preservation per perturbation rather than assuming it. The demonstration adds a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation.

Abstract

Prompt Sensitivity

Explore similar work

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances