Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

Bogdan Kostić; Conor Fallon; Julian Risch; Alexander Löser

cs.CLFeb 19, 2026

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

Authors: Bogdan Kostić, Conor Fallon, Julian Risch, Alexander Löser

Organizations: Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin

Abstract

The rapid advancement of Large Language Models (LLMs) has established standardized evaluation benchmarks as the primary instrument for model comparison. Yet, their reliability is increasingly questioned due to sensitivity to shallow variations in input prompts. This paper examines how controlled, truth-conditionally equivalent lexical and syntactic perturbations affect the absolute performance and relative ranking of 23 contemporary LLMs across three benchmarks: MMLU, SQuAD, and AMEGA. We employ two linguistically principled pipelines to generate meaning-preserving variations: one performing synonym substitution for lexical changes, and another using dependency parsing to determine applicable syntactic transformations. Results show that lexical perturbations consistently induce substantial, statistically significant performance degradation across nearly all models and tasks, while syntactic perturbations have more heterogeneous effects, occasionally improving results. Both perturbation types destabilize model leaderboards on complex tasks. Furthermore, model robustness did not consistently scale with model size, revealing strong task dependence. Overall, the findings suggest that LLMs rely more on surface-level lexical patterns than on abstract linguistic competence, underscoring the need for robustness testing as a standard component of LLM evaluation.

Figures & tables

Figure 1: Illustration of the two linguistically principled, meaning-preserving perturbation pipelines and their average impact on LLM performance on the MMLU benchmark. Yellow and blue annotations mark changed words and moved constituents, respectively. The bar chart quantifies one of the core findings: lexical perturbations induce a substantial average accuracy drop, while the impact of syntactic perturbations is smaller.

Figure 2: Example from MMLU: original item (left), lexically perturbed version (center), and syntactically perturbed version (right). Changed words are marked in yellow and moved constituents in blue.

	MMLU	SQuAD	AMEGA
Task	Multiple-choice QA	Extractive QA	Free-form QA
Domain	Humanities, social sciences, STEM + others	Wikipedia	Clinical
Size	15,858 questions	1,000 questions	135 ques- tions with 1,337 eval. criteria
Metric	Accuracy	EM, F1 & SAS	Guideline adherence score

Table 1: Diverse representative benchmarks to assess LLM robustness to linguistic perturbations.

	MMLU			SQuAD (EM / F1 / SAS)			AMEGA
Model	Orig	$\Delta$ Lex	$\Delta$ Syn	Orig	$\Delta$ Lex	$\Delta$ Syn	Orig	$\Delta$ Lex	$\Delta$ Syn
GPT-5-Nano	$69.6$	$10.2$ ∗∗∗	$2.2$ ∗∗∗	$66.4$ / $83.7$ / $91.0$	$3.5$ ∗∗∗ / $3.6$ / $2.2$	$3.2$ ${}^{\hphantom{*}}$ / $2.6$ / $1.7$	$37.4$	$1.9$ ∗∗	$0.6$
GPT-5-mini	$80.0$	$9.3$ ∗∗∗	$1.8$ ∗∗∗	$74.9$ / $89.5$ / $93.7$	$4.0$ ∗∗∗ / $3.5$ / $1.6$	$3.0$ ${}^{\hphantom{*}}$ / $2.4$ / $1.3$	$39.6$	$2.1$ ∗∗∗	$1.6$ ∗∗
GPT-4.1-Nano	$69.9$	$7.8$ ∗∗∗	$2.0$ ∗∗∗	$76.8$ / $89.7$ / $94.1$	$5.3$ ∗∗∗ / $4.0$ / $2.1$	$3.5$ ∗∗∗ / $3.0$ / $1.5$	$34.1$	$0.7$	$0.6$
GPT-4.1-mini	$80.7$	$8.3$ ∗∗∗	$1.6$ ∗∗∗	$77.2$ / $90.7$ / $94.4$	$4.5$ ∗∗∗ / $3.7$ / $1.9$	$2.0$ ${}^{\hphantom{*}}$ / $2.3$ / $1.3$	$36.0$	$0.3$	$0.9$
GPT-OSS-120b	$86.1$	$9.7$ ∗∗∗	$2.4$ ∗∗∗	$71.9$ / $87.3$ / $93.5$	$4.4$ ∗∗∗ / $3.4$ / $1.9$	$1.6$ ${}^{\hphantom{***}}$ / $2.0$ / $0.9$	$39.8$	$0.5$	$0.7$

Table 2: LLM performance variance under lexical and syntactic perturbations across MMLU, SQuAD, and AMEGA. For each model, we report the original score and the absolute change under lexical and syntactic variants (

\Delta

Lex,

\Delta

Syn). MMLU uses accuracy, SQuAD uses EM / F1 / SAS, and AMEGA uses the guideline adherence score out of 50 as metric. Positive

\Delta

values indicate a drop in performance, and negative values indicate improvements. Grey backgrounds mark the highest value in each column. Asterisks mark significance (*:

p<.05

, **:

p<.01

, ***:

p<.001

) according to McNemar’s test.

Figure 3: Average drop in performance after lexical and syntactic perturbation across 23 LLMs for MMLU, SQuAD, and AMEGA. Lexical perturbations cause larger drops, most notably on MMLU.

Figure 4: Model performance rankings before and after lexical perturbation (top) and syntactic perturbation (bottom) for (a) MMLU, (b) SQuAD, and (c) AMEGA. Rankings are largely preserved on MMLU, while SQuAD and AMEGA show noticeably more movement.

Perturb.	Statistic	MMLU	SQuAD	AMEGA
Lexical	Kendall’s $\tau$	0.98	0.93	0.89
	95% CI (lower)	0.93	0.83	0.80
	Strict agreement ( $\tau>0.9$ )	✓ ∗	✕	✕
	Mod. agreement ( $\tau>0.8$ )	✓ ∗	✓ ∗	✓ ∗
	Score shift	✓ ∗∗∗	✓ ∗∗∗	✓ ∗∗∗
Syntactic	Kendall’s $\tau$	0.99	0.87	0.87

Table 3: Comparison of rankings on original vs. perturbed MMLU, SQuAD, and AMEGA datasets using Kendall’s

\tau

and Wilcoxon signed-rank test. Asterisks denote significance (

{}^{*}\!:\!p<0.05

{}^{**}\!:\!p<0.01

{}^{***}\!:\!p<0.001

). Rankings are stable on MMLU but weaken on SQuAD and AMEGA.

Figure 5: Correlation between log-transformed model size and performance drop on lexically and syntactically perturbed benchmarks (MMLU, SQuAD, and AMEGA). The dashed line in each plot illustrates the Ordinary Least Squares regression fit for the points. Model size correlates positively with performance drop on MMLU and negatively on SQuAD and AMEGA.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Transformation	Required Conditions
Active to	• A nominal subject ( nsubj ).
Passive	• A direct object ( dobj ).
	• The subject is not the pronoun ‘ it ’.
	• The main verb is not ‘ have ’.
Passive to	• A passive nominal subject ( nsubjpass ).
Active	• A passive auxiliary verb ( auxpass ).

Appendix

Table 4: Complete list of syntactic transformations and their corresponding rule-based applicability conditions used in the syntactic perturbation pipeline. Conditions rely on dependency parsing tags.

Prompt Sensitivity

Prompt sensitivity is the extent to which changes in wording, format, or context alter an AI model’s outputs and measured performance.

See all 108 papers

LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances
Sep 7, 2026Vamsi Krishna Kodavali, Rituraj SinghLLM Evaluation Instruction Following
Samsung
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University
When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
Jun 5, 2026Mahdi AlkaeedPrompt Sensitivity Adversarial Attacks on LLMs
Department of Computer Science and Engineering, Doha, Qatar

Sep 7, 2026J/K move · Enter open · S save

LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances

Vamsi Krishna Kodavali, Rituraj Singh

Samsung

LLM Evaluation Instruction Following Language Model Robustness

Open paper AskPDF arXiv

Robustness evaluation of large language models (LLMs) remains a critical challenge, particularly in assessing their sensitivity to perturbations in input data. In this work, we systematically evaluate LLM robustness across multiple dimensions, including word error rate, character repetition and duplication, modifications in choices, and variability in instruction following. To facilitate this evaluation, we construct a synthetic and augmented dataset encompassing a diverse set of LLM benchmarks, specifically targeting multiple-choice question (MCQ) datasets and instruction-following tasks. We conduct extensive experiments on LLMs of varying scales-small, medium, and large-as well as across base and instruction-tuned variants. Our analysis quantifies the variability in model responses under perturbed conditions and highlights discrepancies relative to baseline models. The findings provide insights into the stability of LLMs across different evaluation scenarios contributing to the development of more robust and reliable language models as well as robust evaluation methodologies.

Abstract

Prompt Sensitivity

Explore similar work

LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances

Brittlebench: Quantifying LLM robustness via prompt sensitivity

When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations