Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity

Zoë Prins; Samuele Punzo; Frank Wildenburg; Giovanni Cinà; Sandro Pezzelle

cs.CLMar 31, 2026

Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity

Authors: Zoë Prins, Samuele Punzo, Frank Wildenburg, Giovanni Cinà, Sandro Pezzelle

Organizations: Graduate School of Informatics, University of Amsterdam · College of Informatics, University of Amsterdam · Department of Medical Informatics, Amsterdam University Medical Center · Institute for Logic, Language and Computation (ILLC), University of Amsterdam

Abstract

Standard evaluations of Large language models (LLMs) focus on task performance, offering limited insight into whether correct behavior reflects appropriate underlying mechanisms and risking confirmation bias. We introduce a simple, principled interpretability framework based on token-level perplexity to test whether models rely on linguistically relevant cues. By comparing perplexity distributions over minimal sentence pairs differing in one or a few `pivotal' tokens, our method enables precise, hypothesis-driven analysis without relying on unstable feature-attribution techniques. Experiments on controlled linguistic benchmarks with several open-weight LLMs show that, while linguistically important tokens influence model behavior, they never fully explain perplexity shifts, revealing that models rely on heuristics other than the expected linguistic ones.

Figures & tables

Task	# Sent.	Control	Problematic	Pivotal Tokens
BLIMP - Animacy	1,000	Amanda was respected by some waitresses	Amanda was respected by some picture	[waitresses, picture]
BLIMP - Anaphor	1,000	Katherine can’t help herself	Katherine can’t help himself	[himself, herself]
DUST	2,097	In the storm, the trees fell down and crashed through the roof of my house. Now, I have to get them removed.	In the storm, the tree fell down and crashed through the roof of my house. Now, I have to get it removed.	[tree, it, trees, them]
Stereotypes	1,286	The rich are really ignorant about how to handle the money they do have.	The poor are really ignorant about how to handle the money they do have.	[poor, rich]
Non-sense words	250	af doi broiz day thag plown	af doi broiz oos thag plown	[day, oos]

Table 1: Overview of the tasks, showing the distinction between the 2 sentences of a minimal pair.

Figure 1: Plain proportion of perplexity difference between minimal pairs explained by pivotal token(s) (dots) and accuracy (triangles) of all models on a specific task. Bars represent standard error. Best viewed in color.

Figure 2: Natural logarithm of the number of sentences (y-axis) across models and tasks, given a fixed prompt, in which the proportion of the difference in perplexity is explained by key tokens (x-axis).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Natural logarithm of the number of sentences (y-axis) in the nonsense words task for which a proportion of the difference in perplexity is explained by key tokens (x-axis). We see that the pivotal tokens explain a bigger proportion of the difference in perplexity then other tokens and/or periods, although only a maximum of $\pm 50\%$ of the difference in perplexity is explained by these tokens.

Figure 4: Natural logarithm of the number of sentences (y-axis) in the BLiMP animacy task for which a proportion of the difference in perplexity is explained by key tokens (x-axis). We see that the pivotal tokens explain a bigger proportion of the difference in perplexity than other tokens, with only a small amount being explained by the periods. However, we can notice as also for this task the pivotal tokens are accountable for only a maximum of $\pm 50\%$ of the difference in perplexity.

Figure 5: Natural logarithm of the number of sentences (y-axis) in the BLiMP anaphor agreement task for which a proportion of the difference in perplexity is explained by key tokens (x-axis). We see that the pivotal tokens explain a greater proportion of the difference in perplexity than other tokens, but with a non-negligible amount being explained also by the periods. As for the previous task we can notice how the pivotal tokens account for only a maximum of $\pm 50\%$ of the difference in perplexity.

Figure 6: Natural logarithm of the number of sentences (y-axis) in the stereotypes task for which a proportion of the difference in perplexity is explained by key tokens (x-axis). Here, we can notice how the pivotal tokens along with the periods explains the bigger proportion of the difference in perplexity. Again, as for the previous tasks these tokens accounts for only a maximum of $\pm 50\%$ of the difference in perplexity.

Figure 7: Natural logarithm of the number of sentences (y-axis) in the DUST task for which a proportion of the difference in perplexity is explained by key tokens (x-axis). For this task we can observe how pivotal tokens are almost entirely accountable for the difference in perplexity, also DUST is the only task where these tokens account for the entire difference.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Feb 19, 2026Bogdan Kostić, Conor Fallon, Julian Risch +1Prompt Sensitivity LLM Evaluation
Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin
Probabilistic Attribution For Large Language Models
May 20, 2026Shilpika Shilpika, Carlo Graziani, Bethany Lusch +2LLM Interpretability LLM Uncertainty Estimation
Argonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA · Mathematics and Computer Science Division Argonne National Laboratory Lemont, IL, USA · Department of Computer Science University of Illinois Chicago Chicago, IL, USA
Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Aug 11, 2026Davood Wadi, Mohsen Ghodrat, Matthew PhilpLLM Evaluation Cultural Bias in Language Models
McGill University · University Canada West · Marketing Management +1

Feb 19, 2026J/K move · Enter open · S save

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

Bogdan Kostić, Conor Fallon, Julian Risch, Alexander Löser

Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin

Prompt Sensitivity LLM Evaluation Language Model Robustness Adversarial Perturbations

Open paper AskPDF arXiv

The rapid advancement of Large Language Models (LLMs) has established standardized evaluation benchmarks as the primary instrument for model comparison. Yet, their reliability is increasingly questioned due to sensitivity to shallow variations in input prompts. This paper examines how controlled, truth-conditionally equivalent lexical and syntactic perturbations affect the absolute performance and relative ranking of 23 contemporary LLMs across three benchmarks: MMLU, SQuAD, and AMEGA. We employ two linguistically principled pipelines to generate meaning-preserving variations: one performing synonym substitution for lexical changes, and another using dependency parsing to determine applicable syntactic transformations. Results show that lexical perturbations consistently induce substantial, statistically significant performance degradation across nearly all models and tasks, while syntactic perturbations have more heterogeneous effects, occasionally improving results. Both perturbation types destabilize model leaderboards on complex tasks. Furthermore, model robustness did not consistently scale with model size, revealing strong task dependence. Overall, the findings suggest that LLMs rely more on surface-level lexical patterns than on abstract linguistic competence, underscoring the need for robustness testing as a standard component of LLM evaluation.

Abstract

LLM Interpretability

Explore similar work

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

Probabilistic Attribution For Large Language Models

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases