Sensivity of LLMs' Explanations to the Training Randomness:Context, Class & Task Dependencies

Romain Loncour; Jérémie Bogaert; François-Xavier Standaert

cs.CLMar 9, 2026

Sensivity of LLMs' Explanations to the Training Randomness:Context, Class & Task Dependencies

Authors: Romain Loncour, Jérémie Bogaert, François-Xavier Standaert

Organizations: UCLouvain, Louvain-la-Neuve, Belgium.

Abstract

Transformer models are now a cornerstone in natural language processing. Yet, explaining their decisions remains a challenge. It was shown recently that the same model trained on the same data with a different randomness can lead to very different explanations. In this paper, we investigate how the (syntactic) context, the classes to be learned and the tasks influence this explanations' sensitivity to randomness. We show that they all have statistically significant impact: smallest for the (syntactic) context, medium for the classes and largest for the tasks.

Figures & tables

Figure 1: MCWME estimation with Pearson’s correlation coefficient.

Figure 2: MCWME comparison plot between shuffled and non-shuffled sentences. A red cross on top of the plot highlights a non significant difference

Figure 3: Explanations’ boxplot for a text containing the first name “John”.

Figure 4: MCWME comparison plot. Texts of the left class always contain the first name “John”. Texts on the right contain either “James” or no first name.

Figure 5: Explanations’ boxplot for a text where “John” is replaced by “today”.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
Jul 14, 2026Yanzhe Zhang, Sanmi Koyejo, Diyi YangLLM Evaluation Neural Network Robustness
Georgia Tech · Stanford University
An Empirical Study of Counterfactual Self-Explanations in LLMs
Sep 15, 2026Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis Mastromichalakis +2Counterfactual Explanations Faithfulness of Language Model Explanations
National Technical University of Athens, Athens, Greece · Instituto de Telecomunicações, Lisbon, Portugal
Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales
May 31, 2026Pingjun Hong, Benjamin RothFeature Attribution LLM Interpretability
Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Faculty of Philological and Cultural Studies, University of Vienna, Vienna, Austria

Jul 14, 2026J/K move · Enter open · S save

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

Yanzhe Zhang, Sanmi Koyejo, Diyi Yang

Georgia Tech · Stanford University

LLM Evaluation Neural Network Robustness LLM Reliability Language Model Robustness

Open paper AskPDF arXiv

As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.

Abstract

LLM Interpretability

Explore similar work

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

An Empirical Study of Counterfactual Self-Explanations in LLMs

Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales