Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models
Organizations: Scuola Normale Superiore Pisa, Italy · ISTI-CNR, Italy Pisa, Italy · University of Pisa Pisa, Italy
Abstract
Large-scale foundation models exhibit behavioral shifts when subjected to interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning. Current explainability methods are structurally ill-suited to explain these shifts, because they either treat models as static objects, as traditional eXplainable AI (XAI) approaches do, or merely compare independent explanations across different checkpoints of a model. As a result, these approaches fail to explain the functional transition between two model instances in which a certain behavior has shifted following an intervention. This gap creates significant governance risks across jurisdictions including the EU AI Act, US state legislation, and Chinese AI regulations, which require documenting causal chains for substantial system modifications. This position paper argues that explaining behavioral shifts in large language models requires a principled approach that treats the shift itself as the primary object of explanation: namely, one that explains how and why an intervention transforms a reference model into an updated model with different behavior. To support this claim, we introduce Comparative XAI (XAI), a novel XAI paradigm aimed at explaining the difference between two model checkpoints where a behavior has shifted, together with a set of desiderata specifying what XAI explainers and explanations must satisfy, including comparability, validity, actionability, and monitoring, with the goal of grounding model auditing in explicit, measurable requirements. Finally, we provide preliminary evidence suggesting the need for XAI in practice through illustrative experiments, compiling the resulting findings into a transition report directly usable for governance and incident documentation.
Figures & tables
| Condition | (%) | Reference | (pts) | 95% CI | McNemar |
|---|---|---|---|---|---|
| - | - | - | - | ||
| family | How to use | Insights & strengths | Limitations |
|---|---|---|---|
| Feature Attribution | Compute and contrast input-level attribution scores for and under matched prompts, targets, and decoding. For prompting-based interventions, compute attribution for the tokens in the intervention prompts. | Highlights how reliance on specific tokens or spans changes across checkpoints. Fast, local, and easily communicable; useful for hypothesis generation about input-linked drivers of . | Primarily correlational and method-sensitive; attribution differences depend on scoring and aggregation choices. They may miss internal changes not expressed as shifts in input reliance. |
| Probing | Train and compare probing classifiers to decode a target property from hidden states of multiple checkpoints. | Reveals representational changes and supports layer-wise localization of when a property becomes more linearly accessible. | Requires predefined concepts and labels; probe success does not imply causal use. Sensitive to probe capacity and dataset construction. |
| Representation Similarity | Feed a shared input set through both models and compute layer- or component-wise similarity of activations or subspaces. Use similarity drops to nominate candidate regions of change. | Provides a global divergence map useful for triage and narrowing the search space without labeled concepts. | Quantifies difference but not semantics; similarity may be invariant to rotations or miss small but behaviorally critical changes. |
| Concept discovery | Extract interpretable directions or features from paired activations or weights and compare their presence or effects across checkpoints. | Produces human-meaningful handles on behavioral change and enables downstream steering or projection-based tests. | Concepts may be ill-defined or polysemantic; discovered directions may not generalize across layers or contexts and require careful interpretation. |
| Mechanistic interventions | Use as a reference for targeted interventions on such as activation patching, ablations, or stitching. | Provides the strongest mechanistic evidence, enabling falsifiable causal claims and fine-grained localization of responsible components. | Labor-intensive and architecture-dependent; effects may be distributed and sensitive to intervention design. |
| Data attribution | For interventions like fine-tuning, attribute the observed behavioral difference to influential training examples, updates, or phases, producing ranked data- or update-level evidence. | Connects to plausible training causes, supporting data-level remediation and auditability. | Approximate and noisy at scale; credit assignment is difficult, especially for reinforcement learning or multi-stage training, and requires access to training artifacts. |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| desideratum | Analogue in standard XAI | Changes in |
|---|---|---|
| D1: Aligned Measurement Space | Fidelity and consistent evaluation protocols | Explanations for and must be computed in a shared space, using matched prompts, targets, probes, feature bases, and decoding conditions. Otherwise, an observed difference may reflect measurement misalignment rather than a genuine model change. |
| D2: Localization | Feature, component, or concept localization | The objective is not merely to identify components relevant to one model’s output, but to localize where the behaviorally relevant change between checkpoints is represented or implemented. |
| D3: Contrastive Relevance | Fidelity and contrastive explanation | A checkpoint difference must be associated with the observed behavioral shift . Differences that also appear when behavior remains stable should be deprioritized, even when they distinguish from . |
| D4: Robustness | Stability under input perturbations or explainer variation | Robustness applies to the comparative explanation , which must remain stable across prompt paraphrases, retraining seeds, checkpoint subsampling, and other sources of cross-checkpoint variation. |
| D5: Specificity to the Intervention | No direct single-checkpoint analogue | The explanation must distinguish changes caused by the intervention from generic training drift or unrelated variation. This motivates placebo interventions, off-target behavioral controls, and null distributions over checkpoint pairs. |
| D6: Causal Verifiability | Causal faithfulness, necessity, and sufficiency tests | The relevant question is whether an identified factor causally mediates the change in behavior, rather than merely influencing one model’s output. This enables cross-checkpoint and bidirectional interventions. |
| Model | (%) | (%) | (pts) | 95% CI | McNemar |
|---|---|---|---|---|---|
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 824 papers