Transformer Interpretability

Follow this topic Ask about this topic

Transformer interpretability asks how models process information internally, from language and vision to speech, biology, and time series. Researchers use probes and representation geometry to find encoded features, sparse autoencoders to decompose activations, and circuit tracing, ablation, and activation patching to test causal roles; attribution methods explain inputs, while steering tests how internal signals can control outputs. Work also explores interpretable architectures that expose computation by design and controlled algorithmic tasks where the expected solution is known. A central challenge is that information being readable in a representation does not prove the model uses it, and findings can vary with architecture, task, and intervention, making causal validation and reliable evaluation essential.

Latest papers

340 papers34 papers in the last four weeks, down 8% on the four weeks before. 0.2% of all new papers.

SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding
Mar 9, 2026Jesús Sánchez Ochoa, Enrique Tomás Martínez Beltrán, Alberto Huertas CeldránNeuron-Level Interpretability Transformer Interpretability
Department of Information and Communications Engineering, University of Murcia, Murcia, 30100, Spain
Sensivity of LLMs' Explanations to the Training Randomness:Context, Class & Task Dependencies
Mar 9, 2026Romain Loncour, Jérémie Bogaert, François-Xavier StandaertLLM Interpretability Transformer Interpretability
UCLouvain, Louvain-la-Neuve, Belgium.
Interpretable-by-Design Transformers via Architectural Stream Independence
Mar 8, 2026Clayton Kerce, Alexis FoxTransformer Interpretability Neural Network Interpretability
Georgia Tech Research Institute
Engineering Verifiable Modularity in Transformers via Per-Layer Supervision
Mar 8, 2026J. Clayton KerceTransformer Interpretability Mechanistic Interpretability
Georgia Tech Research Institute
The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling
Mar 8, 2026J. Clayton Kerce, Alexis FoxTransformer Transformer Interpretability
Georgia Tech Research Institute
Class Visualizations and Activation Atlases for Enhancing Interpretability in Deep Learning-Based Computational Pathology
Mar 7, 2026Marco Gustav, Fabian Wolf, Christina Glasner +6Feature Visualization Explainable Medical Image Analysis
Else Kroener Fresenius Center for Digital Health, Faculty of Medicine, TUD Dresden University of Technology, Dresden, Germany · Institute of Pathology, University Medical Center Mainz, Mainz, Germany · Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany +4
Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignment
Mar 6, 2026Jingyuan Feng, Andrew Gambardella, Gouki Minegishi +3LLM Safety Alignment LLM Alignment
The University of Tokyo.
The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
Mar 5, 2026Shangwen Sun, Alfredo Canziani, Yann LeCun +1Massive Activations Attention Sinks
New York University.
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Multimodal Understanding Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT
Mar 3, 2026Ihor KendiukhovGenomic Foundation Models Single-Cell Transcriptomics
Department of Computer Science, University of Tübingen, Tübingen, Germany
What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers
Mar 2, 2026Qin Su, Tie LuoTransformer Interpretability Explainable Image Classification
Department of Electrical and Computer Engineering University of Kentucky, Lexington, KY, USA
GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models
Feb 27, 2026Sebastian Gerstner, Leonor Veloso, Hinrich SchützeNeuron-Level Interpretability Transformer Interpretability
LMU Munich and Munich Center for Machine Learning (MCML), Munich, Germany
MetaOthello: A Controlled Study of Multiple World Models in Transformers
Feb 26, 2026Aviral Chawla, Galen Hall, Juniper LovatoTransformer Interpretability World Model Learning
Vermont Complex Systems Institute, University of Vermont, Burlington, VT · University of Michigan, Ann Arbor, MI
Transformers converge to invariant algorithmic cores
Feb 26, 2026Joshua S. SchiffmanTransformer Interpretability Invariant Representation Learning
New York Genome Center, New York, NY, USA.
How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing
Feb 25, 2026Javier MarínTransformer Interpretability LLM Interpretability
Groundlens-dev
Multi-Dimensional Spectral Geometry of Biological Knowledge in Single-Cell Transformer Representations
Feb 24, 2026Ihor KendiukhovTransformer Interpretability Single-Cell Transcriptomics
Department of Computer Science University of Tübingen Tübingen, Germany
Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection
Feb 24, 2026Trishita Dhara, Siddhesh ShethContent Moderation Neural Network Interpretability
Upper Hand, Indianapolis, IN 46204, USA · Ace Rent a Car, Indianapolis, IN 46204, USA
Beyond Behavioural Trade-Offs: Mechanistic Tracing of Pain-Pleasure Decisions in an LLM
Feb 22, 2026Francesca Bianco, Derek ShillerLLM Decision-Making Emotion Representation in Language Models
Independent Researcher Future Impact Group Fellowship · Rethink Priorities Future Impact Group Mentor
The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
Feb 19, 2026Peter BaloghAttention Head Analysis Attention Head Specialization
A Structural Theory of Position Bias in Transformers
Feb 18, 2026Hanna Herasimchyk, Robin Labryga, Tomislav Prusina +1Lost-in-the-Middle Effect Position Bias
University of Hamburg Hamburg, Germany
Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models
Feb 18, 2026Melkamu Abay Mersha, Jugal KalitaTransformer Interpretability Gradient-Based Attribution
College of Engineering and Applied Science, University of Colorado Colorado Springs, Colorado Springs, 80918, CO, USA
Quantifying LLM Attention-Head Stability: Implications for Circuit Universality
Feb 17, 2026Karan Bali, Jack Stanley, Praneet Suresh +1Transformer Interpretability Attention Head Analysis
Mila - Quebec Artificial Intelligence Institute, Montreal, Canada · McGill University, Montreal, Canada
Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
Feb 16, 2026Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4Transformer Inference Test-Time Scaling
IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, France · Sony Europe Ltd. Stuttgart Technology Center, EUREC, Germany
Central Dogma Transformer II: An AI Microscope for Understanding Cellular Regulatory Mechanisms
Feb 9, 2026Nobuyuki OtaTransformer Interpretability Single-Cell Perturbation Modeling
Independent Researcher, Burlingame, CA, USA
Patches of Nonlinearity: Instruction Vectors in Large Language Models
Feb 8, 2026Irina Bigoulaeva, Jonas Rohweder, Subhabrata Dutta +1Transformer Interpretability LLM Interpretability
Ubiquitous Knowledge Processing Lab (UKP Lab) · Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
Feb 4, 2026Jonas Rohweder, Subhabrata Dutta, Iryna GurevychTransformer Interpretability Latent Variable Models
UKP Lab, TU Darmstadt
Universal Redundancies in Time Series Foundation Models
Feb 2, 2026Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1Transformer Interpretability Kernel Regression
ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin
The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
Jan 9, 2026Nora Graichen, Iria de-Dios-Flores, Gemma BoledaTransformer Interpretability LLM Evaluation
Universitat Pompeu Fabra1 · ICREA2
Probing Spectrum-Like Organization of States of Mind in Transformer Representation Spaces
Dec 23, 2025Sophie ZhaoTransformer Interpretability Neural Representation Geometry
School of Computer Science Georgia Institute of Technology
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer Interpretability Transformer FFNs
University of Cambridge, UK · Google Research

Mar 9, 2026J/K move · Enter open · S save

SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

Jesús Sánchez Ochoa, Enrique Tomás Martínez Beltrán, Alberto Huertas Celdrán

Department of Information and Communications Engineering, University of Murcia, Murcia, 30100, Spain

Neuron-Level Interpretability Transformer Interpretability Neural Network Robustness Mechanistic Interpretability

Open paper AskPDF arXiv

In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework (requiring no retraining of the base model) for understanding and stress-testing the internal behavior of Transformer models across domains. SYNAPSE extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled, repeatable experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. SYNAPSE reveals a consistent organization of internal representations across the two evaluated domains, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable fine-grained, label-aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust and interpretable Transformer models.