Transformer Interpretability

Follow this topic Ask about this topic

Transformer interpretability asks how models process information internally, from language and vision to speech, biology, and time series. Researchers use probes and representation geometry to find encoded features, sparse autoencoders to decompose activations, and circuit tracing, ablation, and activation patching to test causal roles; attribution methods explain inputs, while steering tests how internal signals can control outputs. Work also explores interpretable architectures that expose computation by design and controlled algorithmic tasks where the expected solution is known. A central challenge is that information being readable in a representation does not prove the model uses it, and findings can vary with architecture, task, and intervention, making causal validation and reliable evaluation essential.

Latest papers

340 papers34 papers in the last four weeks, down 8% on the four weeks before. 0.2% of all new papers.

From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers
Apr 25, 2026Jainum SanghaviTransformer Interpretability Visual Spatial Reasoning
Northeastern University
When Chain-of-Thought Fails, the Solution Hides in the Hidden States
Apr 25, 2026Houman Mehrafarin, Amit Parekh, Ioannis KonstasTransformer Interpretability CoT Reasoning
Heriot-Watt University, Edinburgh, United Kingdom · University of Edinburgh, Edinburgh, United Kingdom
Real-Time AttentionBender: Granular Interactive Network Bending of Video Diffusion Transformers
Apr 24, 2026Adam Cole, Rebecca Fiebrink, Mick GriersonTransformer Interpretability Video Diffusion Models
Creative Computing Institute University of the Arts London London, UK
Dissociating Decodability and Causal Use in Bracket-Sequence Transformers
Apr 24, 2026Aryan Sharma, Cutter Dawes, Shivam RavalTransformer Interpretability Self-Attention
1Yale University · 2Independent · 3Harvard University
Evaluating Post-hoc Explanations of the Transformer-based Genome Language Model DNABERT-2
Apr 23, 2026Isabel Kurth, Paulo Yanez Sarmiento, Bernhard Y. RenardTransformer Interpretability Attention Mechanisms
Hasso Plattner Institute University of Potsdam
AttentionBender: Manipulating Cross-Attention in Video Diffusion Transformers as a Creative Probe
Apr 22, 2026Adam Cole, Mick GriersonTransformer Interpretability Video Diffusion Models
University of the Arts London London, UK
Cognitive Alignment At No Cost: Inducing Human Attention Biases For Interpretable Vision Transformers
Apr 21, 2026Ethan KnightsTransformer Interpretability Visual Attention
Cambridge, UK
Decision-Aware Attention Propagation for Vision Transformer Explainability
Apr 20, 2026Sehyeong Jo, Gangjae Jang, Haesol ParkTransformer Interpretability Vision Transformer
Aarhus University · University of Colorado Boulder · KIST
One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them
Apr 18, 2026Ali Holmov, Paul Youssef, Nandi Schoots +1Knowledge Editing Transformer Interpretability
Technical University of Munich · Marburg University · University of Oxford
Structural Instability of Feature Composition
Apr 18, 2026Yunpeng ZhouFeature Superposition Transformer Interpretability
Whiteknights House, Whiteknights, Reading, RG6 6UR
Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures
Apr 17, 2026Yutong Gao, Qinglin Meng, Yuan Zhou +1Transformer Interpretability LLM Interpretability
MOE Key Lab of Computational Linguistics, Peking University · Nanjing University of Science and Technology · Purdue University +1
What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers
Apr 16, 2026Éric JacopinTransformer Interpretability Attention Head Analysis
Cosmic AI
Hallucination as Trajectory Commitment: Causal Evidence for Asymmetric Attractor Dynamics in Transformer Generation
Apr 16, 2026G. Aytug AkarlarTransformer Interpretability Hallucination in Language Models
Chimera Research Initiative Istanbul, Türkiye · Chimera Research Initiative is an independent research effort exploring causal and neuro-symbolic approaches to AI systems.
Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers
Apr 15, 2026Nina Żukowska, Wolfgang Stammer, Bernt Schiele +1Transformer Interpretability Mechanistic Interpretability
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany
Applied Explainability for Large Language Models: A Comparative Study
Apr 15, 2026Venkata Abhinandan KancharlaLLM Interpretability Explainability Evaluation
NRI Institute of Technology
Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision
Apr 14, 2026Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li +4Transformer Interpretability Vision Transformer
Rutgers University · University of California San Diego
Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task
Apr 14, 2026Alicia Curth, Rachel Lawrence, Sushrut Karmalkar +1Adaptive Computation Transformer Interpretability
Microsoft Research Cambridge
Distinct mechanisms underlying in-context learning in transformers
Apr 14, 2026Cole Gibson, Wenping Cui, Gautam ReddyIn-Context Learning Transformer Training Dynamics
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA · Department of Physics, Princeton University, Princeton, NJ 08544, USA
The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts
Apr 13, 2026Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero +1Neural Network Interpretability Mechanistic Interpretability
Rice University · Google Research · Brown University
Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification
Apr 13, 2026Patrick Lutz, Themistoklis Haris, Arjun Chandra +2In-Context Learning Transformer Interpretability
Boston University, Departments of Computer Science and Electrical & Computer Engineering.
Tracing the Thought of a Grandmaster-level Chess-Playing Transformer
Apr 11, 2026Rui Lin, Zhenyu Jin, Guancheng Zhou +7Transformer Interpretability Mechanistic Interpretability
Shanghai Innovation Institute, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China · School of Mathematics and Statistics, Xi’an Jiaotong University.
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Apr 11, 2026Zunhai Su, Hengyuan Zhang, Wei Wu +24Attention Sinks Transformer Attention
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +6
Relational Preference Encoding in Looped Transformer Internal States
Apr 10, 2026Jan KirinPairwise Preference Learning Reward Modeling
Selective Neuron Amplification in Transformer Language Models
Apr 8, 2026Ryyan Akhtar, Payal Pahwa, Monika AroraCausal Interventions in Language Models Activation Steering
Department of Artificial Intelligence and Data Science, Guru Gobind Singh Indraprastha University, BPIT, New Delhi, India
LAG-XAI: A Lie-Inspired Affine Geometric Framework for Interpretable Paraphrasing in Transformer Latent Spaces
Apr 7, 2026Olexander Mazurets, Olexander Barmak, Leonid Bedratyuk +1Transformer Interpretability Representation Geometry in Language Models
Department of Computer Science, Khmelnytskyi National University, Khmelnytskyi, 29016, Ukraine · Department of Theoretical Cybernetics, Taras Shevchenko National University of Kyiv, Kyiv, 03187, Ukraine · Laboratory of Communicative Information Technologies, V.M. Glushkov Institute of Cybernetics of NAS of Ukraine, Kyiv, 03187, Ukraine
Disentangling MLP Neuron Weights in Vocabulary Space
Apr 7, 2026Asaf Avrahamy, Yoav Gur-Arieh, Mor GevaLLM Interpretability Transformer Interpretability
Blavatnik School of Computer Science and AI, Tel Aviv University
Selective Aggregation of Attention Maps Improves Diffusion-Based Visual Interpretation
Apr 7, 2026Jungwon Park, Jungmin Ko, Dongnam Byun +1T2I Diffusion Models Cross-Attention
Department of Intelligence and Information, Seoul National University, Seoul, Republic of Korea · Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea
Learn to Rank: Visual Attribution by Learning Importance Ranking
Apr 7, 2026David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2Feature Attribution Explainable Image Classification
Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA
Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms
Apr 6, 2026James Hu, Mahdi GhelichiTabular ML Tabular ICL
Model Development Innovation, Risk Management TD Bank, Toronto, Canada
Automated Attention Pattern Discovery at Scale in Large Language Models
Apr 4, 2026Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes +2Transformer Interpretability LLM Interpretability
Delft University of Technology

Apr 25, 2026J/K move · Enter open · S save

From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers

Jainum Sanghavi

Northeastern University

Transformer Interpretability Visual Spatial Reasoning Visual Representation Learning Vision Transformer

Open paper AskPDF arXiv

Vision Transformers trained only on image classification routinely transfer to tasks that demand spatial understanding, yet they receive no spatial supervision during pretraining. We ask where and how robustly such structure is encoded. Probing a frozen ViT-B/16 layerwise for two complementary properties, local patch boundaries (BSDS500) and per-patch depth (NYU Depth V2), reveals a clear hierarchy: boundary structure becomes linearly decodable at layers 5-6 (AP = 0.833), while depth, which requires integrating global cues, peaks two to three layers later at layer 8 (MAE = 0.0875). Both signals collapse at the final classification layer, and random-weight controls confirm the encodings are learned rather than architectural. Causal interventions add specificity: ablating the single direction a linear depth probe reads degrades depth decoding by up to 165%, while ablating any other direction changes it by less than 1%. Targeted activation patching along that direction shows the depth signal is partially re-derived at each layer rather than passively carried in the residual stream, with mid-layer interventions persisting most strongly downstream. The result is that a classification-trained ViT develops an actively maintained spatial hierarchy that mirrors the early-to-late progression observed in the primate visual cortex.