Transformer Interpretability

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 180% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 224

All topics

Beyond Linear Superposition: Discovering Climate Features in AI Weather Models with KAN-SAE
May 17, 2026Minjong CheonTransformer Interpretability Climate Science
Department of Computer Science and Engineering Sejong University
Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
May 15, 2026Valeria Ruscio, Eli-Shaoul Khedouri, Keiran ThompsonLanguage Model Pretraining Transformer Interpretability
Intuition Machines
How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning
May 15, 2026Entang Wang, Yiwei Wang, Aleksandra Bakalova +1Transformer Interpretability Task Vectors
Saarland Informatics Campus, Saarland Univesity, Saarbrücken, Germany.
On the Interpretability of Whisper Encodings Using Sparse Autoencoders
May 12, 2026Dan Pluth, Zachary Nicholas Houghton, Yu Zhou +1Transformer Interpretability Mechanistic Interpretability
Vail Systems, Inc. · University of Oregon
From Clever Hans to Scientific Discovery: Interpreting EEG Foundational Transformers with LRP
May 12, 2026Justus Meyer zu Bexten, Nico Scherf, Bogdan Franczyk +1Transformer Interpretability EEG Decoding
Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig University · Neural Data Science and Statistical Computing, Max Planck Institute for Human Cognitive and Brain Sciences · Faculty of Economics, Leipzig University +1
Transformer Interpretability from Perspective of Attention and Gradient
May 12, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenGradient-Based Attribution Transformer Interpretability
Zhejiang University
Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions
May 11, 2026Andrew Lee, Fernanda Viégas, Martin WattenbergTransformer Interpretability Neural Representation Geometry
Harvard University · Google DeepMind* · *Work done at Harvard.
From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models
May 11, 2026Zehao Li, Yasuhiro Yoshikai, Shumpei Nemoto +2Transformer Interpretability Language Modeling
Laboratory of Molecular Pharmacokinetics, Graduate School of Pharmaceutical Sciences, The University of Tokyo, 7-3-1 Hongo, Bunkyo, Tokyo, Japan · The Institute of Statistical Mathematics (ISM), Research Organization of Information and Systems, 190-8562 Tachikawa, Tokyo, Japan
Dissecting Jet-Tagger Through Mechanistic Interpretability
May 11, 2026Saurabh Rai, Sanmay GangulyTransformer Interpretability Attention Head Analysis
Department of Physics, Indian Institute of Technology, Kanpur. Uttar-Pradesh 208016, India
fmxcoders: Factorized Masked Crosscoders for Cross-Layer Feature Discovery
May 10, 2026Andreas D. Demou, Panagiotis Koromilas, James Oldfield +2Transformer Interpretability Sparse Autoencoders
1The Cyprus Institute · University of Athens · 4Archimedes AI/Athena Research Center +2
Attention Sinks in Diffusion Transformers: A Causal Analysis
May 10, 2026Fangzheng Wu, Brian SummaTransformer Interpretability Diffusion Transformer
Department of Computer Science, Tulane University, New Orleans, LA, USA.
Architecture, Not Scale: Circuit Localization in Large Language Models
May 9, 2026Sohan VenkateshTransformer Interpretability Grouped-Query Attention
Manipal Institute of Technology Bengaluru.
Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure
May 9, 2026Nilesh Sarkar, Dawar Jyoti DekaTransformer Interpretability Sparse Autoencoders
Erdős AI Lab
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
May 8, 2026Mingsong Yan, Dongyang Li, Charles Kulick +1Softmax Attention Transformer Interpretability
Department of Mathematics, University of California, Santa Barbara, CA
Belief or Circuitry? Causal Evidence for In-Context Graph Learning
May 8, 2026Katharine Kowalyshyn, Timothy Duggan, Daniel Little +1Transformer Interpretability LLM Reasoning with Graphs
Department of Computer Science, Tufts University, Medford, United States
Context-Gated Associative Retrieval: From Theory to Transformers
May 8, 2026Moulik Choraria, Argyrios Gerogiannis, Vidhata Jayaraman +2Transformer Interpretability In-Context Learning
UIUC · University of Minnesota, Twin Cities · Stony Brook University
Instruction Tuning Changes How Upstream State Conditions Late Readout: A Cross-Patching Diagnostic
May 8, 2026Yifan ZhouTransformer Interpretability Instruction Tuning
University of California, Los Angeles
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation Models Transformer Interpretability
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany
From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features
May 7, 2026Ruben Fernandez-Boullon, Pablo Magariños-Docampo, Javier Perez-RoblesTransformer Interpretability Sparse Autoencoders
University of Vigo
The Metagame of Interpretability and Meta-Attributions
May 7, 2026Hubert Baniecki, Przemyslaw Biecek, Fabian FumagalliTransformer Interpretability Feature Interaction Modeling
University of Warsaw · Centre for Credible AI, Warsaw University of Technology · Bielefeld University +1
Playing the network backward: A Game Theoretic Attribution Framework
May 7, 2026Jakob Paul Zimmermann, Jim Berend, Georg Loho +2Gradient-Based Attribution Transformer Interpretability
Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany. · Technische Universität Berlin, Berlin, Germany. · Freie Universität Berlin, Berlin, Germany. +2
Metonymy in vision models undermines attention-based interpretability
May 7, 2026Ananthu Aniraj, Cassio F. Dantas, Dino Ienco +2Transformer Interpretability Disentangled Representation Learning
Inria, EVERGREEN, University of Montpellier, 34090 Montpellier, France · INRAE, UMR TETIS, University of Montpellier, 34090 Montpellier, France · University of Trento, Trento, Italy
Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
May 7, 2026Hang Chen, Jiaying Zhu, Hongyang Chen +3Supervised Fine-Tuning Transformer Interpretability
School of Computer Science and Technology Xi’an Jiaotong University · School of Computer Science and Engineering The Chinese University of Hong Kong · Shaanxi Co., Ltd(Xi’an 710077, China) China Mobile Group +1
Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
May 6, 2026Alper YıldırımTransformer Interpretability Time Series Forecasting
Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
May 6, 2026Dominik Dahlem, Diego Maniloff, Mac MisiuraTransformer Interpretability Attention Head Analysis
Red Hat AI
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
May 5, 2026Hao Yan, Haolin Yang, Yiqiao ZhongTransformer Interpretability Neural Representation Geometry
University of Wisconsin–Madison · University of Chicago
Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers
May 5, 2026Mathis Immertreu, Fitim Abdullahu, Thomas Kinfe +3Transformer Interpretability Vision-Language Models
Cognitive Computational Neuroscience Group, Patter Recognition Lab, University Erlangen-Nürnberg, Germany · IDS Institut für Data Science, ZHAW School of Engineering, Winterthur, Switzerland · Mannheim Center for Neuromodulation and Neuroprosthetics, University Hospital Mannheim, Heidelberg University, Germany +3
The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It
May 5, 2026Gabriel GarciaNumerical Reasoning in Language Models Transformer Interpretability
Independent Researcher
Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models
May 3, 2026Kotaro Furuya, Takahito TanimuraTransformer Interpretability LLM Evaluation
Research and Development Group, Hitachi, Ltd. 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo 185-8601, Japan

May 17, 2026, cs.LGJ/K move · Enter open · S save

Beyond Linear Superposition: Discovering Climate Features in AI Weather Models with KAN-SAE

Minjong Cheon

Department of Computer Science and Engineering Sejong University

Transformer Interpretability Climate Science Sparse Autoencoders Mechanistic Interpretability Climate Modeling Kolmogorov-Arnold Networks+1

Open paper AskPDF arXiv

Deep learning weather prediction models achieve remarkable predictive skill yet remain largely opaque: we know little about how they represent physical climate phenomena internally. Mechanistic interpretability through Sparse Autoencoders (SAEs) offers a principled route to decomposing these representations, but existing SAEs assume strictly linear feature superposition - a constraint ill-suited for the highly nonlinear atmospheric dynamics encoded in modern transformers. We introduce KAN-SAE, a sparse autoencoder whose encoder replaces the standard ReLU with learnable per-feature B-spline activations drawn from Kolmogorov-Arnold Networks (KANs), allowing each latent dimension to develop its own nonlinear gating profile. Applied to Sonny, KAN-SAE discovers 975 alive features (vs. 566 for a linear baseline, a 72% improvement) with 20% lower inter-feature redundancy and comparable reconstruction fidelity. Without any climate supervision, KAN-SAE identifies an interpretable European heatwave feature spatially concentrated over western Europe, and a western Pacific typhoon tracker confirmed by causal steering experiments. Our results demonstrate that nonlinear activations are essential for mechanistic interpretability of deep learning weather prediction models, recovering climate features that remain invisible to linear baselines.

Momentum

Related topics

Latest papers 224