Transformer Interpretability

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 180% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 224

All topics

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
May 29, 2026Harry Jake Cunningham, Nicola Muca CironeTransformer Interpretability Feature Attribution
Department of Computer Science, University College London, London, UK. · Cartesia.AI, San Francisco, US.
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiTransformer Interpretability Disentangled Representation Learning
Sorbonne Université, CNRS, ISIR, Paris, France · Orange Innovation
Improving Adversarial Robustness of Attribution via Implicit Regularization
May 28, 2026Amir Mehrpanah, Matteo Gamba, Hossein AzizpourTransformer Interpretability Neural Network Robustness
Department of Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Computer Science, Brown University, USA · Science for Life Laboratory, Stockholm, Sweden
Unsupervised Semantic Segmentation Facilitates Model Understanding
May 28, 2026Xiaoyan Yu, Lisa Mais, Jannik Franzen +4Transformer Interpretability Vision Transformer
Max-Delbrück-Center (MDC), Berlin, Germany · Helmholtz Imaging · Humboldt-Universität zu Berlin, Berlin, Germany +2
Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames
May 28, 2026Mazen KobroslyTransformer Interpretability Neural Representation Geometry
Independent Researcher
The Attentional White Bear Effect in Transformer Language Models
May 27, 2026Rebecca Ramnauth, Brian ScassellatiTransformer Interpretability LLM Alignment
Yale University New Haven, CT, USA
Integrated and Cross-Architecture Interpretation of LLM Reasoning
May 27, 2026Leonardo Matthew Yauw, Wei-Bin Kou, Yujiu YangTransformer Interpretability LLM Interpretability
Tsinghua Shenzhen International Graduate School · Harbin Institute of Technology, Shenzhen
ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions
May 27, 2026Prathyush Poduval, Calvin Yeung, Neel Desai +1Transformer Interpretability Activation Sparsity
Department of Computer Science University of California, Irvine Irvine, CA 92697 · Georgia Institute of Technology Atlanta, GA 30332
MechRL: Reinforcement Learning Agents Perform Circuit Discovery for Mechanistic Interpretability
May 25, 2026Barsat KhadkaTransformer Interpretability Circuit Discovery
School of Computing Sciences and Engineering The University of Southern Mississippi
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
May 25, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenTransformer Interpretability Cross-Modal Attention
Zhejiang University
Universal Activation Verbalizer: A Unified Framework for Cross-Model Activation Explanation
May 25, 2026Haiyan Zhao, Zirui He, Guanchu Wang +3Transformer Interpretability LLM Interpretability
New Jersey Institute of Technology · University of North Carolina at Charlotte · Cisco Research +1
Geometric Evolution Maps: Extracting Stable Concept Probes from Transformer Residual Streams
May 25, 2026James HenryTransformer Interpretability Representation Probing
Independent Researcher
Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning
May 25, 2026Han-yu WangTransformer Interpretability Task Vectors
The University of Hong Kong
Analogies between Transformer Layers and Power Method
May 25, 2026Chenglong Li, Claudio AltafiniTransformer Interpretability Transformer
Transformer Field Theory: A Response-Theoretic Approach to Mechanistic Interpretability
May 24, 2026David N. Olivieri, Antonio F. Pérez RodríguezTransformer Interpretability Mechanistic Interpretability
Departamento de Informática, Universidade de Vigo, Spain · Independent Researcher, Spain
The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth
May 24, 2026James HenryTransformer Interpretability Representation Learning
Independent Researcher
CSP-Atlas: Concept-Specific Neural Circuits in a Sparse Python Transformer
May 23, 2026Piotr WilamTransformer Interpretability Circuit Discovery
University College London
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
May 23, 2026Jordan F. McCannTransformer Interpretability Transformer
Independent Researcher
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
May 23, 2026Spandan PratyushTransformer Interpretability Sparse Attention
Independent Researcher, New Delhi 110070, India
Causal Physics Steering in Video World Models via Concept Activation Vectors
May 23, 2026Nahid AlamTransformer Interpretability Activation Steering
Oreon Labs, Cohere Labs Community
Every Component Is a Lookup: One Linear Graph for Interaction, Composition and Attribution
May 22, 2026Po-Kai Chen, Aske Plaat, Niki van SteinTransformer Interpretability Token-Level Credit Assignment
Leiden University
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
Reading Task Failure Off the Activations: A Sparse-Feature Audit of GPT-2 Small on Indirect Object Identification
May 21, 2026Mahdi NasermoghadasiTransformer Interpretability Sparse Autoencoders
Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX
Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a Transformer
May 21, 2026Ishita Darade, Sushrut ThoratNumerical Reasoning in Language Models Transformer Interpretability
MKSSS’s Cummins College of Engineering for Women, Pune 411052, India · Institute of Cognitive Science, Osnabrück University, Osnabrück 49069, Germany
From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models
May 21, 2026Caleb MunigetyTransformer Interpretability Causal Reasoning in Language Models
Independent Researcher
Towards Explainability of SLMs by investigating Token Level Activation
May 21, 2026Sayantani Ghosh, Rajashik Datta, Amit Kumar Das +1Transformer Interpretability Feature Attribution
Information Technology, A.K. Choudhury School of Information Technology, Kolkata, West Bengal, India. · Computer Science & Engineering(Artificial Intelligence), Institute of Engineering & Management, Kolkata, 700091, West Bengal, India. · Computer Science & Engineering, Institute of Engineering & Management, Kolkata, 700091, West Bengal, India. +1
Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy
May 20, 2026Lawhori Chakrabarti, Jennifer Johnson-Leung, Bert Baumgaertner +3Figurative Language Understanding Transformer Interpretability
Department of Computer Science, University of Idaho, Moscow ID 83844, USA · Department of Mathematics and Statistical Science, University of Idaho, Moscow ID 83844, USA · Department of Politics and Philosophy, University of Idaho, Moscow ID 83844, USA +2
Markovian Circuit Tracing for Transformer State Dynamic
May 20, 2026Abdullah XTransformer Interpretability Transformer
Project AWARE and Zephara AI
Chessformer: A Unified Architecture for Chess Modeling
May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2Transformer Interpretability Game-Playing Agents
University of Toronto · Williams College
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
May 18, 2026Jucheng Shen, Barbara Su, Anastasios KyrillidisTransformer Interpretability Recurrent Transformers
Rice University

May 29, 2026, cs.LGJ/K move · Enter open · S save

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Harry Jake Cunningham, Nicola Muca Cirone

Department of Computer Science, University College London, London, UK. · Cartesia.AI, San Francisco, US.

Transformer Interpretability Feature Attribution Self-Attention Attention Sinks

Open paper AskPDF arXiv

Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs). However, this approach has significant limitations as it neglects the geometric properties of the value vectors being aggregated. To address this gap, we introduce \emph{Contribution Weights}, a projection-based metric that quantifies a token's influence by accounting for it's attention weight, value magnitude, and directional alignment with the layer output. We demonstrate that contribution weights provide a more faithful measure of token importance, consistently outperforming attention-based metrics in identifying semantically critical tokens across different decoder-only models, tasks, and datasets. Further, our metric enables novel mechanistic analysis of \emph{attention sinks}. While previous work characterized sinks as passive repositories for excess attention, we reveal they serve an active functional role, suppressing information through a convex relationship between sink rate and output norm, stabilizing representations by opposing the semantic drift of low-confidence tokens.

Momentum

Related topics

Latest papers 224