Mechanistic Interpretability

Mechanistic interpretability aims to understand how neural networks, particularly large language models (LLMs) and image models, perform computations by reverse-engineering their internal mechanisms. Current research focuses on identifying and characterizing "circuits"—minimal subnetworks responsible for specific tasks—within transformer architectures, often using techniques like sparse autoencoders and activation patching to analyze neuron activations and attention mechanisms. This work is crucial for improving model reliability, safety, and trustworthiness, as well as for gaining fundamental insights into the nature of artificial intelligence and its relationship to human cognition.

Papers

July 31, 2024

Comgra: A Tool for Analyzing and Debugging Neural Networks
Florian Dietz, Sophie Fellenz, Dietrich Klakow, Marius Kloft
Neural Network Neural Architecture Pytorch Model BRIO Tool Mechanistic Interpretability Code Debugging

July 29, 2024

Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability
Jorge García-Carrasco, Alejandro Maté, Juan Trujillo
Language Model Adversarial Attack Data Detection Adversarial Sample Security Vulnerability Mechanistic Interpretability

July 19, 2024

July 18, 2024

Mechanistically Interpreting a Transformer-based 2-SAT Solver: An Axiomatic Approach
Nils Palumbo, Ravi Mangal, Zifan Wang, Saranya Vijayakumar, Corina S. Pasareanu, Somesh Jha
Mechanistic Interpretability Abstract Interpretation Automatic Solver Axiomatic Approach Mechanistic Interpretation Program Analysis

July 15, 2024

July 11, 2024

Transformer Circuit Faithfulness Metrics are not Robust
Joseph Miller, Bilal Chughtai, William Saunders
Mechanistic Interpretability Circuit Discovery Circuit Fidelity

July 2, 2024

A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, Ziyu Yao
Inherent Interpretability DCU Insight AQ Transformer Based Language Model Neural Network Model Mechanistic Interpretability

July 1, 2024

Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition
Aliyah R. Hsu, Georgia Zhou, Yeshwanth Cherapanamjeri, Yaxuan Huang, Anobel Y. Odisho, Peter R. Carroll, Bin Yu
Transformer Megatron Decepticons Mechanistic Interpretability Mechanistic Interpretation Interpretation Method

June 25, 2024

Interpreting Attention Layer Outputs with Sparse Autoencoders
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, Neel Nanda
Mechanistic Interpretability Interpretable Feature Sparse Autoencoders Sparse Auto Encoders

June 20, 2024

Finding Safety Neurons in Large Language Models
Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, Juanzi Li
Large Language Model Mechanistic Interpretability Safety Alignment

June 17, 2024

May 27, 2024

May 22, 2024

Automatically Identifying Local and Global Circuits with Linear Computation Graphs
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, Xipeng Qiu
Mechanistic Interpretability Sparse Autoencoders Circuit Analysis Interpretability Scale Computation Graph Circuit Discovery

May 17, 2024

Mechanistic Interpretability

Papers

Comgra: A Tool for Analyzing and Debugging Neural Networks

Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability

Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery

InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques

Mechanistically Interpreting a Transformer-based 2-SAT Solver: An Axiomatic Approach

Mechanistic interpretability of large language models with applications to the financial services industry

Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities

Transformer Circuit Faithfulness Metrics are not Robust

A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models

Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition

Interpreting Attention Layer Outputs with Sparse Autoencoders

Finding Safety Neurons in Large Language Models

Transcoders Find Interpretable LLM Feature Circuits

Compact Proofs of Model Performance via Mechanistic Interpretability

Mechanistic Interpretability of Binary and Ternary Transformers

From Neurons to Neutrons: A Case Study in Interpretability

Automatically Identifying Local and Global Circuits with Linear Computation Graphs

The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks

Using Degeneracy in the Loss Landscape for Mechanistic Interpretability

Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning