Interpretability Research

Interpretability research aims to understand how and why machine learning models, particularly deep neural networks like transformers, make predictions. Current efforts focus on identifying crucial subnetworks (circuits) within models, developing improved methods for causal analysis (e.g., optimal ablation and counterfactual methods), and evaluating the effectiveness of various explanation techniques (e.g., through human studies and automated metrics). This research is vital for building trust in AI systems, improving model design, and facilitating the responsible deployment of AI in high-stakes applications.

Papers

October 17, 2023

Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations
Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, Leilani H. Gilpin
Natural Language Processing Sentiment Analysis Study Feature Explanation Method Interpretability Research LLM Explanation Self Explanation

October 16, 2023

Attribution Patching Outperforms Automated Circuit Discovery
Aaquib Syed, Can Rager, Arthur Conmy
Interpretability Research Circuit Discovery Computational Graph

October 2, 2023

A Framework for Interpretability in Machine Learning for Medical Imaging
Alan Q. Wang, Batuhan K. Karaman, Heejong Kim, Jacob Rosenthal, Rachit Saluja, Sean I. Young, Mert R. Sabuncu
Machine Learning Machine Learning Model New Framework Inherent Interpretability Medical Image Analysis Medical Imaging Interpretability Research Model Transparency

September 29, 2023

PRIME: Prioritizing Interpretability in Failure Mode Extraction
Keivan Rezaei, Mehrdad Saberi, Mazda Moayeri, Soheil Feizi
Inherent Interpretability Interpretable Feature Image Classification Model Failure Analysis Interpretability Research Essence Prime Language

June 21, 2023

Feature Interactions Reveal Linguistic Structure in Language Models
Jaap Jumelet, Willem Zuidema
Language Model Feature Interaction Interpretability Research Linguistic Structure Post Hoc Interpretability

May 31, 2023

Neuron to Graph: Interpreting Language Model Neurons at Scale
Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, Fazl Barez
Graph Drawing Visual Analogue Scale Single Neuron Level Individual Neuron Interpretability Research Interpretable Graph

April 28, 2023

Towards Automated Circuit Discovery for Mechanistic Interpretability
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, Adrià Garriga-Alonso
Transformer Model Mechanistic Interpretability Interpretability Research Circuit Discovery

December 18, 2022

Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, Dan Roth
Large Language Model Language Model Context Learning Integral Role Visual Analogue Scale Interpretability Research

July 27, 2022

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Tilman Räuker, Anson Ho, Stephen Casper, Dylan Hadfield-Menell
Deep Neural Network Timely Survey Inner Structure Interpretability Research Transparent AI Interpretability Tool DNN Interpretation

March 27, 2022

A Unified Study of Machine Learning Explanation Evaluation Metrics
Yipei Wang, Xiaoqian Wang
Line by Line Explanation Unified Framework Evaluation Metric Explanation Method Interpretability Research

February 9, 2022

Discovering Concepts in Learned Representations using Statistical Inference and Interactive Visualization
Adrianna Janik, Kris Sankaran
Latent Space Interactive Visualization Feature Space Learned Representation Statistical Inference Concept Discovery Interpretability Research

December 6, 2021

HIVE: Evaluating the Human Interpretability of Visual Explanations
Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, Olga Russakovsky
Visual Explanation Interpretability Research Interpretability Analysis Visual Interpretation Human Interpretability Interpretable Computer Vision Instructional Visual Editing