Interpretability Illusion

The "interpretability illusion" describes the misleading appearance of understanding in complex machine learning models, particularly deep neural networks, arising from simplification or interpretation methods. Current research focuses on identifying and mitigating these illusions in various model architectures, including transformers and those employing techniques like subspace interventions and activation patching, often using controlled datasets and systematic generalization tests. Understanding and overcoming these illusions is crucial for building trust in AI systems and ensuring the reliable application of machine learning in scientific discovery and other high-stakes domains where accurate causal inference is paramount.

Papers

March 8, 2024

Multiple Instance Learning with random sampling for Whole Slide Image Classification
H. Keshvarikhojasteh, J. P. W. Pluim, M. Veta
Inherent Interpretability Multiple Instance Learning Computational Pathology Model Interpretability Whole Slide Image Classification Random Sampling Interpretability Illusion

January 23, 2024

A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments
Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Icard, Christopher Potts, Noah D. Goodman
Inherent Interpretability User Response Target Argument Optical Illusion Et Al Interpretability Illusion

December 6, 2023

Interpretability Illusions in the Generalization of Simplified Models
Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, Asma Ghandeharioun
Strong Generalization Transformer Model Generalization Gap Systematic Generalization Proxy Dataset Simple Model Interpretability Illusion

November 28, 2023

Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching
Aleksandar Makelov, Georg Lange, Neel Nanda
Inherent Interpretability Mechanistic Interpretability Subspace Prototype Mechanistic Interpretation Interpretability Illusion

October 11, 2023

An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l
James Dao, Yeu-Tong Lau, Can Rager, Jett Janiak
Language Model Adversarial Example Adversarial Prompt Residual Stream Memory Management MLP Layer Interpretability Illusion

February 2, 2023

Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2
Wolfgang Messner
Machine Learning XAI Method Hypothesis Testing Explanatory Inference Effect Size Interpretability Illusion

Interpretability Illusion

Papers

Multiple Instance Learning with random sampling for Whole Slide Image Classification

A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments

Interpretability Illusions in the Generalization of Simplified Models

Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching

An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l

Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2