Mechanistic Interpretability of Deep Neural Networks: Methods, Frameworks, and Perspectives
Abstract
Deep neural networks underpin fundamental tasks in a wide range of signal domains, such as image, audio, and language processing. However, their internal mechanisms are notably difficult to interpret. Internal activations are structured signals transformed across layers that may encode a multitude of interdependent relationships, as individual neurons frequently represent multiple underlying concepts. Mechanistic interpretability aims to uncover the internal mechanisms of neural networks by identifying components whose activations are causally responsible for specific behaviors. In such a context, this article provides an overview of advances in mechanistic interpretability for the analysis of causal inference in deep neural networks.
Figures & tables
| TRADITIONAL INTERPRETABILITY | MECHANISTIC INTERPRETABILITY | |
|---|---|---|
| GOAL | Identify input features that most contribute to the model final output | Identify model components that are causally responsible for the model behavior |
| GRANULARITY | Input-output relationship | Model inner mechanisms |
| COMPUTATION COST | Medium-low (generally a single forward pass and backward pass) | Medium-high (sparse autoencoder training, circuit discovery across all model components) |
| SCALABILITY | Medium-high (analysis on model input and final output) | Medium-low (causal testing iterating across all model components) |
| LIMITATIONS | Approximate, limited analysis of how the model produces internal activations | Poor performance under data distribution shift |
| Lorenzo Cirillo received the B.S. degree in computer and control engineering, the M.S. degree in artificial intelligence and robotics, earning the Excellent Student Award, from the Sapienza University of Rome, Italy, in 2022 and 2024, respectively, where he is currently pursuing the Ph.D. degree in cybersecurity at the AlcorLab. In 2023, he was an intern researcher at the Bank of Italy, working on knowledge graphs and graph databases. He is a member of the EURASIP Biometrics, Data Forensics, and Security Technical Area Committee. His research interests include deep learning, computer vision, adversarial machine learning, and explainable artificial intelligence. |
| Irene Amerini is an Associate Professor at DIAG, Sapienza University of Rome, where she leads the Computer Vision and Multimedia Forensics Research Team at ALCORLab. She was an Assistant Professor at Sapienza and a postdoctoral researcher at the Image Forensics and Security Lab, University of Florence. She received her Ph.D. from the University of Florence in 2011 and held visiting positions at Binghamton University and Charles Sturt University. She has served in IEEE, EURASIP, and IAPR committees, and is Senior Area Editor for IEEE Signal Processing Letters. Her research focuses on trustworthy AI, computer vision, adversarial learning, multimedia forensics. |
| Andrea Cavallaro is a Full Professor at EPFL and a Fellow of the IAPR, ELLIS, and the Higher Education Academy. His research focuses on machine learning for multimodal perception, computer vision, machine listening, and information privacy. He served as an IEEE Signal Processing Society Distinguished Lecturer and as Chair of the IEEE Image, Video and Multidimensional Signal Processing Technical Committee. Furthermore, he has served as an Area, Associate, and Guest Editor for numerous leading journals and magazines, including IEEE Transactions on Image Processing, Signal Processing, and Multimedia, and the IEEE Signal Processing Magazine. |
Neural Network Interpretability
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
See all 354 papers