Explainability Evaluation

Follow this topic Ask about this topic

Momentum

27 papers in the last four weeks, up 50% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 218

All topics

How to train your model organism
Oct 7, 2026Xilin Wang, David Bau, Byron C. WallaceMechanistic Interpretability Explainability Evaluation
Northeastern University Boston, MA 02115, USA
Does Explainability Survive Data Drift?
Oct 4, 2026Samuel OzechiExplainability Evaluation
A Comparative Explainability Framework for DeBERTa-v3 in Zero-Shot Medical Abstract Classification
Oct 1, 2026Javier Diaz Esteban-Herreros, David Muñoz-Valero, Raquel Martínez-España +2Transformer Interpretability Perturbation-Based Feature Attribution
Department of Technologies and Information Systems, Universidad de Castilla-La Mancha, Avenida Carlos III, s/n, Toledo, 45071, Spain · University of Murcia, Avenida Teniente Flomesta, 5, Murcia, 3003, Spain · Murcian Bio-Health Institute (IMIB-Arrixaca), Pabellón Docente del Hospital Clínico Universitario Virgen de la Arrixaca, Murcia, 3120, Spain
Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability
Oct 1, 2026Chuqin Geng, Li Zhang, Haolin Ye +3Circuit Discovery Mechanistic Interpretability
University of Toronto · McGill University
MCIR: A Feature Dependence-Aware Explainability Method with Reliability Guarantees
Oct 1, 2026Poushali Sengupta, Sabita Maharjan, Frank Eliassen +2Feature Attribution Explainability Evaluation
Department of Informatics, University of Oslo Oslo, Norway · Department of Electronic Systems, Aalborg University Aalborg, Denmark
WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing
Sep 30, 2026Dominik Matuszek, Bartosz Zieliński, Tomasz Danel +1Gradient-Based Attribution Explainability Evaluation
Jagiellonian University, Faculty of Mathematics and Computer Science · Jagiellonian University, Doctoral School of Exact and Natural Sciences · Jagiellonian University, Jagiellonian Center for Artificial Intelligence +2
Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?
Sep 28, 2026Li Zhang, Chuqin Geng, Mark Zhang +4Faithfulness of Language Model Explanations Mechanistic Interpretability
University of Toronto · McGill University · Tsinghua University
Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?
Sep 28, 2026Teodor Chiaburu, Franz Motzkus, Frank Haußer +1Linear Representation Hypothesis Sparse Autoencoders
Fraunhofer SIT · ATHENE National Research Center for Applied Cybersecurity · AUMOVIO +3
A Unifying Framework of Concept-based Explainable AI with Completeness Guarantees
Sep 28, 2026Vojtěch Kůr, Adam Kukučka, Tomáš Brázdil +1Concept-Based Explanations Neural Network Interpretability
Masaryk University, Faculty of Informatics, Brno, Czechia
Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations
Sep 25, 2026Marcin Kostrzewa, Maciej ZiębaCounterfactual Evaluation Counterfactual Explanations
Department of Artificial Intelligence Wrocław University of Science and Technology
A Systematic Benchmark of Explainable Methods for Temporal Attribution in Sequential Recommendation Systems
Sep 23, 2026Akash Pandey, Kanisha Shah, Addrish Roy +5Gradient-Based Attribution Sequential Recommendation
Capital One, AI Foundations San Jose, CA, USA · Capital One, AI Foundations New York, NY, USA · Capital One, AI Foundations McLean, VA, USA +1
Diverse by Design: Architectural Constraints for Prototype-Based Interpretability
Sep 23, 2026Xinmiao Lin, Matthew WrightPrototype-Based Classification Prototype Learning
Rochester Institute of Technology
MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models
Sep 22, 2026Anirudh Prabhakaran, Alexandre Rocchi, Gianni FranchiFeature Attribution SHAP Feature Attribution
AMIAD, Pˆole Recherche, Palaiseau
Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability
Sep 20, 2026Aleks Czufarow, Ihor BabinNeural Network Interpretability Cross-Architecture Knowledge Distillation
XIV High School of Stanislaw Staszic,Warsaw,Poland · Ukrainian Catholic University, Lviv, Ukraine
When Do Language-Grounded Explanations Help? A Graph-Bottleneck for Farm Monitoring Interpretable Sheep Facial Pain
Sep 17, 2026Alam Noor, Miguel Guti'errez Gait'anConcept Bottleneck Models Facial Expression Recognition
CISTER Research Center, Porto, Portugal. · Department of Electrical Engineering, Pontificia Universidad Cat´olica de Chile, Santiago 7820436, Chile.
Evaluating Explanation Methods by the Predictors They Induce
Sep 17, 2026Jacob Selbæk, Hugo L. HammerFeature Attribution SHAP Feature Attribution
Department of Computer Science, Oslo Metropolitan University, Oslo, Norway · Department of Holistic Systems, SimulaMet, Oslo, Norway · Department of Plastic and Reconstructive Surgery, Oslo University Hospital, Oslo, Norway
TRIPROBE: Probing Task Separability Beyond Classification for XAI
Sep 16, 2026Amirhossein Sadough, Freek Hens, Aleksa Bokšan +2Multi-Task Learning Representation Probing
Machine Learning and Neural Computing Radboud University Nijmegen, Netherlands · Delft University of Technology Delft, Netherlands · Unconventional Computing Lab University of the West of England (UWE) Bristol, United Kingdom
Beyond Measurement Metrics: A Human-Centered Framework for Semantic Validation of Network Traffic Classification
Sep 15, 2026Igor Cherepanov, David Sessler, Alex Ulmer +2Human-in-the-Loop Evaluation Explainability Evaluation
Fraunhofer IGD, Darmstadt, 64283, Germany. · Technische Universität Darmstadt, Darmstadt, 64289, Germany.
Data storytelling meets interpretable machine learning: Decoding AI decisions for non-experts without revealing sensitive data and model details
Sep 14, 2026Lemen Chao, Zixuan Yang, Anran Fang +2Interpretable ML Explainability Evaluation
a Key Laboratory of Data Engineering and Knowledge Engineering, Ministry of Education, Renmin University of China · b School of Information Resource Management, Renmin University of China
EEG-Xplain: Decoding Neural Black-Boxes of EEG Foundation Models
Sep 14, 2026Hansong Ma, Junxiao WangGradient-Based Attribution Perturbation-Based Feature Attribution
Guangzhou University, China
Calibrating Interpretability Instruments Before Trusting Their Verdicts
Sep 13, 2026Orion Reblitz-RichardsonLLM Interpretability Causal Interpretability
Distiller Labs
LLMs as Post-hoc Auditors of Physiological Plausibility in Symbolic Regression: A Clinician-Evaluated Case Study
Sep 12, 2026Jorge López-Varela, J. Ignacio Hidalgo, José-Manuel Muñoz +6LLM Auditing Explainability Evaluation
Computer Architecture Department, Universidad Complutense de Madrid, Spain. · Bioinspired Intelligence Ltd., Pozuelo de Alarc´on, Spain · Universidad Aut´onoma de Baja California, M´exico +7
XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?
Sep 10, 2026Yanfei Hu Fleischhauer, Alona Zharova, Nadja Klein +1Explainable Artificial Intelligence LLM-as-a-Judge
LMU Munich, Germany · Chair of Information Systems, School of Business and Economics, Humboldt-Universität zu Berlin, Germany · Scientific Computing Center, Karlsruhe Institute of Technology, Germany +1
XAI-Refine: An Automated Explanation-Knowledge Loop for Brain-Age Prediction
Sep 8, 2026Yang Qiao, Junjie Wu, Deqiang Qiu +2Explainable Artificial Intelligence Functional Connectivity
Department of Computer Science Emory University Atlanta, Georgia, USA · Department of Radiology and Imaging Sciences Computational NeuroImaging & Neuroscience Lab, Emory University Atlanta, Georgia, USA · Department of Neurology Emory University School of Medicine Atlanta, Georgia, USA
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Sep 8, 2026Jaewoo Lim, Sungbok Shin, Sanghyun HongLLM Evaluation LLM Interpretability
Oregon State University, Corvallis USA · Sogang University, Seoul, South Korea
A Quantitative Evaluation Framework for Temporal Explainability in Echocardiographic Video Segmentation
Sep 7, 2026Jiyoo Noh, Jonathan H. ChanUltrasound Image Segmentation Explainability Evaluation
University of Toronto, Canada · King Mongkut’s University of Technology Thonburi, Thailand
Understanding the Impact of Model Pruning on Long-Tail Forgetting and Explanation Reliability in Medical Imaging
Sep 7, 2026Nazish Khalid, Tausifa Jan Saleem, Amal Saqib +2Model Compression Explainable Medical Image Analysis
Department of Electrical and Computer Engineering, Missouri University of Science and Technology, Rolla, MO 65409 USA · Computing and Mathematical Sciences Division, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Unraveling the Real Working Mechanism and Inherent Flaws of GAE: A Method for Interpreting Transformer Processes from an Economic Perspective
Sep 7, 2026Yongjin Cui, Xiaohui FanTransformer Interpretability Explainable Artificial Intelligence
Zhejiang University
Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
Sep 4, 2026Urja Pawar, Rajitha Ramanayake, Nabeel Kemal +6Faithfulness of Language Model Explanations LLM Auditing
BNY
Cross-Dataset Transfer and Reliability of Explainable Artificial Intelligence for RhythmFormer Remote Photoplethysmography
Sep 3, 2026Louis Chen, Torbjörn E. M. NordlingRemote Photoplethysmography Feature Attribution
Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan.

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

How to train your model organism

Xilin Wang, David Bau, Byron C. Wallace

Northeastern University Boston, MA 02115, USA

Mechanistic Interpretability Explainability Evaluation Multi-Objective Optimization

Open paper AskPDF arXiv

Model organisms of alignment-relevant behaviors (e.g., backdoors, sycophancy, spurious correlations) have emerged as a key tool for evaluating whitebox interpretability techniques. We argue that the prevailing practice of training model organisms to a single objective of installing the target behavior is insufficient and propose validating model organisms with respect to three objectives with associated metrics: target-behavior installation, general-capability preservation (i.e., parametric knowledge, chat quality), and output naturalness (i.e., CoT and activations). We re-visit two publicly released organism suites using this validation framework and show that (1) chat quality and CoT naturalness degrade substantially across training recipes, and (2) validation metrics predict how well interpretability methods recover the installed behavior, e.g., a logit lens readout covaries with an organism's general capabilities. We introduce a multi-objective training approach based on model merging to train more realistic model organisms. Finally, on a new suite of model organisms targeting demographic biases in clinical reasoning, we compare training recipes and find that DPO training stays closer to the base model than supervised finetuning, and the proposed model optimization approach better preserves capabilities and naturalness. Auditing this suite with an investigator agent, we again observe validation metrics tracking bias recovery. In sum, training methods shape the interpretability conclusions an organism supports, and we argue that one should consider multiple objectives to draw generalizable conclusions about interpretability methods using (realistic) model organisms.

Momentum

Related topics

Latest papers 218