Multimodal Model Interpretability

Ask about this topic

Overview

It asks which modalities, regions, tokens, concepts, or internal circuits support a model’s output, helping people audit errors, shortcuts, bias, and safety. Common approaches include saliency and Shapley attribution, concept bottlenecks and sparse autoencoders, and probing or causal interventions that test what happens when evidence is altered. Researchers evaluate these explanations on visual question answering, hallucination and grounding tasks, cross-modal benchmarks, and clinical settings where inputs may be missing or incomplete. A central tension is that an explanation can look plausible without being faithful: models may encode useful evidence yet ignore it, favor one modality, or rely on shortcuts, and evaluation of genuine cross-modal reasoning remains difficult.

Latest papers

CardsList

130 papers13 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

Apr 18, 2026

Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models

Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference

Xinru Yan, Boxi Cao, Yaojie Lu +4

University of Chinese Academy of Sciences, Beijing, China · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China

Unified Multimodal ModelsNeural Network InterpretabilityMultimodal Foundation ModelsMultimodal Model InterpretabilityMultimodal Model EvaluationMultimodal Understanding
AskPDF
Apr 17, 2026

Real-Time Visual Attribution Streaming in Thinking Model

We present an amortized framework for real-time visual attribution streaming in multimodal thinking models. When these models generate code from a screenshot or solve math problems from images, their long reasoning traces should be grounded in visual evidence. However, verifying this reliance is challenging: faithful causal methods require costly repeated backward passes or perturbations, while raw attention maps offer instant access, they lack causal validity. To resolve this, we introduce an amortized approach that learns to estimate the causal effects of semantic regions directly from the rich signals encoded in attention features. Across five diverse benchmarks and four thinking models, our approach achieves faithfulness comparable to exhaustive causal methods while enabling visual attribution streaming, where users observe grounding evidence as the model reasons, not after. Our results demonstrate that real-time, faithful attribution in multimodal thinking models is achievable through lightweight learning, not brute-force computation.

Seil Kang, Woojung Han, Junhyeok Kim +3

YYonsei University. · AAmazon.

VLM InterpretabilityMultimodal ReasoningVisually Grounded ReasoningMultimodal Model InterpretabilityMultimodal Understanding
AskPDF
Apr 13, 2026

Dynamic Summary Generation for Interpretable Multimodal Depression Detection

Depression remains widely underdiagnosed and undertreated because stigma and subjective symptom ratings hinder reliable screening. To address this challenge, we propose a coarse-to-fine, multi-stage framework that leverages large language models (LLMs) for accurate and interpretable detection. The pipeline performs binary screening, five-class severity classification, and continuous regression. At each stage, an LLM produces progressively richer clinical summaries that guide a multimodal fusion module integrating text, audio, and video features, yielding predictions with transparent rationale. The system then consolidates all summaries into a concise, human-readable assessment report. Experiments on the E-DAIC and CMDC datasets show significant improvements over state-of-the-art baselines in both accuracy and interpretability.

Shiyu Teng, Jiaqing Liu, Hao Sun +6

College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Faculty of Engineering, University of the Ryukyus, Japan

Depression DetectionDepression Severity EstimationMultimodal Model InterpretabilityMedical Report Generation
AskPDF
Apr 13, 2026

Diffusion-CAM: Faithful Visual Explanations for dMLLMs

While diffusion Multimodal Large Language Models (dMLLMs) have recently achieved remarkable strides in multimodal generation, the development of interpretability mechanisms has lagged behind their architectural evolution. Unlike traditional autoregressive models that produce sequential activations, diffusion-based architectures generate tokens via parallel denoising, resulting in smooth, distributed activation patterns across the entire sequence. Consequently, existing Class Activation Mapping (CAM) methods, which are tailored for local, sequential dependencies, are ill-suited for interpreting these non-autoregressive behaviors. To bridge this gap, we propose Diffusion-CAM, the first interpretability method specifically tailored for dMLLMs. We derive raw activation maps by differentiably probing intermediate representations in the transformer backbone, accordingly capturing both latent features and their class-specific gradients. To address the inherent stochasticity of these raw signals, we incorporate four key modules to resolve spatial ambiguity and mitigate intra-image confounders and redundant token correlations. Extensive experiments demonstrate that Diffusion-CAM significantly outperforms SoTA methods in both localization accuracy and visual fidelity, establishing a new standard for understanding the parallel generation process of diffusion multimodal systems.

Haomin Zuo, Yidi Li, Luoxiao Yang +1

Sun Yat-sen University · Northwestern University · Technion - Israel Institute of Technology +1

Multimodal Model InterpretabilityFeature AttributionMultimodal Diffusion Models
AskPDF
Apr 13, 2026

Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models

Unified multimodal models (UMMs) were designed to combine the reasoning ability of large language models (LLMs) with the generation capability of vision models. In practice, however, this synergy remains elusive: UMMs fail to transfer LLM-like reasoning to image synthesis and exhibit divergent response behaviors. We term this phenomenon pseudo-unification. Diagnosing its internal causes is important, but existing probing methods either lack model-internal insight or ignore prompt-response dependencies. To address these limitations, we propose an information-theoretic probing framework that jointly analyzes how UMMs encode inputs and generate outputs. Applied to ten representative UMMs, our framework reveals that pseudo-unification stems from a dual divergence: (i) Modality-Asymmetric Encoding, where vision and language follow different entropy trajectories, and (ii) Pattern-Split Response, where text generation exhibits high-entropy creativity while image synthesis enforces low-entropy fidelity. Only models that unify both sides (e.g., via contextual prediction) achieve more genuine unification, enabling stronger reasoning-based text-to-image generation even with fewer parameters. Our work provides the first model-internal probing of unification, demonstrating that real multimodal synergy requires consistency in information flow, not just shared parameters.

Songlin Yang, Xianghao Kong, Anyi Rao

Unified Multimodal ModelsMultimodal GenerationMultimodal Model InterpretabilityRepresentation Probing
AskPDF
Apr 9, 2026

Tree-of-Evidence: Efficient "System 2" Search for Faithful Multimodal Grounding

Large Multimodal Models (LMMs) achieve state-of-the-art performance in high-stakes domains like healthcare, yet their reasoning remains opaque. Current interpretability methods, such as attention mechanisms or post-hoc saliency, often fail to faithfully represent the model's decision-making process, particularly when integrating heterogeneous modalities like time-series and text. We introduce Tree-of-Evidence (ToE), an inference-time search algorithm that frames interpretability as a discrete optimization problem. Rather than relying on soft attention weights, ToE employs lightweight Evidence Bottlenecks that score coarse groups or units of data (e.g., vital-sign windows, report sentences) and performs a beam search to identify the compact evidence set required to reproduce the model's prediction. We evaluate ToE across six tasks spanning three datasets and two domains: four clinical prediction tasks on MIMIC-IV, cross-center validation on eICU, and non-clinical fault detection on LEMMA-RCA. ToE produces auditable evidence traces while maintaining predictive performance, retaining over 0.98 of full-model AUROC with as few as five evidence units across all settings. Under sparse evidence budgets, ToE achieves higher decision agreement and lower probability fidelity error than other approaches. Qualitative analyses show that ToE adapts its search strategy: it often resolves straightforward cases using only vitals, while selectively incorporating text when physiological signals are ambiguous. ToE therefore provides a practical mechanism for auditing multimodal models by revealing which discrete evidence units support each prediction.

Micky C. Nnamdi, Benoit L. Marteau, Yishan Zhong +2

Georgia Institute of Technology

Multimodal Model InterpretabilityMultimodal GroundingEvidence SelectionEfficient Multimodal Inference
AskPDF
Apr 7, 2026

Beyond Semantics: Disentangling Information Scope in Sparse Autoencoders for CLIP

Sparse Autoencoders (SAEs) have emerged as a powerful tool for interpreting the internal representations of CLIP vision encoders, yet existing analyses largely focus on the semantic meaning of individual features. We introduce information scope as a complementary dimension of interpretability that characterizes how broadly an SAE feature aggregates visual evidence, ranging from localized, patch-specific cues to global, image-level signals. We observe that some SAE features respond consistently across spatial perturbations, while others shift unpredictably with minor input changes, indicating a fundamental distinction in their underlying scope. To quantify this, we propose the Contextual Dependency Score (CDS), which separates positionally stable local scope features from positionally variant global scope features. Our experiments show that features of different information scopes exert systematically different influences on CLIP's predictions and confidence. These findings establish information scope as a critical new axis for understanding CLIP representations and provide a deeper diagnostic view of SAE-derived features.

Yusung Ro, Jaehyun Choi, Junmo Kim

KAIST · Korea AI Safety Institute, ETRI

Neural Network InterpretabilityVLM InterpretabilitySparse AutoencodersMultimodal Model Interpretability
AskPDF
Apr 6, 2026

Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models

Learners' use of video controls in educational videos provides implicit signals of cognitive processing and instructional design quality, yet the lack of scalable and explainable predictive models limits instructors' ability to anticipate such behavior before deployment. We propose a scalable, interpretable pipeline for predicting population-level watching, pausing, skipping, and rewinding behavior as proxies for cognitive load from video content alone. Our approach leverages multimodal large language models (MLLMs) to compute embeddings of short video segments and trains a neural classifier to identify temporally fine-grained interaction peaks. Drawing from multimedia learning theory on instructional design for optimal cognitive load, we code features of the video segments using GPT-5 and employ them as a basis for interpreting model predictions via concept activation vectors. We evaluate our pipeline on 77 million video control events from 66 online courses. Our findings demonstrate that classifiers based on MLLM embeddings reliably predict interaction peaks, generalize to unseen academic fields, and encode interpretable, theory-relevant instructional concepts. Overall, our results show the feasibility of cost-efficient, interpretable pre-screening of educational video design and open new opportunities to empirically examine multimedia learning theory at scale.

Dominik Glandorf, Fares Fawzi, Tanja Käser

EPFL, Switzerland

Video UnderstandingMultimodal Model Interpretability
AskPDF
Apr 4, 2026

MultiPress: A Multi-Agent Framework for Interpretable Multimodal News Classification

With the growing prevalence of multimodal news content, effective news topic classification demands models capable of jointly understanding and reasoning over heterogeneous data such as text and images. Existing methods often process modalities independently or employ simplistic fusion strategies, limiting their ability to capture complex cross-modal interactions and leverage external knowledge. To overcome these limitations, we propose MultiPress, a novel three-stage multi-agent framework for multimodal news classification. MultiPress integrates specialized agents for multimodal perception, retrieval-augmented reasoning, and gated fusion scoring, followed by a reward-driven iterative optimization mechanism. We validate MultiPress on a newly constructed large-scale multimodal news dataset, demonstrating significant improvements over strong baselines and highlighting the effectiveness of modular multi-agent collaboration and retrieval-augmented reasoning in enhancing classification accuracy and interpretability.

Tailong Luo, Hao Li, Rong Fu +7

New York Institute of Technology · University of Arizona · University of Macau +3

Multimodal ClassificationMultimodal AgentsMultimodal FusionMultimodal Model Interpretability
AskPDF
Apr 3, 2026

Do Audio-Visual Large Language Models Really See and Hear?

Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.

Ramaneswaran Selvakumar, Kaousheik Jayakumar, S Sakshi +3

University of Maryland, College Park

Multimodal Large Language ModelsAudio-Visual UnderstandingMultimodal Model Interpretability
AskPDF
Mar 31, 2026

Quantifying Cross-Modal Interactions in Multimodal Glioma Survival Prediction via InterSHAP: Evidence for Additive Signal Integration

Multimodal deep learning for cancer prognosis is commonly assumed to benefit from synergistic cross-modal interactions, yet this assumption has not been directly tested in survival prediction settings. This work adapts InterSHAP, a Shapley interaction index-based metric, from classification to Cox proportional hazards models and applies it to quantify cross-modal interactions in glioma survival prediction. Using TCGA-GBM and TCGA-LGG data (n=575), we evaluate four fusion architectures combining whole-slide image (WSI) and RNA-seq features. Our central finding is an inverse relationship between predictive performance and measured interaction: architectures achieving superior discrimination (C-index 0.64→\to0.82) exhibit equivalent or lower cross-modal interaction (4.8%→\to3.0%). Variance decomposition reveals stable additive contributions across all architectures (WSI≈{\approx}40%, RNA≈{\approx}55%, Interaction≈{\approx}4%), indicating that performance gains arise from complementary signal aggregation rather than learned synergy. These findings provide a practical model auditing tool for comparing fusion strategies, reframe the role of architectural complexity in multimodal fusion, and have implications for privacy-preserving federated deployment.

Iain Swift, JingHua Ye, Ruairi O'Reilly

Department of Computer Science, Munster Technological University, Cork, Ireland

Multimodal FusionMultimodal Survival AnalysisCancer Survival PredictionMultimodal Model InterpretabilityShapley Value Attribution
AskPDF
Mar 28, 2026

Structural Graph Probing of Vision-Language Models

Vision-language models (VLMs) achieve strong multimodal performance, yet how computation is organized across populations of neurons remains poorly understood. In this work, we study VLMs through the lens of neural topology, representing each layer as a within-layer correlation graph derived from neuron-neuron co-activations. This view allows us to ask whether population-level structure is behaviorally meaningful, how it changes across modalities and depth, and whether it identifies causally influential internal components under intervention. We show that correlation topology carries recoverable behavioral signal; moreover, cross-modal structure progressively consolidates with depth around a compact set of recurrent hub neurons, whose targeted perturbation substantially alters model output. Neural topology thus emerges as a meaningful intermediate scale for VLM interpretability: richer than local attribution, more tractable than full circuit recovery, and empirically tied to multimodal behavior. Code is publicly available at https://github.com/he-h/vlm-graph-probing.

Haoyu He, Yue Zhuo, Yu Zheng +1

Northeastern University · Massachusetts Institute of Technology

VLM InterpretabilityMultimodal Model InterpretabilityCausal Interpretability
AskPDF
Mar 26, 2026

Probabilistic Concept Graph Reasoning for Multimodal Misinformation Detection

Multimodal misinformation poses an escalating challenge that often evades traditional detectors, which are opaque black boxes and fragile against new manipulation tactics. We present Probabilistic Concept Graph Reasoning (PCGR), an interpretable and evolvable framework that reframes multimodal misinformation detection (MMD) as structured and concept-based reasoning. PCGR follows a build-then-infer paradigm, which first constructs a graph of human-understandable concept nodes, including novel high-level concepts automatically discovered and validated by multimodal large language models (MLLMs), and then applies hierarchical attention over this concept graph to infer claim veracity. This design produces interpretable reasoning chains linking evidence to conclusions. Experiments demonstrate that PCGR achieves state-of-the-art MMD accuracy and robustness to emerging manipulation types, outperforming prior methods in both coarse detection and fine-grained manipulation recognition.

Ruichao Yang, Wei Gao, Xiaobin Zhu +5

University of Science and Technology Beijing · Singapore Management University · Hong Kong Baptist University

Graph ReasoningMultimodal RobustnessMultimodal Graph LearningMultimodal Model Interpretability
AskPDF
Mar 25, 2026

A Neuro-Symbolic System for Interpretable Multimodal Physiological Signals Integration in Human Fatigue Detection

We propose a neuro-symbolic architecture that learns four interpretable physiological concepts, oculomotor dynamics, gaze stability, prefrontal hemodynamics, and multimodal, from eye-tracking and neural hemodynamics, functional near-infrared spectroscopy, (fNIRS) windows using attention-based encoders, and combines them with differentiable approximate reasoning rules using learned weights and soft thresholds, to address both rigid hand-crafted rules and the lack of subject-level alignment diagnostics. We apply this system to fatigue classification from multimodal physiological signals, a domain that requires models that are accurate and interpretable, with internal reasoning that can be inspected for safety-critical use. In leave-one-subject-out evaluation on 18 participants (560 samples), the method achieves 72.1% +/- 12.3% accuracy, comparable to tuned baselines while exposing concept activations and rule firing strengths. Ablations indicate gains from participant-specific calibration (+5.2 pp), a modest drop without the fNIRS concept (-1.2 pp), and slightly better performance with Lukasiewicz operators than product (+0.9 pp). We also introduce concept fidelity, an offline per-subject audit metric from held-out labels, which correlates strongly with per-subject accuracy (r=0.843, p < 0.0001).

Mohammadreza Jamalifard, Yaxiong Lei, Parasto Azizinezhad +2

School of Computer Science and Electronic Engineering, University of Essex, United Kingdom

Multimodal Model InterpretabilityNeuro-Symbolic ReasoningMultimodal ClassificationMultimodal Sensor Fusion
AskPDF
Mar 24, 2026

From Prediction to Diagnosis: Reasoning-Aware AI for Photovoltaic Defect Inspection

Reliable photovoltaic defect identification is essential for maintaining energy yield, ensuring warranty compliance, and enabling scalable inspection of rapidly expanding solar fleets. Although recent advances in computer vision have improved automated defect detection, most existing systems operate as opaque classifiers that provide limited diagnostic insight for high-stakes energy infrastructure. Here we introduce REVL-PV, a vision-language framework that embeds domain-specific diagnostic reasoning into multimodal learning across electroluminescence, thermal, and visible-light imagery. By requiring the model to link visual evidence to plausible defect mechanisms before classification, the framework produces structured diagnostic reports aligned with professional photovoltaic inspection practice. Evaluated on 1,927 real-world modules spanning eight defect categories, REVL-PV achieves 93% classification accuracy while producing interpretable diagnostic rationales and maintaining strong robustness under realistic image corruptions. A blind concordance study with a certified solar inspection expert shows strong semantic alignment between model explanations and expert assessments across defect identification, root-cause attribution, and visual descriptions. These results demonstrate that reasoning-aware multimodal learning establishes a general paradigm for trustworthy AI-assisted inspection of photovoltaic energy infrastructure.

Dev Mistry, Feng Qiu, Bo Chen +4

Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616, USA · Argonne National Laboratory, Lemont, IL 60439, USA · Smart Grid Emerging Technology Department, Commonwealth Edison, Chicago, IL 60618, USA +2

Fault DetectionIndustrial InspectionMultimodal ReasoningMultimodal Model Interpretability
AskPDF
Mar 20, 2026

OmniTrace: A Unified Framework for Generation-Time Source Attribution in Omni-Modal LLMs

An omni-modal language model can answer a question about a multimodal document, a recording, or a video, but it cannot show which part of the input the answer came from. Asking the model to cite its evidence yields what it says it used; measuring what it used has so far required rerunning it many times with parts of the input removed, a cost that grows with every second of audio and video. We show that the generation pass already holds most of this information. OmniTrace attributes each generated sentence to source units (text chunks, images, and seconds of audio or video) by reading late-layer attention as mass over units with no extra forward pass and no per-modality tuning; head weights can be trained where ablation data are available. With fixed causal test setting, the units OmniTrace selects are causally important for responses and for individual sentences in video, audio, and image-text question answering on Qwen3-Omni-30B, MiniCPM-o 4.5 and Qwen2.5-Omni-7B. With one generation pass, OmniTrace recovers 77-96% of the effect of leave-one-out attribution, which needs up to 41 extra passes. OmniTrace also agrees best with human-marked evidence: its response-level rankings reach an AUROC of 0.83 on documents and 0.64 on video, against 0.76 and 0.60 for the strongest baseline.

Qianqi Yan, Yichen Guo, Ching-Chen Kuo +4

University of California, Santa Barbara · eBay

Multimodal Model InterpretabilityMultimodal Large Language ModelsMultimodal Generation
AskPDF
Mar 20, 2026

DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis

Deep learning models have achieved strong performance in medical image analysis, but their internal decision processes remain difficult to interpret. Concept Bottleneck Models (CBMs) partially address this limitation by structuring predictions through human-interpretable clinical concepts. However, existing CBMs typically overlook the contextual dependencies among concepts. To address these issues, we propose an end-to-end interpretable framework DCG-Net that integrates multimodal alignment with structured concept reasoning. DCG-Net introduces a Dual Cross-Attention module that replaces cosine similarity matching with bidirectional attention between visual tokens and canonicalized textual concept-value prototypes, enabling spatially localized evidence attribution. To capture the relational structure inherent to clinical concepts, we develop a Parametric Concept Graph initialized with Positive Pointwise Mutual Information priors and refined through sparsity-controlled message passing. This formulation models inter-concept dependencies in a manner consistent with clinical domain knowledge. Experiments on white blood cell morphology and skin lesion diagnosis demonstrate that DCG-Net achieves state-of-the-art classification performance while producing clinically interpretable diagnostic explanations.

Getamesay Dagnaw, Xuefei Yin, Muhammad Hassan Maqsood +2

School of Information and Communication Technology, Griffith University, QLD, Australia

Explainable Medical Image AnalysisMedical Image ClassificationConcept Bottleneck ModelsMultimodal Model InterpretabilityGraph Reasoning
AskPDF
Mar 19, 2026

Multimodal Model for Computational Pathology:Representation Learning and Image Compression

Whole slide imaging (WSI) has transformed digital pathology by enabling computational analysis of gigapixel histopathology images. Recent foundation model advances have accelerated progress in computational pathology, facilitating joint reasoning across pathology images, clinical reports, and structured data. Despite this progress, challenges remain: the extreme resolution of WSIs creates computational hurdles for visual learning; limited expert annotations constrain supervised approaches; integrating multimodal information while preserving biological interpretability remains difficult; and the opacity of modeling ultra-long visual sequences hinders clinical transparency. This review comprehensively surveys recent advances in multimodal computational pathology. We systematically analyze four research directions: (1) self-supervised representation learning and structure-aware token compression for WSIs; (2) multimodal data generation and augmentation; (3) parameter-efficient adaptation and reasoning-enhanced few-shot learning; and (4) multi-agent collaborative reasoning for trustworthy diagnosis. We specifically examine how token compression enables cross-scale modeling and how multi-agent mechanisms simulate a pathologist's "Chain of Thought" across magnifications to achieve uncertainty-aware evidence fusion. Finally, we discuss open challenges and argue that future progress depends on unified multimodal frameworks integrating high-resolution visual data with clinical and biomedical knowledge to support interpretable and safe AI-assisted diagnosis.

Peihang Wu, Zehong Chen, Lijian Xu

Shenzhen University of Advanced Technology

Computational PathologyWhole-Slide Image AnalysisMedical Image Representation LearningMultimodal LearningVisual Token CompressionMultimodal Model Interpretability
AskPDF
Mar 19, 2026

Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media

Advances in social media data dissemination enable the provision of real-time information during a crisis. The information comes from different classes, such as infrastructure damages, persons missing or stranded in the affected zone, etc. Existing methods attempted to classify text and images into various humanitarian categories, but their decision-making process remains largely opaque, which affects their deployment in real-life applications. Recent work has sought to improve transparency by extracting textual rationales from tweets to explain predicted classes. However, such explainable classification methods have mostly focused on text, rather than crisis-related images. In this paper, we propose an interpretable-by-design multimodal classification framework. Our method first learns the joint representation of text and image using a visual language transformer model and extracts text rationales. Next, it extracts the image rationales via the mapping with text rationales. Our approach demonstrates how to learn rationales in one modality from another through cross-modal rationale transfer, which saves annotation effort. Finally, tweets are classified based on extracted rationales. Experiments are conducted over CrisisMMD benchmark dataset, and results show that our proposed method boosts the classification Macro-F1 by 2-35% while extracting accurate text tokens and image patches as rationales. Human evaluation also supports the claim that our proposed method is able to retrieve better image rationale patches (12%) that help to identify humanitarian classes. Our method adapts well to new, unseen datasets in zero-shot mode, achieving an accuracy of 80%.

Thi Huyen Nguyen, Koustav Rudra, Wolfgang Nejdl

L3S Research Center, Leibniz University Hannover, Hannover, Germany · Indian Institute of Technology Kharagpur, Kharagpur, India

ClassificationMultimodal ClassificationMultimodal Model InterpretabilityExplainable Image Classification
AskPDF
Mar 17, 2026

Tabular LLMs for Interpretable Few-Shot Alzheimer's Disease Prediction with Multimodal Biomedical Data

Accurate diagnosis of Alzheimer's disease (AD) requires handling tabular biomarker data, yet such data are often small and incomplete, where deep learning models frequently fail to outperform classical methods. Pretrained large language models (LLMs) offer few-shot generalization, structured reasoning, and interpretable outputs, providing a powerful paradigm shift for clinical prediction. We propose TAP-GPT Tabular Alzheimer's Prediction GPT, a domain-adapted tabular LLM framework built on TableGPT2 and fine-tuned for few-shot AD classification using tabular prompts rather than plain texts. We evaluate TAP-GPT across four ADNI-derived datasets, including QT-PAD biomarkers and region-level structural MRI, amyloid PET, and tau PET for binary AD classification. Across multimodal and unimodal settings, TAP-GPT improves upon its backbone models and outperforms traditional machine learning baselines in the few-shot setting while remaining competitive with state-of-the-art general-purpose LLMs. We show that feature selection mitigates degradation in high-dimensional inputs and that TAP-GPT maintains stable performance under simulated and real-world missingness without imputation. Additionally, TAP-GPT produces structured, modality-aware reasoning aligned with established AD biology and shows greater stability under self-reflection, supporting its use in iterative multi-agent systems. To our knowledge, this is the first systematic application of a tabular-specialized LLM to multimodal biomarker-based AD prediction, demonstrating that such pretrained models can effectively address structured clinical prediction tasks and laying the foundation for tabular LLM-driven multi-agent clinical decision-support systems. The source code is publicly available on GitHub: https://github.com/sophie-kearney/TAP-GPT.

Sophie Kearney, Shu Yang, Zixuan Wen +8

University of Pennsylvania, Philadelphia, PA, USA · Stony Brook University, Stony Brook, NY, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA +2

Alzheimer's Disease ClassificationTabular Foundation ModelsMultimodal ClassificationMultimodal Model Interpretability
AskPDF
Mar 14, 2026

SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis

Interpreting the internal reasoning of vision-language models is essential for deploying AI in safety-critical domains. Concept-based explainability provides a human-aligned lens by representing a model's behavior through semantically meaningful components. However, existing methods are largely restricted to images and overlook the cross-modal interactions. Text-image embeddings, such as those produced by CLIP, suffer from a modality gap, where visual and textual features follow distinct distributions, limiting interpretability. Canonical Correlation Analysis (CCA) offers a principled way to align features from different distributions, but has not been leveraged for multi-modal concept-level analysis. We show that the objectives of CCA and InfoNCE are closely related, such that optimizing CCA implicitly optimizes InfoNCE, providing a simple, training-free mechanism to enhance cross-modal alignment without affecting the pre-trained InfoNCE objective. Motivated by this observation, we couple concept-based explainability with CCA, introducing Concept CCA (CoCCA), a framework that aligns cross-modal embeddings while enabling interpretable concept decomposition. We further extend it and propose Sparse Concept CCA (SCoCCA), which enforces sparsity to produce more disentangled and discriminative concepts, facilitating improved activation, ablation, and semantic manipulation. Our approach generalizes concept-based explanations to multi-modal embeddings and achieves state-of-the-art performance in concept discovery, evidenced by reconstruction and manipulation tasks such as concept ablation.

Ehud Gordon, Meir Yossef Levi, Guy Gilboa

Viterbi Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, Israel

Canonical Correlation AnalysisMultimodal Model InterpretabilityConcept-Based ExplanationsModality Gap in VLMsCross-Modal Alignment
AskPDF
Mar 13, 2026

Towards Faithful Multimodal Concept Bottleneck Models

Concept Bottleneck Models (CBMs) are interpretable models that route predictions through a layer of human-interpretable concepts. While widely studied in vision and, more recently, in NLP, CBMs remain largely unexplored in multimodal settings. For their explanations to be faithful, CBMs must satisfy two conditions: concepts must be properly detected, and concept representations must encode only their intended semantics, without smuggling extraneous task-relevant or inter-concept information into final predictions, a phenomenon known as leakage. Existing approaches treat concept detection and leakage mitigation as separate problems, and typically improve one at the expense of predictive accuracy. In this work, we introduce f-CBM, a faithful multimodal CBM framework built on a vision-language backbone that jointly targets both aspects through two complementary strategies: a differentiable leakage loss to mitigate leakage, and a Kolmogorov-Arnold Network prediction head that provides sufficient expressiveness to improve concept detection. Experiments demonstrate that f-CBM achieves the best trade-off between task accuracy, concept detection, and leakage reduction, while applying seamlessly to both image and text or text-only datasets, making it versatile across modalities.

Pierre Moreau, Emeline Pineau Ferrand, Yann Choho +3

Ekimetrics, F-75009 Paris, France · Télécom Paris, F-91120 Palaiseau, France · Sorbonne Université, CNRS, LIP6, F-75005 Paris, France

Concept Bottleneck ModelsMultimodal Model Interpretability
AskPDF
Mar 12, 2026

Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition

Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.

Umberto Cappellazzo, Stavros Petridis, Maja Pantic

Imperial College London, UK · NatWest AI Research, UK

Audio-Visual Speech RecognitionModality ImbalanceMultimodal Model InterpretabilityShapley Value AttributionAudio-Visual Alignment
AskPDF
Mar 12, 2026

FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval

Composed image retrieval (CIR) requires multi-modal models to jointly reason over visual content and semantic modifications presented in text-image input pairs. While current CIR models achieve strong performance on common benchmark cases, their accuracies often degrades in more challenging scenarios where negative candidates are semantically aligned with the query image or text. In this paper, we attribute this degradation to focus imbalances, where models disproportionately attend to one modality while neglecting the other. To validate this claim, we propose FBCIR, a multi-modal focus interpretation method that identifies the most crucial visual and textual input components to a model's retrieval decisions. Using FBCIR, we report that focus imbalances are prevalent in existing CIR models, especially under hard negative settings. Building on the analyses, we further propose a CIR data augmentation workflow that facilitates existing CIR datasets with curated hard negatives designed to encourage balanced cross-modal reasoning. Extensive experiments across multiple CIR models demonstrate that the proposed augmentation consistently improves performance in challenging cases, while maintaining their capabilities on standard benchmarks. Together, our interpretation method and data augmentation workflow provide a new perspective on CIR model diagnosis and robustness improvements.

Chenchen Zhao, Jianhuan Zhuo, Muxi Chen +6

Department of Computer Science and Engineering, The Chinese University of Hong Kong · Tencent AI Data Department · School of Mathematical Sciences, Dalian University of Technology +1

Modality Reliance in Multimodal ModelsComposed Image RetrievalMultimodal LearningMultimodal RobustnessMultimodal Model Interpretability
AskPDF
Mar 4, 2026

Feature-level Interaction Explanations in Multimodal Transformers

Multimodal Transformers often produce predictions without clarifying how different modalities jointly support a decision. Most existing multimodal explainable AI (MXAI) methods extend unimodal saliency to multimodal backbones, highlighting important tokens or patches within each modality, but they rarely pinpoint which cross-modal feature pairs provide complementary evidence (synergy) or serve as reliable backups (redundancy). We present Feature-level I2MoE (FL-I2MoE), a structured Mixture-of-Experts layer that operates directly on token/patch sequences from frozen pretrained encoders and explicitly separates unique, synergistic, and redundant evidence at the feature level. We further develop an expert-wise explanation pipeline that combines attribution with top-K% masking to assess faithfulness, and we introduce Monte Carlo interaction probes to quantify pairwise behavior: the Shapley Interaction Index (SII) to score synergistic pairs and a redundancy-gap score to capture substitutable (redundant) pairs. Across three benchmarks (MMIMDb, ENRICO, and MMHS150K), FL-I2MoE yields more interactionspecific and concentrated importance patterns than a dense Transformer with the same encoders. Finally, pair-level masking shows that removing pairs ranked by SII or redundancy-gap degrades performance more than masking randomly chosen pairs under the same budget, supporting that the identified interactions are causally relevant. Code is available at https://github.com/dut0817/FL-I2MoE.

Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1

University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada

Multimodal UnderstandingFeature AttributionExplainable Artificial IntelligenceTransformer InterpretabilityMultimodal Model InterpretabilityMixture-of-Experts Models+1
AskPDF
Mar 2, 2026

Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction

While multimodal survival prediction models are increasingly more accurate, their complexity often reduces interpretability, limiting insight into how different data sources influence predictions. To address this, we introduce DIMAFx, an explainable multimodal framework for cancer survival prediction that produces disentangled, interpretable modality-specific and modality-shared representations from histopathology whole-slide images and transcriptomics data. Across multiple cancer cohorts, DIMAFx achieves state-of-the-art performance and improved representation disentanglement. Leveraging its interpretable design and SHapley Additive exPlanations, DIMAFx systematically reveals key multimodal interactions and the biological information encoded in the disentangled representations. In breast cancer survival prediction, the most predictive features contain modality-shared information, including one capturing solid tumor morphology contextualized primarily by late estrogen response, where higher-grade morphology aligned with pathway upregulation and increased risk, consistent with known breast cancer biology. Key modality-specific features capture microenvironmental signals from interacting adipose and stromal morphologies. These results show that multimodal models can overcome the traditional trade-off between performance and explainability, supporting their application in precision medicine.

Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur +4

AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands · Department of Metabolic Diseases, Wilhelmina Children’s Hospital, University Medical Center Utrecht, Utrecht, The Netherlands · Regenerative Medicine Center Utrecht, Utrecht, The Netherlands +3

Cancer Survival PredictionMultimodal Survival AnalysisMultimodal Model InterpretabilityMultimodal Disentangled Representation LearningComputational Pathology
AskPDF
Mar 2, 2026

PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification

Accurate classification of pediatric central nervous system tumors remains challenging due to histological complexity and limited training data. While pathology foundation models have advanced whole-slide image (WSI) analysis, they often fail to leverage the rich, complementary information found in clinical text and tissue microarchitecture. To this end, we propose PathMoE, an interpretable multimodal framework that integrates H&E slides, pathology reports, and nuclei-level cell graphs via an interaction-aware mixture-of-experts architecture built on state-of-the-art foundation models for each modality. By training specialized experts to capture modality uniqueness, redundancy, and synergy, PathMoE employs an input-dependent gating mechanism that dynamically weights these interactions, providing sample-level interpretability. We evaluate our framework on two dataset-specific classification tasks on an internal pediatric brain tumor dataset (PBT) and external TCGA datasets. PathMoE improves macro-F1 from 0.762 to 0.799 (+0.037) on PBT when integrating WSI, text, and graph modalities; on TCGA, augmenting WSI with graph knowledge improves macro-F1 from 0.668 to 0.709 (+0.041). These results demonstrate significant performance gains over state-of-the-art image-only baselines while revealing the specific modality interactions driving individual predictions. This interpretability is particularly critical for rare tumor subtypes, where transparent model reasoning is essential for clinical trust and diagnostic validation.

Jian Yu, Joakim Nguyen, Jinrui Fang +10

University of Texas, Austin, TX, USA · Dell Children’s Medical Center, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA +1

Brain Tumor ClassificationMultimodal ClassificationComputational PathologyMultimodal Model InterpretabilityMixture-of-Experts Models
AskPDF
Feb 28, 2026

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

Multimodal large language models (MLLMs) project visual tokens into the embedding space of language models, yet the internal structuring and processing of visual semantics remain poorly understood. In this work, we introduce a two-fold analytical framework featuring a novel probing tool, EmbedLens\textbf{EmbedLens}, to conduct a fine-grained analysis. We uncover a pronounced semantic sparsity at the input level: visual tokens consistently partition into sink, dead, and alive categories. Remarkably, only the alive tokens, comprising ≈60%\approx60\% of the total input, carry image-specific meaning. Furthermore, using a targeted patch-compression benchmark, we demonstrate that these alive tokens already encode rich, fine-grained cues (e.g., objects, colors, and OCR) prior to entering the LLM. Internal visual computations (such as visual attention and feed-forward networks) are redundant for most standard tasks. For the small subset of highly vision-centric tasks that actually benefit from internal processing, we reveal that alive tokens naturally align with intermediate LLM layers rather than the initial embedding space, indicating that shallow-layer processing is unnecessary and that direct mid-layer injection is both sufficient. Ultimately, our findings provide a unified mechanistic view of visual token processing, paving the way for more efficient and interpretable MLLM architectures through selective token pruning, minimized visual computation, and mid-layer injection. The code is released at: https://github.com/EIT-NLP/EmbedLens.

Yingqi Fan, Junlong Tong, Anhao Zhao +1

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +1

Multimodal Large Language ModelsMultimodal Model InterpretabilityVisual Token Pruning
AskPDF
Feb 27, 2026

A multimodal slice discovery framework for systematic failure detection and explanation in medical image classification

Despite advances in machine learning-based medical image classifiers, the safety and reliability of these systems remain major concerns in practical settings. Existing auditing approaches mainly rely on unimodal features or metadata-based subgroup analyses, which are limited in interpretability and often fail to capture hidden systematic failures. To address these limitations, we introduce the first automated auditing framework that extends slice discovery methods to multimodal representations specifically for medical applications. Comprehensive experiments were conducted under common failure scenarios using the MIMIC-CXR-JPG dataset, demonstrating the framework's strong capability in both failure discovery and explanation generation. Our results also show that multimodal information generally allows more comprehensive and effective auditing of classifiers, while unimodal variants beyond image-only inputs exhibit strong potential in scenarios where resources are constrained.

Yixuan Liu, Kanwal K. Bhatia, Ahmed E. Fetit

Department of Computing, Imperial College London, UK · Aival, London, UK

Algorithmic AuditingMedical Image ClassificationMultimodal Model InterpretabilityMultimodal Learning
AskPDF
Feb 24, 2026

AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs

Despite strong performance in audio perception tasks, large audio-language models (AudioLLMs) remain opaque to interpretation. A major factor behind this lack of interpretability is that individual neurons in these models frequently activate in response to several unrelated concepts. We introduce the first mechanistic interpretability framework for AudioLLMs, leveraging sparse autoencoders (SAEs) to disentangle polysemantic activations into monosemantic features. Our pipeline identifies representative audio clips, assigns meaningful names via automated captioning, and validates concepts through human evaluation and steering. Experiments show that AudioLLMs encode structured and interpretable features, enhancing transparency and control. This work provides a foundation for trustworthy deployment in high-stakes domains and enables future extensions to larger models, multilingual audio, and more fine-grained paralinguistic features. Project URL: https://townim-faisal.github.io/AutoInterpret-AudioLLM/

Townim Faisal Chowdhury, Ta Duc Huy, Siqi Pan +2

Australian Institute for Machine Learning, University of Adelaide · Dolby Laboratories · School of Computer and Mathematical Sciences, University of Adelaide, Australia

Multimodal Model InterpretabilityMechanistic InterpretabilityRepresentation Disentanglement
AskPDF