CRANE: Post-Hoc Counterfactual Retrieval for Multimodal Classifiers

Franco Rugolon; Ioanna Miliou; Panagiotis Papapetrou

cs.LGOct 22, 2026

CRANE: Post-Hoc Counterfactual Retrieval for Multimodal Classifiers

Authors: Franco Rugolon, Ioanna Miliou, Panagiotis Papapetrou

Organizations: Dept. of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden

Abstract

Counterfactual explanation methods have been introduced for unimodal settings, with dedicated frameworks for tabular, timeseries, text, and image data, yet no post-hoc method has been designed for classifiers that learn from multimodal inputs. Existing explainability methods for multimodal classifiers apply modality-specific attribution independently to each input, which fails to yield coherent explanations and does not transfer to arbitrary pre-trained multimodal architectures. To address these shortcomings, we present CRANE (Counterfactual Retrieval via Agnostic Nearest-neighbour Explanations), the first post-hoc, model-agnostic framework for nearest-neighbour multimodal counterfactual generation that retrieves real training examples from the target class as candidate explanations, ensuring plausibility by construction. Across four heterogeneous datasets and three fusion strategies, we show that Multimodal Consensus Retrieval (MC-R), which intersects per-modality neighbour sets to identify instances jointly close to the query in every modality, emerges as the strongest multimodal strategy, achieving the best or joint-best outcome on three of four datasets while grounding every explanation in the joint training distribution. Unimodal strategies remain competitive when a single modality dominates the decision boundary.

Figures & tables

Fig. 1: Independently generated per-modality counterfactuals can produce a chimeric explanation whose static features and timeseries trajectories are incoherent. Arrows denote rising (

\uparrow

) or falling (

\downarrow

) biomarkers.

Dataset	Modalities	Task	Train	Test
Sepsis	TS + Tab	Binary	2,085/fold	521/fold
Long COVID Tweets	Text + Tab	4-class	20,121	5,031
Hateful Memes	Text + Image	Binary	7,632	1,908
Real or Fake Jobs	Text + Tab	Binary	14,304	3,576

TABLE I: Summary of datasets used in the experiments.

Dataset	Strategy	Model	Macro F1	Acc	AUROC
Sepsis	Early	Spatiotemporal Attention + GRU	0.691	0.920	0.800
	Intermediate	RF on 32-dim latents	0.746	0.946	0.833
	Late	GRU + MLP branches	0.691	0.912	0.871
Long COVID Tweets	Early	MLP on frozen RoBERTa	0.589	0.650	0.855
	Intermediate	Fine-tuned transformer + MLP	0.717	0.759	0.921
	Late	TF-IDF + LR / GBT branches	0.604	0.658	0.844

TABLE II: Best classifier per fusion strategy and dataset.

Fig. 2: Generator quality across the full benchmark (

k=20

). Each box shows the distribution over all encoder/distance-metric combinations, datasets, and fusion strategies. All metrics are lower-is-better.

Dataset	Generator	Outcome	Proximity	Sparsity	Plausibility
Sepsis	TabularNN	0.056	0.158	0.474	0.143
	TimeSeriesNN	0.021	0.317	0.457	0.196
	EF-NN	0.055	0.131	0.465	0.143
	IF-NN	0.371	0.203	0.488	0.151
	MC-R	0.163	0.119	0.476	0.242
	MPS	0.124	0.093	0.450	0.129

TABLE III: Generator quality per dataset at

k=20

, averaged across all three fusion strategies and encoder/distance-metric combinations (lower is better; best result per metric/dataset in bold).

Fig. 3: Effect of

k

on counterfactual quality, averaged across all datasets and fusion strategies. Unimodal generators are shown with dashed lines, multimodal with solid lines.

Counterfactual Explanations

Counterfactual explanations show how changing an input could alter a model’s prediction, supporting model understanding, auditing, and actionable recourse.

See all 117 papers

Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
Jul 29, 2026Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera +2Multimodal Large Language Models Multimodal Model Interpretability
Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom
A Probabilistic Consensus-Driven Approach for Robust Counterfactual Explanations
Apr 19, 2026Marcin Kostrzewa, Maciej Zięba, Jerzy StefanowskiCounterfactual Explanations Conditional Generative Modeling
Wrocław University of Science and Technology, Wrocław 50-347, Poland · Tooploox, Wrocław 53-601, Poland · Poznań University of Technology, Poznań 61-131, Poland
CounterFlowNet: From Minimal Changes to Meaningful Counterfactual Explanations
Feb 19, 2026Oleksii Furman, Patryk Marszałek, Jan Masłowski +3Counterfactual Explanations Constrained Generative Modeling
Wrocław University of Science and Technology, Wrocław, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland +1

Jul 29, 2026J/K move · Enter open · S save

Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera, David Watson+1

Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom

Multimodal Large Language Models Multimodal Model Interpretability Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) increasingly support high-stakes decision making by combining complementary information from images and text. While existing explainability methods identify influential image regions or text tokens, they cannot answer a fundamental question: which modality drives a prediction? Consequently, a model may produce the correct output while relying on the wrong source of evidence, masking shortcut learning and unsafe reasoning. We formulate modality attribution as a complementary explainability objective for multimodal foundation models and propose Counterfactual Modality Attribution (CMA), the first framework for quantifying modality-level contributions in MLLMs. CMA generates image-only, text-only, and joint multimodal counterfactuals using coupled diffusion priors and converts them into principled modality attribution scores through a cooperative game-theoretic formulation based on Shapley values. We evaluate CMA on controlled synthetic benchmarks with known ground-truth modality reliance and on a real-world multimodal clinical dataset. CMA correctly identifies the decision-driving modality in 98% of controlled cases and consistently outperforms baselines, revealing failures of cross-modal reasoning that remain invisible to predictive accuracy alone. Our results establish modality attribution as a complementary dimension of explainability beyond feature attribution, providing a principled framework for auditing multimodal foundation models in safety-critical applications.

Abstract

Counterfactual Explanations

Explore similar work

Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

A Probabilistic Consensus-Driven Approach for Robust Counterfactual Explanations

CounterFlowNet: From Minimal Changes to Meaningful Counterfactual Explanations