What Improves Multimodal Misinformation Detection? Answers from a Large-Scale Empirical Study

Akshit Sharma; Prashant W. Patil

cs.CVSep 24, 2026

What Improves Multimodal Misinformation Detection? Answers from a Large-Scale Empirical Study

Authors: Akshit Sharma, Prashant W. Patil

Organizations: CVPR Lab, MFSDSAI Indian Institute of Technology Guwahati

Abstract

Multimodal misinformation is increasingly crafted to look convincing by pairing a textual claim with an image that appears to "prove" it. Yet in practice, building effective detectors often hinges on a small set of design choices that are rarely examined in a controlled way. In this paper, we conduct a large-scale study of multimodal design choices for misinformation detection with over 3,375 experiments- spanning three benchmark datasets and a broad range of pre-trained vision and language backbones. Through systematic comparisons and targeted robustness analyses, we distill practical guidance on which design choices help, when do they fail silently, and what aspects of the pipeline most strongly shape model behavior, answering 4 key Research Questions (RQs). We aim to provide a reliable foundation for designing stronger and more dependable multimodal misinformation detection systems, thus contributing to the broader research community.

Figures & tables

Figure 1: Figure showcasing the early fusion architecture used as part of our experiments

Figure 2: Figure showcasing the mid and late fusion architecture used as part of our experiments

Classifier	Metric	MMFakeBench	MiRAGeNews	VLDBench
Logistic Regression	Prec	73.93 ( $\pm$ 1.07)	91.65 ( $\pm$ 0.95)	63.68 ( $\pm$ 0.48)
	AUROC	83.55 ( $\pm$ 1.18)	96.80 ( $\pm$ 0.54)	69.63 ( $\pm$ 0.67)
	Acc	77.87 ( $\pm$ 0.87)	91.65 ( $\pm$ 0.95)	64.73 ( $\pm$ 0.45)
XGBoost	Prec	74.51 ( $\pm$ 0.74)	90.00 ( $\pm$ 0.83)	64.52 ( $\pm$ 0.32)
	AUROC	85.45 ( $\pm$ 0.82)	96.18 ( $\pm$ 0.48)	70.60 ( $\pm$ 0.42)
	Acc	78.29 ( $\pm$ 0.59)	89.99 ( $\pm$ 0.83)	65.53 ( $\pm$ 0.30)

Table 1: Classifier-wise in-domain averages (%) over all encoder pairs and fusion strategies. Values are mean

\pm

95% bootstrap CI half-width across matched configurations.

Fusion	Metric	MMFakeBench	MiRAGeNews	VLDBench
Early	Prec	75.20 ( $\pm$ 1.07)	88.80 ( $\pm$ 1.15)	64.06 ( $\pm$ 0.41)
	AUROC	83.73 ( $\pm$ 1.20)	95.10 ( $\pm$ 0.76)	69.93 ( $\pm$ 0.59)
	Acc	76.79 ( $\pm$ 0.85)	88.77 ( $\pm$ 1.16)	65.06 ( $\pm$ 0.40)
Mid	Prec	72.22 ( $\pm$ 1.07)	88.20 ( $\pm$ 1.41)	62.27 ( $\pm$ 0.71)
	AUROC	82.78 ( $\pm$ 1.26)	94.51 ( $\pm$ 1.02)	67.63 ( $\pm$ 0.98)
	Acc	76.62 ( $\pm$ 0.86)	88.19 ( $\pm$ 1.41)	63.31 ( $\pm$ 0.70)

Table 2: Fusion-wise in-domain averages (%) over all encoder pairs and classifier heads. Values are mean

\pm

95% bootstrap CI half-width across matched configurations.

Dataset	Fusion	Classifier	Full	Text drop	$\Delta_{\text{text}}$	Image drop	$\Delta_{\text{img}}$
MMFakeBench	Early	Logistic Reg.	72.2/82.0/76.6	64.4/75.2/68.2	7.8/6.8/8.5	62.8/71.0/65.7	9.4/11.0/11.0
		XGBoost	76.2/86.2/79.0	71.0/79.2/72.5	5.2/7.1/6.5	63.4/72.9/65.7	12.9/13.3/13.3
		Random Forest	77.2/83.0/74.7	74.2/76.8/72.8	3.0/6.1/1.9	58.4/72.3/71.0	18.8/10.7/3.8
	Mid	Logistic Reg.	74.9/84.9/78.6	66.4/78.0/72.3	8.4/6.9/6.3	69.2/77.7/73.2	5.7/7.2/5.4
		XGBoost	73.8/85.2/78.0	69.5/81.0/73.6	4.3/4.2/4.4	67.2/76.2/61.9	6.6/9.0/16.2
		Random Forest	68.0/78.2/73.2	62.3/70.4/68.5	5.7/7.8/4.8	58.0/65.1/64.2	10.0/13.1/9.0

Table 3: Modality-drop study. Each cell reports Macro-Precision/AUROC/Accuracy (%). “Full” is the complete multimodal model, and

\Delta_{\text{text}}

and

\Delta_{\text{img}}

are performance drops in percentage points when one modality is removed at test time. Detailed entries are rounded to one decimal for readability.

Figure 3: Main encoder-pair heatmap across the three datasets. Each cell reports mean Macro-Precision / Accuracy (%) for a fixed vision–text encoder pair, averaged over the three classifier heads under the given fusion strategy; color encodes AUROC.

Train $\rightarrow$ Test	Macro-Precision	AUROC	Accuracy
MMFakeBench $\rightarrow$ MiRAGeNews	63.25 ( $\pm$ 0.59)	67.70 ( $\pm$ 0.70)	57.84 ( $\pm$ 0.57)
MMFakeBench $\rightarrow$ VLDBench	54.51 ( $\pm$ 0.33)	56.60 ( $\pm$ 0.25)	57.42 ( $\pm$ 0.14)
MiRAGeNews $\rightarrow$ MMFakeBench	62.56 ( $\pm$ 0.53)	69.26 ( $\pm$ 0.70)	55.44 ( $\pm$ 0.70)
MiRAGeNews $\rightarrow$ VLDBench	49.25 ( $\pm$ 0.15)	48.88 ( $\pm$ 0.22)	44.85 ( $\pm$ 0.19)
VLDBench $\rightarrow$ MMFakeBench	52.64 ( $\pm$ 0.26)	54.69 ( $\pm$ 0.46)	48.68 ( $\pm$ 0.39)
VLDBench $\rightarrow$ MiRAGeNews	52.03 ( $\pm$ 0.35)	53.07 ( $\pm$ 0.52)	51.39 ( $\pm$ 0.27)

Table 4: Cross-dataset transfer by source–target direction. Values are mean

\pm

95% bootstrap CI half-width (%).

(a) Classifier-wise transfer averages by target dataset
Classifier	Metric	MMFakeBench	MiRAGeNews	VLDBench
Logistic Regression	Prec	58.22 ( $\pm$ 0.97)	58.77 ( $\pm$ 1.02)	51.64 ( $\pm$ 0.52)
	AUROC	63.10 ( $\pm$ 1.46)	62.02 ( $\pm$ 1.32)	52.80 ( $\pm$ 0.59)
	Acc	52.38 ( $\pm$ 0.85)	55.21 ( $\pm$ 0.73)	51.00 ( $\pm$ 1.05)
XGBoost	Prec	58.22 ( $\pm$ 1.01)	58.45 ( $\pm$ 1.12)	52.26 ( $\pm$ 0.55)
	AUROC	62.62 ( $\pm$ 1.39)	61.89 ( $\pm$ 1.53)	52.82 ( $\pm$ 0.74)

Table 5: Cross-dataset transfer averages by component family and target dataset. Values are mean

\pm

95% bootstrap CI half-width (%).

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 768 papers

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection
Jun 23, 2026Chenhao Dang, Dantong Zhu, Jun Yang +2Multimodal Agents Multimodal Model Evaluation
Shanghai Jiaotong University · Shanghai Artificial Intelligence Laboratory · Central South University +2
Probabilistic Concept Graph Reasoning for Multimodal Misinformation Detection
Mar 26, 2026Ruichao Yang, Wei Gao, Xiaobin Zhu +5Graph Reasoning Multimodal Robustness
University of Science and Technology Beijing · Singapore Management University · Hong Kong Baptist University
MM-VeriAgent: Learning to Use Extensive Tools to Verify Multimodal Misinformation with Reinforcement Learning
Sep 25, 2026Peipei Li, Shuhan Xia, Shengyang Liu +2Multimodal Deception Detection Multimodal Tool Use
Beijing University of Posts and Telecommunications · Minzu University of China · NLPR, Institute of automation, Chinese academy of science, Chinese Academy of Sciences

Jun 23, 2026J/K move · Enter open · S save

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

Chenhao Dang, Dantong Zhu, Jun Yang, Conghui He+1

Shanghai Jiaotong University · Shanghai Artificial Intelligence Laboratory · Central South University+2

Multimodal Agents Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal misinformation detection is increasingly important because viral posts now combine long multilingual narratives, several images, mixed provenance, and subtle text--image framing errors. Existing benchmarks and methods remain poorly matched to this setting: they usually isolate short captions, single images, binary labels, or one manipulation source, while agentic verification remains costly under realistic evidence search. We present ReMMD, a realistic multilingual multi-image agentic verification framework for multimodal misinformation detection. ReMMD includes ReMMDBench, a real-world multimodal misinformation detection benchmark with 500 samples, 2,756 images, five monolingual languages, two cross-lingual settings, three text-length tiers, multi-image posts, five-way veracity labels, eight distortion labels, evidence provenance, and rationales. It also includes ReMMD-Agent, a persistent-memory verifier that decomposes posts into atomic points, builds a reusable evidence set, and predicts structured L1/L2/L3 outputs. Across proprietary systems, open LVLMs, MMD-Agent, and T2-Agent, ReMMD-Agent obtains the best five-way veracity performance, with 41.80% accuracy and 39.12% macro-F1 using GPT-5.2, while reducing cost by 17.5% relative to MMD-Agent and 79.9% relative to T2-Agent. The project is available at https://dang-ai.github.io/ReMMD.

Abstract

Multimodal Fusion

Explore similar work

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

Probabilistic Concept Graph Reasoning for Multimodal Misinformation Detection

MM-VeriAgent: Learning to Use Extensive Tools to Verify Multimodal Misinformation with Reinforcement Learning