SAFE-MR: Evidence Sufficiency Learning for Selective Multimodal Rumor Detection

Shiwen Ni

cs.CLOct 5, 2026

SAFE-MR: Evidence Sufficiency Learning for Selective Multimodal Rumor Detection

Authors: Shiwen Ni

Organizations: Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology

Abstract

Multimodal rumor detectors increasingly rely on retrieved evidence, yet relevant evidence is not necessarily sufficient for verification. Missing provenance, duplicated reports, and unresolved contradictions can produce confident predictions without adequate support. We introduce SAFE-MR, a framework that separates claim veracity from evidence sufficiency. The method decomposes image-text posts into verifiable claims, constructs a relation-aware claim-evidence graph, and aggregates evidence using provenance and contextual compatibility. Separate veracity and sufficiency heads support selective prediction, while evidence interventions encourage stability under irrelevant additions and sensitivity to evidence removal. On NewsCLIPpings, VERITE, and XFacta, SAFE-MR achieves macro-F1 scores of 91.2%, 75.8%, and 85.2%, respectively. Against the matched backbone with evidence, its macro-F1 gains are 2.2, 4.9, and 4.8 percentage points. On the diagnostic selection set, SAFE-MR reduces AURC from 0.105 for maximum-probability rejection to 0.075 and lowers error at 80% coverage from 13.8% to 8.5%. Evidence-perturbation and ablation results support the role of sufficiency learning and intervention training in improving selective verification.

Figures & tables

Panel of Figure 1 — Figure 1: Real imagery with a deliberately constructed incorrect caption. Left: topical evidence does not link the image to the claimed context. Right: the same mosaic’s documented provenance (Mabi, Kurashiki, Japan; July 2018) refutes the attribution to 2025. Decisions illustrate intended behavior, not experimental outputs. Image credit: Geospatial Information Authority of Japan; mosaic assembled by Wikimedia Commons contributor Keikai; CC BY 4.0 . The original mosaic is reproduced without cropping or content alteration. Source record: Wikimedia Commons .

Panel of Figure 2 — Figure 2: SAFE-MR architecture. (a) Claim and evidence representations form a relation-aware graph, followed by reliability-aware aggregation and separate veracity and sufficiency heads. Conflict links connect incompatible evidence statements. Selection combines sufficiency with calibrated predictive uncertainty. (b) Training applies intervention-specific constraints: irrelevant additions preserve predictions, decisive deletions can lower sufficiency, and conflict or pollution retains the underlying truth label.

Dataset	Population and split	Task
NewsCLIPpings	100,000 train / 5,000 validation / 5,000 test; balanced classes; no exclusions	Binary image–text contextual verification
VERITE-full	All 1,000: 338 true / 662 misleading	Full-benchmark binary multimodal misinformation detection
XFacta	2,400 total; development 240 (120/120); test 2,160 (1,080/1,080), true/false	Cross-type multimodal misinformation: image manipulation, image–text mismatch, and misleading text

Table 1: Benchmark roles, evaluation populations, and task definitions. XFacta counts follow arXiv:2508.09999v1. Evidence-packet annotation partitions are separate from these benchmark partitions.

Method	NewsCLIPpings	VERITE-full	XFacta
CLIP feature fusion	80.2/80.0	60.7/58.2	65.7/64.5
MUSE	86.8/86.2	67.5/65.2	71.2/70.6
AITR	88.2/88.0	70.3/68.5	75.3/74.7
SNIFFER	87.4/87.2	70.0/68.8	74.0/73.5
FRAUD-Net	88.5/88.1	71.2/69.7	76.0/75.3
DPOD	88.2/88.0	70.3/68.8	75.3/74.6

Table 2: Accuracy/macro-F1 (%) at 100% coverage for reproductions in this study. NewsCLIPpings uses the study’s 100k/5k/5k train/validation/test split; all local baselines use the same test set. VERITE denotes the full 1,000-example binary mapping.

Condition	AITR	E2LVLM	Backbone	SAFE-MR
Clean	77.4	81.2	80.5	84.6
Key deletion 25%	73.3	78.4	77.6	82.4
Key deletion 50%	69.2	74.6	73.5	80.3
Key deletion 75%	64.1	68.1	68.2	75.6
No external evidence	60.4	62.7	63.5	64.3
Irrelevant addition	72.6	77.5	75.7	82.8

Table 3: Pooled stress-test accuracy (%) at full coverage; no rejected cases are removed.

Selector	AURC	Risk@80%	Risk@60%	Class. ECE	Suff. ECE
Maximum probability	0.105	13.8	9.0	0.090	–
Temperature + probability	0.105	13.8	9.0	0.050	–
Learned selector	0.095	11.5	7.5	–	0.055
SAFE-MR sufficiency + entropy	0.075	8.5	5.5	–	0.040

Table 4: Selective prediction on the pooled diagnostic set. Risk is classification error (%) at matched coverage; AURC and ECE are on the 0–1 scale. Class. ECE evaluates classification confidence; Suff. ECE evaluates sufficiency probability against

a^{*}

. ECE uses each selector’s own accepted set, whereas risk uses the stated coverage. ECE columns and differing accepted populations must not be compared as a common calibration benchmark. Dashes denote unreported metrics.

Variant	Stress F1	Suff. F1	Insuff. recall	AURC
Full SAFE-MR	78.5	80.4	85.2	0.075
Without sufficiency supervision	77.2	65.4	68.6	0.105
Without intervention training	73.4	74.3	76.7	0.095
Without reliability features	74.2	75.6	78.3	0.090
Without atomic claims	75.1	76.4	79.6	0.088
Flat fusion, matched parameters	76.3	77.6	80.5	0.085

Table 5: Component ablations. F1 columns are percentages; AURC uses the 0–1 scale.

Explore similar work

Multimodal rumor detection enhanced by external evidence and forgery features
Jan 21, 2026Han Li, Hua SunMultimodal Fusion Forgeries
Information Engineering School of Dalian Ocean University,Dalian Liaoning, China
RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild
May 11, 2026Danni Xu, Shaojing Fan, Harry Cheng +1Fact-Checking Reference-Guided Multimodal In-Context Verification
School of Computing (SoC), National University of Singapore (NUS), Singapore · Department of Electrical and Computer Engineering (ECE), National University of Singapore (NUS), Singapore
Sparse Evidence Can Suffice: Agentic Evidence Seeking for Multimodal Video Misinformation Detection
Jul 20, 2026Haochen Zhao, Yongxiu Xu, Xinkui Lin +6Misinformation Detection Fine-Grained Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Baidu, Beijing, China

Jan 21, 2026, cs.LGJ/K move · Enter open · S save

Multimodal rumor detection enhanced by external evidence and forgery features

Han Li, Hua Sun

Information Engineering School of Dalian Ocean University,Dalian Liaoning, China

Multimodal Fusion Forgeries Deep Semantic Representations Image-Text Pairs

Open paper AskPDF arXiv

Social media increasingly disseminates information through mixed image text posts, but rumors often exploit subtle inconsistencies and forged content, making detection based solely on post content difficult. Deep semantic mismatch rumors, which superficially align images and texts, pose particular challenges and threaten online public opinion. Existing multimodal rumor detection methods improve cross modal modeling but suffer from limited feature extraction, noisy alignment, and inflexible fusion strategies, while ignoring external factual evidence necessary for verifying complex rumors. To address these limitations, we propose a multimodal rumor detection model enhanced with external evidence and forgery features. The model uses a ResNet34 visual encoder, a BERT text encoder, and a forgery feature module extracting frequency domain traces and compression artifacts via Fourier transformation. While some existing approaches employ large scale generative vision language models for caption generation, their open ended generation tendency produces verbose and stylistically inconsistent descriptions that introduce semantic noise and risk drifting from actual image content. To overcome this, we adopt BLIP specifically pre trained for vision language alignment which generates concise, image faithful descriptions stylistically closer to news text, serving as a reliable semantic bridge across modalities. A BLIP Driven Semantic Alignment Module jointly optimizes text image and text description contrastive losses, capturing inconsistencies at both visual and semantic levels. A gated adaptive feature scaling fusion mechanism dynamically adjusts multimodal fusion and reduces redundancy. Experiments on Weibo and Twitter datasets demonstrate that our model outperforms mainstream baselines in,recall,and F1 score

Abstract

Explore similar work

Multimodal rumor detection enhanced by external evidence and forgery features

RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild

Sparse Evidence Can Suffice: Agentic Evidence Seeking for Multimodal Video Misinformation Detection