Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity

Jaeyoon Jung; Yejun Yoon; Kunwoo Park

cs.CLApr 6, 2026

Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity

Authors: Jaeyoon Jung, Yejun Yoon, Kunwoo Park

Organizations: School of AI Convergence, Soongsil University · MAUM AI Inc. · Department of Intelligent Semiconductors, Soongsil University

Abstract

Automated fact-checking is a crucial task that supports a responsible information ecosystem. While recent research has progressed from text-only to multimodal fact-checking, a prevailing assumption is that incorporating visual evidence universally improves verification accuracy. In this work, we challenge this assumption and show that the indiscriminate use of visual evidence can reduce accuracy. Building on this finding, we propose AMuFC, a modular fact-checking framework that employs two collaborative vision-language models with distinct roles to enable the adaptive use of visual evidence. Experimental results on three datasets, including WebFC, introduced in this study, demonstrate the effectiveness of adaptive visual evidence use in fact-checking.

Figures & tables

Figure 1: An example illustrating that visual evidence is not always necessary for multimodal fact-checking, motivating the research direction of this study.

Dataset	Primary Inputs	Evidence	Task	Label Source	Claim Source	External Knowledge Source	Access
Fakeddit Nakamura et al. (2020)	Img/Txt	-	Mixed	Synthetic	Reddit	✗	✓
NeuralNews Tan et al. (2020)	Img/Txt	-	OOC	Mixed	Grover/GoodNews	✗	✓
NewsCLIPings Luo et al. (2021)	Img/Txt	-	OOC	Mixed	CLIP/VisualNews	✗	✓
Factify Suryavardan et al. (2023)	Img/Txt	Img/Txt	OOC	Mixed	Twitter	✗	✓
VERITE Papadopoulos et al. (2024)	Img/Txt	-	Mixed	Mixed	Snopes/Reuters	✗	✓
Fauxtography Zlatkova et al. (2019)	Img/Txt	Meta	FV	Expert	Snopes/Reuters	✗	✓

Table 1: Datasets for multimodal fact-checking (OOC: out-of-context misinformation; FV: fact verification).

Model	Configuration	ACC	F1
Qwen2-VL	(1) Text	0.509	0.419
	(2) $+$ Image (G)	0.506	0.410
	(3) $+$ Image (R)	0.490	0.395
	(4) Oracle	0.547	0.451
Llama-3.2-V	(1) Text	0.496	0.495
	(2) $+$ Image (G)	0.462	0.445

Table 2: Fact-verification performance across different evidence configuration strategies, indicating that visual evidence is not always necessary. Values in parentheses denote standard errors, which are omitted for open-weight models due to deterministic results (G: Gold, R: Retrieved).

Type	Unnecessary	Necessary
Claim	Marquette University threatened to rescind student’s admission over pro-Trump TikTok video.	In an episode of ’The Simpsons,’ Mayor Quimby says he is canceling a trip to the Bahamas while he’s in the Bahamas, because of an ongoing epidemic.
Visual Evidence
Textual Evidence	• The pro-Trump post was not at issue. Marquette and the people complaining to the university were examining comments on topics like sexuality and immigration in other social media posts. • The student clearly felt her status was in doubt, but she has stopped short of calling it a threat. The school says no threat was made. • The “threat” element is tricky to pin down, since that claim stems from a private conversation between the student and Marquette admissions staffers.	After initially saying he didn’t do anything wrong, Steve Adler, the mayor of Austin, Texas, says he now realizes he ’set a bad example’ by traveling to Cabo San Lucas, Mexico, for vacation last month. An Austin American-Statesman story revealed Wednesday that Adler attended an in-person wedding for his daughter in early November and then flew with others to Cabo for a weeklong vacation. At the same time, Adler was encouraging people to stay home to avoid contracting or spreading COVID-19.

Table 3: Examples for visual evidence types according to their necessity for claim verification.

Figure 2: Distribution of visual evidence types across claim categories, illustrating the association between visual evidence necessity and verification outcomes.

Figure 3: Overall pipeline of AMuFC . Given the retrieved evidence, the two VLMs—Analyzer and Verifier—are responsible for assessing the necessity of visual evidence and predicting the claim’s veracity, respectively.

Figure 4: Prompts used in AMuFC .

Method	Gold		Retrieved
Method	ACC	F1	ACC	F1
AMuFC	0.612	0.600	0.546	0.540
MOCHEG	0.520	0.500	0.456	0.438
LVLM4FV	0.534	0.535	0.451	0.450
HGTMFC	0.541	0.520	0.486	0.468
MetaSumPerceiver	0.556	0.482	0.486	-

Table 4: Verification performance of AMuFC and baseline methods on the MOCHEG test set, indicating the effectiveness of the proposed framework.

Method	Gold		Retrieved
Method	ACC	F1	ACC	F1
AMuFC	0.612	0.600	0.546	0.540
Label-only	0.599	0.581	0.517	0.492
Pre-filtering (Analyzer)	0.574	0.551	0.492	0.460
Pre-filtering (CLIP)	0.569	0.546	0.495	0.463
w/o Analyzer	0.563	0.537	0.477	0.435

Table 5: Ablation results for different Analyzer–Verifier integration strategies, demonstrating the effectiveness of incorporating the Analyzer’s natural-language assessment used in AMuFC .

Method	Gold		Retrieved
Method	ACC	F1	ACC	F1
AMuFC	0.612	0.600	0.546	0.540
Unified Verifier	0.577	0.555	0.531	0.513
Verifier with CoT	0.553	0.511	0.493	0.459

Table 6: Comparison with alternative verifiers, demonstrating the benefit of separating evidence assessment from verdict prediction.

Figure 5: Distribution of confusion patterns for AMuFC compared with the Verifier-only baseline.

Method	FIN-FACT		WebFC
Method	ACC	F1	ACC	F1
AMuFC	0.491	0.461	0.316	0.241
Verifier-only	0.469	0.384	0.309	0.179

Table 7: Test-only evaluation on two datasets, demonstrating the effectiveness of AMuFC across diverse fact-checking scenarios.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Claim ID 28	Claim ID 35
Claim	Google PhoneBook	Barack Obama’s Net Worth
Refined Claim	Entering a phone number into the Google search engine can produce a home address and a map with directions to that address.	Barack Obama’s net worth increased over $10 million from 2008 to 2012.

Appendix

Table A1: Examples of refinement results for incomplete claims from FIN-FACT.

Verifier	Analyzer	Gold		Retrieved
Verifier	Analyzer	ACC	F1	ACC	F1
Qwen2-VL	None	0.563	0.537	0.477	0.435
	Qwen2-VL	0.610	0.598	0.547	0.543
	Llama-3.2-V	0.612	0.600	0.546	0.540
	GPT-4o	0.631	0.620	0.538	0.537
	Gemini-2.5-Pro	0.612	0.606	0.478	0.470

Appendix

Table A2: Comparison of verification performance across different VLM choices for the Analyzer and Verifier.

Method		Verifier-only		AMuFC
Text	Image	Acc.	F1	Acc.	F1
SBERT	CLIP	0.475	0.434	0.530	0.527
Mxbai	SigLIP	0.502	0.476	0.521	0.523
SBERT(FT)	CLIP(FT)	0.477	0.435	0.546	0.540

Appendix

Table A3: Verification performance across different text and image retriever configurations. FT denotes fine-tuned models.

Method	Accuracy	F1	Inference time
AMuFC	0.612	0.600	1.531
Analyzer	-	-	1.420
Verifier	-	-	0.111
Baseline	0.563	0.537	0.112

Appendix

Table A4: Accuracy-efficiency trade-offs, with inference time averaged per sample (seconds).

Method	Gold		Retrieved
Method	ACC	F1	ACC	F1
AMuFC	$0.613\pm 0.005$	$0.600\pm 0.007$	$0.541\pm 0.006$	$0.536\pm 0.003$
Verifier-only	$0.553\pm 0.006$	$0.523\pm 0.008$	$0.476\pm 0.002$	$0.426\pm 0.006$

Appendix

Table A5: MOCHEG verification performance across three fine-tuning seeds (mean

\pm

standard error).

Method	ACC	F1
AMuFC	0.662	0.338
DEFAME	0.478	0.307

Appendix

Table A6: Comparison with DEFAME on WebFC using Llama-4-Scout as the Verifier backbone.

Model	ACC	Macro F1
Fine-tuned Analyzer	0.768	0.721
GPT-4o	0.777	0.738
Majority class	0.714	0.417

Appendix

Table A7: Visual-necessity classification performance on 112 manually annotated validation examples.

Method	Separate Analyzer	Necessity Information	Generated Tokens (Analyzer)	Generated Tokens (Verifier)	ACC	F1
AMuFC	Yes	Assessment	85.9	4.6	0.612	0.600
Label-only	Yes	Label	12.1	4.6	0.599	0.581
Pre-filtering (Analyzer)	Yes	Filter	12.1	4.6	0.574	0.551
w/o Analyzer	No	None	–	4.6	0.563	0.537
Unified Verifier	No	Joint assessment	–	110.3	0.577	0.555
Verifier with CoT	No	None	–	96.6	0.553	0.511

Appendix

Table A8: Average generation length and verification performance across ablation configurations.

Type	Successful	Failed
Claim	A Boeing B-17E bomber from World War II was found in the jungle with coffee still in thermoses.	Students no longer say the Pledge of Allegiance in schools.
Visual Evidence
Textual Evidence	• They found the rear door of the airplane, two aluminum receptacles and a third which contained coffee residue. • Both the impactor and the condensate were stored and transported to the laboratory at 2-8 °C. • Print LONG BEACH, Calif. - A B-17 bomber that lay in a New Guinea swamp for decades after being forced down during a World War II combat mission has been returned to the United States after years of salvage efforts. • As food stores were running low at the airplane, Zerbino and Fito went searching for cadavers and although they found one, they were too tired to bring it back to the fuselage. • Ignited by fire, ammunition was exploding inside the cabin.	• Students still say the Pledge of Allegiance in classrooms, though they don’t have to. • Virtual learning has presented new challenge for how schools say the Pledge of Allegiance.
Analysis	The image evidence is necessary for verifying the claim. While the text evidence provides detailed information about the discovery of the Boeing B-17E bomber in the jungle, including the presence of coffee residue and thermoses, the image directly shows the plane in its jungle location, confirming the physical context of the claim. The visual evidence complements the text by providing a tangible representation of the scene described, which is essential for verifying the authenticity of the claim.	The image evidence is not necessary for verifying the claim. The text evidence already provides sufficient information to address the claim that students no longer say the Pledge of Allegiance in schools. It clarifies that students still say the Pledge of Allegiance in classrooms, though it is not mandatory, and that it is recited at every school. The image of students raising their hands does not add any new or essential information to verify or refute the claim.
Prediction	Ground truth: Supported Baseline: Refuted ✗ AMuFC : Supported ✓	Ground truth: Refuted Baseline: Refuted ✓ AMuFC : NEI ✗

Appendix

Table A9: Examples for successful and failed predictions of AMuFC .

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 139 papers

RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild
May 11, 2026Danni Xu, Shaojing Fan, Harry Cheng +1VLM Evaluation Multimodal Grounding
School of Computing (SoC), National University of Singapore (NUS), Singapore · Department of Electrical and Computer Engineering (ECE), National University of Singapore (NUS), Singapore
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Oct 27, 2025Jiahui Geng, Jonathan Tonglet, Iryna GurevychAutomated Fact-Checking Multimodal Understanding
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Electrical Engineering, KU Leuven, Belgium +2
Checking Fact with Better Retrieval: Dynamic Contrastive Learning for Evidence Retrieval
May 24, 2026Zhongtian Hua, Yi Luo, Meijia Yu +1Automated Fact-Checking Cross-Modal Retrieval
Zhengzhou University, Zhengzhou, Henan, China · Henan University of Science and Technology, Luoyang, Henan, China

May 11, 2026J/K move · Enter open · S save

RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild

Danni Xu, Shaojing Fan, Harry Cheng, Mohan Kankanhalli

School of Computing (SoC), National University of Singapore (NUS), Singapore · Department of Electrical and Computer Engineering (ECE), National University of Singapore (NUS), Singapore

VLM Evaluation Multimodal Grounding Automated Fact-Checking

Open paper AskPDF arXiv

Multimodal misinformation increasingly leverages visual persuasion, where repurposed or manipulated images strengthen misleading text. We introduce RW-Post, a post-aligned text--image benchmark for real-world multimodal fact-checking with auditable annotations: each instance links the original social-media post with reasoning traces and explicitly linked evidence items derived from human fact-check articles via an LLM-assisted extraction-and-auditing pipeline. RW-Post supports controlled evaluation across closed-book, evidence-bounded, and open-web regimes, enabling systematic diagnosis of visual grounding and evidence utilization. We provide AgentFact as a reference verification baseline and benchmark strong open-source LVLMs under unified protocols. Experiments show substantial headroom: current models struggle with faithful evidence grounding, while evidence-bounded evaluation improves both accuracy and faithfulness. Code and dataset will be released at https://github.com/xudanni0927/AgentFact.

Abstract

Automated Fact-Checking

Explore similar work

RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild

M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset

Checking Fact with Better Retrieval: Dynamic Contrastive Learning for Evidence Retrieval