M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency

Abolfazl Ansari; Delvin Ce Zhang; Zhuoyang Zou; Wenpeng Yin; Dongwon Lee

cs.CLApr 1, 2026

M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency

Authors: Abolfazl Ansari, Delvin Ce Zhang, Zhuoyang Zou, Wenpeng Yin, Dongwon Lee

Organizations: The Pennsylvania State University, USA · University of Sheffield, UK

Abstract

Evaluating scientific arguments requires assessing the strict consistency between a claim and its underlying multimodal evidence. However, existing benchmarks lack the scale, domain diversity, and visual complexity needed to evaluate this alignment realistically. To address this gap, we introduce M2-Verify, a large-scale multimodal dataset for checking scientific claim consistency. Sourced from PubMed and arXiv, M2-Verify provides over 469K instances across 16 domains, rigorously validated through expert audits. Extensive baseline experiments show that state-of-the-art models struggle to maintain robust consistency. While top models achieve up to 85.8% Micro-F1 on low-complexity medical perturbations, performance drops to 61.6% on high-complexity challenges like anatomical shifts. Furthermore, expert evaluations expose hallucinations when models generate scientific explanations for their alignment decisions. Finally, we demonstrate our dataset's utility and provide comprehensive usage guidelines.

Figures & tables

Dataset	Domain	Source	Expl.	# Figs	# Dom	Size
Text-only Claim Verification (General Domain)
FEVER ( Thorne et al., 2018 )	General	Wikipedia	No	0	0	185K
PubHealth ( Kotonya and Toni, 2020 )	Public Health	Fact-check sites, News	Yes	0	0	11.8K
Text-only Claim Verification (Scientific Domain)
SciFact ( Wadden et al., 2020 )	Biomedical	Semantic Scholar	Yes	0	1	1.4K
Climate-FEVER ( Diggelmann et al., 2020 )	Climate Science	Wikipedia	Yes	0	1	1.5K

Table 1: Comparison of claim verification datasets across domains. Datasets are grouped by modality and domain. Green indicates scientific coverage or largest instances.

{\dagger}

MOCHEG contains heterogeneous visual evidence and does not define a fixed set of figure types.

Figure 1: Representative examples from M2-Verify -Med and M2-Verify -Gen exemplifying its multimodal diversity. Verifying gastric balloon location (left) and model architecture (right) requires joint reasoning across figures and captions.

Figure 2: Overview of the M2-Verify framework. The pipeline integrates automated claim extraction visual dependency filtering domain specific perturbations and grounded explanations validated by multi phase expert audit.

Type	Description	Example
1. Status Swap	Flip normal and abnormal states	No evidence of X $\rightarrow$ Evidence of X
2. Numeric Change	Alter numeric values plausibly	Tumor size 2 cm $\rightarrow$ 4 cm
3. Attribute Flip	Invert descriptive attributes	significant edema $\leftrightarrow$ minimal edema
4. Directional Flip	Reverse comparative relationships	$A>B$ $\leftrightarrow$ $B>A$
5. Diagnosis Swap	Substitute primary diagnosis	cancer $\leftrightarrow$ infection
6. Location Shift	Shift to adjacent anatomy	thalamus $\leftrightarrow$ basal ganglia

Table 2: Medical perturbation taxonomy with descriptions and examples.

Open-source Vision-Language Models
Model	M2-Verify -Med		M2-Verify -Gen
	Macro-F1	Micro-F1	Macro-F1	Micro-F1
Phi-4-Multimodal	65.8	72.3	45.7	47.4
Pixtral-12B	69.6	75.9	41.9	59.2
LLaMA-3.2-11B-Vision	69.3	70.8	53.8	53.8
Mistral-Small-3.1-24B	71.4	72.7	63.1	64.1

Table 3: Comparison of model performance (%) on the M2-Verify test sets. Dashes indicate models not evaluated on a subset. Best scores in bold , second best underlined .

Model	Low-Complexity		Intermediate-Complexity			High-Complexity		Avg.
	Certainty	Status	Diagnosis	Attribute	Numeric	Location	Directional
InternVL3-8B	50.0	60.0	37.4	35.3	37.7	20.5	21.4	37.5 $\pm$ 13.2
Phi-4-Multimodal	66.7	63.4	32.7	34.7	44.9	19.4	26.8	41.2 $\pm$ 16.7
Pixtral-12B	66.7	63.1	40.5	38.9	38.9	26.0	19.6	42.0 $\pm$ 16.2
LLaMA-3.2-11B-Vision	83.3	86.7	60.7	65.8	68.9	42.5	53.6	65.9 $\pm$ 14.5
Mistral-Small-3.1-24B	83.3	84.9	65.0	70.5	78.4	63.0	50.0	70.7 $\pm$ 11.6

Table 4: Micro-F1 (%) across medical perturbations on the M2-Verify -Med test set.

Model	M2-Verify -Med			M2-Verify -Gen
	ROUGE-L	BLEU-2	METEOR	ROUGE-L	BLEU-2	METEOR
Phi-4-MM-Instruct	31.05	30.60	35.70	27.49	24.56	32.10
Qwen2.5-VL-7B	31.76	32.48	39.11	27.48	21.24	27.72
Pixtral-12B	32.95	31.14	35.96	32.12	28.44	33.93
InternVL3-8B	31.64	32.62	39.39	29.60	24.66	36.92
GPT-4o-mini	32.60	30.81	36.11	27.92	24.68	30.23

Table 5: Explanation generation (%) on M2-Verify -Med and M2-Verify -Gen test sets.

	Model	Entailment	Relevance	Correctness	Completeness	Clarity
M2-Verify -Med	Qwen2.5-VL-7B-Instruct	7.76	9.50	7.19	7.48	8.73
	InternVL3-8B	8.19	9.53	7.70	7.66	8.87
	Phi-4-Multimodal-Instruct	8.10	9.33	7.39	7.30	8.65
	Pixtral-12B	8.51	9.38	7.83	7.32	8.86
	GPT-4o-mini	8.87	9.67	8.46	7.82	9.14
	GPT-5-mini	9.47	9.94	9.43	8.86	9.41

Table 6: LLM-as-a-Judge explanation quality on M2-Verify . Scores average entailment, relevance, correctness, completeness, and clarity on a 1-to-10 scale (

\uparrow

Model	cs	econ	eess	gr-qc	hep	math	physics	q-bio	q-fin	quant-ph	stat
Phi-4-Multimodal	45.2	45.0	45.4	45.4	46.0	44.8	46.5	46.5	43.0	47.1	44.5
Pixtral-12B	42.4	40.3	41.2	42.1	41.2	42.9	41.4	41.5	41.8	41.2	41.2
Mistral-Small-3.1-24B	51.7	44.3	53.0	53.4	53.0	50.8	54.0	53.7	46.1	53.6	52.3
LLaMA-3.2-11B-Vision	54.5	47.7	55.7	52.2	52.4	48.8	53.8	52.2	52.9	53.1	50.3
InternVL3-8B	67.2	65.6	65.5	61.4	59.3	61.8	58.1	63.3	70.3	60.9	60.8
Qwen2.5-VL-7B	70.3	70.4	70.4	67.7	65.0	65.4	65.9	71.0	72.9	66.7	65.6

Table 7: Domain-wise macro-F1 (%) performance on the M2-Verify -Gen test set.

Model	Setting	M2-Verify -Med		M2-Verify -Gen
		Macro-F1	Micro-F1	Macro-F1	Micro-F1
Qwen2.5-VL-7B	w/o image	68.1	71.0	52.2	53.2
	w/o text	64.0	65.4	50.4	50.7
	Multimodal	70.4	71.3	68.1	70.4
GPT-4o-mini	w/o image	56.4	56.5	43.6	46.6
	w/o text	50.9	51.1	56.4	64.8

Table 8: Ablation study of modality contributions. Results are in percentage.

Panel of Figure 3 — Figure 3: Radar plots comparing zero-shot baselines (blue) and SFT claim verification performance (red) across diverse scientific domains on the M2-Verify benchmark.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Short Form	Full Name
astro	Astrophysics
cond	Condensed Matter Physics
cs	Computer Science
econ	Economics
eess	Electrical Engineering and Systems Science
grqc	General Relativity and Quantum Cosmology

Appendix

Table 9: arXiv subject categories: short-form codes and full names.

Claim Verification

Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.

See all 59 papers

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning
Jul 17, 2026Binglin Zhou, Peng Shi, Ryo Kamoi +2Tool Use in VLMs Scientific Claim Verification
1The Pennsylvania State University · University of Waterloo
SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Grounding
Jun 18, 2026Yueming Wang, Tianshi Zheng, Jiaxin Bai +3Multimodal Grounding AI for Science
The Hong Kong University of Science and Technology Hong Kong SAR, China · NVIDIA AI Technology Center Santa Clara, USA
VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus
Aug 11, 2026Rohit Sinha, Kunal Tilaganji, Tanuja Ganu +3Multimodal Large Language Models Multimodal CoT Reasoning
Indian Institute of Technology, Hyderabad · Microsoft Research

Jul 17, 2026J/K move · Enter open · S save

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

Binglin Zhou, Peng Shi, Ryo Kamoi, Nan Zhang+1

1The Pennsylvania State University · University of Waterloo

Tool Use in VLMs Scientific Claim Verification Multimodal Tool Use

Open paper AskPDF arXiv

Multimodal Scientific Claim Verification (MSCV) requires models to verify scientific claims using visually grounded evidence from papers, including figures, tables, charts, and textual context. However, existing methods often fail because they struggle to locate decisive visual evidence, accurately read structured scientific visuals, and integrate multimodal observations into reliable reasoning. We introduce ToolSciVer, the first tool-augmented framework for MSCV to our knowledge. ToolSciVer equips a VLM with three type-aware visual tools, table row/column focus, chart-to-structure parsing, and high-resolution region zoom, which convert dense scientific visuals into explicit, claim-facing evidence, and trains the policy with Group Relative Policy Optimization (GRPO) under a composite reward of answer correctness, format validity, length control, tool-use efficiency, and tool-validity penalties. Experiments on SciVer and MuSciClaims datasets on five VLMs from three model families (Qwen, InternVL, Gemma) demonstrate that our method achieves superior performance compared to four competitive baselines including prompting-based and RL-based tool-use methods, highlighting the effectiveness of learned, type-aware tool use for scientific claim verification.

Abstract

Claim Verification

Explore similar work

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Grounding

VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus