Exposing Cross-Modal Consistency for Fake News Detection in Short-Form Videos

Chong Tian; Yu Wang; Chenxu Yang; Junyi Guan; Zheng Lin; Yuhan Liu; Xiuying Chen; Qirong Ho

cs.AIMar 16, 2026

Exposing Cross-Modal Consistency for Fake News Detection in Short-Form Videos

Authors: Chong Tian, Yu Wang, Chenxu Yang, Junyi Guan, Zheng Lin, Yuhan Liu, Xiuying Chen, Qirong Ho

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Institute of Information Engineering, Chinese Academy of Sciences (CAS)

Abstract

Short-form video platforms are major channels for news but also fertile ground for multimodal misinformation where each modality appears plausible alone yet cross-modal relationships are subtly inconsistent, like mismatched visuals and captions. On two benchmark datasets, FakeSV (Chinese) and FakeTT (English), we observe a clear asymmetry: real videos exhibit high text-visual but moderate text-audio consistency, while fake videos show the opposite pattern. Moreover, a single global consistency score forms an interpretable axis along which fake probability and prediction errors vary smoothly. Motivated by these observations, we present MAGIC3 (Modal-Adversarial Gated Interaction and Consistency-Centric Classifier), a detector that explicitly models and exposes cross-tri-modal consistency signals at multiple granularities. MAGIC3 combines explicit pairwise and global consistency modeling with token- and frame-level consistency signals derived from cross-modal attention, incorporates multi-style LLM rewrites to obtain style-robust text representations, and employs an uncertainty-aware classifier for selective VLM routing. Using pre-extracted features, MAGIC3 consistently outperforms the strongest non-VLM baselines on FakeSV and FakeTT. While matching VLM-level accuracy, the two-stage system achieves 18-27x higher throughput and 93% VRAM savings, offering a strong cost-performance tradeoff.

Figures & tables

Figure 1: Illustration of cross-modal consistency patterns. In real news short videos, text, visuals, and audio are contextually aligned (Consistent). In fake news, a “semantic gap” often exists between the sensational claims (text/audio) and the actual visual content. MAGIC 3 acts as a consistency lens to quantify these multimodal relationships for fake news detection.

Figure 2: MAGIC 3 Overview. Frozen encoders provide text, visual, audio, and rewrite features. The Cross-Modal Consistency Gate outputs pairwise and global consistency scores; Consistency Field Estimator converts cross-modal attention into token- and frame-level consistency fields; Temporal Cross-Modal Inconsistency computes a temporal inconsistency score. Adversarial Aware Rewrite Fusion fuses original text with LLM rewrites into a style-robust representation; Hierarchical Multimodal Transformer performs hierarchical multimodal fusion; and the classifier outputs fake probability and uncertainty. The framework is trained via Contrastive Adversarial Joint Learning.

Zero-shot VLMs
Method	FakeSV						FakeTT
	Acc	macF1	Real		Fake		Acc	macF1	Real		Fake
	Acc	macF1	Pre	Rec	Pre	Rec	Acc	macF1	Pre	Rec	Pre	Rec
GPT-4o-mini	68.08	68.05	65.35	73.97	74.41	64.93	61.54	61.20	81.62	52.00	47.28	79.80
GPT-4.1-mini	70.30	70.25	68.12	73.53	73.10	68.21	49.16	48.54	77.42	48.00	37.50	71.40
Qwen2.5-VL	64.21	60.79	62.50	60.54	66.60	62.50	45.82	45.31	72.10	45.00	41.28	65.84

Table 1: Main results on FakeSV and FakeTT. Best values in bold , second best underlined . MAGIC 3 +VLM routes 25% of samples to VLM (Figure 5 ) while achieving comparable or better accuracy, offering the best cost–performance tradeoff : VLM-level or higher accuracy at up to 18–27

\times

higher throughput than VLM-only.

Panel of Figure 3 — Figure 3: Cross-modal consistency distributions of text–visual, text–audio, visual–audio, and global consistency scores for real vs. fake videos. Real news shows high text–visual but moderate text–audio consistency; fake news flips this pattern, while visual–audio consistency remains high. Left: FakeSV; right: FakeTT.

FakeSV
	Variant	Acc	F1	AUC
	MAGIC 3 (full)	86.71	86.84	92.23
Consistency	w/o CMCG	84.90	84.85	91.77
Consistency	w/o CFE	85.69	85.61	92.30
	w/o TCMI	85.75	85.89	91.94
Fusion	w/o AARF	84.49	84.71	91.43

Table 2: Ablation study. Consistency modules (CMCG, CFE, TCMI) and AARF are critical.

Panel of Figure 4 — Figure 4: Consistency–prediction analysis. Global consistency is strongly (negatively) correlated with fake probability; errors concentrate in the middle-consistency band, while $c_{\mathrm{global}}$ vs. entropy is non-linear with uncertainty peaking at mid-range consistency. Left: FakeSV; right: FakeTT.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Description and typical patterns
Entity & context inconsistency	Entities (persons, organizations, places) or contexts (e.g., country, city, venue) mentioned in text do not match those present in video or audio. A typical pattern is subtitles claiming “thousands protest in Paris” while the video shows a small rally in a different city.
Event-level semantic inconsistency	Modalities disagree on the core event: what happened, at what scale, or with which participants. For example, the caption claims “violent clashes erupt” but the video shows a peaceful gathering, or the audio describes a different incident.
Affective inconsistency	Sentiment or emotion conveyed by audio contradicts text or visual content, such as tragic subtitles overlaid on upbeat background music or calm narration over highly dramatic visuals.
Temporal audio–visual inconsistency	Audio and visual streams are misaligned in time, reused from different footage, or re-ordered. This includes dubbed videos, re-edited footage, or mis-timed sound effects.
Stylistic / pragmatic inconsistency	Rhetorical style, framing, or level of sensationalism in one modality is inconsistent with the others, e.g., a sensational subtitle for a relatively mundane video, or highly emotional text with neutral audio.

Appendix

Table 3: Task-oriented taxonomy of inconsistencies in short-form news videos. These categories serve as design targets for MAGIC 3 ’s consistency-centric architecture.

Component	Function	Consistency level	Role in findings / interpretability
Consistency Computation
CMCG	Computes pairwise consistency scores $[c_{\mathrm{tv}},c_{\mathrm{ta}},c_{\mathrm{va}}]$ and global score $c_{\mathrm{global}}$	Pairwise $\rightarrow$ global	Reveals asymmetric text–visual vs. text–audio patterns; defines the global consistency axis (Figures 3 – 4 ).
CFE	Derives token/frame-level consistency fields from cross-modal attention	Local (token / frame)	Localizes suspicious phrases or frames; supports heatmap visualization (Figure 6 ).
TCMI	Computes temporal audio–visual inconsistency score $c_{\mathrm{temp}}$	Temporal	Detects coarse audio–visual synchronization mismatches common in repurposed videos.
Cross-Modal Fusion
AARF	Fuses original text with neutral/formal/sensational LLM rewrites via gated weights	Text-internal	Builds style-robust text embeddings; used to study style perturbation effects (Finding 4).

Appendix

Table 4: MAGIC 3 component overview. Components are grouped by function: consistency computation (CMCG, CFE, TCMI), cross-modal fusion (AARF, HMT), and training/prediction (CAJL, Classifier). Together they provide the signals used in our four findings and the two-stage VLM pipeline.

Panel of Figure 5 — Figure 5: Uncertainty and two-stage routing behaviour of MAGIC 3 . Top: entropy/confidence distributions and calibration plots. Bottom: split between direct predictions and routed samples, and accuracies for routed vs. non-routed subsets when using a VLM-based stage-2 detector. Left: FakeSV; right: FakeTT.

Panel of Figure 6 — Figure 6: CFE consistency fields. Colors indicate per-token consistency with the visual stream; sharp high-score regions suggest localized support, while diffuse/low scores indicate potential inconsistencies.

Panel of Figure 7 — Figure 7: Style sensitivity heatmaps (Finding 4). Rows: consistency types; columns: style rewrites vs. original. Fake samples show larger shifts than real ones.

Dataset	Score	Real	Fake	$p$ -value
FakeSV	$c_{\mathrm{tv}}$	$0.82\!\pm\!0.32$	$0.16\!\pm\!0.27$	$p\ll 0.001$
	$c_{\mathrm{ta}}$	$0.50\!\pm\!0.20$	$0.88\!\pm\!0.13$	$p\ll 0.001$
	$c_{\mathrm{va}}$	$0.89\!\pm\!0.02$	$0.89\!\pm\!0.02$	$p=0.82$
	$c_{\mathrm{global}}$	$0.47\!\pm\!0.04$	$0.38\!\pm\!0.04$	$p\ll 0.001$
FakeTT	$c_{\mathrm{tv}}$	$0.84\!\pm\!0.29$	$0.31\!\pm\!0.35$	$p\ll 0.001$
	$c_{\mathrm{ta}}$	$0.39\!\pm\!0.23$	$0.77\!\pm\!0.18$	$p\ll 0.001$

Appendix

Table 5: Finding 1 statistics: consistency scores by class (mean

\pm

std) and two-sample

t

-test

p

-values. Text–visual vs. text–audio patterns flip between real and fake videos on both datasets.

Dataset	Spear. $\rho$	Pear. $r$	$R^{2}$
FakeSV	$-0.91$ ( $p\ll 0.001$ )	$-0.99$ ( $p\ll 0.001$ )	$0.99$
FakeTT	$-0.96$ ( $p\ll 0.001$ )	$-0.96$ ( $p\ll 0.001$ )	$0.91$

Appendix

Table 6: Finding 2: correlation between

c_{\mathrm{global}}

and predicted fake probability.

$c_{\mathrm{global}}$ bin	FakeSV rate	FakeTT rate
Lowest (Q1)	0.93	0.83
Q2	0.89	0.55
Q3	0.76	0.17
Q4	0.19	0.03
Highest (Q5)	0.05	0.07

Appendix

Table 7: Finding 2: fake rates in five quantile bins of

c_{\mathrm{global}}

(lower scores = more inconsistent).

Dataset	Score	Real var	Fake var	$p$
FakeSV	$c_{\mathrm{tv}}$	0.0336	0.0815	$p<10^{-16}$
	$c_{\mathrm{ta}}$	0.0109	0.0148	$p<10^{-2}$
	$c_{\mathrm{va}}$	0.0	0.0	n.s.
	$c_{\mathrm{global}}$	$6.19{\times}10^{-4}$	$1.44{\times}10^{-3}$	$p<10^{-15}$
FakeTT	$c_{\mathrm{tv}}$	0.0194	0.0801	$p<10^{-21}$
	$c_{\mathrm{ta}}$	0.0081	0.0170	$p<10^{-7}$

Appendix

Table 8: Finding 4: variance of consistency scores under style rewrites (real vs. fake). “n.s.” denotes non-significant differences (

p\geq 0.05

Dataset	Routed Ratio	MAGIC 3 +VLM Acc	Routing Strategy
FakeSV	25.1%	90.93%	Entropy-based
FakeTT	25.1%	89.52%	Difficulty score

Appendix

Table 9: Finding 3: routing strategies and routed ratios used to construct the two-stage systems. Both strategies outperform the standalone VLM detector (FakeSV: +1.33pp, FakeTT: +0.62pp).

Statistic	FakeSV	FakeTT
Time range	2017/10–2022/02	2019/05–2024/03
Avg. duration (s)	39.88	47.69
# Fake videos	1,810	1,172
# Real videos	1,814	819
# Total	3,624	1,991

Appendix

Table 10: Statistics of FakeSV and FakeTT. All videos are short-form clips on real-world platforms.

Hyperparameter	Value
Learning rate	$5\times 10^{-5}$
Batch size	32
Epochs	50
Warmup ratio	0.1
Hidden dimension	256
Attention heads (HMT)	8

Appendix

Table 11: Key hyperparameter settings for MAGIC 3 .

Fake News Detection

Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.

See all 98 papers

RASR: Retrieval-Augmented Semantic Reasoning for Fake News Video Detection
Apr 8, 2026Hui Li, Peien Ding, Jun Li +5Multimodal RAG Fake News Detection
School of Informatics, Xiamen University, Xiamen, China · School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China · School of Computer and Big Data, Minjiang University, Fuzhou, China +2
Detecting and Explaining Fake News Short Videos with Multimodal Content and Real-World Evidence
Sep 11, 2026Yifeng Luo, Yupeng Li, Ming Tang +2Fake News Detection Automated Fact-Checking
Department of Interactive Media; Hong Kong Baptist University · AI and Social Good Lab, AI Media Centre; Hong Kong Baptist University · Department of Computer Science and Engineering, Southern University of Science and Technology +1
HyperClaim: Fine-Grained Cross-Modal Hypergraph Reasoning for Video Misinformation Detection
Jul 30, 2026Xiangbo Wang, Jiasheng Zhang, Xingtong Yu +2Fake News Detection Automated Fact-Checking
School of Communication Engineering, Hangzhou Dianzi University · School of Computer Science and Technology, Xidian University · School of Computing and Information Systems, Singapore Management University +1

Apr 8, 2026J/K move · Enter open · S save

RASR: Retrieval-Augmented Semantic Reasoning for Fake News Video Detection

Hui Li, Peien Ding, Jun Li, Guoqi Ma+4

School of Informatics, Xiamen University, Xiamen, China · School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China · School of Computer and Big Data, Minjiang University, Fuzhou, China+2

Multimodal RAG Fake News Detection Multimodal Reasoning Multimodal Fusion

Open paper AskPDF arXiv

Multimodal fake news video detection is a crucial research direction for maintaining the credibility of online information. Existing studies primarily verify content authenticity by constructing multimodal feature fusion representations or utilizing pre-trained language models to analyze video-text consistency. However, these methods still face the following limitations: (1) lacking cross-instance global semantic correlations, making it difficult to effectively utilize historical associative evidence to verify the current video; (2) semantic discrepancies across domains hinder the transfer of general knowledge, lacking the guidance of domain-specific expert knowledge. To this end, we propose a novel Retrieval-Augmented Semantic Reasoning (RASR) framework. First, a Cross-instance Semantic Parser and Retriever (CSPR) deconstructs the video into high-level semantic primitives and retrieves relevant associative evidence from a dynamic memory bank. Subsequently, a Domain-Guided Multimodal Reasoning (DGMP) module incorporates domain priors to drive an expert multimodal large language model in generating domain-aware, in-depth analysis reports. Finally, a Multi-View Feature Decoupling and Fusion (MVDFF) module integrates multi-dimensional features through an adaptive gating mechanism to achieve robust authenticity determination. Extensive experiments on the FakeSV and FakeTT datasets demonstrate that RASR significantly outperforms state-of-the-art baselines, achieves superior cross-domain generalization, and improves the overall detection accuracy by up to 0.93%.

Abstract

Fake News Detection

Explore similar work

RASR: Retrieval-Augmented Semantic Reasoning for Fake News Video Detection

Detecting and Explaining Fake News Short Videos with Multimodal Content and Real-World Evidence

HyperClaim: Fine-Grained Cross-Modal Hypergraph Reasoning for Video Misinformation Detection