Organizations: Soochow University, China · Hong Kong University of Science and Technology, China · Computer Science and Engineering, The Chinese University of Hong Kong, China · The Hong Kong University of Science and Technology, China · FiT, Tencent, China · Alibaba Group, China · School of Computer Science and Engineering, Southeast University, China · School of Education, Zhejiang Normal University, China
Multimodal fake news detection (MFND) aims to verify news credibility by jointly exploiting textual and visual evidence. However, real-world news dissemination frequently suffers from missing modality due to deleted images, corrupted screenshots, and similar issues. Thus, robust detection in this scenario requires preserving strong verification ability for each modality, which is challenging in MFND due to insufficient learning of the low-contribution modality and scarce unimodal annotations. To address this issue, we propose Head-wise Modality Specialization within Multimodal Large Language Models (MLLMs) for robust MFND under missing modality. Specifically, we first systematically study attention heads in MLLMs and their relationship with performance under missing modality, showing that modality-critical heads serve as key carriers of unimodal verification ability through their modality specialization. Based on this observation, to better preserve verification ability for the low-contribution modality, we introduce a head-wise specialization mechanism that explicitly allocates these heads to different modalities and preserves their specialization through lower-bound attention constraints. Furthermore, to better exploit scarce unimodal annotations, we propose a Unimodal Knowledge Retention strategy that prevents these heads from drifting away from the unimodal knowledge learned from limited supervision. Experiments show that our method improves robustness under missing modality while preserving performance with full multimodal input.
Figures & tables
Figure 1. Illustration of unequal modality contribution and veracity change under missing modality.
Figure 2. Overview of the proposed framework. (a) Modality head identification. We first run the pretrained MLLM on multimodal training data, compute image and text attention shares for all heads, rank them by modality allocation, and obtain the selected image and text head assignments used in later training. (b) Stage-wise training loop. Each epoch takes the current MLLM together with the modality-head assignments and performs three stages, namely image-only training, text-only training, and multimodal training. In each stage, Head-wise Modality Specialization applies modality-specific lower-bound constraints to the selected heads, while Unimodal Knowledge Retention shrinks updates on mismatched modality-specific heads.
Setting
Ov.
Jac.
Mean(ov)
Med.(ov)
B → M (Img)
0.42
0.27
+0.260
+0.260
B → M (Txt)
0.40
0.25
+0.212
+0.177
B → I (Img)
0.30
0.18
+0.403
+0.449
B → T (Txt)
0.44
0.28
+0.186
+0.156
M → I (Img)
0.52
0.35
+0.131
+0.132
M → T (Txt)
0.54
0.37
+0.022
-0.004
Table 1. Summary statistics of head-level modality allocation under different training settings. All statistics are computed on the top-50 heads.
Figure 3. Head masking analysis on DGM4. For each modality, we progressively mask the top- K ranked heads identified by modality allocation and compare the resulting unimodal performance drop with masking the same number of randomly selected heads. Results are reported on a fixed 5k subset of the test set.
Figure 4. Head-layer attention-share heatmaps comparing the pretrained model and multimodal finetuning. (a) and (b) show image attention share before and after multimodal finetuning, respectively; (c) and (d) show the corresponding text attention share. Compared with the pretrained model, multimodal finetuning activates substantially stronger modality-related head responses for both image and text.
Figure 5. Head-layer attention-share heatmaps comparing unimodal and multimodal finetuning. (a) and (b) compare image-side specialization under image-only and multimodal finetuning, respectively; (c) and (d) compare text-side specialization under text-only and multimodal finetuning, respectively. In both modalities, unimodal finetuning shows more concentrated high-response heads, whereas multimodal finetuning exhibits a comparatively broader and more diffuse specialization pattern.
Figure 6. Ranked attention-share distributions of top-ranked heads under different training settings. (a) Image modality; (b) text modality. Finetuning strengthens modality-related top-ranked heads relative to the pretrained model, while unimodal finetuning yields a more concentrated distribution than multimodal finetuning, indicating a less diffuse specialization pattern.
Dataset
Weibo
DGM4
Language
Chinese
English
Full test total
1,995
50,705
Full test (real / fake)
996 / 999
16,876 / 33,829
Image-only test (real / fake)
134 / 46
23,801 / 26,904
Text-only test (real / fake)
106 / 74
36,312 / 14,393
Table 2. Statistics of the datasets and evaluation settings.
Method
Weibo
DGM4 (0.25%)
DGM4 (1%)
DGM4 (5%)
Avg. DGM4
multi
img
text
multi
img
text
multi
img
text
multi
img
text
multi
img
text
MTS
0.8951
0.4268
0.8543
0.6652
0.6585
0.6635
0.6594
0.6868
0.6272
0.6605
0.6900
0.6500
0.6617
0.6784
0.6469
COOLANT
0.8992
0.2035
0.8755
0.5530
0.5450
0.4393
0.5413
0.5212
0.3401
0.5406
0.5797
0.5197
0.5449
0.5486
0.4330
MoMKE
0.8471
0.5592
0.8374
0.5777
0.5329
0.5892
0.5792
0.5424
0.5818
0.5830
0.5649
0.5883
0.5800
0.5467
0.5864
Qwen + SFT
0.9429
0.3187
0.8924
0.7925
0.7762
0.6533
0.7947
0.7823
0.6588
0.7919
0.7772
0.6669
0.7930
0.7786
0.6597
Ours
0.9353
0.6063
0.8820
0.7966
0.7860
0.6529
0.7914
0.7868
0.6613
0.7934
0.7874
0.6682
0.7938
0.7867
0.6608
Table 3. Main results in terms of macro F1 under full and missing modality inference. For Weibo, the scarce unimodal supervision budget is 100 samples. For DGM4, we report results under three supervision budgets together with their average. The best result in each column is marked in bold, and the second best is underlined.
Figure 7. Sensitivity analysis on DGM4. (a) Effect of τ . (b) Effect of γ with τ=0.2 . Results are averaged over the three unimodal supervision budgets.
Model Variant
multi
img
text
Ours
0.7938
0.7867
0.6608
w/o critical heads
0.7560
0.7558
0.6388
w/o HMS
0.7934
0.7833
0.6430
w/o UKR
0.7891
0.7880
0.6514
w/o HMS & UKR
0.7930
0.7786
0.6597
Table 4. Ablation results on DGM4 in terms of macro F1, averaged over the three unimodal supervision budgets.
Multimodal fake news detection is crucial for mitigating societal disinformation. Existing approaches attempt to address this by fusing multimodal features or leveraging Large Language Models (LLMs) for advanced reasoning. However, these methods suffer from serious limitations, including a lack of comprehensive multi-view judgment and fusion, and prohibitive reasoning inefficiency due to the high computational costs of LLMs. To address these issues, we propose LLM-Guided Multi-View Reasoning Distillation for Fake News Detection ( LLM-MRD), a novel teacher-student framework. The Student Multi-view Reasoning module first constructs a comprehensive foundation from textual, visual, and cross-modal perspectives. Then, the Teacher Multi-view Reasoning module generates deep reasoning chains as rich supervision signals. Our core Calibration Distillation mechanism efficiently distills this complex reasoning-derived knowledge into the efficient student model. Experiments show LLM-MRD significantly outperforms state-of-the-art baselines. Notably, it demonstrates a comprehensive average improvement of 5.19% in ACC and 6.33% in F1-Fake when evaluated across all competing methods and datasets. Our code is available at https://github.com/Nasuro55/LLM-MRD
Weilin Zhou, Shanwen Tan, Enhao Gu +1
Xinjiang University, Urumqi, China · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University, Urumqi, China · Sichuan University, Chengdu, China
In recent years, multimodal multidomain fake news detection has garnered increasing attention. Nevertheless, this direction presents two significant challenges: (1) Failure to Capture Cross-Instance Narrative Consistency: existing models usually evaluate each news in isolation, fail to capture cross-instance narrative consistency, and thus struggle to address the spread of cluster based fake news driven by social media; (2) Lack of Domain Specific Knowledge for Reasoning: conventional models, which rely solely on knowledge encoded in their parameters during training, struggle to generalize to new or data-scarce domains (e.g., emerging events or niche topics). To tackle these challenges, we introduce Retrieval-Augmented Multimodal Model for Fake News Detection (RAMM). First, RAMM employs a Multimodal Large Language Model (MLLM) as its backbone to capture cross-modal semantic information from news samples. Second, RAMM incorporates an Abstract Narrative Alignment Module. This component adaptively extracts abstract narrative consistency from diverse instances across distinct domains, aggregates relevant knowledge, and thereby enables the modeling of high-level narrative information. Finally, RAMM introduces a Semantic Representation Alignment Module, which aligns the model's decision-making paradigm with that of humans - specifically, it shifts the model's reasoning process from direct inference on multimodal features to an instance-based analogical reasoning process. Extensive experimental results on three public datasets validate the efficacy of our proposed approach. Our code is available at the following link: https://github.com/li-yiheng/RAMM
Yiheng Li, Weihai Lu, Hanyi Yu +1
University of International Business and Economics · Beijing, China · Peking University +4
The rapid dissemination of multimodal content has intensified the spread of fabricated news, presenting a substantial threat to social integrity. A formidable challenge for current detection systems is identifying misinformation related to novel events in zero-shot scenarios. Prevailing zero-shot methods typically assess news items in isolation via semantic matching, a strategy that fails to recognize the recycled disinformation tactics from past campaigns and lacks the sophisticated reasoning needed to identify subtle, cross-modal discrepancies. To surmount these deficiencies, we introduce MRAFnd, a novel Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection. MRAFnd emulates a collaborative team of analysts to verify news veracity. The framework initiates with Multimodal Similarity-based News Retrieval to assemble a corpus of contextually analogous articles from an unlabeled reference database. Subsequently, during the Bifurcated Evidential Reasoning stage, agents perform a dual-directional analysis to extract critical patterns from the retrieved evidence. Finally, a Multi-Agent Collaborative Debate, involving Analyst and Arbiter agents, engages in a structured discourse to arrive at a definitive and robust conclusion. Comprehensive experiments on three benchmark datasets reveal that MRAFnd markedly surpasses state-of-the-art baselines, achieving an accuracy gain of up to 2.35% on the demanding Weibo-21 dataset.
Lehan Zhang, Yinlei Cheng, Shiqi Hu Yiheng Zhou +2
Beijing Institute of Fashion Technology, Beijing, China · Shenyang University of Technology, Shenyang, China