Head-wise Modality Specialization within MLLMs for Robust Fake News Detection under Missing Modality

Kai Qian; Weijie Shi; Jiaqi Wang; Mengze Li; Hao Chen; Yue Cui; Hanghui Guo; Ziyi Liu

cs.CVApr 8, 2026

Head-wise Modality Specialization within MLLMs for Robust Fake News Detection under Missing Modality

Authors: Kai Qian, Weijie Shi, Jiaqi Wang, Mengze Li, Hao Chen, Yue Cui, Hanghui Guo, Ziyi Liu,

Organizations: Soochow University, China · Hong Kong University of Science and Technology, China · Computer Science and Engineering, The Chinese University of Hong Kong, China · The Hong Kong University of Science and Technology, China ·

Abstract

Multimodal fake news detection (MFND) aims to verify news credibility by jointly exploiting textual and visual evidence. However, real-world news dissemination frequently suffers from missing modality due to deleted images, corrupted screenshots, and similar issues. Thus, robust detection in this scenario requires preserving strong verification ability for each modality, which is challenging in MFND due to insufficient learning of the low-contribution modality and scarce unimodal annotations. To address this issue, we propose Head-wise Modality Specialization within Multimodal Large Language Models (MLLMs) for robust MFND under missing modality. Specifically, we first systematically study attention heads in MLLMs and their relationship with performance under missing modality, showing that modality-critical heads serve as key carriers of unimodal verification ability through their modality specialization. Based on this observation, to better preserve verification ability for the low-contribution modality, we introduce a head-wise specialization mechanism that explicitly allocates these heads to different modalities and preserves their specialization through lower-bound attention constraints. Furthermore, to better exploit scarce unimodal annotations, we propose a Unimodal Knowledge Retention strategy that prevents these heads from drifting away from the unimodal knowledge learned from limited supervision. Experiments show that our method improves robustness under missing modality while preserving performance with full multimodal input.

Figures & tables

Figure 2. Overview of the proposed framework. (a) Modality head identification. We first run the pretrained MLLM on multimodal training data, compute image and text attention shares for all heads, rank them by modality allocation, and obtain the selected image and text head assignments used in later training. (b) Stage-wise training loop. Each epoch takes the current MLLM together with the modality-head assignments and performs three stages, namely image-only training, text-only training, and multimodal training. In each stage, Head-wise Modality Specialization applies modality-specific lower-bound constraints to the selected heads, while Unimodal Knowledge Retention shrinks updates on mismatched modality-specific heads.

Setting	Ov.	Jac.	Mean(ov)	Med.(ov)
B $\rightarrow$ M (Img)	0.42	0.27	+0.260	+0.260
B $\rightarrow$ M (Txt)	0.40	0.25	+0.212	+0.177
B $\rightarrow$ I (Img)	0.30	0.18	+0.403	+0.449
B $\rightarrow$ T (Txt)	0.44	0.28	+0.186	+0.156
M $\rightarrow$ I (Img)	0.52	0.35	+0.131	+0.132
M $\rightarrow$ T (Txt)	0.54	0.37	+0.022	-0.004

Table 1. Summary statistics of head-level modality allocation under different training settings. All statistics are computed on the top-50 heads.

Figure 3. Head masking analysis on DGM4. For each modality, we progressively mask the top-

K

ranked heads identified by modality allocation and compare the resulting unimodal performance drop with masking the same number of randomly selected heads. Results are reported on a fixed 5k subset of the test set.

Panel of Figure 4 — Figure 4. Head-layer attention-share heatmaps comparing the pretrained model and multimodal finetuning. (a) and (b) show image attention share before and after multimodal finetuning, respectively; (c) and (d) show the corresponding text attention share. Compared with the pretrained model, multimodal finetuning activates substantially stronger modality-related head responses for both image and text.

Panel of Figure 5 — Figure 5. Head-layer attention-share heatmaps comparing unimodal and multimodal finetuning. (a) and (b) compare image-side specialization under image-only and multimodal finetuning, respectively; (c) and (d) compare text-side specialization under text-only and multimodal finetuning, respectively. In both modalities, unimodal finetuning shows more concentrated high-response heads, whereas multimodal finetuning exhibits a comparatively broader and more diffuse specialization pattern.

Panel of Figure 6 — Figure 6. Ranked attention-share distributions of top-ranked heads under different training settings. (a) Image modality; (b) text modality. Finetuning strengthens modality-related top-ranked heads relative to the pretrained model, while unimodal finetuning yields a more concentrated distribution than multimodal finetuning, indicating a less diffuse specialization pattern.

Dataset	Weibo	DGM4
Language	Chinese	English
Full test total	1,995	50,705
Full test (real / fake)	996 / 999	16,876 / 33,829
Image-only test (real / fake)	134 / 46	23,801 / 26,904
Text-only test (real / fake)	106 / 74	36,312 / 14,393

Table 2. Statistics of the datasets and evaluation settings.

Method	Weibo			DGM4 (0.25%)			DGM4 (1%)			DGM4 (5%)			Avg. DGM4
Method	multi	img	text	multi	img	text	multi	img	text	multi	img	text	multi	img	text
MTS	0.8951	0.4268	0.8543	0.6652	0.6585	0.6635	0.6594	0.6868	0.6272	0.6605	0.6900	0.6500	0.6617	0.6784	0.6469
COOLANT	0.8992	0.2035	0.8755	0.5530	0.5450	0.4393	0.5413	0.5212	0.3401	0.5406	0.5797	0.5197	0.5449	0.5486	0.4330
MoMKE	0.8471	0.5592	0.8374	0.5777	0.5329	0.5892	0.5792	0.5424	0.5818	0.5830	0.5649	0.5883	0.5800	0.5467	0.5864
Qwen + SFT	0.9429	0.3187	0.8924	0.7925	0.7762	0.6533	0.7947	0.7823	0.6588	0.7919	0.7772	0.6669	0.7930	0.7786	0.6597
Ours	0.9353	0.6063	0.8820	0.7966	0.7860	0.6529	0.7914	0.7868	0.6613	0.7934	0.7874	0.6682	0.7938	0.7867	0.6608

Table 3. Main results in terms of macro F1 under full and missing modality inference. For Weibo, the scarce unimodal supervision budget is 100 samples. For DGM4, we report results under three supervision budgets together with their average. The best result in each column is marked in bold, and the second best is underlined.

Figure 7. Sensitivity analysis on DGM4. (a) Effect of

\tau

. (b) Effect of

\gamma

with

\tau=0.2

. Results are averaged over the three unimodal supervision budgets.

Model Variant	multi	img	text
Ours	0.7938	0.7867	0.6608
w/o critical heads	0.7560	0.7558	0.6388
w/o HMS	0.7934	0.7833	0.6430
w/o UKR	0.7891	0.7880	0.6514
w/o HMS & UKR	0.7930	0.7786	0.6597

Table 4. Ablation results on DGM4 in terms of macro F1, averaged over the three unimodal supervision budgets.

Fake News Detection

Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.

See all 93 papers

LLM-MRD: LLM-Guided Multi-View Reasoning Distillation for Fake News Detection
Mar 10, 2026Weilin Zhou, Shanwen Tan, Enhao Gu +1Fake News Detection Reasoning Distillation
Xinjiang University, Urumqi, China · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University, Urumqi, China · Sichuan University, Chengdu, China
Retrieval-Augmented Multimodal Model for Fake News Detection
Apr 20, 2026Yiheng Li, Weihai Lu, Hanyi Yu +1Fake News Detection Text Classification
University of International Business and Economics · Beijing, China · Peking University +4
MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection
Aug 2, 2026Lehan Zhang, Yinlei Cheng, Shiqi Hu Yiheng Zhou +2Social Media Analysis Fake News Detection
Beijing Institute of Fashion Technology, Beijing, China · Shenyang University of Technology, Shenyang, China

Mar 10, 2026J/K move · Enter open · S save

LLM-MRD: LLM-Guided Multi-View Reasoning Distillation for Fake News Detection

Weilin Zhou, Shanwen Tan, Enhao Gu, Yurong Qian

Xinjiang University, Urumqi, China · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University, Urumqi, China · Sichuan University, Chengdu, China

Fake News Detection Reasoning Distillation Efficient Multimodal Inference Cross-Modal Knowledge Distillation

Open paper AskPDF arXiv

Multimodal fake news detection is crucial for mitigating societal disinformation. Existing approaches attempt to address this by fusing multimodal features or leveraging Large Language Models (LLMs) for advanced reasoning. However, these methods suffer from serious limitations, including a lack of comprehensive multi-view judgment and fusion, and prohibitive reasoning inefficiency due to the high computational costs of LLMs. To address these issues, we propose LLM-Guided Multi-View Reasoning Distillation for Fake News Detection ( LLM-MRD), a novel teacher-student framework. The Student Multi-view Reasoning module first constructs a comprehensive foundation from textual, visual, and cross-modal perspectives. Then, the Teacher Multi-view Reasoning module generates deep reasoning chains as rich supervision signals. Our core Calibration Distillation mechanism efficiently distills this complex reasoning-derived knowledge into the efficient student model. Experiments show LLM-MRD significantly outperforms state-of-the-art baselines. Notably, it demonstrates a comprehensive average improvement of 5.19% in ACC and 6.33% in F1-Fake when evaluated across all competing methods and datasets. Our code is available at https://github.com/Nasuro55/LLM-MRD

Abstract

Fake News Detection

Explore similar work

LLM-MRD: LLM-Guided Multi-View Reasoning Distillation for Fake News Detection

Retrieval-Augmented Multimodal Model for Fake News Detection

MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection