Can Multimodal Large Language Models Generate and Detect Multimodal Social Media Fake News?

Jiyao Yang; Yang Liu; Zhenyue Qin; Qingyu Chen; Xiuzhen Zhang

cs.CLSep 20, 2026

Can Multimodal Large Language Models Generate and Detect Multimodal Social Media Fake News?

Authors: Jiyao Yang, Yang Liu, Zhenyue Qin, Qingyu Chen, Xiuzhen Zhang

Organizations: Independent Researcher · Carnegie Mellon University · RMIT University · Yale University

Abstract

The rapid advancement of generative AI raises concerns about the misuse of Multimodal LLMs (MLLMs) for large-scale disinformation campaigns on social media. Despite existing research on textual disinformation, a fundamental question remains unanswered: can MLLMs be exploited to fabricate realistic multimodal fake news, and can they reliably detect it? We introduce a multi-agent framework in which a story agent, an image agent, and a critic agent collaborate to produce fake social media posts that plausibly counter true news. We apply the framework to generate over 9,000 paired multimodal news posts across science, health, and entertainment domains, and benchmark 16 open- and closed-source MLLMs for automated detection. We find that most models fall substantially short of human-level accuracy and fail critically on identifying image authenticity. Our research provides a foundation for developing robust defenses against social media fake news. Code and data are available at https: //github.com/xiuzhenzhang/Multimodal.

Figures & tables

Panel of Figure 1 — Figure 1: True and fake news posts about the event “Rosetta landing on Comet 67P”. The generated fake news preserves the event frame but reverses the successful “deliberate soft crash” narrative into a misleading failure story involving a navigation malfunction.

Figure 2: An agent framework for generating multimodal social media fake news

Figure 3: Text statistics of original articles, true-news posts and fake-news posts.

		NIH			Nature			Snopes			Average
Model	Mode	F1	Acc	AUC	F1	Acc	AUC	F1	Acc	AUC	F1	Acc	AUC
Open-source models
InternVL2.5-8B Chen et al. (2023)	Direct	45.53	52.40	53.49	50.54	53.64	53.23	44.56	44.98	45.05	46.87	50.34	50.59
InternVL2.5-8B Chen et al. (2023)	CoT	45.76	46.66	47.08	56.74	57.02	56.90	55.18	55.86	55.97	52.56	53.18	53.32
InternVL3-8B Chen et al. (2023)	Direct	68.54	70.69	71.52	76.92	77.66	77.38	51.28	54.12	54.31	65.58	67.49	67.74
InternVL3-8B Chen et al. (2023)	CoT	59.43	59.45	59.61	65.30	65.53	65.67	53.95	58.18	58.37	59.56	61.05	61.22

Table 1: Performance of LLMs for fake news detection grouped by source. The best results per group are highlighted in bold with a light red background , and the second-best results are highlighted with a light yellow background. Results (%) report macro F1-score (F1), Accuracy (Acc), and Area Under the Curve (AUC).

Model	True News					Fake News					Overall CoT Agreement					Final Accuracy
Model	Fact.	Lang.	Rel.	Auth.	Avg.	Fact.	Lang.	Rel.	Auth.	Avg.	Fact.	Lang.	Rel.	Auth.	Avg.	True	Fake	All
GPT-5.5 CoT	100.0	100.0	94.1	85.3	94.9	65.0	57.1	100.0	97.2	79.8	87.0	84.2	97.1	91.4	89.9	100.0	100.0	100.0
GPT-4o CoT	88.6	97.3	91.4	82.9	90.0	66.7	66.7	100.0	5.0	59.6	79.7	85.2	95.9	41.3	75.5	83.8	69.8	76.2
Qwen3-VL-8B CoT	76.5	94.4	73.5	82.4	81.7	58.3	54.2	74.4	45.0	58.0	69.0	78.3	74.0	62.2	70.9	55.6	90.7	74.7
Qwen2-VL-7B CoT	97.1	100.0	91.4	91.4	95.0	41.7	54.2	100.0	5.0	50.2	74.6	82.0	95.9	45.3	74.5	91.9	16.3	51.2

Table 2: CoT agreement and final prediction accuracy on the human-annotated subset. Human-annotation majority vote is used as the ground truth for the four verification dimensions: Fact. factual accuracy, Lang. misleading language, Rel. image–text relevance, and Auth. image authenticity. CoT Avg. is the average agreement across the four dimensions. Final Accuracy reports true/fake prediction accuracy against the dataset ground-truth labels. Best and second-best results are highlighted in light red and light yellow, respectively.

Figure 4: An example failure case in multimodal fake-news detection.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Check	Allowed values
factual_error	Whether the text appears to contain erroneous or conflicting factual details.	yes : contains errors; no : appears factual
language_issue	Whether the text contains awkward wording, broken grammar, unnatural style, machine-generated tone, or exaggerated/biased wording.	yes : has issues; no : no obvious issues
image_relevance	Whether the image is relevant to the news text.	yes : relevant; no : not relevant
image_authenticity	Whether the image appears visually authentic rather than AI-generated or fake.	yes : authentic; no : not authentic

Appendix

Table 3: Diagnostic checks in the structured CoT prompt.

Panel of Figure 5 — Figure 5: Examples of true and fake news posts generated by our framework

Figure 6: Human annotation interface used for multimodal fake-news review. Each sample presents the news image, source information, headline, and displayed text, followed by four required diagnostic checks and a final true/fake judgment. The four checks cover factual errors, language issues, image–text relevance, and image authenticity, matching the diagnostic dimensions used in the structured CoT evaluation prompt.

Fake News Detection

Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.

See all 93 papers

LLM-MRD: LLM-Guided Multi-View Reasoning Distillation for Fake News Detection
Mar 10, 2026Weilin Zhou, Shanwen Tan, Enhao Gu +1Fake News Detection Reasoning Distillation
Xinjiang University, Urumqi, China · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University, Urumqi, China · Sichuan University, Chengdu, China
MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection
Aug 2, 2026Lehan Zhang, Yinlei Cheng, Shiqi Hu Yiheng Zhou +2Social Media Analysis Fake News Detection
Beijing Institute of Fashion Technology, Beijing, China · Shenyang University of Technology, Shenyang, China
Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation
Apr 10, 2026Xinyu Wang, Sai Koneru, Wenbo Zhang +3Fake News Detection Synthetic Benchmark Generation
Pennsylvania State University University Park, PA 16802, USA

Mar 10, 2026J/K move · Enter open · S save

LLM-MRD: LLM-Guided Multi-View Reasoning Distillation for Fake News Detection

Weilin Zhou, Shanwen Tan, Enhao Gu, Yurong Qian

Xinjiang University, Urumqi, China · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University, Urumqi, China · Sichuan University, Chengdu, China

Fake News Detection Reasoning Distillation Efficient Multimodal Inference Cross-Modal Knowledge Distillation

Open paper AskPDF arXiv

Multimodal fake news detection is crucial for mitigating societal disinformation. Existing approaches attempt to address this by fusing multimodal features or leveraging Large Language Models (LLMs) for advanced reasoning. However, these methods suffer from serious limitations, including a lack of comprehensive multi-view judgment and fusion, and prohibitive reasoning inefficiency due to the high computational costs of LLMs. To address these issues, we propose LLM-Guided Multi-View Reasoning Distillation for Fake News Detection ( LLM-MRD), a novel teacher-student framework. The Student Multi-view Reasoning module first constructs a comprehensive foundation from textual, visual, and cross-modal perspectives. Then, the Teacher Multi-view Reasoning module generates deep reasoning chains as rich supervision signals. Our core Calibration Distillation mechanism efficiently distills this complex reasoning-derived knowledge into the efficient student model. Experiments show LLM-MRD significantly outperforms state-of-the-art baselines. Notably, it demonstrates a comprehensive average improvement of 5.19% in ACC and 6.33% in F1-Fake when evaluated across all competing methods and datasets. Our code is available at https://github.com/Nasuro55/LLM-MRD

Abstract

Fake News Detection

Explore similar work

LLM-MRD: LLM-Guided Multi-View Reasoning Distillation for Fake News Detection

MRAFnd: Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation