Can Multimodal Large Language Models Generate and Detect Multimodal Social Media Fake News?
Organizations: Independent Researcher · Carnegie Mellon University · RMIT University · Yale University
Abstract
The rapid advancement of generative AI raises concerns about the misuse of Multimodal LLMs (MLLMs) for large-scale disinformation campaigns on social media. Despite existing research on textual disinformation, a fundamental question remains unanswered: can MLLMs be exploited to fabricate realistic multimodal fake news, and can they reliably detect it? We introduce a multi-agent framework in which a story agent, an image agent, and a critic agent collaborate to produce fake social media posts that plausibly counter true news. We apply the framework to generate over 9,000 paired multimodal news posts across science, health, and entertainment domains, and benchmark 16 open- and closed-source MLLMs for automated detection. We find that most models fall substantially short of human-level accuracy and fail critically on identifying image authenticity. Our research provides a foundation for developing robust defenses against social media fake news. Code and data are available at https: //github.com/xiuzhenzhang/Multimodal.
Figures & tables
| NIH | Nature | Snopes | Average | ||||||||||
| Model | Mode | F1 | Acc | AUC | F1 | Acc | AUC | F1 | Acc | AUC | F1 | Acc | AUC |
| Open-source models | |||||||||||||
| InternVL2.5-8B Chen et al. (2023) | Direct | 45.53 | 52.40 | 53.49 | 50.54 | 53.64 | 53.23 | 44.56 | 44.98 | 45.05 | 46.87 | 50.34 | 50.59 |
| CoT | 45.76 | 46.66 | 47.08 | 56.74 | 57.02 | 56.90 | 55.18 | 55.86 | 55.97 | 52.56 | 53.18 | 53.32 | |
| InternVL3-8B Chen et al. (2023) | Direct | 68.54 | 70.69 | 71.52 | 76.92 | 77.66 | 77.38 | 51.28 | 54.12 | 54.31 | 65.58 | 67.49 | 67.74 |
| CoT | 59.43 | 59.45 | 59.61 | 65.30 | 65.53 | 65.67 | 53.95 | 58.18 | 58.37 | 59.56 | 61.05 | 61.22 | |
| Model | True News | Fake News | Overall CoT Agreement | Final Accuracy | ||||||||||||||
| Fact. | Lang. | Rel. | Auth. | Avg. | Fact. | Lang. | Rel. | Auth. | Avg. | Fact. | Lang. | Rel. | Auth. | Avg. | True | Fake | All | |
| GPT-5.5 CoT | 100.0 | 100.0 | 94.1 | 85.3 | 94.9 | 65.0 | 57.1 | 100.0 | 97.2 | 79.8 | 87.0 | 84.2 | 97.1 | 91.4 | 89.9 | 100.0 | 100.0 | 100.0 |
| GPT-4o CoT | 88.6 | 97.3 | 91.4 | 82.9 | 90.0 | 66.7 | 66.7 | 100.0 | 5.0 | 59.6 | 79.7 | 85.2 | 95.9 | 41.3 | 75.5 | 83.8 | 69.8 | 76.2 |
| Qwen3-VL-8B CoT | 76.5 | 94.4 | 73.5 | 82.4 | 81.7 | 58.3 | 54.2 | 74.4 | 45.0 | 58.0 | 69.0 | 78.3 | 74.0 | 62.2 | 70.9 | 55.6 | 90.7 | 74.7 |
| Qwen2-VL-7B CoT | 97.1 | 100.0 | 91.4 | 91.4 | 95.0 | 41.7 | 54.2 | 100.0 | 5.0 | 50.2 | 74.6 | 82.0 | 95.9 | 45.3 | 74.5 | 91.9 | 16.3 | 51.2 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Check | Allowed values |
| factual_error | Whether the text appears to contain erroneous or conflicting factual details. | yes : contains errors; no : appears factual |
| language_issue | Whether the text contains awkward wording, broken grammar, unnatural style, machine-generated tone, or exaggerated/biased wording. | yes : has issues; no : no obvious issues |
| image_relevance | Whether the image is relevant to the news text. | yes : relevant; no : not relevant |
| image_authenticity | Whether the image appears visually authentic rather than AI-generated or fake. | yes : authentic; no : not authentic |
Fake News Detection
Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.
See all 93 papers