As generative AI advances, the distinction between authentic and synthetic media is increasingly blurred, challenging the integrity of online information. In this study, we present CONVEX, a large-scale dataset of multimodal misinformation involving miscaptioned, edited, and AI-generated visual content, comprising over 150K multimodal posts with associated notes and engagement metrics from X's Community Notes. We analyze how multimodal misinformation evolves in terms of virality, engagement, and consensus dynamics, with a focus on synthetic media. Our results show that while AI-generated content achieves disproportionate virality, its spread is driven primarily by passive engagement rather than active discourse. Despite slower initial reporting, AI-generated content reaches community consensus more quickly once flagged. Moreover, our evaluation of specialized detectors and vision-language models reveals a consistent decline in performance over time in distinguishing synthetic from authentic images as generative models evolve. These findings highlight the need for continuous monitoring and adaptive strategies in the rapidly evolving digital information environment.
Figures & tables
Figure 1: Example of an AI-generated image post on X, with the Community Note and external sources used to verify it.
Figure 2: Monthly Community Notes volume for both modalities and releases of popular generative AI tools.
Figure 3: Virality Share across both modalities.
Figure 4: Monthly Engagement Index for both modalities.
Image
Video
Type
Notes / Tweet
Helpful (%)
Consensus Probability
First Note (h)
Notes to Consensus
Notes / Tweet
Helpful (%)
Consensus Probability
First Note (h)
Notes to Consensus
Miscaptioned
1.89
66.8
0.284
9.41
1.47
1.85
71.1
0.295
9.06
1.41
Edited
1.85
70.5
0.304
8.47
1.42
1.83
71.2
0.299
9.76
1.40
AI-Generated
1.78
69.4
0.305
11.69
1.41
1.61
81.4
0.362
11.30
1.26
Table 1: Consensus metrics across misinformation categories for image and video subsets.
Image
Video
Type
Model
General
Any
Model
General
Any
Miscaptioned
0.94
1.58
2.48
0.98
1.72
2.68
Edited
1.35
1.47
2.80
0.92
1.63
2.53
AI-generated
9.47
56.75
60.58
12.25
47.07
53.04
Table 2: AI-related mentions in Community Notes. Values represent the percentage of notes containing general references to AI or to specific a model .
Figure 5: Temporal performance of AI-image detection systems from 2023 to 2025. Panels show TPR, FPR, and Accuracy across six-month evaluation periods. TPR declines consistently across models, while FPR remains relatively stable, suggesting that performance degradation is driven by reduced sensitivity to newer AI-generated images.
Model
TPR ↑
FPR ↓
Precision ↑
F1 ↑
Accuracy ↑
SPAI
41.48
24.66
62.90
49.99
58.34
RINE
52.75
30.54
63.75
57.73
61.03
BFree
52.72
27.81
65.65
58.47
62.42
Gemma 3 27B
70.21
31.00
69.51
69.86
69.60
Grok 4.1 Fast
52.28
21.16
71.35
60.35
65.51
GPT-5-mini
55.00
15.06
78.64
64.73
69.91
Table 3: Overall benchmark performance on the image dataset. Bold denotes best values and underlined the second-best.
Figure 6: Examples of AI-generated images with corresponding predictions from SIDs and VLMs.
The rapid advancement of generative AI raises concerns about the misuse of Multimodal LLMs (MLLMs) for large-scale disinformation campaigns on social media. Despite existing research on textual disinformation, a fundamental question remains unanswered: can MLLMs be exploited to fabricate realistic multimodal fake news, and can they reliably detect it? We introduce a multi-agent framework in which a story agent, an image agent, and a critic agent collaborate to produce fake social media posts that plausibly counter true news. We apply the framework to generate over 9,000 paired multimodal news posts across science, health, and entertainment domains, and benchmark 16 open- and closed-source MLLMs for automated detection. We find that most models fall substantially short of human-level accuracy and fail critically on identifying image authenticity. Our research provides a foundation for developing robust defenses against social media fake news. Code and data are available at https: //github.com/xiuzhenzhang/Multimodal.
Jiyao Yang, Yang Liu, Zhenyue Qin +2
Independent Researcher · Carnegie Mellon University · RMIT University +1
Recent generative models can now produce visual artifacts with realistic embedded text and layouts, creating a new misinformation threat: synthetic credibility. We introduce SYNCRED-Bench, a benchmark of 600 AI-generated misinformation images balanced across six credible-form categories and seven fine-grained circulation styles, together with FP450, a real-image negative set for measuring false positives. Extensive evaluation shows that existing systems remain unreliable: under a 5% false-positive-rate constraint, 15 MLLMs achieve only 10.5% true positive rate (TPR), open-source AIGC detectors achieve less than 5%, and commercial APIs reach 57.6%. Human annotators also struggled to identify synthetic credibility, reaching only 63% TPR. These findings establish synthetic credibility as a severe and underexplored visual misinformation challenge, and provide a benchmark for developing detectors that reason beyond superficial credibility cues.
Junxiao Yang, Minghao Zhang, Xiaoce Wang +4
The Conversational AI (CoAI) group, DCST, Tsinghua University
Generative AI has enabled the creation of photorealistic images and videos that are increasingly disseminated on social media, often used for spam, misinformation, manipulation, and fraud. Existing AI-generated content (AIGC) detection methods face challenges including poor generalization to new generation models, reliance on single modalities, and lack of interpretable explanations. We present our pipeline that mitigates these issues by continuously curating diverse multi-modal social media data and training a compact vision-language model for detection and explanation. Our model achieves state-of-the-art detection performance on public benchmarks and demonstrates robust detection and explanation capabilities on internal social media datasets across multiple platforms. We deployed our model for post recommendation on social media platforms and observed positive downstream impacts on user engagement, demonstrating that it is feasible to perform effective AIGC detection in dynamic, real-world social media environments.