cs.CVMar 26, 2026
SaveTHEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics
Organizations: Beijing University of Posts and Telecommunications
Abstract
We present THEMIS, a novel multi-task benchmark designed to comprehensively evaluate multimodal large language models (MLLMs) on visual fraud reasoning within real-world academic scenarios. Compared to existing benchmarks, THEMIS introduces three major advances. (1) Real-World Scenarios and Complexity: Our benchmark comprises over 4,000 questions spanning seven scenarios, derived from authentic retracted-paper cases and carefully curated multimodal synthetic data. With 60.47% complex-texture images, THEMIS bridges the critical gap between existing benchmarks and the complexity of real-world academic fraud. (2) Fraud-Type Diversity and Granularity: THEMIS systematically covers five challenging fraud types and introduces 16 fine-grained manipulation operations. On average, each sample undergoes multiple stacked manipulation operations, with the diversity and difficulty of these manipulations demanding a high level of visual fraud reasoning from the models. (3) Multi-Dimensional Capability Evaluation: We establish a mapping from fraud types to five core visual fraud reasoning capabilities, thereby enabling an evaluation that reveals the distinct strengths and specific weaknesses of different models across these core capabilities. Experiments on 16 leading MLLMs show that even the best-performing model, GPT-5, achieves an overall performance of only 56.15%, demonstrating that our benchmark presents a stringent test. We expect THEMIS to advance the development of MLLMs for complex, real-world fraud reasoning tasks.
Figures & tables
Figure 1: Overview of THEMIS. (1) Real-World Scenarios and Complexity , comprising over 4,000 questions across 7 representative scenarios; (2) Fraud-Type Diversity and Granularity , covering 5 challenging fraud methods with 16 fine-grained manipulation operations; (3) Multi-Dimensional Capability Evaluation , mapping fraud types to 5 core reasoning capabilities.
Table 1: Comparison of THEMIS and other related benchmarks. Mod. : Modality; Manip. : Manipulation Operations; Real : Contains real fraud cases; I : Image; T-I : Cross Text–Image; Task : BC : Binary Classification; IP-L : Image Partial Localization. SMF , CMO , and CMI denote Single-Mode Forgery, Composite Manipulation Operations, and Cross-Modal Inconsistency, respectively, with suffixes -I and -L indicating Identification and Localization ( -IL for both). Fraud Type : 1 Splicing, 2 Copy-Move, 3 AI-Generated, 4 Duplication, and 5 Text–Image Inconsistency.
Figure 2: Statistics of THEMIS. (a) Distribution of fraud methods. (b) Distribution of manipulation operations (synthetic data). See Table 5 for real cases. IIF : Image Inference Forgery; TRE : Targeted Region Editing; CT : Color Temperature; DR : Direct Reuse; HF : Horizontal Flip; VF : Vertical Flip.
Figure 3: Dataset construction pipeline of THEMIS. The dataset is built through 2 stages: Stage 1: Extraction and Parsing , where figures, captions, and related sentences are parsed from scientific PDFs and segmented into panels; Stage 2: Fraud Data Generation , where 5 major fraud types ( Splicing , Copy-Move , AI-Generated , Duplication , and Text–Image Inconsistency ) are applied to construct challenging tasks.
Figure 4: Evaluation task design of THEMIS. A principled mapping from 5 fraud types to 5 core reasoning capabilities ( Expert Knowledge Utilization , Visual Recognition , Spatial Reasoning , Region Localization , and Comparative Reasoning ). The capability distribution bars on the right of each box illustrate the reasoning skills involved and their relative emphasis, with the darkest color highlighting the primary capability being evaluated.
| Model | Single-Mode Forgery Identification (Id Score) | Single-Mode Forgery Localization (Loc Score) | Composite Manipulation Operations Identification (Id Score) | Cross-Modal Inconsistency Identification & Localization | BRI | |||||||
| SPL | CM | AIG | Avg. | SPL | CM | AIG | Avg. | DUP | TII (300) | |||
| (807) | (242) | (897) | (807) | (242) | (897) | (2,079) | Id Score | Loc Score | ||||
| Proprietary MLLMs | ||||||||||||
| GPT-5 | 43.51 | 72.73 | 44.26 | 53.50 | 16.67 | 36.41 | 19.33 | 24.14 | 33.32 | 60.67 | 27.44 | 56.15 |
| OpenAI o4-mini-high | 41.49 | 77.89 | 35.67 | 51.68 | 10.44 | 29.78 | 19.79 | 20.00 | 30.34 | 66.33 | 32.22 | 52.34 |
| Qwen-VL-Max | 30.37 | 87.40 | 35.43 | 51.07 | 40.07 | 48.34 | 35.85 | 41.42 | 23.33 | 56.00 | 15.36 | 49.83 |
Table 2: Model performance across tasks on THEMIS synthetic data. We report the Id Score (Identification Score) and Loc Score (Localization Score) for 5 fraud methods: SPL (Splicing), CM (Copy-Move), AIG (AI-Generated), DUP (Duplication), and TII (Text–Image Inconsistency), with BRI (Balanced Robustness Index). The best and second-best scores are highlighted.
Figure 5: Impact of different factors on model performance using synthetic data. SPL : Splicing; CM : Copy-Move; AIG : AI-Generated; DUP : Duplication; DUP-Id : Id Score for Duplication; TII-Id : Id Score for Text–Image Inconsistency; TII-Loc : Loc Score for Text–Image Inconsistency; SMF-Id : Average Id Score of Single-Mode Forgery Identification; SMF-Loc : Average Loc Score of Single-Mode Forgery Localization.
| Model | SMF-IL | CMO-I | |
| Id Score | Loc Score | Id Score | |
| Qwen2.5-VL-72B | 50.00 | 45.41 | 22.39 |
| Gemini 2.5 Flash | 44.55 | 43.07 | 49.58 |
| GPT-5 | 43.64 | 28.68 | 23.22 |
| OpenAI o4-mini-high | 34.55 | 21.67 | 36.04 |
| Gemma 3 27B | 20.00 | 16.15 | 42.18 |
Table 3: Performance comparison on real cases. We report the results for Single-Mode Forgery Identification and Localization ( SMF-IL ) and Composite Manipulation Operations Identification ( CMO-I ) tasks.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distribution and visualization of 7 scenarios.
| Scope | Fraud Type | Manipulation Operation | # Samples |
| Intra-Element | Copy-Move | Copy-Move | 92 |
| AI-Generated | Image Inference Forgery | 74 | |
| Targeted Region Editing | 550 | ||
| Inter-Element | Splicing | Splicing | 534 |
| Duplication | Global Direct Reuse | 104 | |
| Global 180° Rotation | 283 |
Table 6: Detailed distribution of fraud types and manipulation operations across the synthetic data.
Figure 7: Stage 1: Extraction and Parsing. The regions highlighted with red boxes correspond to figures , while those highlighted with blue boxes correspond to panels . Green boxes are used to verify whether the visualization is effective.
| Provider | Version | Parameters | Access Links |
| Proprietary MLLMs | |||
| OpenAI | GPT-5 | N/A | https://openrouter.ai/openai/gpt-5 |
| OpenAI o4-mini-high | N/A | https://openrouter.ai/openai/o4-mini-high | |
| Gemini 2.5 Flash | N/A | https://openrouter.ai/google/gemini-2.5-flash | |
| Gemini 2.5 Pro | N/A | https://openrouter.ai/google/gemini-2.5-pro | |
| Alibaba | Qwen-VL-Max | N/A | https://openrouter.ai/qwen/qwen-vl-max |
Table 7: Detailed information on benchmarked MLLMs.
| Model | ||||
| GPT-5 | 64.56 | 56.15 | 53.35 | 47.75 |
| OpenAI o4-mini-high | 63.57 | 52.34 | 48.60 | 41.11 |
| Qwen-VL-Max | 57.31 | 49.83 | 47.34 | 42.35 |
| Gemini 2.5 Flash | 57.67 | 44.70 | 40.36 | 31.71 |
| Doubao-Seed-1.6-thinking | 48.43 | 37.14 | 33.37 | 25.85 |
| Doubao-Seed-1.6-vision | 46.17 | 33.47 | 29.24 | 20.78 |
Table 8: Sensitivity of the BRI under different values of .
| Model | Splicing | ||
| ACC | IoU | Set-based F1 | |
| Proprietary MLLMs | |||
| GPT-5 | 43.51 | 16.67 | 20.27 |
| OpenAI o4-mini-high | 41.49 | 10.44 | 12.81 |
| Qwen-VL-Max | 30.37 | 40.07 | 52.84 |
| Gemini 2.5 Flash | 63.39 | 56.72 | 65.08 |
Table 9: Experimental results on THEMIS synthetic data for Splicing. We report the ACC , IoU , and Set-based F1 . The best and second-best scores are highlighted.
| Model | Copy-Move | ||
| ACC | IoU | Set-based F1 | |
| Proprietary MLLMs | |||
| GPT-5 | 72.73 | 36.41 | 46.11 |
| OpenAI o4-mini-high | 77.89 | 29.78 | 39.32 |
| Qwen-VL-Max | 87.40 | 48.34 | 60.94 |
| Gemini 2.5 Flash | 67.56 | 46.98 | 57.90 |
Table 10: Experimental results on THEMIS synthetic data for Copy-Move. We report the ACC , IoU , and Set-based F1 . The best and second-best scores are highlighted.
| Model | AI-Generated | ||
| ACC | IoU | Set-based F1 | |
| Proprietary MLLMs | |||
| GPT-5 | 44.26 | 19.33 | 22.21 |
| OpenAI o4-mini-high | 35.67 | 19.79 | 22.86 |
| Qwen-VL-Max | 35.43 | 35.85 | 43.87 |
| Gemini 2.5 Flash | 35.45 | 38.87 | 44.76 |
Table 11: Experimental results on THEMIS synthetic data for AI-Generated. We report the ACC , IoU , and Set-based F1 . The best and second-best scores are highlighted.
| Model | Duplication | ||
| Set-based F1 | EM | SPC | |
| Proprietary MLLMs | |||
| GPT-5 | 33.32 | 19.05 | 22.72 |
| OpenAI o4-mini-high | 30.34 | 17.12 | 21.35 |
| Qwen-VL-Max | 23.33 | 12.13 | 15.29 |
| Gemini 2.5 Flash | 24.96 | 15.78 | 17.41 |
Table 12: Experimental results on THEMIS synthetic data for Duplication. We report the Set-based F1 , EM (Exact Match), and SPC (Subset Partial Credit). EM requires strict equality with the ground truth; SPC awards credit for correct subsets only if no incorrect elements (false positives) are predicted. The best and second-best scores are highlighted.
| Model | Text–Image Inconsistency | ||
| ACC | Macro-F1 | Text-based F1 | |
| Proprietary MLLMs | |||
| GPT-5 | 60.67 | 44.95 | 27.44 |
| OpenAI o4-mini-high | 66.33 | 48.58 | 32.22 |
| Qwen-VL-Max | 56.00 | 33.74 | 15.36 |
| Gemini 2.5 Flash | 36.33 | 25.30 | 28.24 |
Table 13: Experimental results on THEMIS synthetic data for Text–Image Inconsistency. We report the ACC , Macro-F1 , and Text-based F1 . The best and second-best scores are highlighted.
Figure 8: Success case by GPT-5 on Splicing.
Figure 9: Failure case by GPT-5 on Splicing. The model correctly identified the forgery type but mislocalized it.
Figure 10: Failure case by GPT-5 on Splicing. The model misidentified and mislocalized.
Figure 11: Success case by GPT-5 on Copy-Move.
Figure 12: Failure case by GPT-5 on Copy-Move. The model correctly identified the forgery type but mislocalized it.
Figure 13: Failure case by GPT-5 on Copy-Move. The model correctly localized the forgery region but misidentified it.
Figure 14: Failure case by GPT-5 on Copy-Move. The model misidentified and mislocalized.
Figure 15: Success case by GPT-5 on AI-Generated. The panel is entirely forged via Image Inference Forgery.
Figure 16: Failure case by GPT-5 on AI-Generated. The model misidentified the forgery type. The panel is entirely forged via Image Inference Forgery.
Figure 17: Success case by GPT-5 on AI-Generated. The panel is partially forged via Targeted Region Editing.
Figure 18: Failure case by GPT-5 on AI-Generated. The model correctly identified the forgery type but mislocalized it. The panel is partially forged via Targeted Region Editing.
Figure 19: Failure case by GPT-5 on AI-Generated. The model correctly localized the forgery region but misidentified it. The panel is partially forged via Targeted Region Editing.
Figure 20: Failure case by GPT-5 on AI-Generated. The model misidentified and mislocalized. The panel is partially forged via Targeted Region Editing.
Figure 21: Success case by GPT-5 on Duplication. These panels resulted from Global Duplication with flip.
Figure 22: Failure case by GPT-5 on Duplication. These panels resulted from Global Duplication with rotation and parameter modification.
Figure 23: Failure case by GPT-5 on Duplication. These panels resulted from Local Duplication with rotation. The duplicated region is indicated by the yellow boxes.
Figure 24: Failure case by GPT-5 on Duplication. These panels resulted from Local Duplication with parameter modification. The duplicated region is indicated by the yellow boxes.
Figure 25: Success case by GPT-5 on Text–Image Numerical Inconsistency.
Figure 26: Failure case by GPT-5 on Text–Image Numerical Inconsistency. The model identified but mislocalized.
Figure 27: Failure case by GPT-5 on Text–Image Numerical Inconsistency. The model misidentified and mislocalized.
Figure 28: Failure case by GPT-5 on Text–Image Trend Inconsistency. The model identified but mislocalized.
Figure 29: Failure case by GPT-5 on Text–Image Trend Inconsistency. The model misidentified and mislocalized.
Figure 30: Left: Extraction and Parsing by MinerU. The green highlighted regions indicate MinerU’s panel-level parsing outputs, while the blue regions represent its caption parsing results. MinerU fails to extract complete figures and frequently mismatches image–caption pairs. Right: Extraction and Parsing by our pipeline. The regions highlighted with red boxes correspond to figures , while those highlighted with blue boxes correspond to panels . ( Green boxes are used solely to verify the effectiveness of the visualization.) Our extraction pipeline correctly identifies both full figures and their corresponding panels without structural or visual degradation.
Figure 31: One missing panel highlighted with a yellow box.
Figure 32: Two missing panels highlighted with yellow boxes. Two extra panels highlighted with red boxes.
Figure 33: Three missing panels highlighted with yellow boxes. One extra panel highlighted with a red box.
More in this topic
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 986 papers