cs.CVAug 28, 2026
SaveFUSED: Forensic-Semantic Mixture-of-Experts for AI Inpainting Detection and Localization
Organizations: Informatics Institute, University of Amsterdam Amsterdam, The Netherlands
Abstract
Diffusion-based inpainting models modify only a localized part of an image, while many AI-image detectors rely on global artifacts and do not localize. These artifacts vary across generators, limiting detector transfer under distribution shifts. Recent work shows that restoring the authentic pixels outside the inpainted region removes these cues and can degrade pretrained detectors. To address this, we present FUSED, a unified framework for the joint detection and localization of AI-generated inpainting. FUSED combines low-level forensic cues with high-level semantic features using a sparsely-gated Mixture-of-Experts architecture, enabling the model to adaptively prioritize the most relevant signal for each token. For each input, FUSED predicts both an image-level manipulation score and a pixel-level mask of the inpainted area. On the OpenSDID cross-generator benchmark, FUSED achieves the best average detection and localization, with the largest gains on unseen generators. The same model transfers directly to the held-out AutoSplice and CocoGlide benchmarks, more than doubling localization performance. Evaluating each held-out benchmark with and without the global generator artifact further shows that all evaluated methods, ours included, partly read the artifact as evidence of manipulation, and FUSED remains the strongest under both conditions. Code and pretrained models are available at https://github.com/AntonNuzhdin/FUSED.
Figures & tables
Figure 1: Prior forensic-semantic detectors combine forensic and semantic features using a global image-level fusion. FUSED instead routes each token to a sparse set of experts, enabling token-level fusion and improving robustness to unseen generators.
Figure 2: The FUSED architecture. In contrast to the single image-level fusion illustrated in Figure 1 , FUSED represents the forensic and semantic streams as token sets, concatenates them into a joint sequence, and routes each token through a sparse Mixture-of-Experts before cross-stream self-attention. A classification head predicts the image-level manipulation score, while a segmentation bridge combines the forensic feature map with the post-fusion semantic tokens to decode the tampering mask.
| Method | SD1.5 | SD2.1 | SDXL | SD3 | Flux.1 | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | |
| MVSS-Net ( Dong et al. 2022 ) | 57.9 | 65.3 | 44.9 | 51.8 | 14.7 | 18.5 | 26.9 | 32.7 | 4.8 | 6.4 | 29.8 | 34.9 |
| CAT-Net ( Kwon et al. 2022 ) | 66.4 | 74.8 | 54.6 | 62.3 | 25.5 | 30.7 | 35.6 | 42.1 | 5.0 | 6.6 | 37.4 | 43.3 |
| PSCC-Net ( Liu et al. 2022 ) | 54.7 | 64.2 | 36.7 | 44.8 | 19.7 | 26.1 | 29.3 | 37.3 | 8.2 | 11.6 | 29.7 | 36.8 |
| ObjectFormer ( Wang et al. 2022 ) | 51.2 | 65.7 | 47.4 | 41.4 | 7.4 | 9.8 | 9.4 | 12.6 | 5.3 | 7.3 | 24.1 | 27.4 |
| TruFor ( Guillaro et al. 2023 ) | 63.4 | 71.0 | 54.7 | 61.9 | 26.6 | 31.9 | 32.3 | 38.5 | 7.6 | 9.7 | 36.9 | 42.6 |
Table 1: Pixel-level localization on the OpenSDID test generators (per-image pixel F1 and IoU). Bold is best, underline second-best. FUSED achieves the best average localization, with its largest margins on the three most distant generators.
| Method | SD1.5 | SD2.1 | SDXL | SD3 | Flux.1 | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | |
| CNNDetection ( Wang et al. 2020 ) | 84.6 | 85.0 | 71.6 | 75.9 | 59.7 | 68.7 | 56.3 | 67.1 | 35.7 | 57.6 | 61.6 | 70.9 |
| GramNet ( Liu et al. 2020 ) | 80.5 | 80.4 | 74.0 | 76.7 | 65.3 | 70.8 | 64.4 | 70.3 | 52.0 | 63.4 | 67.2 | 72.3 |
| FreqNet ( Tan et al. 2024a ) | 75.9 | 77.7 | 61.0 | 68.4 | 53.2 | 64.0 | 53.5 | 64.4 | 38.5 | 57.1 | 56.4 | 66.3 |
| NPR ( Tan et al. 2024b ) | 79.4 | 79.3 | 81.7 | 81.8 | 72.1 | 74.3 | 73.4 | 75.5 | 67.6 | 71.4 | 74.9 | 76.4 |
| UniFD ( Ojha et al. 2023 ) | 77.5 | 77.6 | 80.6 | 81.9 | 70.7 | 74.8 | 71.1 | 75.2 | 61.1 | 69.1 | 72.2 | 75.7 |
Table 2: Image-level detection on the OpenSDID test generators (F1 and accuracy at a fixed threshold of ). Bold indicates the best result, and underlining indicates the second-best. FUSED achieves the best average detection.
Figure 3: Qualitative localization results on the four unseen OpenSDID generators, one per row. On unseen generators, several baselines produce empty or fragmented masks, while FUSED more consistently localizes the manipulated region.
| Model | AutoSplice | CocoGlide | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Pix. F1 | IoU | F1 | AUC | Acc. | Pix. F1 | IoU | F1 | AUC | Acc. | |
| MVSS-Net ( Dong et al. 2022 ) | 7.7 | 5.4 | 42.6 | 75.2 | 53.7 | 3.7 | 2.9 | 14.7 | 65.5 | 53.5 |
| PSCC-Net ( Liu et al. 2022 ) | 7.8 | 5.5 | 14.2 | 58.8 | 42.4 | 11.3 | 9.0 | 14.0 | 64.4 | 53.1 |
| IML-ViT ( Ma et al. 2023 ) | 15.8 | 11.3 | 27.9 | 50.7 | 43.7 | 6.0 | 4.7 | 1.9 | 76.0 | 50.1 |
| ObjectFormer ( Wang et al. 2022 ) | 11.0 | 7.9 | 18.3 | 50.4 | 42.0 | 4.1 | 3.1 | 3.8 | 57.2 | 50.3 |
| TruFor ( Guillaro et al. 2023 ) | 16.5 | 11.9 | 18.1 | 69.5 | 43.0 | 23.8 | 19.8 | 2.7 | 79.5 | 50.5 |
Table 3: Zero-shot transfer of OpenSDID-trained models to the held-out AutoSplice and CocoGlide benchmarks. Dagger marks a degenerate predictor excluded from ranking. FUSED achieves the best result on every metric of both benchmarks.
| Model | INP: Real vs. Inpaint | INP-X: Real vs. Exchanged | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Pix. F1 | IoU | F1 | AUC | Acc. | Pix. F1 | IoU | F1 | AUC | Acc. | |
| MVSS-Net ( Dong et al. 2022 ) | 10.6 | 7.5 | 61.0 | 92.1 | 74.5 | 6.9 | 5.7 | 23.0 | 69.2 | 60.5 |
| PSCC-Net ( Liu et al. 2022 ) | 15.3 | 10.4 | 75.2 | 87.6 | 79.3 | 14.0 | 9.8 | 39.5 | 69.3 | 60.7 |
| IML-ViT ( Ma et al. 2023 ) | 16.8 | 11.8 | 57.3 | 81.2 | 68.4 | 13.2 | 10.0 | 34.8 | 65.7 | 58.4 |
| ObjectFormer ( Wang et al. 2022 ) | 6.2 | 4.4 | 17.3 | 63.3 | 58.4 | 5.7 | 4.2 | 12.8 | 60.0 | 57.1 |
| TruFor ( Guillaro et al. 2023 ) | 16.0 | 11.4 | 47.6 | 89.6 | 69.1 | 12.2 | 9.9 | 11.3 | 68.5 | 57.8 |
Table 4: Zero-shot transfer of OpenSDID-trained models to the paired INP-X benchmark, with the global VAE artifact present (INP) and removed (INP-X). FUSED achieves the best detection accuracy and pixel F1 under both conditions.
| Model | Localization: pixel F1 | Detection: F1 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SD1.5 | SD2.1 | SDXL | SD3 | Flux.1 | Avg. | SD1.5 | SD2.1 | SDXL | SD3 | Flux.1 | Avg. | |
| FUSED (Ours) | 70.5 | 63.9 | 48.3 | 62.1 | 31.8 | 55.3 | 93.9 | 95.4 | 90.2 | 89.2 | 71.2 | 88.0 |
| w/o MoE (dense FFN) | 70.2 | 63.8 | 47.2 | 60.6 | 29.0 | 54.2 | 94.7 | 91.0 | 81.5 | 80.8 | 62.4 | 82.1 |
| w/o seg-bridge | 57.9 | 44.6 | 27.0 | 35.0 | 16.8 | 36.2 | 96.0 | 91.8 | 84.9 | 86.2 | 70.2 | 85.8 |
| w/o ConvNeXt | 29.3 | 19.9 | 16.0 | 17.7 | 12.8 | 19.1 | 86.7 | 83.1 | 66.3 | 76.3 | 70.5 | 76.6 |
| w/o SparseViT | 61.6 | 60.5 | 39.2 | 58.1 | 28.1 | 49.5 | 96.2 | 92.6 | 83.1 | 83.0 | 63.4 | 83.7 |
Table 5: Ablation of FUSED on the OpenSDID test generators (pixel F1 and detection F1). The blocks remove one component at a time, vary the frozen semantic backbone, and vary the routing sparsity. Every variant is retrained on the SD1.5 split with the same recipe, and the FUSED row (ConvNeXt-XXL, top- ) is the shared reference. Bold marks the best and the underlined are second-best value in each column across all rows. FUSED achieves the best average on both tasks.
| Model | Pix. F1 | IoU | F1 | AUC | Acc. |
|---|---|---|---|---|---|
| MaskCLIP | 28.3 | 20.5 | 87.7 | 89.3 | 82.4 |
| FUSED (Ours) | 42.8 | 33.6 | 89.2 | 89.7 | 83.9 |
Table 6: Transfer to So-Fake-OOD after training on the multi-generator So-Fake-Set. FUSED improves on every metric, with the largest gains in localization.
Figure 4: Per-token utilization of the MoE experts ( , top- ) for the SparseViT token group on the left and the ConvNeXt token group on the right. Each bar series corresponds to one generator, and the dashed line denotes uniform utilization, . All experts remain active, while utilization differs between branches and varies across generators.
Figure 5: Pixel-F1 robustness to Gaussian blur and JPEG compression on in-domain SD1.5 and unseen SD3. FUSED maintains its SD3 advantage across all severity levels, showing that its unseen-generator transfer remains robust even when the input is degraded.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Performance by manipulated-area fraction on the OpenSDID test set. Mean per-image localization IoU (left) and detection recall on manipulated images (right), binned by the ground-truth manipulated-area fraction. FUSED exceeds MaskCLIP in every bin on both tasks. Localization improves from the smallest to the medium and large partial-edit bins, while recall remains high across partial edits.
Figure 7: Qualitative ablation results. Predictions of the full model and the four component-ablation variants from Table 5 of the main paper on OpenSDID test examples. Removing the segmentation bridge, SparseViT or ConvNeXt branch produces the largest qualitative changes.
Figure 8: Detection outcomes on the OpenSDID test set. Images are grouped by their actual and predicted real/fake labels. The off-diagonal cells show real images classified as fake and fake images classified as real.
Figure 9: Ten OpenSDID examples, two per generator, for which FUSED obtains an IoU of . Each triplet contains the input image, ground-truth mask, and predicted mask.
Figure 10: Qualitative localization results on So-Fake-OOD for GPT-4o (left) and Ideogram 3.0 (right). Each group shows the input image, ground-truth mask, MaskCLIP prediction, and FUSED prediction. MaskCLIP frequently produces empty or near-full masks, whereas FUSED more often localizes the manipulated region, although some predictions remain fragmented or over-segmented.
Figure 11: Additional qualitative results on OpenSDID. Ten randomly selected test examples. Each row contains the input image, ground-truth mask, and predictions from all compared methods. FUSED responds across all examples and generally recovers the coarse manipulated region, although some predictions remain over-segmented.
Figure 12: Qualitative results on fully synthetic images. The ground-truth mask covers the entire image because every pixel is generated. FUSED predicts a full-frame mask in all five examples, whereas the compared baselines are less consistent.
Figure 13: Qualitative results for small manipulated regions. The edited region occupies only a small fraction of each image. FUSED generally localizes the target region, although its predictions may include additional disconnected areas.
More in this topic
AI-Generated Image Detection
AI-generated image detection identifies synthetic or AI-edited images and develops methods that generalize across generators, domains, and image transformations.
See all 194 papers