cs.CVApr 14, 2026
SaveBridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization
Organizations: Sun Yat-sen University, School of Computer Science and Engineering, Guangzhou, Guangdong, China
Abstract
As generative image editing advances, image manipulation localization (IML) must handle both traditional manipulations with conspicuous forensic artifacts and diffusion-generated edits that appear locally realistic. Existing methods typically rely on either low-level forensic cues or high-level semantics alone, leading to a fundamental micro--macro gap. To bridge this gap, we propose FASA, a unified framework for localizing both traditional and diffusion-generated manipulations. Specifically, we extract manipulation-sensitive frequency cues through an adaptive dual-band DCT module and learn manipulation-aware semantic priors via patch-level contrastive alignment on frozen CLIP representations. We then inject these priors into a hierarchical frequency pathway through a semantic-frequency side adapter for multi-scale feature interaction, and employ a prototype-guided, frequency-gated mask decoder to integrate semantic consistency with boundary-aware localization for tampered region prediction. Extensive experiments on OpenSDI and multiple traditional manipulation benchmarks demonstrate state-of-the-art localization performance, strong cross-generator and cross-dataset generalization, and robust performance under common image degradations.
Figures & tables
Figure 1. Comparison of manipulation traces. (a) Traditional manipulations leave conspicuous artifacts. (b) Diffusion-generated edits produce locally realistic content with coherent low-level statistics. A side-by-side visual comparison of image manipulations. On the left, traditional manipulations show conspicuous microscopic artifacts such as unnatural edges and noise. On the right, diffusion-generated edits display locally realistic content that blends seamlessly with the background, showing coherent low-level statistics without obvious visible artifacts.
Figure 2. Localization performance on the OpenSDI benchmark measured by Pixel F1. The proposed framework consistently outperforms previous traditional IML methods, including CAT-Net and TruFor, as well as the recent diffusion-oriented state-of-the-art method MaskCLIP across all evaluated generators. A bar chart comparing the Pixel F1 scores of various IML methods on the OpenSDI benchmark across different diffusion generators. The methods evaluated include traditional baselines such as CAT-Net and TruFor, as well as the diffusion-specific MaskCLIP. The bars for the proposed framework are noticeably taller across all generator categories, indicating superior localization performance compared to all other baseline methods.
Figure 3. Overview of FASA. ADB-DCT extracts microscopic frequency cues, while a frozen CLIP encoder and PSA learn manipulation-aware semantic priors. SF-SA injects these priors into the hierarchical frequency pathway, and PG-FMD combines prototype-guided semantic responses with frequency gating for precise localization. A detailed block diagram of the Frequency-Aware Semantic Alignment (FASA) framework. An input image is processed through two parallel branches. The top branch uses an Adaptive Dual-Band DCT (ADB-DCT) to extract high and low-frequency features. The bottom branch uses a frozen CLIP encoder and Patch-Level Semantic Alignment (PSA) to extract macroscopic semantic priors and optimize real and fake prototypes. A Semantic-Frequency Side Adapter (SF-SA) injects these semantic priors into the frequency branch at multiple scales. Finally, a Prototype-Guided Frequency-Gated Mask Decoder (PG-FMD) dynamically fuses the aligned features to output a predicted pixel-wise manipulation mask.
| Method | SD1.5 | SD2.1 | SDXL | SD3 | Flux.1 | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | |
| MVSS-Net ( Chen et al., 2021 ) | 0.6420 | 0.5656 | 0.5011 | 0.4299 | 0.2202 | 0.1758 | 0.3569 | 0.2950 | 0.0889 | 0.0674 | 0.3618 | 0.3067 |
| PSCC-Net ( Liu et al., 2022a ) | 0.6243 | 0.5156 | 0.5029 | 0.4055 | 0.3323 | 0.2514 | 0.4135 | 0.3215 | 0.1238 | 0.0850 | 0.3994 | 0.3158 |
| CAT-Net ( Kwon et al., 2022 ) | 0.7460 | 0.6589 | 0.6254 | 0.5450 | 0.3464 | 0.2861 | 0.4459 | 0.3764 | 0.0777 | 0.0583 | 0.4483 | 0.3849 |
| TruFor ( Guillaro et al., 2023 ) | 0.7626 | 0.6862 | 0.6278 | 0.5540 | 0.3195 | 0.2636 | 0.3733 | 0.3101 | 0.1160 | 0.0891 | 0.4398 | 0.3806 |
| APSC-Net ( Qu et al., 2024 ) | 0.7556 | 0.6909 | 0.6110 | 0.5464 | 0.3287 | 0.2809 | 0.4256 | 0.3690 | 0.1181 | 0.0949 | 0.4478 | 0.3964 |
Table 1. Quantitative comparison of pixel-level localization performance on the OpenSDI benchmark. Pixel F1 and IoU are reported for each diffusion generator, and Average denotes the mean performance across all generators. Bold indicates the best result, and underline indicates the second-best result.
| Method | SD1.5 | SD2.1 | SDXL | SD3 | Flux.1 | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | |
| CNNDet ( Wang et al., 2020 ) | 0.8460 | 0.8504 | 0.7156 | 0.7594 | 0.5970 | 0.6872 | 0.5627 | 0.6708 | 0.3572 | 0.5757 | 0.6157 | 0.7087 |
| GramNet ( Liu et al., 2020 ) | 0.8051 | 0.8035 | 0.7401 | 0.7666 | 0.6528 | 0.7076 | 0.6435 | 0.7029 | 0.5200 | 0.6337 | 0.6723 | 0.7229 |
| UniFD ( Ojha et al., 2023 ) | 0.7745 | 0.7760 | 0.8062 | 0.8192 | 0.7074 | 0.7483 | 0.7109 | 0.7517 | 0.6110 | 0.6906 | 0.7220 | 0.7572 |
| FreqNet ( Tan et al., 2024a ) | 0.7588 | 0.7770 | 0.6097 | 0.6837 | 0.5315 | 0.6402 | 0.5350 | 0.6437 | 0.3847 | 0.5708 | 0.5639 | 0.6631 |
| NPR ( Tan et al., 2024b ) | 0.7941 | 0.7928 | 0.8167 | 0.8184 | 0.7212 | 0.7428 | 0.7343 | 0.7547 | 0.6762 | 0.7136 | 0.7485 | 0.7645 |
Table 2. Quantitative comparison of image-level detection performance on the OpenSDI benchmark. Image F1 and Accuracy are reported for each diffusion generator, and Average denotes the mean performance across all generators. The upper group contains image-level detection methods, and the lower group contains pixel-level localization methods. Bold indicates the best result, and underline indicates the second-best result.
| Method | CASIAv1 | Columbia | Coverage | NIST16 | Average | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| F1 | AUC | F1 | AUC | F1 | AUC | F1 | AUC | F1 | AUC | |
| MVSS-Net ( Chen et al., 2021 ) | 0.5775 | 0.9117 | 0.7274 | 0.9237 | 0.4663 | 0.8632 | 0.3301 | 0.7883 | 0.5253 | 0.8717 |
| PSCC-Net ( Liu et al., 2022a ) | 0.5690 | 0.9188 | 0.8555 | 0.9458 | 0.4034 | 0.8838 | 0.3655 | 0.8279 | 0.5484 | 0.8941 |
| CAT-Net ( Kwon et al., 2022 ) | 0.8081 | 0.9804 | 0.9150 | 0.9457 | 0.4273 | 0.9168 | 0.2522 | 0.8216 | 0.6007 | 0.9161 |
| TruFor ( Guillaro et al., 2023 ) | 0.8208 | 0.9742 | 0.8754 | 0.8996 | 0.4506 | 0.9424 | 0.3359 | 0.8782 | 0.6207 | 0.9236 |
| APSC-Net ( Qu et al., 2024 ) | 0.8494 | 0.9772 | 0.9299 | 0.9558 | 0.4539 | 0.9284 | 0.2597 | 0.8535 | 0.6232 | 0.9287 |
Table 3. Quantitative comparison of pixel-level localization performance on four traditional manipulation benchmarks. Pixel F1 and AUC are reported for each dataset, and Average denotes the mean performance across all evaluated datasets. Bold indicates the best result, and underline indicates the second-best result.
Figure 4. Qualitative comparison of localization results on diffusion-generated and traditional manipulation datasets. Existing methods often suffer from missed detections or over-segmentation, whereas the proposed framework produces more complete masks and more precise boundaries across diverse manipulation types. A grid of images demonstrating qualitative localization results. The rows show different image samples from diffusion-generated and traditional manipulation datasets. The columns display the original image, ground truth mask, and predicted masks from various baseline methods and the proposed framework. The baselines frequently exhibit severe missed detections (showing blank masks) or over-segmentation (showing noisy false positives), whereas the proposed framework consistently produces precise and complete localization masks that closely align with the ground truth.
Figure 5. Robustness comparison of different methods under Gaussian blur and JPEG compression. The first row reports results on OpenSDI with SD1.5 and SD3, while the second row reports results on the IML benchmark with CASIAv1 and Coverage. Pixel-level F1 is used for evaluation, and higher values indicate better robustness. Four line graphs evaluating robustness against Gaussian blur and JPEG compression at varying intensity levels. The x-axis represents degradation severity, and the y-axis represents the Pixel-level F1 score. The top row shows results on the OpenSDI dataset (SD1.5 and SD3), and the bottom row shows results on traditional datasets (CASIAv1 and Coverage). In all four graphs, the curve representing the proposed framework starts higher and decays at a significantly slower rate than the curves of competing baseline models as the degradation severity increases.
| Model Variation | Pixel F1 | Pixel IoU |
|---|---|---|
| Baseline | 0.4303 | 0.3758 |
| + ADB-DCT | 0.4607 | 0.4039 |
| + PSA | 0.4917 | 0.4289 |
| + SF-SA | 0.5128 | 0.4388 |
| + PG-FMD (Full FASA) | 0.5783 | 0.5009 |
Table 4. Ablation study of main components in FASA on the OpenSDI benchmark (average performance).
More in this topic
Image Forgery Localization
Image forgery localization identifies manipulated or AI-generated regions in images, often alongside image-level authenticity detection.
See all 79 papers