FICGen: Frequency-Inspired Contextual Disentanglement for Layout-driven Degraded Image Generation
Authors: Wenzhuang Wang, Yifan Zhao, Mingcan Ma, Ming Liu, Zhonglin Jiang, Yong Chen, Jia Li
Organizations: State Key Laboratory of Virtual Reality Technology and Systems, SCSE&QRI, Beihang University · Geely Automobile Research Institute (Ningbo) Co., Ltd
Layout-to-image (L2I) generation has exhibited promising results in natural domains, but suffers from limited generative fidelity and weak alignment with user-provided layouts when applied to degraded scenes (i.e., low-light, underwater). We primarily attribute these limitations to the "contextual illusion dilemma" in degraded conditions, where foreground instances are overwhelmed by context-dominant frequency distributions. Motivated by this, our paper proposes a new Frequency-Inspired Contextual Disentanglement Generative (FICGen) paradigm, which seeks to transfer frequency knowledge of degraded images into the latent diffusion space, thereby facilitating the rendering of degraded instances and their surroundings via contextual frequency-aware guidance. To be specific, FICGen consists of two major steps. Firstly, we introduce a learnable dual-query mechanism, each paired with a dedicated frequency resampler, to extract contextual frequency prototypes from pre-collected degraded exemplars in the training set. Secondly, a visual-frequency enhanced attention is employed to inject frequency prototypes into the degraded generation process. To alleviate the contextual illusion and attribute leakage, an instance coherence map is developed to regulate latent-space disentanglement between individual instances and their surroundings, coupled with an adaptive spatial-frequency aggregation module to reconstruct spatial-frequency mixed degraded representations. Extensive experiments on 5 benchmarks involving a variety of degraded scenarios-from severe low-light to mild blur-demonstrate that FICGen consistently surpasses existing L2I methods in terms of generative fidelity, alignment and downstream auxiliary trainability.
Figures & tables
Figure 2: Frequency spectrum comparison between natural and degraded images.
Figure 3: The overview pipeline of our FICGen , consisting of (a) Frequency Perceiver Resamplers, (b) Visual-Frequency Enhanced Attention, and (c) Adaptive Spatial-Frequency Aggregation.
Detector
Method
FID ↓
mAP ↑
AP_50 ↑
AP_75 ↑
DIOR-H [ 15 ]
Faster R-CNN
Oracle
-
33.4
55.6
35.0
Faster R-CNN
MIGC [ 60 ]
31.64
21.8
38.4
17.5
Faster R-CNN
CC-Diff [ 58 ]
30.88
23.6
42.4
21.4
Faster R-CNN
FICGen (ours)
31.25
27.6
48.7
27.6
RUOD [ 7 ]
Table 1: Quantitative comparison of fidelity and alignment on various degraded conditions. The performance is evaluated by off-the-shelf detectors Faster R-CNN [ 32 ] and Cascade R-CNN [ 1 ] on synthetic test set. “ Oracle ” denotes the real test baseline. The best results are in red and the second best results are in underline .
Figure 4: Qualitative comparison of degraded images generated by different L2I methods across various scenarios. Zoom in for better details.
Method
FID ↓
YOLO Score ↑
mAP
AP_50
AP_75
Oracle
-
39.2
57.1
43.0
LostGAN [ 39 ]
57.10
14.3
27.3
15.2
LayoutDiffusion [ 59 ]
45.31
20.0
37.4
19.3
ReCo [ 52 ]
42.56
21.1
40.7
23.1
MIGC [ 60 ]
31.64
24.5
41.8
25.6
Table 2: Quantitative comparison on the DIOR-H test set, with detection performance evaluated using a pre-trained YOLOv8 [ 42 ] .
Method
ExDARK [ 19 ]
RUOD [ 7 ]
mAP ↑
AP_50 ↑
AP_75 ↑
mAP ↑
AP_50 ↑
AP_75 ↑
Oracle
38.3
67.8
39.8
58.4
85.5
64.8
MIGC [ 60 ]
28.6
59.2
24.1
21.1
43.9
17.9
CC-Diff [ 58 ]
31.3
61.8
28.8
29.7
57.8
28.0
FICGen (ours)
38.5
68.5
39.5
37.1
67.1
36.7
Table 3: Quantitative comparison on ExDARK and RUOD test set, evaluated by a pre-trained Deformable-DETR [ 61 ] .
Detector
Backbone
Training Set
Object Detection (AP) for Sampled Classes /%
mAP
AP_50
AP_75
# Real
# Synthetic
bicycle
bus
motorbike
boat
car
cup
bottle
dog
Train with Full Real Data
RetinaNet
PVT-M
R:5.1 k
-
44.8
58.2
31.3
31.1
36.0
25.5
28.7
47.8
35.2
65.4
33.9
Faster R-CNN
X101
R:5.1 k
-
49.0
60.8
35.6
30.1
39.5
30.9
34.1
49.3
38.6
69.3
39.6
Train with Pure Synthetic Data
RetinaNet
PVT-M
-
MIGC (S: 5.1 k )
32.8
46.7
20.4
19.9
22.5
14.1
16.0
33.4
23.5
49.0
19.1
Table 4: Quantitative comparison of trainability on the ExDARK dataset, evaluated by RetinaNet [ 17 ] (PVT-M [ 44 ] ) and Faster R-CNN [ 32 ] (X101 [ 48 ] ) under three training settings.
Method
DIOR-H [ 15 ]
DAWN [ 14 ]
ExDARK [ 19 ]
VOC 2012 [ 6 ]
airport ↑
mAP ↑
AP_50 ↑
AP_75 ↑
bus ↑
mAP ↑
AP_50 ↑
AP_75 ↑
motorbike ↑
mAP ↑
AP_50 ↑
AP_75 ↑
boat ↑
mAP ↑
AP_50 ↑
AP_75 ↑
Oracle
32.2
33.4
55.6
35.0
21.8
24.8
46.0
23.0
33.3
35.8
67.4
34.5
42.9
48.3
76.8
52.5
MIGC [ 60 ]
36.9
34.1
56.1
36.1
21.6
24.9
43.3
24.6
34.4
37.1
67.8
36.6
42.5
49.4
77.2
54.2
CC-Diff [ 58 ]
37.6
34.3
56.1
36.3
26.2
25.8
46.1
25.8
34.1
37.0
67.7
36.6
44.4
49.6
77.3
54.3
FICGen (ours)
38.1
35.5
56.8
37.6
27.5
25.9
44.5
26.5
37.1
37.7
68.0
37.7
47.3
50.5
77.4
55.4
Table 5: Quantitative comparison of trainability on the DIOR, DAWN and ExDARK and natural VOC 2012 datasets, with detection evaluated using the Faster R-CNN (R50) object detector.
Figure 5: Visualization results generated by our FICGen on unseen geometric layouts augmented via flipping and translation.
Figure 6: Performance convergence curves when generated degraded images are incorporated into downstream detection.
Figure 7: Generalization results of FICGen on mildly blurred and natural scenes, as well as under oriented layout control.
FPR
VFEA
ASFA
mAP
AP_50
AP_75
✗
✗
✗
15.7
34.8
12.0
✗
✔
✗
23.2
44.6
22.7
✔
✗
✔
18.5
36.3
13.9
✔
✔
✗
25.5
46.3
25.1
✗
✔
✔
25.6
46.9
25.3
✔
✔
✔
27.6
48.7
27.6
Table 6: Ablation study of FPR, VFEA, and ASFA modules, evaluated by the pre-trained Faster R-CNN (R50) on the DIOR test set.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Train
Test
Total
Instances
Classes
Degradation mode
ExDARK [ 19 ]
5,145
2,218
7,363
23,710
12
low-light
RUOD [ 7 ]
9,800
4,200
14,000
74,904
10
underwater
DIOR [ 15 ]
5,862
11,738
17,600
192,472
20
aerial
DAWN [ 14 ]
5,544
410
5,954
43,869
5
adverse weather
VOC 2012 [ 6 ]
10,582
1,449
12,031
33,149
20
blur
Appendix
Table 7: An overview of five degraded datasets.
Detector
Method
FID ↓
Object Detection (AP) for Sampled Classes /%
mAP ↑
AP_50 ↑
AP_75 ↑
DIOR-H [ 15 ]
windmill
airport
stadium
ballfield
airplane
golffield
Faster R-CNN
Oracle
-
29.0
32.2
26.7
50.2
33.5
34.7
33.4
55.6
35.0
Faster R-CNN
MIGC [ 60 ]
31.64
3.4
33.6
18.0
32.4
6.8
43.8
21.8
38.4
17.5
Faster R-CNN
CC-Diff [ 58 ]
30.88
5.6
31.4
21.4
44.0
9.5
50.6
23.6
42.4
21.4
Faster R-CNN
FICGen (ours)
31.25
16.0
33.5
16.9
50.0
23.8
51.2
27.6
48.7
27.6
RUOD [ 7 ]
starfish
echinus
fish
scallop
corals
turtle
Appendix
Table 8: Quantitative comparison of generative fidelity and alignment on five degraded datasets (DIOR-H [ 15 ] , ExDARK [ 19 ] , RUOD [ 7 ] , DAWN [ 14 ] , and blurred VOC 2012 [ 6 ] ) and one natural image dataset (VOC 2012 [ 6 ] ). The performance is evaluated using off-the-shelf detectors Faster R-CNN (R50) [ 32 ] and Cascade R-CNN (R50) [ 1 ] on synthetic test images generated by different L2I methods. “ Oracle ” denotes the real test set baseline ( i.e . , upper bound). The top-2 performers are marked in red and underlined .
Figure 8: Qualitative comparison of natural images (VOC 2012 [ 6 ] ) generated by different L2I methods. Zoom in for more detail.
category
Oracle
MIGC [ 60 ]
CC-Diff [ 58 ]
FICGen
category
Oracle
MIGC [ 60 ]
CC-Diff [ 58 ]
FICGen
bicycle
46.4
48.3
49.6
47.8
chair
25.4
26.8
26.8
27.5
boat
30.2
31.1
29.7
31.7
cup
28.6
28.2
28.7
29.6
bottle
31.8
31.6
31.1
32.8
dog
44.1
47.9
47.6
48.5
bus
56.8
60.8
59.8
59.9
motorbike
33.3
34.4
34.1
37.1
car
38.1
38.3
38.4
38.8
people
32.9
32.5
32.4
34.0
cat
41.8
41.4
42.0
42.4
table
20.6
24.1
24.1
22.5
Appendix
Table 9: Detection accuracy (%) per class on the ExDARK test set using Faster R-CNN (R50) [ 32 ] , trained on 5.1k real and 5.1k synthetic images generated by different L2I methods.
category
Oracle
MIGC [ 60 ]
CC-Diff [ 58 ]
FICGen
motorcycle
18.0
23.0
20.0
22.9
person
19.6
19.5
19.5
20.2
bus
21.8
21.6
27.8
26.2
truck
23.4
19.6
21.6
20.5
car
41.1
39.4
40.0
39.6
All ( mAP % )
24.8
24.6
25.8
25.9
Appendix
Table 10: Detection accuracy (%) per class on the DAWN test set using Faster R-CNN (R50) [ 32 ] , trained on 5.5k real and 5.5k synthetic images generated by different L2I methods.
category
Oracle
MIGC [ 60 ]
CC-Diff [ 58 ]
FICGen
category
Oracle
MIGC [ 60 ]
CC-Diff [ 58 ]
FICGen
aeroplane
57.1
58.0
59.6
59.2
diningtable
42.8
44.5
42.3
45.2
bicycle
51.6
54.3
53.1
53.6
dog
59.6
59.8
60.9
61.9
bird
52.2
53.6
54.1
55.7
horse
54.9
55.7
55.2
55.4
boat
42.9
42.5
44.4
47.3
motorbike
56.2
58.7
59.1
57.8
bottle
33.2
34.7
36.3
34.9
person
48.1
47.9
48.1
48.9
bus
59.4
60.1
60.1
62.2
pottedplant
26.9
25.8
26.0
27.9
Appendix
Table 11: Detection accuracy (%) per class on natural VOC 2012 test set using Faster R-CNN (R50) [ 32 ] , trained on 10.6k real and 10.6k synthetic images generated by different L2I methods.
Dataset
Method
Sparse ( 33% / 39% / 23% / 27% )
Partial ( 21% / 15% / 18% / 18% )
Heavy ( 46% / 46% / 59% / 55% )
mAP
AP_50
AP_75
mAP
AP_50
AP_75
mAP
AP_50
AP_75
DIOR-H [ 15 ]
Oracle
45.9
70.2
50.9
35.2
53.3
38.6
28.7
47.6
30.7
MIGC [ 60 ]
33.1
64.4
30.7
19.1
38.8
16.7
16.1
33.9
12.9
CC-Diff [ 58 ]
38.7
66.3
40.7
24.4
42.9
25.0
19.5
36.5
19.1
FICGen
44.8
72.4
48.8
30.1
49.2
32.5
24.1
42.6
24.9
ExDARK [ 19 ]
Oracle
55.6
87.4
65.7
42.6
73.4
46.3
28.7
58.6
25.3
Appendix
Table 12: Quantitative comparison of alignment across four datasets under three occlusion degrees, with detection evaluated by Faster R-CNN (R50). The numbers in brackets represent the proportion of each occlusion level within the dataset.
Method
Trainable Params (M)
Inf.Time (s/img)
Inf. Mem (GB)
mAP
CC-Diff
∼ 261
8
∼ 12.2
26.4
AeroGen
∼ 905
8
∼ 13.9
29.8
FICGen (ours)
∼ 304
10
∼ 12.6
31.2
Appendix
Table 13: Comparison of trainable parameters and inference efficiency with AeroGen [ 41 ] and CC-Diff [ 58 ] .
Figure 9: Further qualitative comparison with CC-Diff on ExDARK and generalization to oriented layout inputs [ 4 ] .
Figure 10: Visualization results generated by FICGen conditioned on augmented geometric layouts.
Figure 11: Visualization results of the continuous generation of instance interactivity ( i.e . , addition, removal, transform and weather change) by our FICGen on the DAWN dataset.
Figure 12: Architectural details of the proposed Adaptive Spatial-Frequency Aggregation Module (ASFA).
Fusion
airplane
mAP
AP_50
AP_75
Depth-wise
23.8
27.6
48.7
27.6
Point-wise
16.0
26.4
48.1
26.4
Appendix
Table 14: The ablation results of different fusion strategies for SAM and FAM outputs.
Figure 13: Detection results of Faster R-CNN (R50) trained on real images vs. real & FICGen-generated images.
Figure 14: Failure cases of FICGen, where red dashed boxes denote the missing instances.
Figure 15: Visualization results generated by our FICGen under the mild blur (blurred VOC 2012), with geometric layouts and corresponding object categories superimposed on the generated images.
Figure 16: Visualization results generated by our FICGen under low-light conditions (ExDARK [ 19 ] ), with geometric layouts and corresponding object categories superimposed on the generated images.
Figure 17: Visualization results generated by our FICGen under the underwater scene (RUOD [ 7 ] ), with geometric layouts and corresponding object categories superimposed on the generated images.
Figure 18: Visualization results generated by our FICGen under the remote sensing scene (DIOR [ 15 ] ), with geometric layouts and corresponding object categories superimposed on the generated images.
Figure 19: Visualization results generated by our FICGen under the adverse weather condition (DAWN [ 14 ] ), with geometric layouts and corresponding object categories superimposed on the generated images.
The layout-to-image (L2I) task enables fine-grained control over image generation via object categories and spatial layouts. However, existing L2I methods yield fragmented and distorted generations under few-shot atypical settings. We term this failure as representation fragmentation, arising from a granularity mismatch that entangles semantic identity with visual details. To address this issue, we propose a representation-driven framework that disentangles semantics from primitives for robust few-shot adaptation. Specifically, Semantic Anchoring aggregates categorical semantics into anchors for stable identity, while Primitive Imbuing models recomposable primitives for robust local detail modeling. Conceptual Steering further regulates optimization with a saliency-aware objective to preserve foreground semantic consistency. Extensive experiments demonstrate consistent improvements in the 5-shot regime over state-of-the-art L2I methods in both visual fidelity and alignment across diverse atypical domains. The source code is publicly available at https://github.com/iCVTEAM/DSP.
Nan Bao, Yifan Zhao, Wenzhuang Wang +1
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering and Qingdao Research Institute, Beihang University, China.
Pixel-space diffusion has re-emerged as a promising alternative to latent-space generation because it avoids the representation bottleneck introduced by VAEs. Yet most existing methods still treat image generation as a frequency-homogeneous process, overlooking the distinct roles and learning dynamics of low- and high-frequency components. To address this, we propose FREPix, a FREquency-heterogeneous flow matching framework for Pixel-space image generation. FREPix explicitly decomposes generation into low- and high-frequency components, assigns them separate transport paths, predicts them with a factorized network, and trains them with a frequency-aware objective. In this way, coarse-to-fine generation becomes an explicit design principle rather than an implicit behavior. On ImageNet class-to-image generation, FREPix achieves competitive results among pixel-space generation models, reaching 1.91 FID at 256×256 and 2.38 FID at 512×512, with particularly strong behavior in the low-NFE regime.
Different visual patterns appear with different frequencies in the world: e.g., beach balls appear on sand more often than they do on a road. These statistics are reflected in vision datasets, and as a result trained models more easily recognize objects in common scenarios. However, recognizing a beach ball on a road may arguably be even more important than recognizing it on sand. We study how to mitigate this discrepancy. Since collecting uncommon images in the real world may be difficult, we explore whether generating images with less frequent contexts can serve as effective training augmentation. A key challenge is guiding generations to remain close to the original dataset distribution while creating diverse images with uncommon contexts. We introduce Decoupling Contextual Patterns with Generations (DecoupleGen), a method that personalizes text-to-image diffusion models to facilitate coherent synthesis of images with rare contexts while preserving original visual details. The generated images contain semantically meaningful content and remain visually aligned with the original datasets. We further apply verification constraints to ensure relevance of the augmented data. We evaluate our approach on object classification and recognition tasks on complex scene datasets. Our experiments demonstrate consistent improvements over previous approaches, and our analyses identify factors underlying these improvements.
Xinran Liang, Esin Tureci, Prachi Sinha +3
Department of Computer Science, Princeton University · LIX, CNRS, ´Ecole Polytechnique