Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications, and Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China. · Visual Computing Center, King Abdullah University of Science and Technology, Jeddah, Saudi Arabia. · China Telecom Artificial Intelligence Technology Co. Ltd, Beijing 100034, China.
Semantic segmentation takes pivotal roles in various applications such as autonomous driving and medical image analysis. When deploying segmentation models in practice, it is critical to test their behaviors in varied and complex scenes in advance. In this paper, we construct an automatic data generation pipeline Gen4Seg to stress-test semantic segmentation models by generating various challenging samples with different attribute changes. Beyond previous evaluation paradigms focusing solely on global weather and style transfer, we investigate variations in both appearance and geometry attributes at the object and image level. These include object color, material, size, position, as well as image-level variations such as weather and style. To achieve this, we propose to edit visual attributes of existing real images with precise control of structural information, empowered by diffusion models. In this way, the existing segmentation labels can be reused for the edited images, which greatly reduces the labor costs. Using our pipeline, we construct two new benchmarks, Pascal-EA and COCO-EA. We benchmark a wide variety of semantic segmentation models, spanning from closed-set models to open-vocabulary large models. We have several key findings: 1) advanced open-vocabulary models do not exhibit greater robustness compared to closed-set methods under geometric variations; 2) data augmentation techniques, such as CutOut and CutMix, are limited in enhancing robustness against appearance variations; 3) our pipeline can also be employed as a data augmentation tool and improve both in-distribution and out-of-distribution performances. Our work suggests the potential of generative models as effective tools for automatically analyzing segmentation models, and we hope our findings will assist practitioners and researchers in developing more robust and reliable segmentation models.
Figures & tables
Fig. 1 : Gen4Seg expands existing images with various appearance and geometry attributes changes for evaluating the robustness of semantic segmentation models.
Fig. 2 : Our Gen4Seg generates appearance and geometry attributes variations to existing images, with collaboration of diffusion models, VLMs and LLMs. For appearance editing, we employ a VLM to edit text description of target object, and use proposed Mask-Guidance Diffusion model with ControlNet and designed energy function. For geometry editing, we first manipulate (move, rescale) the intent edit object, and use a VLM and ControlNet to recover background content by a diffusion-based inpainting model. At last, we apply established metrics to discard low-quality images and further employ a surrogate segmentation model to eliminate harmful regions from evaluation.
Fig. 3 : Illustration of our Mask-Guidance Diffusion. We propose the Mask-Guidance Energy function to optimize cross-attention maps region of target edit token.
Fig. 4 : Illustration of our Geometry Attributes Editing pipeline. We first apply a rigid transform, then employ a diffusion inpainting model and ControlNet [ 18 ] .
Fig. 5 : Examples of our generated datasets Pascal-EA and COCO-EA with diverse appearance and geometry variations.
Type
Snow
Rain
Fog
Night
CLIP Acc ( ↑)
FID ( ↓ )
CLIP Acc ( ↑)
FID ( ↓ )
CLIP Acc ( ↑ )
FID ( ↓ )
CLIP Acc ( ↑ )
FID ( ↓ )
ACDC [ICCV2021] [ 3 ]
real
1.000
0.000
1.000
0.000
1.000
0.000
1.000
0.000
Multi-weather [ICCV2021] [ 5 ]
generative
0.163
197.48
0.852
189.85
-
-
0.906
154.30
Fog Cityscapes [IJCV2018] [ 20 ]
simulator
-
-
-
-
0.940
164.00
-
-
SHIFT [CVPR2022] [ 4 ]
simulator
-
-
0.980
242.28
0.955
272.58
0.914
274.45
GenVal [ECCV2024] [ 22 ]
generative
0.770
164.59
0.870
172.04
0.896
155.09
0.713
181.57
TABLE I : Comparison of our generated images with previous benchmarks under four adverse weather conditions. “ generative ” and “ simulator ” refer to the images generated by generative methods and the simulator, respectively. The symbol “-” indicates that the benchmark does not support the required setting or functionality.
Fig. 6 : Qualitative Comparison of GenVal [ 22 ] and Ours. We can better maintain the spatial structures of original scenes.
Method
Backbone
Pascal-EA
COCO-EA
Original
Recon
Color
Material
Weather
Style
mR
Original
Recon
Color
Material
Weather
Style
mR
PSPNet [CVPR2017] [ 73 ]
ResNet50
77.02
76.10
72.00
45.14
69.97
63.83
0.82
36.20
35.54
32.8
29.62
30.25
27.23
0.84
DeepLabV3 [CVPR2018] [ 74 ]
ResNet50
75.33
74.26
71.66
45.49
69.21
63.7
0.84
36.98
36.25
33.68
30.68
31.28
28.27
0.85
GCNet [TPAMI2020] [ 75 ]
ResNet50
76.78
75.53
70.83
43.09
69.19
61.79
0.81
37.59
37.00
35.07
29.65
28.34
26.03
0.80
OCRNet [ECCV2020] [ 76 ]
HRNet48
78.29
77.00
73.16
49.37
73.69
69.08
0.86
40.55
39.74
37.69
33.94
30.04
29.16
0.82
Segmenter [ICCV2021] [ 77 ]
ViT-B/16
82.50
81.19
78.69
60.29
78.43
77.04
0.91
46.93
46.38
45.28
42.64
41.75
39.39
0.91
TABLE II : Benchmarking the robustness of 13 state-of-the-art methods under four appearance attributes changes on our Pascal-EA and COCO-EA datasets. The results are reported using mIoU ( ↑ ). Recon denotes results on reconstructed images by our Gen4Seg without any attribute variations. We also report the average mIoU on the last row.
Method
Backbone
Pascal-EA
COCO-EA
Original
Size 0.2
Size 0.4
Position 0.2
Position 0.4
mR
Original
Size 0.2
Size 0.4
Position 0.2
Position 0.4
mR
PSPNet [CVPR2017] [ 73 ]
ResNet50
67.41
64.97
62.83
64.98
63.67
0.95
22.18
20.89
19.75
21.44
20.87
0.93
DeepLabV3+ [CVPR2018] [ 85 ]
ResNet50
66.69
64.12
62.37
64.68
62.98
0.95
22.07
20.67
19.64
21.45
20.80
0.94
GCNet [TPAMI2020] [ 75 ]
ResNet50
67.85
65.69
63.60
64.95
63.5
0.95
22.78
21.11
19.46
21.74
21.36
0.92
OCRNet [ECCV2020] [ 76 ]
HRNet48
68.15
67.26
65.33
66.61
65.08
0.97
23.03
21.22
19.50
21.01
20.68
0.89
Segmenter [ICCV2021] [ 77 ]
ViT-B/16
70.87
70.34
69.54
69.58
68.25
0.97
23.65
22.02
19.71
22.81
22.15
0.92
TABLE III : Benchmarking the robustness of 13 state-of-the-art methods under two object geometry attribute changes on our Pascal-EA and COCO-EA datasets. The results are reported using mIoU ( ↑ ).
Method
Appearance
Geometry
Recon
Color
Material
Weather
Style
mR
Original
Size 0.2
Size 0.4
Position 0.2
Position 0.4
mR
Mask2Former [CVPR2022] [ 80 ]
87.52
85.35
61.52
85.30
79.09
0.89
77.11
73.56
71.92
72.87
72.04
0.94
+CutOut [Arxiv2017] [ 89 ]
89.56
84.20
48.47
85.17
75.68
0.82
76.82
74.98
72.44
72.58
71.61
0.94
+CutMix [ICCV2019] [ 90 ]
90.64
86.44
52.42
84.34
77.51
0.83
77.56
76.62
74.66
77.01
74.47
0.98
+Hendrycks et. al. [ICLR2019] [ 92 ]
91.19
83.27
55.63
86.21
80.05
0.84
75.90
72.45
70.89
71.14
68.93
0.93
+AugMix [ICLR2020] [ 91 ]
91.58
85.54
56.79
87.03
80.77
0.84
75.71
73.9
71.25
71.33
69.38
0.94
TABLE IV : mIoU ( ↑ ) of four augmentation algorithms on our Pascal-EA. The best results are in bold.
HRNet [ 100 ]
Segformer [ 78 ]
Method
Cityscapes
Rain
Fog
Snow
Night
Avg.
Cityscapes
Rain
Fog
Snow
Night
Avg.
Baseline †
70.47
44.15
58.68
44.20
18.90
41.48
67.90
50.22
60.52
48.86
28.56
47.04
CutOut [Arxiv2017] [ 89 ]
71.39
40.29
57.70
43.98
16.55
39.63
68.93
47.68
60.34
46.98
26.49
45.37
CutMix † [ICCV2019] [ 90 ]
72.68
42.48
58.63
44.50
17.07
40.67
69.23
49.53
61.58
47.42
27.77
46.57
Hendrycks et. al. † [ICLR2019] [ 92 ]
69.25
50.78
60.82
38.34
22.82
43.19
67.41
54.02
64.74
49.57
28.50
49.21
StyleMix † [CVPR2021] [ 101 ]
57.40
40.59
49.11
39.14
19.34
37.04
65.30
53.54
63.86
49.98
28.93
49.08
TABLE V : Comparison of different methods from Cityscapes [ 70 ] (source) to ACDC [ 3 ] (target) using the mIoU ( ↑ ) metric. The results are reported on the Cityscapes validation set, four individual scenarios of ACDC, and the average (Avg). † means results are directly from [ 99 ] . Oracle indicates the supervised training, serving as performance upper bounds.
Mask2Former
+CutOut
+CutMix
+AugMix
+ Our
Pascal VOC
87.52
89.56
90.64
91.58
91.77
COCO Stuff
45.06
45.98
46.52
46.84
47.03
TABLE VI : The results of different data augmentation techniques on Pascal VOC [ 68 ] and COCO-Stuff 164k [ 102 ] .
Fig. 7 : Qualitative results of our method and existing text-driven image editing methods.
Method
Material
Color
DINO Dist ( ↓ )
Masked LPIPS ( ↓ )
DINO Dist ( ↓ )
Masked LPIPS ( ↓ )
BLD [SIGGRAPH2023] [ 106 ]
0.081
0.413
0.092
0.384
DiffEdit [ICLR2023] [ 105 ]
0.068
0.407
0.061
0.372
InstructPix2Pix [CVPR2022] [ 104 ]
0.059
0.393
0.043
0.359
P2P [CVPR2023] [ 15 ]
0.038
0.255
0.014
0.272
PnP [CVPR2023] [ 16 ]
0.047
0.329
0.021
0.295
TABLE VII : Quantitative results of existing image editing models. We compare them with our method by editing object color and material on the Pascal VOC [ 68 ] dataset. MGD is the conference version of our method.
Fig. 8 : Distributions measured by GradNorm [ 107 ] of in-distribution images (Pascal VOC [ 68 ] dataset) and counterparts with object size and position changes, generated by our method and the ImageNet-E [ 11 ] .
Fig. 9 : Visualization of cross-attention maps. There is considerable improvement after applying our strategies.
Our Strategies
DINO Dist ( ↓ )
CLIP Acc ( ↑ )
Loss (Eq. 5 )
Blend (Eq. 7 )
ControlNet
0.052
0.59
✓
0.036
0.62
✓
0.010
0.63
✓
0.048
0.61
✓
✓
0.007
0.70
TABLE VIII : Ablation results of appearance attributes editing.
Our Strategies
FID ( ↓ )
LPIPS ( ↓ )
DINO Dist ( ↓ )
VLM [ 54 ]
ControlNet [ 18 ]
36.08
0.478
0.045
✓
27.92
0.334
0.022
✓
27.55
0.346
0.028
✓
✓
24.60
0.295
0.016
TABLE IX : Ablation results of geometry attributes editing.
Fig. 10 : Visualization of synthetic images and the filtered regions (white masks in the label). The black masks are denoted as “background” category.
TABLE I : Summary of existing benchmarks for semantic segmentation robustness evaluation. Our benchmarks cover both global and local variations. We extend geometry attributes compared to our conference version.
Attribute
Instruction prompt
Examples
Color
I want to change the color of the object in the source image. Please generate all possible target text prompts given the source text prompt describing the source image. For example, the source is “a cat”, you can generate “a blue cat”.
Input: a white and red train. Output: a white and blue train.
Material
I want to change the material of the object in the source image. Please generate all possible target text prompts given the source text prompt describing the source image. For example, source is “a cat”, you can generate “a wooden cat sculpture”.
Input: an airplane on the ground. Output: a wooden airplane model on the ground.
Style:
I want to change the image style of source images without perturbing the content. Please generate all possible target text prompts given the source text prompt describing the source image. For example, source is ’a cat’, you can generate ’a watercolor cat’.
Input: A brown leather couch is sitting outside on a sidewalk. Output: A digital painting of a brown leather couch is sitting outside on a sidewalk.
Weather:
I want to change the weather or season condition of the source image. Please generate all possible target text prompts given the source text prompt that describes the source image, only changing the weather conditions, or adding a description of the weather if not already present.
Input: Two sheep lying in the grass. Output: Two sheep laying in the grass under a heavy downpour .
Appendix
TABLE II : Illustrations of text prompts used to instruct LLaMA3 [ 57 ] and edited text examples.
Fig. 1 : Average mIoU ( ↑ ) of segmentation methods under two different attribute variations and their combinations.
Method
Backbone
Pascal-EA
COCO-EA
Original
Size 0.6
Size 0.8
mR
Original
Size 0.6
Size 0.8
mR
DeepLabV3+ [CVPR2018] [ 85 ]
ResNet50
66.69
30.21
10.25
0.30
22.07
17.12
10.45
0.62
Segmenter [ICCV2021] [ 77 ]
ViT-B/16
70.87
35.58
12.79
0.34
23.65
15.29
11.79
0.57
Mask2Former [CVPR2022] [ 80 ]
Swin-B
77.11
36.70
18.22
0.36
21.81
13.99
10.33
0.56
OVSeg [CVPR2023] [ 81 ]
Swin-B
79.46
40.29
20.85
0.38
23.42
15.79
11.74
0.59
SEEM [NeurIPS2023] [ 84 ]
Focal-T
76.24
37.29
20.53
0.38
27.38
16.38
13.22
0.54
Appendix
TABLE III : Benchmarking the robustness of several methods under greater object size attribute changes on our Pascal-EA and COCO-EA datasets. The results are reported using mIoU ( ↑ ).
Method
PASCAL VOC
COCO Stuff 164k
Filtered/Total
CLIP Accuracy
Filtered/Total
CLIP Accuracy
UperNet [ 110 ]
14/200
0.879
12/200
0.882
OCRNet [ 76 ]
14/200
0.871
12/200
0.890
Segmenter [ 77 ]
19/200
0.884
15/200
0.896
Mask2Former [ 80 ]
23/200
0.903
18/200
0.915
Appendix
TABLE IV : Comparison of different surrogate models used for noise filtering.
Model
CLIP Acc ( ↑ )
DINO Dist ( ↓ )
Stylized-Object [ECCV2022] [ 6 ]
AdaIN
0.574
0.075
Ours
Diffusion
0.906
0.002
Stylized-Background [ECCV2022] [ 6 ]
AdaIN
0.678
0.084
Ours
Diffusion
0.975
0.004
Appendix
TABLE V : Comparison of our pipeline with Stylized COCO [ 6 ] in local object and background changes. Our generated images have superior image reality and fidelity.
Caption
Length (#tokens)
Appearance
Geometry
DINO Dist( ↓ )
Masked LPIPS( ↓ )
CLIP Score( ↑ )
FID( ↓ )
HPSv2( ↑ )
Short
≈ 12
0.109
0.192
0.512
16.84
22.79
Medium
≈ 30
0.118
0.207
0.395
17.92
19.38
Long
≈ 60
0.242
0.346
0.171
35.38
18.26
Appendix
TABLE VI : Quantitative results of edit performance under different prompt lengths. The best results are in bold.
Fig. 2 : Two examples of captions with varying levels of detail. From top to bottom are short, medium, and long captions, respectively.
Fig. 3 : Visualization results under appearance variations on our proposed Pascal-EA benchmarks.
Fig. 4 : Visualization results under appearance variations on our proposed Pascal-EA benchmarks.
Fig. 5 : Visualization results under object geometry (size and position) variations.
Collecting and annotating datasets for pixel-level semantic segmentation tasks are highly labor-intensive. Data augmentation provides a viable solution by enhancing model generalization without additional real-world data collection. Traditional augmentation techniques, such as translation, scaling, and color transformations, create geometric variations but fail to generate new structures. While generative models have been employed to extend semantic information of datasets, they often struggle to maintain consistency between the original and generated images, particularly for pixel-level tasks. In this work, we propose a novel synthetic data augmentation pipeline that integrates controllable diffusion models. Our approach balances diversity and reliability data, effectively bridging the gap between synthetic and real data. We utilize class-aware prompting and visual prior blending to improve image quality further, ensuring precise alignment with segmentation labels. By evaluating benchmark datasets such as PASCAL VOC and BDD100K, we demonstrate that our method significantly enhances semantic segmentation performance, especially in data-scarce scenarios, while improving model robustness in real-world applications. Our code is available at https://github.com/chequanghuy/Enhanced-Generative-Data-Augmentation-for-Semantic-Segmentation-via-Stronger-Guidance.
Huy Che, Dinh-Duy Phan, Duc-Khai Lam
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
Data augmentation is a standard component of modern semantic segmentation pipelines, but most augmentation techniques allocate transformations uniformly across training samples or adapt to a single difficulty signal such as loss. This ignores the fact that segmentation difficulty is multi-factorial, since ambiguous predictions, persistent optimization errors, rare classes, and complex object boundaries can each make a sample informative in different ways. This paper introduces Difficulty-Aware Sample Allocation (DASA), an architecture-agnostic framework that assigns stronger augmentation to samples estimated to be more difficult. DASA combines prediction ambiguity, training loss, class rarity, and boundary complexity into a normalized difficulty score, then maps that score to sample-specific augmentation strength during iterative training. Experiments on Oxford-IIIT Pet and binary Pascal VOC segmentation with U-Net, DeepLabV3, and SegFormer-B0 show that DASA improves over standard training and is competitive with or stronger than single-signal adaptive baselines. On Oxford-IIIT Pet, DASA improves DeepLabV3 from 0.633 to 0.740 mIoU. On binary Pascal VOC, DASA obtains the best foreground IoU for all three evaluated architectures. These results attest to the value of multi-factor difficulty estimation as a practical mechanism for directing augmentation where it is most useful.
Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.
Haoxiao Wang, Antao Xiang, Haiyang Sun +8
Zhejiang University · South China University of Technology · Nanjing University +1