Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing

Zijin Yin; Bing Li; Kongming Liang; Hao Sun; Zhongjiang He; Zhanyu Ma; Jun Guo

cs.CVMar 2, 2026

Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing

Authors: Zijin Yin, Bing Li, Kongming Liang, Hao Sun, Zhongjiang He, Zhanyu Ma, Jun Guo

Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications, and Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China. · Visual Computing Center, King Abdullah University of Science and Technology, Jeddah, Saudi Arabia. · China Telecom Artificial Intelligence Technology Co. Ltd, Beijing 100034, China.

Abstract

Semantic segmentation takes pivotal roles in various applications such as autonomous driving and medical image analysis. When deploying segmentation models in practice, it is critical to test their behaviors in varied and complex scenes in advance. In this paper, we construct an automatic data generation pipeline Gen4Seg to stress-test semantic segmentation models by generating various challenging samples with different attribute changes. Beyond previous evaluation paradigms focusing solely on global weather and style transfer, we investigate variations in both appearance and geometry attributes at the object and image level. These include object color, material, size, position, as well as image-level variations such as weather and style. To achieve this, we propose to edit visual attributes of existing real images with precise control of structural information, empowered by diffusion models. In this way, the existing segmentation labels can be reused for the edited images, which greatly reduces the labor costs. Using our pipeline, we construct two new benchmarks, Pascal-EA and COCO-EA. We benchmark a wide variety of semantic segmentation models, spanning from closed-set models to open-vocabulary large models. We have several key findings: 1) advanced open-vocabulary models do not exhibit greater robustness compared to closed-set methods under geometric variations; 2) data augmentation techniques, such as CutOut and CutMix, are limited in enhancing robustness against appearance variations; 3) our pipeline can also be employed as a data augmentation tool and improve both in-distribution and out-of-distribution performances. Our work suggests the potential of generative models as effective tools for automatically analyzing segmentation models, and we hope our findings will assist practitioners and researchers in developing more robust and reliable segmentation models.

Figures & tables

Fig. 1 : Gen4Seg expands existing images with various appearance and geometry attributes changes for evaluating the robustness of semantic segmentation models.

Fig. 2 : Our Gen4Seg generates appearance and geometry attributes variations to existing images, with collaboration of diffusion models, VLMs and LLMs. For appearance editing, we employ a VLM to edit text description of target object, and use proposed Mask-Guidance Diffusion model with ControlNet and designed energy function. For geometry editing, we first manipulate (move, rescale) the intent edit object, and use a VLM and ControlNet to recover background content by a diffusion-based inpainting model. At last, we apply established metrics to discard low-quality images and further employ a surrogate segmentation model to eliminate harmful regions from evaluation.

Fig. 3 : Illustration of our Mask-Guidance Diffusion. We propose the Mask-Guidance Energy function to optimize cross-attention maps region of target edit token.

Fig. 4 : Illustration of our Geometry Attributes Editing pipeline. We first apply a rigid transform, then employ a diffusion inpainting model and ControlNet [ 18 ] .

Fig. 5 : Examples of our generated datasets Pascal-EA and COCO-EA with diverse appearance and geometry variations.

	Type	Snow		Rain		Fog		Night
	Type	CLIP Acc ( $\uparrow)$	FID ( $\downarrow$ )	CLIP Acc ( $\uparrow)$	FID ( $\downarrow$ )	CLIP Acc ( $\uparrow$ )	FID ( $\downarrow$ )	CLIP Acc ( $\uparrow$ )	FID ( $\downarrow$ )
ACDC [ICCV2021] [ 3 ]	real	1.000	0.000	1.000	0.000	1.000	0.000	1.000	0.000
Multi-weather [ICCV2021] [ 5 ]	generative	0.163	197.48	0.852	189.85	-	-	0.906	154.30
Fog Cityscapes [IJCV2018] [ 20 ]	simulator	-	-	-	-	0.940	164.00	-	-
SHIFT [CVPR2022] [ 4 ]	simulator	-	-	0.980	242.28	0.955	272.58	0.914	274.45
GenVal [ECCV2024] [ 22 ]	generative	0.770	164.59	0.870	172.04	0.896	155.09	0.713	181.57

TABLE I : Comparison of our generated images with previous benchmarks under four adverse weather conditions. “ generative ” and “ simulator ” refer to the images generated by generative methods and the simulator, respectively. The symbol “-” indicates that the benchmark does not support the required setting or functionality.

Fig. 6 : Qualitative Comparison of GenVal [ 22 ] and Ours. We can better maintain the spatial structures of original scenes.

Method	Backbone	Pascal-EA							COCO-EA
Method	Backbone	Original	Recon	Color	Material	Weather	Style	$mR$	Original	Recon	Color	Material	Weather	Style	$mR$
PSPNet [CVPR2017] [ 73 ]	ResNet50	77.02	76.10	72.00	45.14	69.97	63.83	0.82	36.20	35.54	32.8	29.62	30.25	27.23	0.84
DeepLabV3 [CVPR2018] [ 74 ]	ResNet50	75.33	74.26	71.66	45.49	69.21	63.7	0.84	36.98	36.25	33.68	30.68	31.28	28.27	0.85
GCNet [TPAMI2020] [ 75 ]	ResNet50	76.78	75.53	70.83	43.09	69.19	61.79	0.81	37.59	37.00	35.07	29.65	28.34	26.03	0.80
OCRNet [ECCV2020] [ 76 ]	HRNet48	78.29	77.00	73.16	49.37	73.69	69.08	0.86	40.55	39.74	37.69	33.94	30.04	29.16	0.82
Segmenter [ICCV2021] [ 77 ]	ViT-B/16	82.50	81.19	78.69	60.29	78.43	77.04	0.91	46.93	46.38	45.28	42.64	41.75	39.39	0.91

TABLE II : Benchmarking the robustness of 13 state-of-the-art methods under four appearance attributes changes on our Pascal-EA and COCO-EA datasets. The results are reported using mIoU (

\uparrow

). Recon denotes results on reconstructed images by our Gen4Seg without any attribute variations. We also report the average mIoU on the last row.

Method	Backbone	Pascal-EA						COCO-EA
Method	Backbone	Original	Size 0.2	Size 0.4	Position 0.2	Position 0.4	$mR$	Original	Size 0.2	Size 0.4	Position 0.2	Position 0.4	$mR$
PSPNet [CVPR2017] [ 73 ]	ResNet50	67.41	64.97	62.83	64.98	63.67	0.95	22.18	20.89	19.75	21.44	20.87	0.93
DeepLabV3+ [CVPR2018] [ 85 ]	ResNet50	66.69	64.12	62.37	64.68	62.98	0.95	22.07	20.67	19.64	21.45	20.80	0.94
GCNet [TPAMI2020] [ 75 ]	ResNet50	67.85	65.69	63.60	64.95	63.5	0.95	22.78	21.11	19.46	21.74	21.36	0.92
OCRNet [ECCV2020] [ 76 ]	HRNet48	68.15	67.26	65.33	66.61	65.08	0.97	23.03	21.22	19.50	21.01	20.68	0.89
Segmenter [ICCV2021] [ 77 ]	ViT-B/16	70.87	70.34	69.54	69.58	68.25	0.97	23.65	22.02	19.71	22.81	22.15	0.92

TABLE III : Benchmarking the robustness of 13 state-of-the-art methods under two object geometry attribute changes on our Pascal-EA and COCO-EA datasets. The results are reported using mIoU (

\uparrow

Method	Appearance						Geometry
Method	Recon	Color	Material	Weather	Style	$mR$	Original	Size 0.2	Size 0.4	Position 0.2	Position 0.4	$mR$
Mask2Former [CVPR2022] [ 80 ]	87.52	85.35	61.52	85.30	79.09	0.89	77.11	73.56	71.92	72.87	72.04	0.94
+CutOut [Arxiv2017] [ 89 ]	89.56	84.20	48.47	85.17	75.68	0.82	76.82	74.98	72.44	72.58	71.61	0.94
+CutMix [ICCV2019] [ 90 ]	90.64	86.44	52.42	84.34	77.51	0.83	77.56	76.62	74.66	77.01	74.47	0.98
+Hendrycks et. al. [ICLR2019] [ 92 ]	91.19	83.27	55.63	86.21	80.05	0.84	75.90	72.45	70.89	71.14	68.93	0.93
+AugMix [ICLR2020] [ 91 ]	91.58	85.54	56.79	87.03	80.77	0.84	75.71	73.9	71.25	71.33	69.38	0.94

TABLE IV : mIoU (

\uparrow

) of four augmentation algorithms on our Pascal-EA. The best results are in bold.

	HRNet [ 100 ]						Segformer [ 78 ]
Method	Cityscapes	Rain	Fog	Snow	Night	Avg.	Cityscapes	Rain	Fog	Snow	Night	Avg.
Baseline †	70.47	44.15	58.68	44.20	18.90	41.48	67.90	50.22	60.52	48.86	28.56	47.04
CutOut [Arxiv2017] [ 89 ]	71.39	40.29	57.70	43.98	16.55	39.63	68.93	47.68	60.34	46.98	26.49	45.37
CutMix † [ICCV2019] [ 90 ]	72.68	42.48	58.63	44.50	17.07	40.67	69.23	49.53	61.58	47.42	27.77	46.57
Hendrycks et. al. † [ICLR2019] [ 92 ]	69.25	50.78	60.82	38.34	22.82	43.19	67.41	54.02	64.74	49.57	28.50	49.21
StyleMix † [CVPR2021] [ 101 ]	57.40	40.59	49.11	39.14	19.34	37.04	65.30	53.54	63.86	49.98	28.93	49.08

TABLE V : Comparison of different methods from Cityscapes [ 70 ] (source) to ACDC [ 3 ] (target) using the mIoU (

\uparrow

) metric. The results are reported on the Cityscapes validation set, four individual scenarios of ACDC, and the average (Avg). † means results are directly from [ 99 ] . Oracle indicates the supervised training, serving as performance upper bounds.

	Mask2Former	+CutOut	+CutMix	+AugMix	+ Our
Pascal VOC	87.52	89.56	90.64	91.58	91.77
COCO Stuff	45.06	45.98	46.52	46.84	47.03

TABLE VI : The results of different data augmentation techniques on Pascal VOC [ 68 ] and COCO-Stuff 164k [ 102 ] .

Fig. 7 : Qualitative results of our method and existing text-driven image editing methods.

Method	Material		Color
Method	DINO Dist ( $\downarrow$ )	Masked LPIPS ( $\downarrow$ )	DINO Dist ( $\downarrow$ )	Masked LPIPS ( $\downarrow$ )
BLD [SIGGRAPH2023] [ 106 ]	0.081	0.413	0.092	0.384
DiffEdit [ICLR2023] [ 105 ]	0.068	0.407	0.061	0.372
InstructPix2Pix [CVPR2022] [ 104 ]	0.059	0.393	0.043	0.359
P2P [CVPR2023] [ 15 ]	0.038	0.255	0.014	0.272
PnP [CVPR2023] [ 16 ]	0.047	0.329	0.021	0.295

TABLE VII : Quantitative results of existing image editing models. We compare them with our method by editing object color and material on the Pascal VOC [ 68 ] dataset. MGD is the conference version of our method.

Fig. 8 : Distributions measured by GradNorm [ 107 ] of in-distribution images (Pascal VOC [ 68 ] dataset) and counterparts with object size and position changes, generated by our method and the ImageNet-E [ 11 ] .

Fig. 9 : Visualization of cross-attention maps. There is considerable improvement after applying our strategies.

Our Strategies			DINO Dist ( $\downarrow$ )	CLIP Acc ( $\uparrow$ )
Loss (Eq. 5 )	Blend (Eq. 7 )	ControlNet	DINO Dist ( $\downarrow$ )	CLIP Acc ( $\uparrow$ )
			0.052	0.59
✓			0.036	0.62
	✓		0.010	0.63
		✓	0.048	0.61
✓	✓		0.007	0.70

TABLE VIII : Ablation results of appearance attributes editing.

Our Strategies		FID ( $\downarrow$ )	LPIPS ( $\downarrow$ )	DINO Dist ( $\downarrow$ )
VLM [ 54 ]	ControlNet [ 18 ]	FID ( $\downarrow$ )	LPIPS ( $\downarrow$ )	DINO Dist ( $\downarrow$ )
		36.08	0.478	0.045
✓		27.92	0.334	0.022
	✓	27.55	0.346	0.028
✓	✓	24.60	0.295	0.016

TABLE IX : Ablation results of geometry attributes editing.

Fig. 10 : Visualization of synthetic images and the filtered regions (white masks in the label). The black masks are denoted as “background” category.

Fig. 11 : Failure cases of our proposed image editing pipeline.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Type	Scene	Global variation	Local variation
Fog Cityscapes [IJCV2018] [ 20 ]	simulator	cityscape	fog	-
ACDC [ICCV2021] [ 3 ]	real	cityscape	weather, daytime	-
Dark Zurich [TPAMI2020] [ 108 ]	real	cityscape	daytime	-
Multi-weather [ICCV2021] [ 5 ]	GAN	cityscape	weather	-
SHIFT [CVPR2022] [ 4 ]	simulator	cityscape	weather, daytime	-
Stylized COCO [ECCV2022] [ 6 ]	AdaIN	general	-	object texture

Appendix

TABLE I : Summary of existing benchmarks for semantic segmentation robustness evaluation. Our benchmarks cover both global and local variations. We extend geometry attributes compared to our conference version.

Attribute	Instruction prompt	Examples
Color	I want to change the color of the object in the source image. Please generate all possible target text prompts given the source text prompt describing the source image. For example, the source is “a cat”, you can generate “a blue cat”.	Input: a white and red train. Output: a white and blue train.
Material	I want to change the material of the object in the source image. Please generate all possible target text prompts given the source text prompt describing the source image. For example, source is “a cat”, you can generate “a wooden cat sculpture”.	Input: an airplane on the ground. Output: a wooden airplane model on the ground.
Style:	I want to change the image style of source images without perturbing the content. Please generate all possible target text prompts given the source text prompt describing the source image. For example, source is ’a cat’, you can generate ’a watercolor cat’.	Input: A brown leather couch is sitting outside on a sidewalk. Output: A digital painting of a brown leather couch is sitting outside on a sidewalk.
Weather:	I want to change the weather or season condition of the source image. Please generate all possible target text prompts given the source text prompt that describes the source image, only changing the weather conditions, or adding a description of the weather if not already present.	Input: Two sheep lying in the grass. Output: Two sheep laying in the grass under a heavy downpour .

Appendix

TABLE II : Illustrations of text prompts used to instruct LLaMA3 [ 57 ] and edited text examples.

Fig. 1 : Average mIoU (

\uparrow

) of segmentation methods under two different attribute variations and their combinations.

Method	Backbone	Pascal-EA				COCO-EA
Method	Backbone	Original	Size 0.6	Size 0.8	$mR$	Original	Size 0.6	Size 0.8	$mR$
DeepLabV3+ [CVPR2018] [ 85 ]	ResNet50	66.69	30.21	10.25	0.30	22.07	17.12	10.45	0.62
Segmenter [ICCV2021] [ 77 ]	ViT-B/16	70.87	35.58	12.79	0.34	23.65	15.29	11.79	0.57
Mask2Former [CVPR2022] [ 80 ]	Swin-B	77.11	36.70	18.22	0.36	21.81	13.99	10.33	0.56
OVSeg [CVPR2023] [ 81 ]	Swin-B	79.46	40.29	20.85	0.38	23.42	15.79	11.74	0.59
SEEM [NeurIPS2023] [ 84 ]	Focal-T	76.24	37.29	20.53	0.38	27.38	16.38	13.22	0.54

Appendix

TABLE III : Benchmarking the robustness of several methods under greater object size attribute changes on our Pascal-EA and COCO-EA datasets. The results are reported using mIoU (

\uparrow

Method	PASCAL VOC		COCO Stuff 164k
Method	Filtered/Total	CLIP Accuracy	Filtered/Total	CLIP Accuracy
UperNet [ 110 ]	14/200	0.879	12/200	0.882
OCRNet [ 76 ]	14/200	0.871	12/200	0.890
Segmenter [ 77 ]	19/200	0.884	15/200	0.896
Mask2Former [ 80 ]	23/200	0.903	18/200	0.915

Appendix

TABLE IV : Comparison of different surrogate models used for noise filtering.

	Model	CLIP Acc ( $\uparrow$ )	DINO Dist ( $\downarrow$ )
Stylized-Object [ECCV2022] [ 6 ]	AdaIN	0.574	0.075
Ours	Diffusion	0.906	0.002
Stylized-Background [ECCV2022] [ 6 ]	AdaIN	0.678	0.084
Ours	Diffusion	0.975	0.004

Appendix

TABLE V : Comparison of our pipeline with Stylized COCO [ 6 ] in local object and background changes. Our generated images have superior image reality and fidelity.

Caption	Length (#tokens)	Appearance			Geometry
Caption	Length (#tokens)	DINO Dist( $\downarrow$ )	Masked LPIPS( $\downarrow$ )	CLIP Score( $\uparrow$ )	FID( $\downarrow$ )	HPSv2( $\uparrow$ )
Short	$\approx$ 12	0.109	0.192	0.512	16.84	22.79
Medium	$\approx$ 30	0.118	0.207	0.395	17.92	19.38
Long	$\approx$ 60	0.242	0.346	0.171	35.38	18.26

Appendix

TABLE VI : Quantitative results of edit performance under different prompt lengths. The best results are in bold.

Fig. 2 : Two examples of captions with varying levels of detail. From top to bottom are short, medium, and long captions, respectively.

Fig. 3 : Visualization results under appearance variations on our proposed Pascal-EA benchmarks.

Fig. 4 : Visualization results under appearance variations on our proposed Pascal-EA benchmarks.

Fig. 5 : Visualization results under object geometry (size and position) variations.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentation
Mar 19, 2026Huy Che, Dinh-Duy Phan, Duc-Khai LamSynthetic Data Augmentation Semantic Segmentation
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
Difficulty-Aware Sample Allocation for Adaptive Data Augmentation in Semantic Segmentation
Aug 26, 2026Olasimbo Ayodeji Arigbabu, Abimbola Ismail ArigbabuData Augmentation Semantic Segmentation
Independent Researcher, Germany. · Department of Science and Technology Education, Olabisi Onabanjo University, Ago-Iwoye, Ogun State, Nigeria.
Diffusion Model as a Generalist Segmentation Learner
Apr 27, 2026Haoxiao Wang, Antao Xiang, Haiyang Sun +8Image Segmentation Text-Guided Image Segmentation
Zhejiang University · South China University of Technology · Nanjing University +1

Mar 19, 2026J/K move · Enter open · S save

R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentation

Huy Che, Dinh-Duy Phan, Duc-Khai Lam

University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Synthetic Data Augmentation Semantic Segmentation Diffusion-Based Image Generation

Open paper AskPDF arXiv

Collecting and annotating datasets for pixel-level semantic segmentation tasks are highly labor-intensive. Data augmentation provides a viable solution by enhancing model generalization without additional real-world data collection. Traditional augmentation techniques, such as translation, scaling, and color transformations, create geometric variations but fail to generate new structures. While generative models have been employed to extend semantic information of datasets, they often struggle to maintain consistency between the original and generated images, particularly for pixel-level tasks. In this work, we propose a novel synthetic data augmentation pipeline that integrates controllable diffusion models. Our approach balances diversity and reliability data, effectively bridging the gap between synthetic and real data. We utilize class-aware prompting and visual prior blending to improve image quality further, ensuring precise alignment with segmentation labels. By evaluating benchmark datasets such as PASCAL VOC and BDD100K, we demonstrate that our method significantly enhances semantic segmentation performance, especially in data-scarce scenarios, while improving model robustness in real-world applications. Our code is available at https://github.com/chequanghuy/Enhanced-Generative-Data-Augmentation-for-Semantic-Segmentation-via-Stronger-Guidance.

Abstract

Semantic Segmentation

Explore similar work

R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentation

Difficulty-Aware Sample Allocation for Adaptive Data Augmentation in Semantic Segmentation

Diffusion Model as a Generalist Segmentation Learner