R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentation

Huy Che; Dinh-Duy Phan; Duc-Khai Lam

cs.CVMar 19, 2026

R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentation

Authors: Huy Che, Dinh-Duy Phan, Duc-Khai Lam

Organizations: University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Abstract

Collecting and annotating datasets for pixel-level semantic segmentation tasks are highly labor-intensive. Data augmentation provides a viable solution by enhancing model generalization without additional real-world data collection. Traditional augmentation techniques, such as translation, scaling, and color transformations, create geometric variations but fail to generate new structures. While generative models have been employed to extend semantic information of datasets, they often struggle to maintain consistency between the original and generated images, particularly for pixel-level tasks. In this work, we propose a novel synthetic data augmentation pipeline that integrates controllable diffusion models. Our approach balances diversity and reliability data, effectively bridging the gap between synthetic and real data. We utilize class-aware prompting and visual prior blending to improve image quality further, ensuring precise alignment with segmentation labels. By evaluating benchmark datasets such as PASCAL VOC and BDD100K, we demonstrate that our method significantly enhances semantic segmentation performance, especially in data-scarce scenarios, while improving model robustness in real-world applications. Our code is available at https://github.com/chequanghuy/Enhanced-Generative-Data-Augmentation-for-Semantic-Segmentation-via-Stronger-Guidance.

Figures & tables

Figure 1: Our proposed synthetic data augmentation pipeline utilizes the real dataset $\mathcal{D}_{0}$ to create two synthetic datasets, $\mathcal{D}_{1}^{gen}$ and $\mathcal{D}_{2}^{gen}$ . The annotations for the synthetic data are directly copied from the labels of the real dataset.

Figure 2: Some examples of text prompt selection for input images show that simple text prompts are often too simplistic, while generated captions may miss some labeled classes. Class-prompt appending addresses this but can lead to incoherent prompts. In contrast, conditional image captioning creates coherent prompts that accurately describe the image and include all labeled classes.

Panel of Figure 3 — Figure 3: Image generation using the Img2Img Controllable Diffusion Model.

Panel of Figure 4 — Figure 4: Image generation using the Controllable Inpainting Diffusion Model.

Dataset			VOC7	VOC12
Number images			209	92	183	366	732	1464
DeepLabV3+	Resnet50	$\mathcal{D}_{0}$	63.75	48.19	58.44	65.84	70.55	72.19
		$\mathcal{D}_{0}$ $\cup$ [ 11 ]	64.02	51.83	59.37	65.98	69.14	72.16
		$\mathcal{D}_{0}$ $\cup$ ours	64.47	53.67	59.98	67.52	71.06	72.96
	Resnet101	$\mathcal{D}_{0}$	67.61	54.06	62.88	67.85	73.06	76.19
		$\mathcal{D}_{0}$ $\cup$ [ 11 ]	68.79	56.01	63.09	68.89	73.05	75.68

Table 1: Comparison of mIoU (%) on the validation set when training models on the original dataset (

\mathcal{D}_{0}

) and when merging it with the synthetic dataset (

\mathcal{D}_{0}

\cup

[ 11 ] vs.

\mathcal{D}_{0}

\cup

ours), using DeepLabV3+ and Mask2Former model architectures.

	Real images		Synthetic images
	VOC (5k)	VOC (1,5k)	DiffuMask [ 10 ] (60k)	DD [ 9 ] (40k)	Attn2Mask [ 27 ]	SG [ 11 ] (1,5k)	Ours (2,9k)	mIoU (%)
	$\checkmark$							83.4
(1)		$\checkmark$						81.8
			$\checkmark$					70.6
				$\checkmark$				67.6
					$\checkmark$			71.0

Table 2: The table shows the performance of the Mask2Former (Swin-B) model when trained on (1) the real dataset, (2) the synthetic dataset, and (3) fine-tuned on the real dataset after pre-training on the synthetic dataset. The compared methods include generating synthetic data with pseudo-labels [ 10 , 9 , 27 ] and synthetic data based on the original dataset [ 11 ]

Condition	Ours	Number	Lane Line		Drivable Area
Condition	Ours	Number	Accuracy (%)	IoU(%)	mIoU(%)
Foggy		130	56.9	4.5	72.6
Foggy	$\checkmark$	390	67.8 / 67.5	8.3 / 8.7	79.9 / 80.4
Tunnel		129	80.5	9.3	73.4
Tunnel	$\checkmark$	387	84.4 / 85.2	15.2 / 14.9	86.0 / 87.3
Gas Station		27	48.9	0.2	67.1

Table 3: Evaluation of multi-task segmentation model performance across different environmental conditions when integrating our method with real data via merging/fine-tuning.

Train with Pure Real Data	$\checkmark$			R: 209	76.19	R: 1464	81.8
				VOC7		VOC12
	$\mathcal{D}_{0}$	$\mathcal{D}^{gen}_{1}$	$\mathcal{D}^{gen}_{2}$	Number	mIoU (%)	Number	mIoU (%)
		$\checkmark$		S: 209	74.32	R: 1464	74.22
			$\checkmark$	S: 209	73.85	R: 1464	75.18
Train with Pure Synthetic Data		$\checkmark$	$\checkmark$	S: 418	74.67	R: 2928	76.27
	$\checkmark$	$\checkmark$		R: 209 S: 209	77.58 78.53	R: 1464 S: 1464	81.91 83.01

Table 5: Comparison of semantic segmentation performance (mIoU%) using different training strategies. Results are reported for Mask2Former models.

Table 10

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 449 papers

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models
Jun 30, 2026Nikolai Röhrich, Julian Gleißner, Ahmed H. A. Ibrahim +2Synthetic Data Augmentation Semantic Segmentation
1XITASO GmbH · 2Zuse School of Excellence in Reliable AI · 3Technische Hochschule Ingolstadt +1
What Makes Synthetic Data Effective in Image Segmentation
May 19, 2026Jinjin Zhang, Xiefan Guo, Yizhou Jin +2Synthetic Data Augmentation Image Segmentation
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China
Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation
Mar 28, 2026Guohuan Xie, Xin He, Dingying Fan +3Few-Shot Semantic Segmentation Semantic Segmentation
College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. +2

Jun 30, 2026J/K move · Enter open · S save

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models

Nikolai Röhrich, Julian Gleißner, Ahmed H. A. Ibrahim, Silvan Mertes+1

1XITASO GmbH · 2Zuse School of Excellence in Reliable AI · 3Technische Hochschule Ingolstadt+1

Synthetic Data Augmentation Semantic Segmentation

Open paper AskPDF arXiv

Semantic segmentation models struggle with data sparsity and rare or visually diverse regions, e.g., dense regions or small objects in aerial or autonomous mobility data. While synthetic augmentation is an appealing solution, directly generating new labeled data risks misalignment of labels and generated pixels. Existing solutions to this problem often rely on external models, or employ coarse heuristics such as indiscriminately augmenting all foreground objects or entire backgrounds, which wastes capacity on uninformative pixels. To address this, we propose an uncertainty-guided synthetic context augmentation strategy that strictly preserves label validity and efficiently maximizes pixel informativeness per synthetic sample - no external guardrails required. Using a baseline segmenter's predictive entropy, we identify uncertain semantic regions and inpaint only the complementary visual context. When fine-tuning the segmenter on this synthetic data, we compute the loss only over the original pixels, excluding inpainted regions. This focuses learning on the unmodified, uncertain regions while presenting them in novel contexts. We demonstrate substantial mIoU gains on Cityscapes, UAVID, and BDD100K with the largest gains on rare and difficult classes such as buses, trains, or (from the aerial perspective) cars. Our results demonstrate that uncertainty-guided context augmentation is a highly effective lever to improve segmentation performance on complex datasets, with code provided at https://github.com/XITASO/Preserve-the-Hard-Regenerate-the-Rest.

Abstract

Synthetic Data Augmentation

Explore similar work

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models

What Makes Synthetic Data Effective in Image Segmentation

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation