Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

Guohuan Xie; Xin He; Dingying Fan; Le Zhang; Ming-Ming Cheng; Yun Liu

cs.CVMar 28, 2026

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

Authors: Guohuan Xie, Xin He, Dingying Fan, Le Zhang, Ming-Ming Cheng, Yun Liu

Organizations: College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. · VCIP, College of Computer Science, and Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin 300350, China ·

Abstract

Generalized few-shot semantic segmentation (GFSS) is fundamentally limited by the coverage of novel-class appearances under scarce annotations. While diffusion models can synthesize novel-class images at scale, practical gains are often hindered by insufficient coverage and noisy supervision when masks are unavailable or unreliable. We propose Syn4Seg, a generation-enhanced GFSS framework designed to expand novel-class coverage while improving pseudo-label quality. Syn4Seg first maximizes prompt-space coverage by constructing an embedding-deduplicated prompt bank for each novel class, yielding diverse yet class-consistent synthetic images. It then performs support-guided pseudo-label estimation via a two-stage refinement that i) filters low-consistency regions to obtain high-precision seeds and ii) relabels uncertain pixels with image-adaptive prototypes that combine global (support) and local (image) statistics. Finally, we refine only boundary-band and unlabeled pixels using a constrained SAM-based update to improve contour fidelity without overwriting high-confidence interiors. Extensive experiments on PASCAL- $5^i$ and COCO- $20^i$ demonstrate consistent improvements in both 1-shot and 5-shot settings, highlighting synthetic data as a scalable path for GFSS with reliable masks and precise boundaries.

Figures & tables

Fig. 1: Pipelines of classical GFSS works and our Syn4Seg framework. (a) Classical GFSS methods follow three stages: base class learning, novel class registration, and evaluation, where novel classes rely on limited manual annotations, resulting in low diversity and constrained performance. (b) Our Syn4Seg framework synthesizes diverse, realistic novel class images and pseudo labels guided by support annotations. These synthesized samples, together with base data, are used to train a semantic segmentation model, enabling robust recognition of both base and novel classes during evaluation.

Fig. 2: (a), (b) Given several novel classes and their support set, we first use HDIG to synthesize high-quality, diverse novel class images proportional to the base class images. Guided by support images, APE refines these synthesized images to generate accurate masks through two stages: Adaptive Pseudo-label Filtering (APF) and Adaptive Pseudo-label Relabeling (APR). Specifically, APF filters noisy regions in the initial mask $M$ to obtain $M_{1}$ ; APR relabels these unlabeled regions to yield $M_{2}$ . Finally, SAM-based Boundary Refinement (SABR) further refines the mask boundaries, yielding the final $M_{3}$ . (c), (d) illustrate the details of APF and APR.

Fig. 3: Qualitative comparison between images generated directly using class-name prompts (a) and those generated with our HDIG method (b).

Fig. 4: Quantitative ablation results showing the progressive improvements brought by the mask-quality refinement modules: APF, APR, and SABR.

Methods	PASCAL- $5^{i}$
	1-shot				5-shot
	Base	Novel	Mean	Harmonic	Base	Novel	Mean	Harmonic
CANet [ 49 ] (CVPR’2019)	8.73	2.42	5.58	3.79	9.05	1.52	5.29	2.60
PFENet [ 50 ] (TPAMI’2020)	8.32	2.67	5.50	4.04	8.83	1.89	5.36	3.11
PANet [ 51 ] (ICCV’2019)	31.88	11.25	21.57	16.63	32.95	15.25	24.10	20.85
SCL [ 52 ] (CVPR’2021)	8.88	2.44	5.66	3.83	9.11	1.83	5.47	3.05

TABLE I: Comparison with state-of-the-art methods on PASCAL-

5^{i}

and COCO-

20^{i}

under the GFSS setting. Bold and underlined indicate the best and second best methods. Mean denotes the average of the mIoU over base and novel classes, while Harmonic represents their harmonic mean. All methods use a ResNet-50 backbone and follow the same training/test split.

Fig. 5: Qualitative results of the proposed Syn4Seg and state-of-the-art approach under the 1-shot setting. The left panel illustrates results on PASCAL- $5^{i}$ , and the right panel on COCO- $20^{i}$ . The first row shows the input images and their corresponding ground-truth masks, followed by the results of VP and BCM in the second and third rows, respectively. The fourth row presents the results of our Syn4Seg.

ID	HDIG	APF	APR	SABR	Base	Novel	Mean	Harmonic
(1)					73.68	41.75	57.22	52.48
(2)	$\checkmark$				74.97	43.47	59.22	55.03
(3)	$\checkmark$	$\checkmark$	–	–	74.74	44.46	59.10	54.96
(4)	$\checkmark$	$\checkmark$	$\checkmark$	–	74.64	45.65	60.15	56.65
(5)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	74.80	46.13	60.47	57.07

TABLE II: Ablation study of different components in Syn4Seg, where rows (1)–(5) progressively add each component. Bold and underlined indicate best and the second best method.

$\mathbf{\lambda}$	Base	Novel	Mean	Harmonic
0.2	74.50	45.53	60.02	56.52
0.4	74.43	45.66	60.05	56.60
0.6	74.80	46.13	60.47	57.07
0.8	74.55	24.84	49.70	37.26
1.0	74.36	0.00	37.18	0.00

TABLE III: Ablation study on the influence of hyperparameters

\lambda

. Bold and underlined indicate best and the second best method.

$\beta$	Base	Novel	Mean	Harmonic
0.1	73.05	45.31	59.18	55.93
0.3	74.47	46.11	60.29	56.95
0.5	74.95	45.79	60.37	56.85
0.7	74.80	46.13	60.47	57.07
0.9	74.64	45.53	60.09	56.56

TABLE IV: Ablation study on the influence of hyperparameter

\beta

. Bold and underlined indicate best and the second best method.

$\delta$	Base	Novel	Mean	Harmonic
1	74.70	45.83	60.27	56.81
3	75.14	45.73	60.44	56.86
5	74.80	46.13	60.47	57.07
7	75.18	45.65	60.42	56.81
9	74.49	46.11	60.30	56.96

TABLE V: Ablation study on the influence of hyperparameter

\delta

. Bold and underlined indicate best and the second best method.

Backbone	Base	Novel	Mean	Harmonic
ResNet50	74.80	46.13	60.47	57.07
ResNet101	75.41	46.83	61.12	57.78

TABLE VI: Ablation study on the influence of the backbone of Mask2Former. The best and second-best results for each setting are highlighted.

Fig. 6: t-SNE visualization of pooled ResNet-50 features extracted from images generated by class-only prompts and HDIG for eight categories.

Few-Shot Semantic Segmentation

Few-shot semantic segmentation labels pixels in new image classes using only a small set of annotated examples.

See all 32 papers

What Makes Synthetic Data Effective in Image Segmentation
May 19, 2026Jinjin Zhang, Xiefan Guo, Yizhou Jin +2Synthetic Data Augmentation Image Segmentation
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China
ZeroDSS: Semantic Segmentation from Class Names Alone
Oct 17, 2026Tian Ming, Marzi Heidari, Yuhong GuoSemantic Segmentation Synthetic Data Augmentation
School of Computer Science, Carleton University, Ottawa, Canada
Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models
Jun 30, 2026Nikolai Röhrich, Julian Gleißner, Ahmed H. A. Ibrahim +2Synthetic Data Augmentation Semantic Segmentation
1XITASO GmbH · 2Zuse School of Excellence in Reliable AI · 3Technische Hochschule Ingolstadt +1

May 19, 2026J/K move · Enter open · S save

What Makes Synthetic Data Effective in Image Segmentation

Jinjin Zhang, Xiefan Guo, Yizhou Jin, Nan Zhou+1

State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Synthetic Data Augmentation Image Segmentation Semantic Segmentation Synthetic Data Evaluation Diffusion Models

Open paper AskPDF arXiv

Driven by rapid advances in large-scale generative models, synthetic data has emerged as a promising solution for visual understanding. While modern diffusion models achieve remarkable photorealistic image synthesis, their potential in complex visual segmentation tasks remains underexplored. In this work, we conduct a systematic analysis of synthetic images from state-of-the-art diffusion models to uncover the factors governing their utility. In particular, synthetic images characterized by dense scene composition and fine instance fidelity demonstrate distinctive benefits, yielding significantly more discriminative spatial representations. Building on these insights, we propose SENSE, a unified framework that leverages flexible and scalable synthetic data to substantially enhance segmentation performance. Notably, SENSE is model-agnostic, compatible with diverse architectures (e.g., DPT and Mask2Former), and scales effectively across models with varying parameter capacities. Extensive experiments on Cityscapes, COCO, and ADE20K validate the effectiveness and generalization capability of our approach. Code is available at https://github.com/zhang0jhon/SENSE.

Abstract

Few-Shot Semantic Segmentation

Explore similar work

What Makes Synthetic Data Effective in Image Segmentation

ZeroDSS: Semantic Segmentation from Class Names Alone

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models