ZeroDSS: Semantic Segmentation from Class Names Alone

Tian Ming; Marzi Heidari; Yuhong Guo

cs.CVOct 17, 2026

ZeroDSS: Semantic Segmentation from Class Names Alone

Authors: Tian Ming, Marzi Heidari, Yuhong Guo

Organizations: School of Computer Science, Carleton University, Ottawa, Canada

Abstract

Semantic segmentation depends heavily on real images with pixel-level annotations, making dense supervision a major obstacle to deployment in new domains. We study task-specific data-free semantic segmentation, where training uses generic pretrained generative priors but has access to no target-domain images, masks, or captions--only a predefined category set. We introduce Zero-Data Semantic Segmentation (ZeroDSS), a feedback-driven approach that treats synthetic supervision as an adaptively controllable training distribution rather than a direct surrogate for real data. ZeroDSS constructs a semantically structured prompt distribution through LLM-guided compositional generation and adaptive class rebalancing, then converts the prompts into image-mask pairs using diffusion attention. Its feedback-driven distribution refinement consists of a two-phase closed-loop training procedure with a single loop-back step. In Phase I, a segmenter is trained using confidence-weighted supervision and dual-scale consistency. In Phase II, model feedback is used to adaptively target underperforming categories and refine diffusion-derived masks using model predictions. After this refined initial training, a teacher-student framework is utilized in the final phase to iteratively improve the fused supervision and segmenter through exponential moving average (EMA) updates. Using synthetic training data only, ZeroDSS achieves 35.0% mIoU on MS COCO with DeepLabV3, and 66.3% and 63.5% mIoU on PASCAL VOC with DeepLabV3 and Mask2Former, respectively. These results show that explicitly constructing and iteratively refining the synthetic supervision distribution can substantially reduce the synthetic-to-real gap in dense prediction.

Figures & tables

Figure 1: ZeroDSS constructs a structured synthetic distribution from class names through LLM-guided prompts and diffusion attention. Phase I trains with confidence-weighted supervision and dual-scale consistency. Phase II regenerates deficient categories, fuses diffusion masks with model predictions, and loops back to Phase I for continued training on the refined distribution. Phase III further enhance the model using a teacher-student framework, where the fixed teacher is updated by EMA between iterative rounds.

Panel of Figure 2 — Figure 2: Examples of segmentation-oriented prompts generated by the LLM and the corresponding attention-derived masks.

Training Setting	Method	COCO(DLV3)	VOC(DLV3)	VOC(M2F)
Fully Supervised Reference: Real Images & Annotations	GT training	48.9	79.3	78.8
Open-Vocabulary Segmentation Models	CLIPSeg [ 10 ]	23.4	54.2
	MaskCLIP [ 5 ]	25.1	56.9
Synthetic Data with Real Dataset Captions	Dataset Diffusion † [ 19 ]	28.6	58.0	57.8
	DiffuMask [ 33 ]	–	–	57.4
	SDS [ 28 ]	31.0	60.4	59.8

Table 1: Comparison across supervision settings (mIoU). ZeroDSS uses approximately 82k COCO or 42k VOC synthetic training pairs after Phase II expansion and is evaluated on real validation sets. † denotes reproduced results; open-vocabulary methods retain their original architectures.

Table 4

Pseudo-label source	mIoU(%)
Original attention mask only	33.5
Teacher prediction only	34.4
Class-wise fusion	35.0

Table 4: Effect of the Phase III pseudo-label source on COCO with DeepLabV3-R50. All variants evaluate the final EMA teacher.

Table 6

VOC + 5 Novel				VOC + 10 Novel
Subset	Dataset Diffusion	CLIPSeg	ZeroDSS (Phase I-II)	Subset	Dataset Diffusion	CLIPSeg	ZeroDSS (Phase I-II)
VOC (21)	58.1	55.0	60.5	VOC (21)	57.5	55.8	59.9
Novel(5)	1.02	23.3	27.1	Novel(10)	0.38	21.9	25.9
All(26)	47.1	49.0	54.7	All(31)	39.1	44.9	48.9

Table 7: Novel category expansion with ZeroDSS(Phase I-II) model (mIoU (%)). Extra COCO categories are added to VOC without real training images.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Structure of the instruction used for segmentation-oriented prompt generation.

Panel of Figure 4 — Figure 4: Effect of semantic grouping and token order on diffusion-based image generation and attention-derived masks.

Figure 5: Qualitative examples of mask refinement across four stages: (1) synthetic image, (2) diffusion cross-attention mask, (3) segmentation model prediction, and (4) fused mask after class-wise refinement. Each column corresponds to one example.

Figure 6: Effect of $\gamma$ on category selection quality. The y-axis reports the number of selected top- $K$ deficient categories that correspond to below-average categories on the real COCO validation set. Moderate values of $\gamma$ achieve the best alignment between synthetic-domain deficiency estimation and real-domain difficulty.

Figure 7: Three examples of weak-category (fork, chair, necktie) regeneration in Phase II For each category, the left column shows the original synthetic sample and attention-derived mask, while the right column shows the regenerated sample and mask after adaptive refinement.

Figure 8: Sensitivity analysis of the consistency weight $\lambda$ on COCO (DeepLabV3-R50).

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 399 papers

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation
Mar 28, 2026Guohuan Xie, Xin He, Dingying Fan +3Few-Shot Semantic Segmentation Semantic Segmentation
College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. +2
D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation
May 24, 2026Wenjie Zheng, Haoji Hu, Jiali Lu +2Diffusion Model Guidance Dataset Distillation
Zhejiang University, China
What Makes Synthetic Data Effective in Image Segmentation
May 19, 2026Jinjin Zhang, Xiefan Guo, Yizhou Jin +2Synthetic Data Augmentation Image Segmentation
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Mar 28, 2026J/K move · Enter open · S save

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

Guohuan Xie, Xin He, Dingying Fan, Le Zhang+2

College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China.+2

Few-Shot Semantic Segmentation Semantic Segmentation Synthetic Data Augmentation Pseudo-Labeling

Open paper AskPDF arXiv

Generalized few-shot semantic segmentation (GFSS) is fundamentally limited by the coverage of novel-class appearances under scarce annotations. While diffusion models can synthesize novel-class images at scale, practical gains are often hindered by insufficient coverage and noisy supervision when masks are unavailable or unreliable. We propose Syn4Seg, a generation-enhanced GFSS framework designed to expand novel-class coverage while improving pseudo-label quality. Syn4Seg first maximizes prompt-space coverage by constructing an embedding-deduplicated prompt bank for each novel class, yielding diverse yet class-consistent synthetic images. It then performs support-guided pseudo-label estimation via a two-stage refinement that i) filters low-consistency regions to obtain high-precision seeds and ii) relabels uncertain pixels with image-adaptive prototypes that combine global (support) and local (image) statistics. Finally, we refine only boundary-band and unlabeled pixels using a constrained SAM-based update to improve contour fidelity without overwriting high-confidence interiors. Extensive experiments on PASCAL-

5^i

and COCO-

20^i

demonstrate consistent improvements in both 1-shot and 5-shot settings, highlighting synthetic data as a scalable path for GFSS with reliable masks and precise boundaries.

Abstract

Semantic Segmentation

Explore similar work

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation

What Makes Synthetic Data Effective in Image Segmentation