cs.CVOct 17, 2026

ZeroDSS: Semantic Segmentation from Class Names Alone

Authors: Tian Ming, Marzi Heidari, Yuhong Guo

Organizations: School of Computer Science, Carleton University, Ottawa, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Semantic segmentation depends heavily on real images with pixel-level annotations, making dense supervision a major obstacle to deployment in new domains. We study task-specific data-free semantic segmentation, where training uses generic pretrained generative priors but has access to no target-domain images, masks, or captions--only a predefined category set. We introduce Zero-Data Semantic Segmentation (ZeroDSS), a feedback-driven approach that treats synthetic supervision as an adaptively controllable training distribution rather than a direct surrogate for real data. ZeroDSS constructs a semantically structured prompt distribution through LLM-guided compositional generation and adaptive class rebalancing, then converts the prompts into image-mask pairs using diffusion attention. Its feedback-driven distribution refinement consists of a two-phase closed-loop training procedure with a single loop-back step. In Phase I, a segmenter is trained using confidence-weighted supervision and dual-scale consistency. In Phase II, model feedback is used to adaptively target underperforming categories and refine diffusion-derived masks using model predictions. After this refined initial training, a teacher-student framework is utilized in the final phase to iteratively improve the fused supervision and segmenter through exponential moving average (EMA) updates. Using synthetic training data only, ZeroDSS achieves 35.0% mIoU on MS COCO with DeepLabV3, and 66.3% and 63.5% mIoU on PASCAL VOC with DeepLabV3 and Mask2Former, respectively. These results show that explicitly constructing and iteratively refining the synthetic supervision distribution can substantially reduce the synthetic-to-real gap in dense prediction.

Figures & tables

Figure 1: ZeroDSS constructs a structured synthetic distribution from class names through LLM-guided prompts and diffusion attention. Phase I trains with confidence-weighted supervision and dual-scale consistency. Phase II regenerates deficient categories, fuses diffusion masks with model predictions, and loops back to Phase I for continued training on the refined distribution. Phase III further enhance the model using a teacher-student framework, where the fixed teacher is updated by EMA between iterative rounds.
Figure 2: Examples of segmentation-oriented prompts generated by the LLM and the corresponding attention-derived masks.
Training SettingMethodCOCO(DLV3)VOC(DLV3)VOC(M2F)
Fully Supervised Reference: Real Images & AnnotationsGT training48.979.378.8
Open-Vocabulary Segmentation ModelsCLIPSeg [ 10 ]23.454.2
 MaskCLIP [ 5 ]25.156.9
Synthetic Data with Real Dataset CaptionsDataset Diffusion † [ 19 ]28.658.057.8
 DiffuMask [ 33 ]––57.4
 SDS [ 28 ]31.060.459.8
Table 1: Comparison across supervision settings (mIoU). ZeroDSS uses approximately 82k COCO or 42k VOC synthetic training pairs after Phase II expansion and is evaluated on real validation sets. † denotes reproduced results; open-vocabulary methods retain their original architectures.
Table 4
Pseudo-label sourcemIoU(%)
Original attention mask only33.5
Teacher prediction only34.4
Class-wise fusion35.0
Table 4: Effect of the Phase III pseudo-label source on COCO with DeepLabV3-R50. All variants evaluate the final EMA teacher.
Table 6
VOC + 5 NovelVOC + 10 Novel
SubsetDataset DiffusionCLIPSegZeroDSS (Phase I-II)SubsetDataset DiffusionCLIPSegZeroDSS (Phase I-II)
VOC (21)58.155.060.5VOC (21)57.555.859.9
Novel(5)1.0223.327.1Novel(10)0.3821.925.9
All(26)47.149.054.7All(31)39.144.948.9
Table 7: Novel category expansion with ZeroDSS(Phase I-II) model (mIoU (%)). Extra COCO categories are added to VOC without real training images.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Structure of the instruction used for segmentation-oriented prompt generation.
Figure 4: Effect of semantic grouping and token order on diffusion-based image generation and attention-derived masks.
Figure 5: Qualitative examples of mask refinement across four stages: (1) synthetic image, (2) diffusion cross-attention mask, (3) segmentation model prediction, and (4) fused mask after class-wise refinement. Each column corresponds to one example.
Figure 6: Effect of γ\gamma on category selection quality. The y-axis reports the number of selected top- KK deficient categories that correspond to below-average categories on the real COCO validation set. Moderate values of γ\gamma achieve the best alignment between synthetic-domain deficiency estimation and real-domain difficulty.
Figure 7: Three examples of weak-category (fork, chair, necktie) regeneration in Phase II For each category, the left column shows the original synthetic sample and attention-derived mask, while the right column shows the regenerated sample and mask after adaptive refinement.
Figure 8: Sensitivity analysis of the consistency weight λ\lambda on COCO (DeepLabV3-R50).
More in this topic

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 399 papers

Explore similar work

CardsList
Mar 28, 2026

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

Generalized few-shot semantic segmentation (GFSS) is fundamentally limited by the coverage of novel-class appearances under scarce annotations. While diffusion models can synthesize novel-class images at scale, practical gains are often hindered by insufficient coverage and noisy supervision when masks are unavailable or unreliable. We propose Syn4Seg, a generation-enhanced GFSS framework designed to expand novel-class coverage while improving pseudo-label quality. Syn4Seg first maximizes prompt-space coverage by constructing an embedding-deduplicated prompt bank for each novel class, yielding diverse yet class-consistent synthetic images. It then performs support-guided pseudo-label estimation via a two-stage refinement that i) filters low-consistency regions to obtain high-precision seeds and ii) relabels uncertain pixels with image-adaptive prototypes that combine global (support) and local (image) statistics. Finally, we refine only boundary-band and unlabeled pixels using a constrained SAM-based update to improve contour fidelity without overwriting high-confidence interiors. Extensive experiments on PASCAL-5i5^i and COCO-20i20^i demonstrate consistent improvements in both 1-shot and 5-shot settings, highlighting synthetic data as a scalable path for GFSS with reliable masks and precise boundaries.

Guohuan Xie, Xin He, Dingying Fan +3

College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. +2

Few-Shot Semantic SegmentationSemantic SegmentationSynthetic Data AugmentationPseudo-Labeling
AskPDF
May 24, 2026

D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation

Dataset distillation (DD) aims to compress large-scale datasets into compact synthetic sets while preserving training efficacy. However, existing studies mainly focus on image classification, leaving dense prediction tasks such as semantic segmentation largely underexplored. In this work, we identify three key challenges for segmentation DD: (i) long-tailed class imbalance, (ii) the need for strict pixel-wise alignment between images and dense labels, and (iii) the high computational cost of optimizing high-resolution data with complex models. To address these challenges, we propose D3S2, a Diffusion-guided Dataset Distillation framework for Semantic Segmentation. Our method adopts a two-stage design. In Class-Balanced Mask Selection, we construct a representative mask set via a greedy strategy that prioritizes underrepresented classes. In Diffusion-Guided Image Synthesis, we employ a pretrained layout-to-image diffusion model to generate images conditioned on the selected masks, naturally ensuring spatial alignment. To further enhance the training utility of synthesized data, we introduce guided diffusion sampling with two complementary objectives: a segmentation-consistency loss for pixel-level alignment, and a class-wise feature matching loss for aligning per-class feature statistics across layers. Extensive experiments demonstrate the superiority of D3S2. Notably, at an extremely compression rate of 1%, our method achieves 24.99% and 35.49% mIoU on ADE20K and COCO-Stuff with Mask2Former (Swin-S), outperforming random selection by 9.34% and 5.70%, respectively. Our code is available at https://github.com/zwj084/D3S2.

Wenjie Zheng, Haoji Hu, Jiali Lu +2

Zhejiang University, China

Diffusion Model GuidanceDataset DistillationSemantic Segmentation
AskPDF
May 19, 2026

What Makes Synthetic Data Effective in Image Segmentation

Driven by rapid advances in large-scale generative models, synthetic data has emerged as a promising solution for visual understanding. While modern diffusion models achieve remarkable photorealistic image synthesis, their potential in complex visual segmentation tasks remains underexplored. In this work, we conduct a systematic analysis of synthetic images from state-of-the-art diffusion models to uncover the factors governing their utility. In particular, synthetic images characterized by dense scene composition and fine instance fidelity demonstrate distinctive benefits, yielding significantly more discriminative spatial representations. Building on these insights, we propose SENSE, a unified framework that leverages flexible and scalable synthetic data to substantially enhance segmentation performance. Notably, SENSE is model-agnostic, compatible with diverse architectures (e.g., DPT and Mask2Former), and scales effectively across models with varying parameter capacities. Extensive experiments on Cityscapes, COCO, and ADE20K validate the effectiveness and generalization capability of our approach. Code is available at https://github.com/zhang0jhon/SENSE.

Jinjin Zhang, Xiefan Guo, Yizhou Jin +2

State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Synthetic Data AugmentationImage SegmentationSemantic SegmentationSynthetic Data EvaluationDiffusion Models
AskPDF