Open-world semantic segmentation presently relies significantly on extensive image-text pair datasets, which often suffer from a lack of fine-grained pixel annotations on sufficient categories. The acquisition of such data is rendered economically prohibitive due to the substantial investments of both human labor and time. In light of the formidable image generation capabilities of diffusion models, we introduce a novel diffusion model-driven pipeline for automatically generating datasets tailored to the needs of open-world semantic segmentation, named "MagicSeg". Our MagicSeg initiates from class labels and proceeds to generate high-fidelity textual descriptions, which in turn serve as guidance for the diffusion model to generate images. Rather than only generating positive samples for each label, our process encompasses the simultaneous generation of corresponding negative images, designed to serve as paired counterfactual samples for contrastive training. Then, to provide a self-supervised signal for open-world segmentation pretraining, our MagicSeg integrates an open-vocabulary detection model and an interactive segmentation model to extract object masks as precise segmentation labels from images based on the provided category labels. By applying our dataset to the contrastive language-image pretraining model with the pseudo mask supervision and the auxiliary counterfactual contrastive training, the downstream model obtains strong performance on open-world semantic segmentation. We evaluate our model on PASCAL VOC, PASCAL Context, and COCO, achieving SOTA with performance of 62.9%, 26.7%, and 40.2%, respectively, demonstrating our dataset's effectiveness in enhancing open-world semantic segmentation capabilities. Project website: https://github.com/ckxhp/magicseg.
Figures & tables
Fig. 1: Comparison between the previous method and MagicSeg. Previous methods [ 1 , 2 ] involved constructing dedicated segmentation datasets for specific downstream tasks. In contrast, MagicSeg leverages a large-scale vocabulary to build a dataset applied in open-world segmentation. Additionally, we generate counterfactual images to assist for segmentation task.
Fig. 2: Visualization of synthetic image and mask from MagicSeg. With ChatGPT, we generate richly descriptive text from class names, which are used to create diverse images and corresponding masks.
Fig. 3: The overall Counterfactual Diffusion-based Generation framework of MagicSeg: creating a large-scale open-world segmentation dataset with pixel-level annotations and counterfactual images for a wide range of categories.
Fig. 4: The overall open-world segmentation model training framework of MagicSeg: through the category random sampling strategy and counterfactual contrastive training, we apply the constructed dataset to CLIP-based segmentation model, enhancing its capability in open-world segmentation.
Fig. 5: Visualization of the diversity of MagicSeg’s dataset with class “cat”. It can be seen that MagicSeg can generate diverse images for each category.
Fig. 6: Visualization of counterfactual image pairs with mask annotations.
Fig. 7: Text prompt generation conditions for ChatGPT. We design a series of conditions to control the diversity of generated text.
Method
Val (mIoU(%))
PASCAL VOC
LVIS
Grounding DINO + SAM
61.8
0.5
MagicSeg
62.9
4.9
TABLE I: Performance comparison with baseline Grounding DINO [ 11 ] + SAM [ 12 ] (Grounded SAM) on PASCAL VOC and LVIS.
Pre-training
Validation (mIoU (%))
Method
Dataset
Supervision
PASCAL VOC
PASCAL Context
COCO
ODISE [ 54 ]
COCO
mask
82.7
55.3
52.4
OpenSeg [ 55 ]
COCO
mask
-
42.1
36.1
CLIP-MAE ∗ [ 53 ]
LAION-20M
text & self
-
16.8
11.8
MaskCLIP [ 53 ]
LAION-20M
text & self
-
17.7
11.8
ViewCo [ 19 ]
CC12M+YFCC
text & self
52.4
23.0
23.5
TABLE II: Performance comparison on PASCAL VOC, PASCAL Context, and COCO with open-world semantic segmentation methods.”20k” and ”380k” indicate the size of the dataset. ”20c” represents that the dataset is composed of only 20 categories of VOC, while ”1205c” means we use the full vocabulary V to design the dataset. The superscript ∗ denotes the results are from [ 53 ] . Different from MagicSeg, ODISE [ 54 ] and OpenSeg [ 55 ] use real data with manual annotations. PT stands for pre-training, while FT stands for fine-tuning.
Method
p-mIoU (%)
OVSeg [ 59 ]
22.5
SegCLIP [ 56 ]
32.1
GroupViT [ 18 ]
36.5
CLIP [ 25 ]
27.6
MaskCLIP [ 26 ]
42.0
Groundingdino + SAM [ 11 , 12 ]
53.3
TABLE III: Comparison on zero-shot point prediction on OpenImages-V7.
Segmenter
PASCAL VOC
COCO
Dataset
Class Number
Val (mIoU(%))
Dataset
Class Number
Val (mIoU (%))
DeepLabV3 (R50)
VOC’s training (real: 11.5k)
20
77.4
COCO’s training (real: 118k)
80
48.9
DeepLabV3 (R101)
20
79.9
80
54.9
Mask2Former (R50)
20
77.3
80
57.8
Mask2Former (R50)
DiffuMask [ 1 ] (synthetic: 60k)
20
57.4
-
-
-
DeepLabV3 (R50)
Dataset Diffusion [ 2 ] (synthetic: 40k)
20
61.6
Dataset Diffusion [ 2 ] (synthetic: 80k)
80
32.4
TABLE IV: Performance comparison on PASCAL VOC and COCO with works for synthetic dataset. ”R50” and ”R101” mean resnet50 and restnet101 respectively. Our dataset generation pipeline is primarily designed for open-world segmentation without specific settings for different datasets.
Table 12
Fig. 8: Semantic segmentation results on PASCAL VOC. Compared to GroupViT, MagicSeg handles details such as mask edges more effectively.
Fig. 9: Semantic segmentation results on COCO. Compared to GroupViT, MagicSeg shows a clear advantage in fine-grained category recognition and detail handling. The quality of the masks has significantly improved, reducing the gap with supervised training.
prompt
counterfactual prompt
Simple
A photo of bus .
A photo of nothing .
Ours
In a peaceful countryside setting, a yellow school bus stops by a quaint picket fence, …..
In a peaceful countryside setting, a yellow school nothing stops by a quaint picket fence, …..
TABLE VII: Examples of simple text prompt and MagicSeg’s prompt.
Fig. 10: Semantic segmentation results on images from network with rare classes.
Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.
Christoph Hümmer, Joachim Sicking, Fabian Hüger +1
Institute of Mathematics, Technical University of Berlin · CARIAD SE, Volkswagen Group
Open world image segmentation aims to achieve precise segmentation and semantic understanding of targets within images by addressing the infinitely open set of object categories encountered in the real world. However, traditional closed-set segmentation approaches struggle to adapt to complex open world scenarios, while foundation segmentation models such as SAM exhibit notable discrepancies between their strong segmentation capabilities and relatively weaker semantic understanding. To bridge these discrepancies, we propose WOW-Seg, a Word-free Open World Segmentation model for segmenting and recognizing objects from open-set categories. Specifically, WOW-Seg introduces a novel visual prompt module, Mask2Token, which transforms image masks into visual tokens and ensures their alignment with the VLLM feature space. Moreover, we introduce the Cascade Attention Mask to decouple information across different instances. This approach mitigates inter-instance interference, leading to a significant improvement in model performance. We further construct an open world region recognition test benchmark: the Region Recognition Dataset (RR-7K). With 7,662 classes, it represents the most extensive category-rich region recognition dataset to date. WOW-Seg attains strong results on the LVIS dataset, achieving a semantic similarity of 89.7 and a semantic IoU of 82.4. This performance surpasses the previous SOTA while using only one-eighth the parameter count. These results underscore the strong open world generalization capabilities of WOW-Seg. The code and related resources are available at https://github.com/AAwcAA/WOW-Seg-Meta.
Danyang Li, Tianhao Wu, Bin Li +5
2VCIP, CS, Nankai University · 1NKIARI, Shenzhen Futian · 3AAIS, Nankai University +2
Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.
Haoxiao Wang, Antao Xiang, Haiyang Sun +8
Zhejiang University · South China University of Technology · Nanjing University +1