Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.
Figures & tables
Figure 1: Diffuse2Seg: Multi-granular pseudo-labels from diffusion-based self-attention. Our method produces smooth masks with dense coverage via diffusion self-attention, outperforming prior “things”- biased DINO-based state-of-the-art UnSAM in pseudo-label average recall (AR). Training models on our diffusion-based labels improves prior state-of-the-art performance, particularly on dense segmentation datasets covering both “stuff” and “things”.
Figure 2: Schematic depiction of our unsupervised Diffuse2Seg method. Diffuse2Seg generates multi-granular instance masks, without supervision, from a pre-trained diffusion model in three steps. Step 1: self-attention extraction via single-step extraction encodes an input image using the SD2 VAE and performs single-step denoising with the SD2 UNet to collect and aggregate multi-layer self-attention matrices into a single affinity matrix A . Step 2: prompt-based segment generation initializes a grid of equidistant one-hot prompts and propagates them over the affinity into soft object maps using non-linear edge-stopping p-Laplacian propagation. We then produce multi-granular instance pseudo-labels through map merging, deduplication, and boundary refinement. Step 3: two-round model training trains a lightweight Mask2Former model on the generated pseudo-labels for one round, followed by self-training via crop-based prediction refinement to correct errors and discover fine-grained instances in the second round.
Figure 3: Generation of soft object maps by prompt propagation. An equidistant grid of one-hot point prompts is placed across the latent token grid (Hf×Wf) as initial seeds (grey dots). Each prompt is propagated over the self-attention affinity graph A via edge-preserving non-linear p-Laplacian propagation ( Equation 2 ) until it reaches ∥ft+1−ft∥22≤τprop , yielding continuous soft object maps. In this example, we show the propagation of a single prompt (green dot).
Method
Backbone
AR S
AR M
AR L
AR 1000
DINO-based labeling
UnSAM ( Wang et al., 2024 )
DINO ViT-B/8
5.7
13.9
26.0
16.4
SOHES ( Cao et al., 2024 )
DINO ViT-B/8
2.8
10.6
15.7
11.0
Diffusion-based labeling
DiffSeg ( Tian et al., 2024 )
SD2
0.3
12.4
29.9
16.0
M2N2 ( Karmann and Urfalioglu, 2025 )
SD2
0.0
2.8
26.5
9.7
Table 1: Comparison of DINO- and diffusion-based pseudo-labeling methods on the SA-1B validation set. We report class-agnostic mask average recall (AR) by object size, with 1000 proposals per image. Methods are grouped by backbone type. The diffusion-based labels of Diffuse2Seg achieve significantly higher average recall compared to previous methods.
Method
Backbone
AR 1000 Across Domains
SA-1B
ADE20K
EntitySeg
UVO
PACO
Training-based labeling
CutLER ( Wang et al., 2023 )
DINO R50
17.0
24.8
22.1
32.3
10.7
DINO-based labeling
UnSAM ( Wang et al., 2024 )
DINO ViT-B/8
16.4
16.0
19.3
22.8
9.3
SOHES ( Cao et al., 2024 )
DINO ViT-B/8
11.0
11.6
11.6
19.3
5.2
Table 2: Cross-domain pseudo-label quality , as measured by class-agnostic mask AR 1000 against the ground truth of the respective dataset. Apart from SA-1B, we probe generalization to scene parsing (ADE20K), entity segmentation (EntitySeg), and open-world video objects (UVO). The diffusion features produce more robust labels across domains, as indicated by consistently higher recall values. We keep for all methods the same hyperparameters as for SA-1B. We mainly focus on the performance of the training-free approaches and report CutLER because of its integration into UnSAM and as a widely used training-based reference.
Method
Detector Supervision
# Images
Things
Stuff + Things
Parts
COCO
LVIS
Avg.
ADE
Entity
SA-1B
Avg.
PACO
Supervised
SAM
✗
11M
49.6
46.1
47.9
45.8
45.9
60.8
50.8
18.1
Unsupervised - label generation without a pre-trained detector
CutLER
✗
1.3M
28.1
20.2
24.2
26.3
23.1
17.0
22.1
8.9
SOHES
✗
0.2M
30.5
29.1
29.8
31.1
33.5
33.3
32.6
17.1
Table 3: Comparison of instance segmentation methods with and without auxiliary detector supervision. While the previous state-of-the-art method UnSAM relies on an ImageNet pre-trained detector, our method can also be trained without auxiliary supervision purely on diffusion-model-based labels. Combining our method with CutLER supervision positions it as complementary to the object-biased labels of UnSAM. In fact, dense supervision leads to significant improvements on stuff+things datasets while performing on par on things-only datasets.
Figure 4: Semi-supervised extensions of UnSAM and Diffuse2Seg. Mean average recall vs. the number of labeled fine-tuning images. 5k images suffice for Diffuse2Seg to match the 100k supervised UnSAM reference.
Figure 5: Impact of propagation exponent and denoising timestep on Diffuse2Seg performance . Panel : mAR@1000 peaks at p=1.2 , while mAP@1000 peaks at the selected p=1.6 . Panel : each metric is normalized to its own best timestep; recall, particularly for small objects, degrades earlier than mAP. Dashed lines mark the selected p=1.6 and t=150 . All metrics use at most 1000 detections.
Table 4: Diffuse2Seg mAR as a function of number of hierarchy levels and NMS threshold for deduplication. We find that a higher number of hierarchy levels and a conservative τIoU in our mask merging and deduplication consistently boosts recall.
Table 5: Impact of label sources and training stages. (a) Starting from global Diffuse2Seg pseudo-labels, the rows add crop-refined child masks and CutLER masks, respectively. Crop refinement adds locally re-inferred, parent-contained child masks. All rows use a 0.1 M image budget. (b) Self-training with our presented strategies significantly improves the model performance by correcting and filling gaps of the initial pseudo-labels.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Parameter sweeps of Diffuse2Seg label generation on a SA-1B holdout set. Each panel varies a single parameter with all others held at a reference configuration. AR1000 (blue, left axis) and mAP (purple, right axis) are shown at their true values, using identical scales across all panels to enable direct comparison of parameter sensitivities. In some cases a shared optimum for both metrics ( e.g., τatt ) can be observed. However, often there is tradeoff, for instance recall being stable but precision decreasing ( e.g., w1 , L ), and trade-offs of in-domain recall and precision ( e.g., p ). Moreover, increasing some parameters incurs higher computational cost, such as grid spacing s and resolution r .
Figure 7: Qualitative comparison of semi-supervised instance segmentation on ADE20K. Each column (a–d) shows a different indoor/outdoor scene. Top: raw input image, middle: predictions of the UnSAM checkpoint fine-tuned on 10k labeled images, bottom: predictions of our Diffuse2Seg model fine-tuned on 10k labeled images. Under the same 10k-label budget, initializing from our diffusion-based unsupervised model yields higher mask coverage, crisper object boundaries, and better segmentation of fine structures ( e.g., furniture legs, windows, and facade details).
Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.
Haoxiao Wang, Antao Xiang, Haiyang Sun +8
Zhejiang University · South China University of Technology · Nanjing University +1
Open-world semantic segmentation presently relies significantly on extensive image-text pair datasets, which often suffer from a lack of fine-grained pixel annotations on sufficient categories. The acquisition of such data is rendered economically prohibitive due to the substantial investments of both human labor and time. In light of the formidable image generation capabilities of diffusion models, we introduce a novel diffusion model-driven pipeline for automatically generating datasets tailored to the needs of open-world semantic segmentation, named "MagicSeg". Our MagicSeg initiates from class labels and proceeds to generate high-fidelity textual descriptions, which in turn serve as guidance for the diffusion model to generate images. Rather than only generating positive samples for each label, our process encompasses the simultaneous generation of corresponding negative images, designed to serve as paired counterfactual samples for contrastive training. Then, to provide a self-supervised signal for open-world segmentation pretraining, our MagicSeg integrates an open-vocabulary detection model and an interactive segmentation model to extract object masks as precise segmentation labels from images based on the provided category labels. By applying our dataset to the contrastive language-image pretraining model with the pseudo mask supervision and the auxiliary counterfactual contrastive training, the downstream model obtains strong performance on open-world semantic segmentation. We evaluate our model on PASCAL VOC, PASCAL Context, and COCO, achieving SOTA with performance of 62.9%, 26.7%, and 40.2%, respectively, demonstrating our dataset's effectiveness in enhancing open-world semantic segmentation capabilities. Project website: https://github.com/ckxhp/magicseg.
Kaixin Cai, Pengzhen Ren, Jianhua Han +4
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · PengCheng Laboratory · Huawei Noah’s Ark Lab
We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance (dAHD) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at https://github.com/GML-FMGroup/llamaseg.
Jiru Deng, Tengjin Weng, Tianyu Yang +3
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2