Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

Christoph Hümmer; Joachim Sicking; Fabian Hüger; Hanno Gottschalk

cs.CVSep 6, 2026

Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

Authors: Christoph Hümmer, Joachim Sicking, Fabian Hüger, Hanno Gottschalk

Organizations: Institute of Mathematics, Technical University of Berlin · CARIAD SE, Volkswagen Group

Abstract

Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.

Figures & tables

Figure 1: Diffuse2Seg: Multi-granular pseudo-labels from diffusion-based self-attention. Our method produces smooth masks with dense coverage via diffusion self-attention, outperforming prior “things”- biased DINO-based state-of-the-art UnSAM in pseudo-label average recall (AR). Training models on our diffusion-based labels improves prior state-of-the-art performance, particularly on dense segmentation datasets covering both “stuff” and “things”.

Figure 2: Schematic depiction of our unsupervised Diffuse2Seg method. Diffuse2Seg generates multi-granular instance masks, without supervision, from a pre-trained diffusion model in three steps. Step 1: self-attention extraction via single-step extraction encodes an input image using the SD2 VAE and performs single-step denoising with the SD2 UNet to collect and aggregate multi-layer self-attention matrices into a single affinity matrix $\mathbf{A}$ . Step 2: prompt-based segment generation initializes a grid of equidistant one-hot prompts and propagates them over the affinity into soft object maps using non-linear edge-stopping p-Laplacian propagation. We then produce multi-granular instance pseudo-labels through map merging, deduplication, and boundary refinement. Step 3: two-round model training trains a lightweight Mask2Former model on the generated pseudo-labels for one round, followed by self-training via crop-based prediction refinement to correct errors and discover fine-grained instances in the second round.

Figure 3: Generation of soft object maps by prompt propagation. An equidistant grid of one-hot point prompts is placed across the latent token grid $(H_{f}\times W_{f})$ as initial seeds (grey dots). Each prompt is propagated over the self-attention affinity graph A via edge-preserving non-linear p-Laplacian propagation ( Equation 2 ) until it reaches $\|\mathbf{f}^{t+1}-\mathbf{f}^{t}\|_{2}^{2}\leq\tau_{\text{prop}}$ , yielding continuous soft object maps. In this example, we show the propagation of a single prompt (green dot).

Method	Backbone	AR S	AR M	AR L	AR 1000
DINO-based labeling
UnSAM ( Wang et al., 2024 )	DINO ViT-B/8	5.7	13.9	26.0	16.4
SOHES ( Cao et al., 2024 )	DINO ViT-B/8	2.8	10.6	15.7	11.0
Diffusion-based labeling
DiffSeg ( Tian et al., 2024 )	SD2	0.3	12.4	29.9	16.0
M2N2 ( Karmann and Urfalioglu, 2025 )	SD2	0.0	2.8	26.5	9.7

Table 1: Comparison of DINO- and diffusion-based pseudo-labeling methods on the SA-1B validation set. We report class-agnostic mask average recall (AR) by object size, with 1000 proposals per image. Methods are grouped by backbone type. The diffusion-based labels of Diffuse2Seg achieve significantly higher average recall compared to previous methods.

Method	Backbone	AR 1000 Across Domains
Method	Backbone	SA-1B	ADE20K	EntitySeg	UVO	PACO
Training-based labeling
CutLER ( Wang et al., 2023 )	DINO R50	17.0	24.8	22.1	32.3	10.7
DINO-based labeling
UnSAM ( Wang et al., 2024 )	DINO ViT-B/8	16.4	16.0	19.3	22.8	9.3
SOHES ( Cao et al., 2024 )	DINO ViT-B/8	11.0	11.6	11.6	19.3	5.2

Table 2: Cross-domain pseudo-label quality , as measured by class-agnostic mask AR 1000 against the ground truth of the respective dataset. Apart from SA-1B, we probe generalization to scene parsing (ADE20K), entity segmentation (EntitySeg), and open-world video objects (UVO). The diffusion features produce more robust labels across domains, as indicated by consistently higher recall values. We keep for all methods the same hyperparameters as for SA-1B. We mainly focus on the performance of the training-free approaches and report CutLER because of its integration into UnSAM and as a widely used training-based reference.

Method	Detector Supervision	# Images	Things			Stuff + Things				Parts
Method	Detector Supervision	# Images	COCO	LVIS	Avg.	ADE	Entity	SA-1B	Avg.	PACO
Supervised
SAM	✗	11M	49.6	46.1	47.9	45.8	45.9	60.8	50.8	18.1
Unsupervised - label generation without a pre-trained detector
CutLER	✗	1.3M	28.1	20.2	24.2	26.3	23.1	17.0	22.1	8.9
SOHES	✗	0.2M	30.5	29.1	29.8	31.1	33.5	33.3	32.6	17.1

Table 3: Comparison of instance segmentation methods with and without auxiliary detector supervision. While the previous state-of-the-art method UnSAM relies on an ImageNet pre-trained detector, our method can also be trained without auxiliary supervision purely on diffusion-model-based labels. Combining our method with CutLER supervision positions it as complementary to the object-biased labels of UnSAM. In fact, dense supervision leads to significant improvements on stuff+things datasets while performing on par on things-only datasets.

Figure 4: Semi-supervised extensions of UnSAM and Diffuse2Seg. Mean average recall vs. the number of labeled fine-tuning images. 5k images suffice for Diffuse2Seg to match the 100k supervised UnSAM reference.

Panel of Figure 5 — Figure 5: Impact of propagation exponent and denoising timestep on Diffuse2Seg performance . Panel : mAR@1000 peaks at $p=1.2$ , while mAP@1000 peaks at the selected $p=1.6$ . Panel : each metric is normalized to its own best timestep; recall, particularly for small objects, degrades earlier than mAP. Dashed lines mark the selected $p=1.6$ and $t=150$ . All metrics use at most 1000 detections.

Table 4: Diffuse2Seg mAR as a function of number of hierarchy levels and NMS threshold for deduplication. We find that a higher number of hierarchy levels and a conservative

\tau_{\mathrm{IoU}}

in our mask merging and deduplication consistently boosts recall.

Table 5: Impact of label sources and training stages. (a) Starting from global Diffuse2Seg pseudo-labels, the rows add crop-refined child masks and CutLER masks, respectively. Crop refinement adds locally re-inferred, parent-contained child masks. All rows use a

0.1

M image budget. (b) Self-training with our presented strategies significantly improves the model performance by correcting and filling gaps of the initial pseudo-labels.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Parameter sweeps of Diffuse2Seg label generation on a SA-1B holdout set. Each panel varies a single parameter with all others held at a reference configuration.

\mathrm{AR}_{1000}

(blue, left axis) and

\mathrm{mAP}

(purple, right axis) are shown at their true values, using identical scales across all panels to enable direct comparison of parameter sensitivities. In some cases a shared optimum for both metrics ( e.g.,

\tau_{\mathrm{att}}

) can be observed. However, often there is tradeoff, for instance recall being stable but precision decreasing ( e.g.,

w_{1}

L

), and trade-offs of in-domain recall and precision ( e.g.,

p

). Moreover, increasing some parameters incurs higher computational cost, such as grid spacing

s

and resolution

r

Figure 7: Qualitative comparison of semi-supervised instance segmentation on ADE20K. Each column (a–d) shows a different indoor/outdoor scene. Top: raw input image, middle: predictions of the UnSAM checkpoint fine-tuned on 10k labeled images, bottom: predictions of our Diffuse2Seg model fine-tuned on 10k labeled images. Under the same 10k-label budget, initializing from our diffusion-based unsupervised model yields higher mask coverage, crisper object boundaries, and better segmentation of fine structures ( e.g., furniture legs, windows, and facade details).

Explore similar work

Diffusion Model as a Generalist Segmentation Learner
Apr 27, 2026Haoxiao Wang, Antao Xiang, Haiyang Sun +8Image Segmentation Text-Guided Image Segmentation
Zhejiang University · South China University of Technology · Nanjing University +1
MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation
Mar 20, 2026Kaixin Cai, Pengzhen Ren, Jianhua Han +4Semantic Segmentation Open-Vocabulary Semantic Segmentation
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · PengCheng Laboratory · Huawei Noah’s Ark Lab
LlamaSeg: Image Segmentation via Autoregressive Mask Generation
May 26, 2025Jiru Deng, Tengjin Weng, Tianyu Yang +3Image Segmentation Text-Guided Image Segmentation
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2

Apr 27, 2026J/K move · Enter open · S save

Diffusion Model as a Generalist Segmentation Learner

Haoxiao Wang, Antao Xiang, Haiyang Sun, Peilin Sun+7

Zhejiang University · South China University of Technology · Nanjing University+1

Image Segmentation Text-Guided Image Segmentation Open-Vocabulary Semantic Segmentation Semantic Segmentation Diffusion Models

Open paper AskPDF arXiv

Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.

Abstract

Explore similar work

Diffusion Model as a Generalist Segmentation Learner

MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation

LlamaSeg: Image Segmentation via Autoregressive Mask Generation