Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generator

Kumju Jo; Heesun Jung; Sungyong Baik

cs.CVAug 27, 2026

Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generator

Authors: Kumju Jo, Heesun Jung, Sungyong Baik

Organizations: Dept. of Artificial Intelligence, Hanyang University · Dept. of Data Science, Hanyang University

Abstract

Open-vocabulary semantic segmentation (OVSS) aims to segment image regions corresponding to arbitrary text queries. Although the Segment Anything Model (SAM) is a powerful foundation model for segmentation, its standalone performance on OVSS remains limited. Existing methods therefore often use SAM to refine coarse masks predicted by other models, but this strategy is unreliable when the initial masks are inaccurate. In this work, we argue that more reliable segmentation can be achieved by exploiting SAM as a region expansion module guided by accurate object points (i.e., seeds) rather than inaccurate coarse masks. Inspired by classical seeded segmentation, we reformulate OVSS as text-guided seed localization followed by seed-based region expansion. To realize this idea, we propose Text-to-Seed (T2S), a training-free framework that leverages the text-to-region correspondence of Stable Diffusion to generate attention-based seed points for target categories described by text. These sparse seeds are then used as point prompts for SAM to produce full object masks. Without task-specific training or additional annotations, T2S achieves strong performance on standard OVSS benchmarks, demonstrating the effectiveness of combining semantic grounding with seed-driven spatial segmentation.

Figures & tables

Figure 1: Overview of our framework, dubbed T2S. We extract seeds from attention maps of Stable Diffusion. Then, seeds are fed into SAM to perform region expansion.

Figure 2: (a) The simplified overview of our framework . (b) Initialization of seed generator is performed via Stable Diffusion (SD) with EoT token weighting, followed by aggregating attention maps. Once attention maps are aggregated, SD is no longer needed. (c) Seed generation and Spreading (SGS) block extracts and spreads seeds from aggregated attention maps to localize a given category. (d) Iterative Seed Generation and Spreading process iteratively applies SGS block that extracts both positive seeds for categories and negative seeds for irrelevant or background, which are iteratively refined to high resolution. Since this process is performed on already aggregated attention maps, it can be performed efficiently. Then, (e) Region Expansion process via SAM is performed by feeding positive seeds and negative seeds into SAM as point prompts to perform region expansion to get a final mask.

Figure 3: Visualization of how aggregated cross-attention (ACA) maps change with different starting denoising steps.

Figure 4: Visualization of cross-attention aggregation (ACA) maps at different resolutions in Stable Diffusion.

					mIoU
Model	training-free	supervision	preprocess	SAM	VOC-20	VOC-21	Cityscapes	ADE	Objects	Pascal59	Pascal60
Weakly-supervised
ReCo [ 89 ]	✗	✓	✗	✗	57.7	25.1	21.1	11.2	15.7	22.3	19.9
GroupViT [ 114 ]	✗	✓	✗	✗	79.7	50.4	11.1	9.2	27.5	23.4	17.7
MaskCLIP [ 133 ]	✗	✓	✗	✗	74.9	38.8	12.6	9.8	20.6	26.4	23.6
TCL [ 18 ]	✗	✓	✗	✗	83.2	55.0	24.0	17.1	31.6	33.9	30.4

Table 1: Comparison with open-vocabulary semantic segmentation models on Pascal VOC [ 31 ] , Cityscapes [ 25 ] , ADE20K [ 132 ] , COCO objects [ 67 ] and Pascal Context [ 75 ] . The performance of other models is reported as in the respective papers. Best and second best results are highlighted in bold and underlined , respectively.

Figure 5: Illustration of the process that iteratively aggregates masks before generating the final mask.

Figure 6: Visualization of how aggregated self-attention mask ${{\mathcal{M}}}_{s_{k}}$ evolves over the iterations in iterative seed generation and spreading. Through the progress, ${{\mathcal{M}}}_{s_{k}}$ is shown to exhibit more fine-grained and sparse attention masks. Thus, at the end of the iterative seed generation and spreading, we can extract precise seeds at a resolution of $64\times 64$ , which are then passed to SAM for region expansion.

Figure 7: Qualitative comparison of T2S with the state-of-the-art method (CaR) under open-vocabulary scenarios

Figure 8: Qualitative results of T2S under multi subject scenarios.

Init	SE		ISSS	VOC
Token weight	Seed generation	Negative seed	Iterative SSS	mIoU
-	✓	-	-	73.1
✓	✓	-	-	76.8
✓	✓	✓	-	82.3
✓	✓	✓	✓	92.1

Table 2: Ablation on our proposed module. Init denotes the Initialization of Seed Generator, SG denotes Seed Generator, ISSS denotes Iterative Seeded Semantic Segmentation from our method. All experiments are conducted on Pascal VOC 20.

	3/4 T	1/2 T	1/4 T
mIoU	90.4	92.1	89.7

Table 3: Ablation study on the selection of the timestep at which to initiate the denoising process.

Resolutions	mIoU
8	88.2
8, 16	92.1
8, 16, 32	82.4
8, 16, 32, 64	76.3

Table 4: mIoU by aggregated cross-attention (ACA) map resolutions used for seed initialization in SGS.

$\alpha$	mIoU
0.3	87.2
0.4	91.7
0.5	92.1
0.6	90.3
0.7	84.2

Table 5: mIoU by seed generation threshold

\alpha

	$R=1$	$R=5$	Ours
mIoU	82.3	86.8	92.1

Table 6: Ablation study comparing fixed iteration counts (

R

) and our adaptive rejection-based strategy using CLIP.

	CaR+SAM	T2S (Ours)
mIoU	81.3	83.2

Table 7: Generalization performance on ISPRS Potsdam dataset.

Metric	Value
Inference time	188 s/image (3.13 min)
Peak GPU memory	$\sim$ 20 GB
Average number of iterations	9.1/class
SAM image-encoder calls	1/image
SAM mask-decoder calls	311.7/image
CLIP mask-verification calls	21.8/image

Table 8: Computational cost of T2S. Runtime, GPU memory, and model-call counts are reported per image, while the average number of iterations is reported per queried class. CLIP calls correspond to mask verification.

	$n=1$	$n=2$	$n=3$	$n>5$
Seed Generator	3623	3630	3618	3651

Table 9: The analysis of computational complexity with respect to the number of objects, measured in GFLOPs.

	GPU Memory	GFLOPs
CaR [ 97 ]	8.6 GB	338
FreeDA [ 8 ]	23.4 GB	588
OVDiff [ 48 ]	21.0 GB	1492
T2S (Ours)	20.0 GB	3851

Table 10: Peak GPU memory and computational-cost comparison with representative training-free OVSS baselines on VOC-20. Memory usage is reported in GB, and computational cost is reported in GFLOPs.

Figure 9: Qualitative results of our method on background classes in the Pascal59 dataset.

Figure 10: Qualitative comparison of T2S and CaR on small objects and occlusion.

Figure 11: Qualitative comparison of T2S with simpler alternatives.

Figure 12: Qualitative analysis across object scales and scene complexity.

Figure 13: Qualitative results with failure analysis . (a) Successful case shows the very fine-grained segmentation output. (b) The failure cases show the object missed by T2S (highlighted by a dotted red box). T2S may fail in scenes with severe occlusion, overlapping target and non-target regions, or weak object boundaries. These cases can lead to missed instances or leakage into nearby objects.

Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation labels image pixels using text-defined categories beyond a fixed training vocabulary.

See all 87 papers

Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation
Aug 4, 2026Wanli Ma, Jiangwen Lu, Qinmu Peng +1Open-Vocabulary Semantic Segmentation Vision-Language Alignment
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · National Anti-counterfeit Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China · School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China
dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
Mar 19, 2026Saikat Dutta, Biplab Banerjee, Hamid RezatofighiOpen-Vocabulary Semantic Segmentation Vision-Language Models
IITB-Monash Research Academy · IIT Bombay · Monash University
SegRAG: Retrieval Augmented Spatial Prompting for Open Vocabulary Semantic Segmentation
May 17, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedImage Segmentation Promptable Image Segmentation
Khalifa University, United Arab Emirates.

Aug 4, 2026J/K move · Enter open · S save

Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

Wanli Ma, Jiangwen Lu, Qinmu Peng, Xinge You

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · National Anti-counterfeit Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China · School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China

Open-Vocabulary Semantic Segmentation Vision-Language Alignment VLM Calibration

Open paper AskPDF arXiv

Training-free open-vocabulary semantic segmentation (OVSS) partitions an image into semantically distinct regions based on arbitrary text descriptions, without learning any additional parameters. However, existing methods typically focus on improving visual representations while treating text embeddings that encode only generic category concepts as fixed classification references. The resulting semantic gap between these generic concepts and the visual representations that capture the specific appearances of target instances often causes incomplete masks and erroneous predictions in non-target regions. Inspired by the symbol-percept correspondence underlying perceptual anchoring, we propose Prototype-Guided Text Calibration (PTC) for training-free OVSS. In the Perceiving stage, PTC selects reliable visual evidence based on initial matching scores to construct category-specific visual prototypes. In the Anchoring stage, PTC uses these prototypes to calibrate their corresponding text embeddings, with the calibration strength adaptively adjusted based on the amount of visual evidence. Consequently, the calibrated text embeddings align more accurately with instance-specific visual representations while preserving generic category semantics and open-vocabulary generalization. Moreover, PTC requires neither additional training nor external models and can serve as a plug-and-play module for existing methods. Extensive experiments across eight benchmarks show that PTC significantly enhances the performance of six representative methods and yields more complete and accurate segmentation results. These results validate PTC as a simple and effective approach to improving visual-text alignment.

Abstract

Open-Vocabulary Semantic Segmentation

Explore similar work

Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3

SegRAG: Retrieval Augmented Spatial Prompting for Open Vocabulary Semantic Segmentation