Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

Jiahao Li; Yang Lu; Yachao Zhang; Fangyong Wang; Yuan Xie; Yanyun Qu

cs.CVApr 9, 2026

Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

Authors: Jiahao Li, Yang Lu, Yachao Zhang, Fangyong Wang, Yuan Xie, Yanyun Qu

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University. · Hanjiang National Laboratory. · East China Normal University.

Abstract

Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, \ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.

Figures & tables

Figure 1: Compared with the logits-optimization methods seeking optimal solutions, we reformulate the problem as an analytical solution form. This reformulation confers three key advantages: independence from GT annotations, elimination of time-consuming training, and freedom from model-specific modulation.

Figure 2: Overview of the proposed method. The pipeline begins by computing the cosine similarity (Cos.) between vision-language features to obtain logits, followed by non-maximum suppression (NMS) and normalization (Norm.) to filter low-confidence patches. The core step involves calculating the distribution discrepancy between the normalized logits and the degenerate distribution map. This solved discrepancy is then upsampled to the original image resolution via joint bilateral upsampling (JBU) to obtain the discrepancy map. Finally, the semantic map is generated by applying an $\arg\max$ operation to the discrepancy map.

Figure 3: Reformulation analysis results. (a) Performance comparison targeting the degenerate distribution $\mathcal{S}$ and the GT distribution $\mathcal{P}$ demonstrates that the equivalence of the two distributions. (b) KL divergence visualization across three transmission scenarios confirms distribution discrepancy’s capacity to capture semantic information.

Model	Logits Model	Paradigm	VOC21	Context60	COCO-Obj	VOC20	Context59	COCO-Stuff	Cityscapes	ADE20K	Avg.
FreeDa [ 2 ]	CLIP ViT-B/16	M.M.	-	-	-	85.6	43.1	27.8	36.7	22.4	-
CLIP [ 35 ]	CLIP ViT-B/16	I.T.	18.6	7.8	6.5	49.1	11.2	7.2	6.7	3.2	13.8
ReCo [ 40 ]	CLIP ViT-B/16	M.M.	25.1	19.9	15.7	57.7	22.3	14.8	21.6	11.2	23.5
MaskCLIP [ 58 ]	CLIP ViT-B/16	I.T.	38.3	23.6	20.6	74.9	26.4	16.4	12.6	9.8	27.9
GroupViT [ 51 ]	CLIP ViT-B/16	I.T.	50.4	18.7	27.5	79.7	23.4	15.3	11.1	9.2	29.4
CLIPtrase [ 37 ]	CLIP ViT-B/16	M.M.	50.9	29.9	43.6	81.0	33.8	22.8	21.3	16.4	32.7

Table 1: Quantitative evaluation on standard benchmarks (unit: %).

\dagger

: Re-implementation with OpenAI’s weights.

\ddagger

: Re-implementation with its DINO-B/16 variant. M.M. denotes model-specific modulation. I.T. denotes iterative training. Here, the best results are shown in bold and the second-best results are underlined.

Figure 4: Visualization of outputs from each stage of our pipeline. We present the logits and distribution discrepancies for each category across the input image, segmentation maps, and GT. Predictions $1$ and $2$ represent the segmentation maps obtained by solving the optimal path and maximum velocity formulations, respectively. Visualization of both distribution discrepancies demonstrates that the optimal path mode is more sensitive to high-frequency regions than the maximum velocity mode. For instance, the visualization results from the optimal path mode in the second image can clearly depict the internal contours of the dog.

Figure 5: Quantitative evaluation of SD2 time step, attention resolution, threshold $\tau$ and foundation model on standard benchmarks is presented, with each metric represented by a distinct color (unit: %). Here, SD2 time step denotes the denoising step in SD2, and attention resolution refers to the input size of SD2. All comparisons were performed using the maximum velocity mode.

	Components	VOC21	COCO-Stuff	Cityscapes	ADE20K	Avg.
(I)	Baseline	18.6	7.2	6.7	3.2	8.9
(II)	(I) + KL	44.2	12.1	8.6	6.4	17.8
(III)	(II) + NMS	45.9	13.0	9.6	7.7	19.1
(IV)	(III) + JBU	46.3	13.3	10.1	8.8	19.6
(V)	(IV) + O.P.	66.9	28.6	41.7	22.8	40.0
(VI)	(IV) + M.V.	67.8	28.9	43.3	23.0	40.8

Table 2: Effect of different components (unit: %). Baseline : apply

\arg\max

to the raw logits without optimization or analytical solution. KL : Kullback-Leibler (KL) divergence. O.P. : solving the optimal path. M.V. : solving the maximum velocity.

	Variants	VOC21	COCO-Stuff	Cityscapes	ADE20K	Avg.
self-attention weight combination ( $\text{down}_{0},\text{down}_{1},\text{up}_{0},\text{up}_{1},\text{up}_{2}$ )
(I)	( $1,0,0,0,0$ )	62.1	24.6	37.5	17.6	35.5
(II)	( $0,1,0,0,0$ )	63.3	26.1	40.1	18.5	37.0
(III)	( $0,0,1,0,0$ )	65.0	26.7	41.9	19.6	38.3
(IV)	( $0,0,0,1,0$ )	66.2	27.8	42.8	21.8	39.7
(V)	( $0,0,0,0,1$ )	65.3	26.9	42.4	20.2	38.7

Table 3: Effect of different strategy for our components (unit: %). q-q mean : the average attention tensor between queries across all layers. k-k mean : the average attention tensor between keys across all layers. All comparisons were performed using the maximum velocity mode.

Figure 6: Quantitative evaluation of SD2 time step, attention resolution, and foundation model on standard benchmarks is presented, with each metric represented by a distinct color (unit: %). Here, SD2 time step denotes the denoising step in SD2, and attention resolution refers to the input size of SD2. All comparisons were performed using the optimal path mode.

	Variants	VOC21	COCO-Stuff	Cityscapes	ADE20K	Avg.
self-attention weight combination ( $\text{down}_{0},\text{down}_{1},\text{up}_{0},\text{up}_{1},\text{up}_{2}$ )
(I)	( $1,0,0,0,0$ )	61.1	23.5	36.4	16.5	34.4
(II)	( $0,1,0,0,0$ )	62.2	25.1	38.2	17.6	35.8
(III)	( $0,0,1,0,0$ )	64.0	25.6	40.6	18.3	37.1
(IV)	( $0,0,0,1,0$ )	65.1	26.4	41.5	20.8	38.5
(V)	( $0,0,0,0,1$ )	64.1	25.4	41.3	19.2	37.5

Table 4: Effect of different strategy for our components (unit: %). q-q mean : the average attention tensor between queries across all layers. k-k mean : the average attention tensor between keys across all layers. All comparisons were performed using the optimal path mode.

Model	FLOPs(G) $\downarrow$	Params(M) $\downarrow$	Inference time(sec.) $\downarrow$	mIoU(%) $\uparrow$
CLIP	52.2	149.6	0.08	18.6
ProxyCLIP	103.2	235.4	0.16	59.1
CASS	1675.5	265.7	3.11	65.8
Ours (O.P.)	351.3	1006.8	0.59 (-2.52)	66.9 (+1.1)
Ours (M.V.)	339.2	1006.8	0.57 (-2.54)	67.8 (+2.0)
the inference time for each component

Table 11

Model	VOC21	Context60	ADE20K	COCO-Stuff	Cityscapes
original CLIP-B/16	18.6	7.8	3.2	7.2	6.7
integrating self-attention via maximum velocity
- CLIP-B/16	58.5 (+39.9)	30.1 (+22.3)	17.3 (+14.1)	24.1 (+16.9)	35.7 (+29.0)
- DINO-B/8	64.8	36.3	20.4	26.3	41.3
- DINOv2-B/14	66.9	36.9	21.3	27.9	42.7
- DINOv2-B/14 w/ Registers	69.8	38.0	22.3	29.3	44.6

Table 12

Model	Scale	VOC21	Context60	COCO-Obj	VOC20	Context59	COCO-Stuff	Cityscapes	ADE20K
CASS	B/16	65.8	36.7	37.8	87.8	40.2	26.7	39.4	20.4
w/ O.P.		68.2 (+2.4)	38.7 (+2.0)	40.1 (+2.3)	89.6 (+1.8)	45.5 (+5.3)	29.4 (+2.7)	42.8 (+3.4)	22.9 (+2.5)
w/ M.V.		69.5 (+3.7)	39.1 (+2.4)	40.6 (+2.8)	90.0 (+2.2)	45.9 (+5.7)	29.8 (+3.1)	44.4 (+5.0)	23.1 (+2.7)
SC-CLIP	B/16	64.6	36.8	37.7	84.3	40.1	26.6	41.0	20.1
w/ O.P.		68.1 (+3.5)	38.7 (+1.9)	40.0 (+2.3)	89.2 (+4.9)	45.3 (+5.2)	29.3 (+2.7)	42.9 (+1.9)	22.8 (+2.7)
w/ M.V.		69.3 (+4.7)	39.1 (+2.3)	40.6 (+2.9)	89.6 (+5.3)	45.7 (+5.6)	29.8 (+3.2)	44.6 (+3.6)	23.0 (+2.9)

Table 14

Figure 7: Visualization of segmentation maps on COCO-Stuff benchmark dataset.

Figure 8: Visualization of segmentation maps on Pascal VOC benchmark dataset.

Figure 9: Visualization of segmentation maps on ADE150k benchmark dataset.

Figure 10: Visualization of segmentation maps on Pascal Context benchmark dataset.

Figure 11: Visualization of maximum velocity maps across different threshold.

Figure 12: Visualization of optimal path maps across different iterations.

Explore similar work

Sep 28, 2026

Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement

Open-vocabulary semantic segmentation (OVSS) enables pixel-level prediction over arbitrary text-specified vocabularies and has shown strong generalization on common benchmarks. However, OVSS performance often degrades in specialized domains such as medical imaging, remote sensing, and industrial inspection, where dense pixel-level masks for adaptation are costly to obtain and require domain-specific expertise. We propose a preference-guided adaptation framework that replaces dense mask supervision with binary preferences. We observe that different prompt templates produce systematically different segmentations for the same image, a phenomenon we call prompt disagreement, and we repurpose it as a built-in source of preference supervision. Building on this, we mine localized preference queries from regions of high cross-template uncertainty, and adapt the OVSS model with Region-Localized Preference Optimization (RLPO) together with consistency regularization that stabilizes updates outside the queried region. Across extensive experiments on the MESS benchmark, the proposed method achieves consistent gains across diverse OVSS backbones without any pixel-level annotation, and remains effective under noisy preferences. Our code is available at https://github.com/blue-531/pref-ovss.

Hyun-Kurl Jang, Jihun Kim, Kuk-Jin Yoon

Visual Intelligence Lab. KAIST

Weakly Supervised Semantic Segmentation Preference Optimization Open-Vocabulary Semantic Segmentation

Ask PDF

Apr 9, 2026

OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentation

Training-free open-vocabulary semantic segmentation(TF-OVSS) has recently attracted attention for its ability to perform dense prediction by leveraging the pretrained knowledge of large vision and vision-language models, without requiring additional training. However, due to the limited input resolution of these pretrained encoders, existing TF-OVSS methods commonly adopt a sliding-window strategy that processes cropped sub-images independently. While effective for managing high-resolution inputs, this approach prevents global attention over the full image, leading to fragmented feature representations and limited contextual reasoning. We propose OV-Stitcher, a training-free framework that addresses this limitation by stitching fragmented sub-image features directly within the final encoder block. By reconstructing attention representations from fragmented sub-image features, OV-Stitcher enables global attention within the final encoder block, producing coherent context aggregation and spatially consistent, semantically aligned segmentation maps. Extensive evaluations across eight benchmarks demonstrate that OV-Stitcher establishes a scalable and effective solution for open-vocabulary segmentation, achieving a notable improvement in mean Intersection over Union(mIoU) from 48.7 to 50.7 compared with prior training-free baselines.

Seungjae Moon, Seunghyun Oh, Youngmin Ro

Machine Intelligence Laboratory, University of Seoul, Korea

Open-Vocabulary Semantic Segmentation

Ask PDF

Feb 26, 2026

Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?

Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.

Tilemachos Aravanis, Vladan Stojnić, Bill Psomas +2

VRG, FEE, Czech Technical University in Prague · University of Crete · Archimedes, Athena RC +1

Open-Vocabulary Semantic Segmentation Test-Time Adaptation of VLMs Few-Shot Semantic Segmentation

Ask PDF