cs.CVApr 9, 2026
SaveDirect Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University. · Hanjiang National Laboratory. · East China Normal University.
Abstract
Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, \ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.
Figures & tables
Figure 1: Compared with the logits-optimization methods seeking optimal solutions, we reformulate the problem as an analytical solution form. This reformulation confers three key advantages: independence from GT annotations, elimination of time-consuming training, and freedom from model-specific modulation.
Figure 2: Overview of the proposed method. The pipeline begins by computing the cosine similarity (Cos.) between vision-language features to obtain logits, followed by non-maximum suppression (NMS) and normalization (Norm.) to filter low-confidence patches. The core step involves calculating the distribution discrepancy between the normalized logits and the degenerate distribution map. This solved discrepancy is then upsampled to the original image resolution via joint bilateral upsampling (JBU) to obtain the discrepancy map. Finally, the semantic map is generated by applying an operation to the discrepancy map.
Figure 3: Reformulation analysis results. (a) Performance comparison targeting the degenerate distribution and the GT distribution demonstrates that the equivalence of the two distributions. (b) KL divergence visualization across three transmission scenarios confirms distribution discrepancy’s capacity to capture semantic information.
| Model | Logits Model | Paradigm | VOC21 | Context60 | COCO-Obj | VOC20 | Context59 | COCO-Stuff | Cityscapes | ADE20K | Avg. |
| FreeDa [ 2 ] | CLIP ViT-B/16 | M.M. | - | - | - | 85.6 | 43.1 | 27.8 | 36.7 | 22.4 | - |
| CLIP [ 35 ] | CLIP ViT-B/16 | I.T. | 18.6 | 7.8 | 6.5 | 49.1 | 11.2 | 7.2 | 6.7 | 3.2 | 13.8 |
| ReCo [ 40 ] | CLIP ViT-B/16 | M.M. | 25.1 | 19.9 | 15.7 | 57.7 | 22.3 | 14.8 | 21.6 | 11.2 | 23.5 |
| MaskCLIP [ 58 ] | CLIP ViT-B/16 | I.T. | 38.3 | 23.6 | 20.6 | 74.9 | 26.4 | 16.4 | 12.6 | 9.8 | 27.9 |
| GroupViT [ 51 ] | CLIP ViT-B/16 | I.T. | 50.4 | 18.7 | 27.5 | 79.7 | 23.4 | 15.3 | 11.1 | 9.2 | 29.4 |
| CLIPtrase [ 37 ] | CLIP ViT-B/16 | M.M. | 50.9 | 29.9 | 43.6 | 81.0 | 33.8 | 22.8 | 21.3 | 16.4 | 32.7 |
Table 1: Quantitative evaluation on standard benchmarks (unit: %). : Re-implementation with OpenAI’s weights. : Re-implementation with its DINO-B/16 variant. M.M. denotes model-specific modulation. I.T. denotes iterative training. Here, the best results are shown in bold and the second-best results are underlined.
Figure 4: Visualization of outputs from each stage of our pipeline. We present the logits and distribution discrepancies for each category across the input image, segmentation maps, and GT. Predictions and represent the segmentation maps obtained by solving the optimal path and maximum velocity formulations, respectively. Visualization of both distribution discrepancies demonstrates that the optimal path mode is more sensitive to high-frequency regions than the maximum velocity mode. For instance, the visualization results from the optimal path mode in the second image can clearly depict the internal contours of the dog.
Figure 5: Quantitative evaluation of SD2 time step, attention resolution, threshold and foundation model on standard benchmarks is presented, with each metric represented by a distinct color (unit: %). Here, SD2 time step denotes the denoising step in SD2, and attention resolution refers to the input size of SD2. All comparisons were performed using the maximum velocity mode.
| Components | VOC21 | COCO-Stuff | Cityscapes | ADE20K | Avg. | |
| (I) | Baseline | 18.6 | 7.2 | 6.7 | 3.2 | 8.9 |
| (II) | (I) + KL | 44.2 | 12.1 | 8.6 | 6.4 | 17.8 |
| (III) | (II) + NMS | 45.9 | 13.0 | 9.6 | 7.7 | 19.1 |
| (IV) | (III) + JBU | 46.3 | 13.3 | 10.1 | 8.8 | 19.6 |
| (V) | (IV) + O.P. | 66.9 | 28.6 | 41.7 | 22.8 | 40.0 |
| (VI) | (IV) + M.V. | 67.8 | 28.9 | 43.3 | 23.0 | 40.8 |
Table 2: Effect of different components (unit: %). Baseline : apply to the raw logits without optimization or analytical solution. KL : Kullback-Leibler (KL) divergence. O.P. : solving the optimal path. M.V. : solving the maximum velocity.
| Variants | VOC21 | COCO-Stuff | Cityscapes | ADE20K | Avg. | |
| self-attention weight combination ( ) | ||||||
| (I) | ( ) | 62.1 | 24.6 | 37.5 | 17.6 | 35.5 |
| (II) | ( ) | 63.3 | 26.1 | 40.1 | 18.5 | 37.0 |
| (III) | ( ) | 65.0 | 26.7 | 41.9 | 19.6 | 38.3 |
| (IV) | ( ) | 66.2 | 27.8 | 42.8 | 21.8 | 39.7 |
| (V) | ( ) | 65.3 | 26.9 | 42.4 | 20.2 | 38.7 |
Table 3: Effect of different strategy for our components (unit: %). q-q mean : the average attention tensor between queries across all layers. k-k mean : the average attention tensor between keys across all layers. All comparisons were performed using the maximum velocity mode.
Figure 6: Quantitative evaluation of SD2 time step, attention resolution, and foundation model on standard benchmarks is presented, with each metric represented by a distinct color (unit: %). Here, SD2 time step denotes the denoising step in SD2, and attention resolution refers to the input size of SD2. All comparisons were performed using the optimal path mode.
| Variants | VOC21 | COCO-Stuff | Cityscapes | ADE20K | Avg. | |
| self-attention weight combination ( ) | ||||||
| (I) | ( ) | 61.1 | 23.5 | 36.4 | 16.5 | 34.4 |
| (II) | ( ) | 62.2 | 25.1 | 38.2 | 17.6 | 35.8 |
| (III) | ( ) | 64.0 | 25.6 | 40.6 | 18.3 | 37.1 |
| (IV) | ( ) | 65.1 | 26.4 | 41.5 | 20.8 | 38.5 |
| (V) | ( ) | 64.1 | 25.4 | 41.3 | 19.2 | 37.5 |
Table 4: Effect of different strategy for our components (unit: %). q-q mean : the average attention tensor between queries across all layers. k-k mean : the average attention tensor between keys across all layers. All comparisons were performed using the optimal path mode.
| Model | FLOPs(G) | Params(M) | Inference time(sec.) | mIoU(%) |
| CLIP | 52.2 | 149.6 | 0.08 | 18.6 |
| ProxyCLIP | 103.2 | 235.4 | 0.16 | 59.1 |
| CASS | 1675.5 | 265.7 | 3.11 | 65.8 |
| Ours (O.P.) | 351.3 | 1006.8 | 0.59 (-2.52) | 66.9 (+1.1) |
| Ours (M.V.) | 339.2 | 1006.8 | 0.57 (-2.54) | 67.8 (+2.0) |
| the inference time for each component | ||||
| Model | VOC21 | Context60 | ADE20K | COCO-Stuff | Cityscapes |
| original CLIP-B/16 | 18.6 | 7.8 | 3.2 | 7.2 | 6.7 |
| integrating self-attention via maximum velocity | |||||
| - CLIP-B/16 | 58.5 (+39.9) | 30.1 (+22.3) | 17.3 (+14.1) | 24.1 (+16.9) | 35.7 (+29.0) |
| - DINO-B/8 | 64.8 | 36.3 | 20.4 | 26.3 | 41.3 |
| - DINOv2-B/14 | 66.9 | 36.9 | 21.3 | 27.9 | 42.7 |
| - DINOv2-B/14 w/ Registers | 69.8 | 38.0 | 22.3 | 29.3 | 44.6 |
| Model | Scale | VOC21 | Context60 | COCO-Obj | VOC20 | Context59 | COCO-Stuff | Cityscapes | ADE20K |
| CASS | B/16 | 65.8 | 36.7 | 37.8 | 87.8 | 40.2 | 26.7 | 39.4 | 20.4 |
| w/ O.P. | 68.2 (+2.4) | 38.7 (+2.0) | 40.1 (+2.3) | 89.6 (+1.8) | 45.5 (+5.3) | 29.4 (+2.7) | 42.8 (+3.4) | 22.9 (+2.5) | |
| w/ M.V. | 69.5 (+3.7) | 39.1 (+2.4) | 40.6 (+2.8) | 90.0 (+2.2) | 45.9 (+5.7) | 29.8 (+3.1) | 44.4 (+5.0) | 23.1 (+2.7) | |
| SC-CLIP | B/16 | 64.6 | 36.8 | 37.7 | 84.3 | 40.1 | 26.6 | 41.0 | 20.1 |
| w/ O.P. | 68.1 (+3.5) | 38.7 (+1.9) | 40.0 (+2.3) | 89.2 (+4.9) | 45.3 (+5.2) | 29.3 (+2.7) | 42.9 (+1.9) | 22.8 (+2.7) | |
| w/ M.V. | 69.3 (+4.7) | 39.1 (+2.3) | 40.6 (+2.9) | 89.6 (+5.3) | 45.7 (+5.6) | 29.8 (+3.2) | 44.6 (+3.6) | 23.0 (+2.9) |
Figure 7: Visualization of segmentation maps on COCO-Stuff benchmark dataset.
Figure 8: Visualization of segmentation maps on Pascal VOC benchmark dataset.
Figure 9: Visualization of segmentation maps on ADE150k benchmark dataset.
Figure 10: Visualization of segmentation maps on Pascal Context benchmark dataset.
Figure 11: Visualization of maximum velocity maps across different threshold.
Figure 12: Visualization of optimal path maps across different iterations.