A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at https://github.com/2btlFe/GLA-CLIP.
Figures & tables
Figure 1: Comparison of segmentation consistency near window boundaries. (a) We evaluate segmentation inconsistency using the Boundary Error Rate (BER). BER is defined as the proportion of pixels near adjacent window boundaries where predicted labels differ despite identical ground-truth. ProxyCLIP yields high BER due to its lack of cross-window interaction, whereas BER is significantly reduced in ours by incorporating global context into attention process. More details in Appendix A . (b) ProxyCLIP exhibits grid artifacts (marked with white circles), caused by the limited receptive field within individual windows. In contrast, ours mitigates these artifacts by leveraging contextual information beyond local windows.
Figure 2: Overview of our proposed framework. The input image is first divided into overlapping windows and processed using frozen backbones: a Vision Foundation Model (VFM, e.g., DINO) and CLIP. We introduce a Key-Value Token Extension, where VFM features from the current window Fvfm serve as query tokens, while key tokens are gathered from all windows to provide global context. The corresponding value tokens V are extracted from the final transformer layer of CLIP. Cross-attention is then applied, followed by a projection layer to generate the final visual features Fvisual . To stabilize attention across windows, each query token is replaced with a semantically representative proxy anchor. Finally, a dynamic normalization scheme adjusts attention strength based on object size, approximated by the number of positive samples associated with each proxy anchor.
Figure 3: Visualization of attention maps for an anchor query token. Proxy-based attention enhances focus on semantically relevant regions across both inner- and outer-window areas. The subsequent dynamic normalization further suppresses irrelevant responses, especially from noisy tokens, yielding sharper and more semantically consistent attention distributions.
Table 2: Ablation study of each component in our method. KVE: Key-Value Extension, Proxy: Proxy Anchor-based Attention, Norm: Normalization (Best, Dynamic). Dataset-specific tuning is applied for ‘Best’ normalization configuration, with detailed results in Tab. 1 .
Figure 5: Class-wise object scale and 1/ w in Cityscapes
Figure 6: Hyperparameter sensitivity experiment.
CLIP
Model
V21
PC59
C-Stf
Avg.
ViT-B/32
ProxyCLIP
57.3
35.2
23.6
38.7
CASS
58.2
36.5
24.4
39.7
Ours
61.8
37.1
25.2
41.4
ViT-L/14
ProxyCLIP
60.7
38.2
26.2
41.7
CASS
62.1
39.1
26.3
42.5
Ours
63.5
39.1
26.7
43.1
Table 3: Segmentation results of different CLIP backbones.
VFM
Model
V21
PC59
C-Stf
Avg.
DINOv2 [ 33 ]
ProxyCLIP
59.6
37.3
25.3
40.7
+GLA
60.8
37.8
25.7
41.4
DINOv3 [ 41 ]
ProxyCLIP
62.8
38.8
26.1
42.6
+GLA
63.6
38.9
26.3
42.9
Table 4: Segmentation results of different VFM backbones.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qualitative Results with Sample-wise mIoU and BER.
Figure 8: Effect of Smoothing on ClearCLIP attention map
Metric
V21
PC59
C-Stf
City
ADE
Avg.
Precision (%)
98.7
96.3
94.2
95.6
93.9
96.2
Appendix
Table 5: High confidence token classification precision.
Figure 9: Effect of Key-Value Extension and u .
Figure 10: Object scale (GT vs high Confidence).
Model
V21
PC 60
C- Obj
V20
PC 59
C- Stf
City
ADE
Avg
Trident
67.1
38.6
41.1
84.5
42.2
28.3
42.9
21.9
45.8
Trident + GLA
67.4
38.9
41.4
84.8
42.4
28.3
43.6
21.7
46.1
Appendix
Table 6: Trident adaptation with GLA.
Model
V21
PC 60
C- Obj
V20
PC 59
C- Stf
City
ADE
Avg
Ours
66.3
36.1
37.7
84.2
39.9
26.9
40.8
20.0
44.0
Ours + Fusion
67.9
37.0
38.1
84.1
40.9
27.5
43.2
20.3
44.9
Appendix
Table 7: Effect of multi-layer feature fusion.
Model
V21
PC 60
C- Obj
V20
PC 59
C- Stf
City
ADE
Avg
CASS
65.8
36.7
37.8
87.8
40.2
26.7
39.4
20.4
44.4
ProxyCLIP + GLA
66.3
36.1
37.7
84.2
40.8
26.9
39.9
20.0
44.0
CASS + GLA
67.2
36.8
36.2
88.5
39.7
27.2
40.6
21.0
44.7
Appendix
Table 8: CASS adaptation with GLA
Model
Vaih.
UAVid
VDD
Avg
ClearCLIP
27.3
36.2
39.3
34.3
ProxyCLIP
30.6
41.4
44.3
38.8
ProxyCLIP + GLA
31.9 ( +1.3 )
41.9 ( +0.5 )
45.0 ( +0.7 )
39.6 ( +0.8 )
Appendix
Table 9: Performance comparison across remote sensing datasets.
Model
Web (8)
Vaih.
UAVid
VDD
Avg (3)
Ours
41.9
31.9
41.9
45.0
39.6
CLIP-DINOiser
40.3
18.6
33.3
37.7
29.9
Appendix
Table 10: Domain-shift evaluation on remote sensing datasets.
Context59 (336px×497px)
#Win
Lat.
Mem.
mIoU
CASS
1
5799 ms
7.6 GB
40.2
ProxyCLIP (Baseline)
8
429 ms
1.8 GB
38.8
Ours (Stride=224)
6
415 ms
1.7 GB
39.5
Ours (Stride=112)
8
431 ms
1.9 GB
39.9
Ours (Stride=98)
12
441 ms
2.4 GB
39.9
Appendix
Table 11: Latency & memory usage comparison on PC59
VOC21
Default
Crop size
Stride
Img resize
Bg. th.
Rename
224→
112→
336→
0.1→
O →
280
224
448
0.2
X
ProxyCLIP
63.3
64.4
59.7
61.8
61.0
60.4
Ours
66.3
66.5
63.6
65.3
64.6
63.3
Appendix
Table 12: Setting ablation on VOC21 (mIoU).
Figure 11: Qualitative results on ADE20K [ 52 ] with 150 categories. We compare GLA-CLIP with ProxyCLIP [ 24 ] , CASS [ 22 ] .
Figure 12: Qualitative results on Pascal Context60 [ 32 ] with 60 categories.. We compare GLA-CLIP with ProxyCLIP [ 24 ] , CASS [ 22 ] .
Figure 13: Qualitative results on Cityscapes [ 12 ] with 19 categories. We compare GLA-CLIP with ProxyCLIP [ 24 ] , CASS [ 22 ] .
Open-vocabulary semantic segmentation requires assigning pixel-level semantic labels while supporting an open and unrestricted set of categories. Training-free CLIP-based approaches preserve strong zero-shot generalization but typically rely on a single inference mechanism, limiting their ability to jointly address unreliable local tokens and insufficient spatial coherence. We propose DouC, a training-free dual-branch CLIP framework that decomposes dense prediction into two complementary components. OG-CLIP improves patch-level reliability via lightweight, inference-time token gating, while FADE-CLIP injects external structural priors through proxy attention guided by frozen vision foundation models. The two branches are fused at the logit level, enabling local token reliability and structure-aware patch interactions to jointly influence final predictions, with optional instance-aware correction applied as post-processing. DouC introduces no additional learnable parameters, requires no retraining, and preserves CLIP's zero-shot generalization. Extensive experiments across eight benchmarks and multiple CLIP backbones demonstrate that DouC consistently outperforms prior training-free methods and scales favorably with model capacity.
Mohamad Zamini, Diksha Shukla
Department of Electrical Engineering and Computer Science, University of Wyoming, Laramie, WY, USA.
Training-free open-vocabulary semantic segmentation(TF-OVSS) has recently attracted attention for its ability to perform dense prediction by leveraging the pretrained knowledge of large vision and vision-language models, without requiring additional training. However, due to the limited input resolution of these pretrained encoders, existing TF-OVSS methods commonly adopt a sliding-window strategy that processes cropped sub-images independently. While effective for managing high-resolution inputs, this approach prevents global attention over the full image, leading to fragmented feature representations and limited contextual reasoning. We propose OV-Stitcher, a training-free framework that addresses this limitation by stitching fragmented sub-image features directly within the final encoder block. By reconstructing attention representations from fragmented sub-image features, OV-Stitcher enables global attention within the final encoder block, producing coherent context aggregation and spatially consistent, semantically aligned segmentation maps. Extensive evaluations across eight benchmarks demonstrate that OV-Stitcher establishes a scalable and effective solution for open-vocabulary segmentation, achieving a notable improvement in mean Intersection over Union(mIoU) from 48.7 to 50.7 compared with prior training-free baselines.
Seungjae Moon, Seunghyun Oh, Youngmin Ro
Machine Intelligence Laboratory, University of Seoul, Korea
Training-free open-vocabulary segmentation remains limited by a missing inference abstraction. Frozen vision-language features are produced at patch level, yet dense prediction requires a unit that simultaneously governs feature interaction, spatial support, contextual recovery, and retrieval-based correction. We present SCI-CLIP, a segment-centric inference framework built around the principle that the same region abstraction should organize all stages of dense open-vocabulary prediction. SCI-CLIP first induces a region-consistent interaction graph over frozen visual tokens, then reconstructs dense features by propagating values over this graph, augmenting them with selective cross-window support only where local evidence is insufficient. The same segment abstraction is subsequently used to construct and query an offline reference memory, aligning exemplar retrieval with the units on which prediction is made. SCI-CLIP turns frozen CLIP-style features into spatially coherent, context-aware, and retrieval-compatible dense predictions without any training. SCI-CLIP consistently improves the structural quality of dense predictions, the robustness of contextual reasoning, and the alignment of exemplar-based correction, yielding stronger open-vocabulary segmentation across eight benchmarks. Project code is available at: https://github.com/mzamini92/SCICLIP.