A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at https://github.com/2btlFe/GLA-CLIP.
Figures & tables
Figure 1: Comparison of segmentation consistency near window boundaries. (a) We evaluate segmentation inconsistency using the Boundary Error Rate (BER). BER is defined as the proportion of pixels near adjacent window boundaries where predicted labels differ despite identical ground-truth. ProxyCLIP yields high BER due to its lack of cross-window interaction, whereas BER is significantly reduced in ours by incorporating global context into attention process. More details in Appendix A . (b) ProxyCLIP exhibits grid artifacts (marked with white circles), caused by the limited receptive field within individual windows. In contrast, ours mitigates these artifacts by leveraging contextual information beyond local windows.
Figure 2: Overview of our proposed framework. The input image is first divided into overlapping windows and processed using frozen backbones: a Vision Foundation Model (VFM, e.g., DINO) and CLIP. We introduce a Key-Value Token Extension, where VFM features from the current window Fvfm serve as query tokens, while key tokens are gathered from all windows to provide global context. The corresponding value tokens V are extracted from the final transformer layer of CLIP. Cross-attention is then applied, followed by a projection layer to generate the final visual features Fvisual . To stabilize attention across windows, each query token is replaced with a semantically representative proxy anchor. Finally, a dynamic normalization scheme adjusts attention strength based on object size, approximated by the number of positive samples associated with each proxy anchor.
Figure 3: Visualization of attention maps for an anchor query token. Proxy-based attention enhances focus on semantically relevant regions across both inner- and outer-window areas. The subsequent dynamic normalization further suppresses irrelevant responses, especially from noisy tokens, yielding sharper and more semantically consistent attention distributions.
Table 2: Ablation study of each component in our method. KVE: Key-Value Extension, Proxy: Proxy Anchor-based Attention, Norm: Normalization (Best, Dynamic). Dataset-specific tuning is applied for ‘Best’ normalization configuration, with detailed results in Tab. 1 .
Figure 5: Class-wise object scale and 1/ w in Cityscapes
Figure 6: Hyperparameter sensitivity experiment.
CLIP
Model
V21
PC59
C-Stf
Avg.
ViT-B/32
ProxyCLIP
57.3
35.2
23.6
38.7
CASS
58.2
36.5
24.4
39.7
Ours
61.8
37.1
25.2
41.4
ViT-L/14
ProxyCLIP
60.7
38.2
26.2
41.7
CASS
62.1
39.1
26.3
42.5
Ours
63.5
39.1
26.7
43.1
Table 3: Segmentation results of different CLIP backbones.
VFM
Model
V21
PC59
C-Stf
Avg.
DINOv2 [ 33 ]
ProxyCLIP
59.6
37.3
25.3
40.7
+GLA
60.8
37.8
25.7
41.4
DINOv3 [ 41 ]
ProxyCLIP
62.8
38.8
26.1
42.6
+GLA
63.6
38.9
26.3
42.9
Table 4: Segmentation results of different VFM backbones.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qualitative Results with Sample-wise mIoU and BER.
Figure 8: Effect of Smoothing on ClearCLIP attention map
Metric
V21
PC59
C-Stf
City
ADE
Avg.
Precision (%)
98.7
96.3
94.2
95.6
93.9
96.2
Appendix
Table 5: High confidence token classification precision.
Figure 9: Effect of Key-Value Extension and u .
Figure 10: Object scale (GT vs high Confidence).
Model
V21
PC 60
C- Obj
V20
PC 59
C- Stf
City
ADE
Avg
Trident
67.1
38.6
41.1
84.5
42.2
28.3
42.9
21.9
45.8
Trident + GLA
67.4
38.9
41.4
84.8
42.4
28.3
43.6
21.7
46.1
Appendix
Table 6: Trident adaptation with GLA.
Model
V21
PC 60
C- Obj
V20
PC 59
C- Stf
City
ADE
Avg
Ours
66.3
36.1
37.7
84.2
39.9
26.9
40.8
20.0
44.0
Ours + Fusion
67.9
37.0
38.1
84.1
40.9
27.5
43.2
20.3
44.9
Appendix
Table 7: Effect of multi-layer feature fusion.
Model
V21
PC 60
C- Obj
V20
PC 59
C- Stf
City
ADE
Avg
CASS
65.8
36.7
37.8
87.8
40.2
26.7
39.4
20.4
44.4
ProxyCLIP + GLA
66.3
36.1
37.7
84.2
40.8
26.9
39.9
20.0
44.0
CASS + GLA
67.2
36.8
36.2
88.5
39.7
27.2
40.6
21.0
44.7
Appendix
Table 8: CASS adaptation with GLA
Model
Vaih.
UAVid
VDD
Avg
ClearCLIP
27.3
36.2
39.3
34.3
ProxyCLIP
30.6
41.4
44.3
38.8
ProxyCLIP + GLA
31.9 ( +1.3 )
41.9 ( +0.5 )
45.0 ( +0.7 )
39.6 ( +0.8 )
Appendix
Table 9: Performance comparison across remote sensing datasets.
Model
Web (8)
Vaih.
UAVid
VDD
Avg (3)
Ours
41.9
31.9
41.9
45.0
39.6
CLIP-DINOiser
40.3
18.6
33.3
37.7
29.9
Appendix
Table 10: Domain-shift evaluation on remote sensing datasets.
Context59 (336px×497px)
#Win
Lat.
Mem.
mIoU
CASS
1
5799 ms
7.6 GB
40.2
ProxyCLIP (Baseline)
8
429 ms
1.8 GB
38.8
Ours (Stride=224)
6
415 ms
1.7 GB
39.5
Ours (Stride=112)
8
431 ms
1.9 GB
39.9
Ours (Stride=98)
12
441 ms
2.4 GB
39.9
Appendix
Table 11: Latency & memory usage comparison on PC59
VOC21
Default
Crop size
Stride
Img resize
Bg. th.
Rename
224→
112→
336→
0.1→
O →
280
224
448
0.2
X
ProxyCLIP
63.3
64.4
59.7
61.8
61.0
60.4
Ours
66.3
66.5
63.6
65.3
64.6
63.3
Appendix
Table 12: Setting ablation on VOC21 (mIoU).
Figure 11: Qualitative results on ADE20K [ 52 ] with 150 categories. We compare GLA-CLIP with ProxyCLIP [ 24 ] , CASS [ 22 ] .
Figure 12: Qualitative results on Pascal Context60 [ 32 ] with 60 categories.. We compare GLA-CLIP with ProxyCLIP [ 24 ] , CASS [ 22 ] .
Figure 13: Qualitative results on Cityscapes [ 12 ] with 19 categories. We compare GLA-CLIP with ProxyCLIP [ 24 ] , CASS [ 22 ] .