OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance
Authors: Haoxi Zeng, Qiankun Liu, Yi Bin, Haiyue Zhang, Yujuan Ding, Guoqing Wang, Deqiang Ouyang, Heng Tao Shen
Organizations: Tongji University Shanghai, China · Hong Kong Polytechnic University Hong Kong, China · University of Electronic Science and Technology of China Chengdu, China · Chongqing University Chongqing, China
Open-Vocabulary Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).
Figures & tables
Figure 1. Overall performance comparison. We evaluate our proposed method against state-of-the-art weakly-supervised approaches. The results illustrate that while existing methods struggle with fine-grained segmentation on challenging datasets such as Cityscapes ( Cordts et al., 2016 ) and COCO-Stuff ( Caesar et al., 2018 ) , our OVS-DINO consistently achieves superior performance across all benchmarks, with particularly significant improvements on complex scenarios.
Figure 2. Frequency-domain analysis of DINO layers. (a) The high-to-low frequency energy ratio exhibits a consistent downward trend across layers, indicating a progressive transition from high-frequency local details to low-frequency global semantics during inference. (b) The azimuthally averaged power spectrum E(r) shows that at higher frequencies (large r ), deeper layers exhibit lower energy compared to shallow ones, indicating that shallow layers effectively preserve more fine-grained edge information.
Figure 3. Layer-wise PCA visualization. DINO exhibits strong object contour awareness as early as its initial layers, indicating that structural information is captured during the early stages of feature extraction ( Caron et al., 2021 ) .
Figure 4. CKA heatmap results of DINOv2 and SAM, obtained from experiments conducted on the COCO dataset ( Caesar et al., 2018 ) .
Figure 5. Overall architecture of OVS-DINO. We first encode the multi-level features from DINO using the Structure-Aware Encoder (SAE) and align them with SAM features. The aligned features are then passed through the Structure-Modulated Decoder (SMD) and projection layer to map them into the CLIP text space for subsequent segmentation. The alignment loss LAlign updates only the SAE, SMD, and projector, while the segmentation loss LSeg further updates the last three layers of DINO in addition to these modules. For the training data, we use NLTK ( Bird et al., 2009 ) to extract nouns from natural language descriptions, followed by refinement and filtering. Grounding DINO ( Liu et al., 2024 ) and SAM are then applied to generate the corresponding pseudo-masks. PG is an abbreviation for Preservation Gate.
Model
Backbone
VOC21
Context60
Object
VOC20
Context59
Stuff
City
ADE
Avg
GroupViT (CVPR2022 ( Xu et al., 2022 ) )
CLIP S/16
50.4
18.7
27.5
79.7
23.4
15.3
11.1
9.2
29.4
ReCo (NeurIPS2022 ( Shin et al., 2022 ) )
CLIP B/16
25.1
19.9
15.7
57.7
22.3
14.8
21.1
11.2
23.5
TCL (CVPR2023 ( Cha et al., 2023 ) )
CLIP B/16
51.2
24.3
30.4
77.5
30.3
19.6
23.1
14.9
33.9
CoDe (CVPR2024 ( Wu et al., 2024 ) )
CLIP B/16
46.0
23.1
25.4
64.9
28.2
17.6
24.4
12.9
30.3
CLIP-DINOiser (ECCV2024 ( Wysoczańska et al., 2024 ) )
CLIP B/16
62.1
32.4
34.8
80.9
35.9
24.6
31.1
20.0
40.2
PixelCLIP-ViT (NeurIPS2024 ( Shin et al., 2024 ) )
CLIP B/16
–
–
–
85.9
37.9
23.6
27.2
18.7
–
Table 1. Comparison of different methods across multiple datasets. We report results on benchmarks with background classes (VOC21 ( Everingham et al., 2010 ) , Context60 ( Mottaghi et al., 2014 ) , and Object ( Caesar et al., 2018 ) ) as well as without background classes (VOC20 ( Everingham et al., 2010 ) , Context59 ( Mottaghi et al., 2014 ) , etc.). We annotate the backbone used by each method and compare their average performance to reflect overall capability. Notably, our method achieves state-of-the-art performance on all four datasets as well as in terms of overall average.
Method
V21
C60
Object
V20
C59
Stuff
City
ADE
Avg
w/o SA
60.1
40.0
38.3
75.1
47.4
32.9
39.4
19.7
44.1
w/o Encoder
55.2
36.5
32.5
72.2
42.6
27.6
37.7
18.3
40.3
w/o PG
37.4
31.0
26.5
49.4
36.9
25.9
38.3
16.5
32.7
Ours
64.0
42.0
38.6
83.5
49.4
34.2
42.9
20.6
46.9
Table 2. Ablation study of the effect of designed model. SA and PG are abbreviations for SAM alignment and Preservation Gate.
Method
V21
C60
Object
V20
C59
Stuff
City
ADE
Avg
four layers
61.2
41.0
38.9
80.2
48.7
34.4
41.4
19.8
45.7
frozen DINO
56.3
37.0
33.8
73.4
43.5
28.2
40.9
18.1
41.4
Ours
64.0
42.0
38.6
83.5
49.4
34.2
42.9
20.6
46.9
Table 3. Ablation study of the layers selection and fine-tuning strategy.
Figure 6. Visualization results of different models.
Feature
V21
C60
Object
V20
C59
Stuff
City
ADE
Avg
SAM
63.93
41.96
38.62
83.49
49.35
34.16
42.9
20.64
46.88
DINO
63.95
41.97
38.62
83.51
49.35
34.16
42.88
20.64
46.89
Table 4. Ablation study of the validation of activation from SAM.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2. Visualization of success cases.
Dataset
DINO
DINOv2 (reg)
DINOv2 (Ours)
VOC21
35.1
62.9
64.0
Context60
26.6
41.9
42.0
Object
18.4
39.4
38.6
VOC20
46.5
82.5
83.5
Context59
30.3
49.7
49.4
Stuff
18.2
34.8
34.2
Appendix
Table 1. Ablation study of using different visual backbones.
Figure 3. Visualization of failure cases.
Figure 4. Additional qualitative results on the ADE20K ( Zhou et al., 2019 ) dataset.
Figure 5. Additional qualitative results on the Cityscapes ( Cordts et al., 2016 ) dataset.
Figure 6. Additional qualitative results on the COCO-Stuff ( Caesar et al., 2018 ) dataset.