Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference. Although modern vision-language models (VLMs) support strong open-vocabulary recognition, their representations learned through global contrastive objectives remain suboptimal for dense prediction, prompting many OVSS methods to depend on limited adaptation or refinement of image-text similarity maps. This, in turn, restricts spatial precision and robustness in complex, cluttered scenes. We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS. Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work. Second, we jointly leverage text embeddings aligned with both the global [CLS] token and local patch-level visual features from ViT-based encoder, effectively combining semantic discrimination with fine-grained spatial locality. Third, unlike prior approaches that rely primarily on post hoc similarity refinement, we perform early refinement of visual representations prior to image-text interaction, followed by late refinement of the resulting image-text correlation features, enabling more accurate and robust dense predictions in cluttered scenes. Finally, we propose a high-resolution local-global inference strategy based on sliding-window aggregation, which preserves spatial detail while maintaining global context. We conduct extensive experiments on five widely adopted OVSS benchmarks to evaluate our approach. The results demonstrate its effectiveness and robustness, consistently outperforming current state-of-the-art methods.
Figures & tables
Figure 1: Visualization of final visual features across different models. Pretrained models exhibit noisy features, while dinov3.txt produces comparatively cleaner features with more structural information than CLIP. Fine-tuned dinov3.txt features yield sharper boundaries, but fail to capture fine-grained details. CLIP-based OVSS approach CAT-Seg recovers more detail in the features, yet produces less well-defined boundaries. In contrast, our method produces features with sharp boundaries and rich fine-grained detail.
Figure 2: Overall architecture of proposed dinov3.seg . Given an input image (and sub-images at inference time), the dinov3.txt backbone extracts dense visual features and multiple textual embeddings. The dense visual features are passed to the Early Refinement Module — via the Local-Global Aggregation module at inference time, or directly during training. The refined visual features interact with the textual embeddings to yield image–text correlation features, which are subsequently enriched with auxiliary semantic guidance from the Semantic Prior Encoder via the Late Refinement Module. The final segmentation map is generated by the Upsampling Decoder.
Figure 3: Effect of Early Refinement on dinov3.txt visual features.
Table 1: Quantitative comparison with state-of-the-art OVSS methods. Best and second-best results are highlighted in red and blue , respectively. Avg denotes the mean mIoU over A-847, PC-459, A-150, PC-59, and PAS-20.
A-847
PC-459
A-150
PC-59
Method
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
SED [ 40 ]
31.67
12.23
40.22
15.86
47.02
28.07
61.73
40.35
MAFT+ [ 13 ]
38.11
13.40
39.71
14.01
48.47
29.11
61.11
29.37
CAT-Seg [ 7 ]
38.67
13.80
42.83
16.40
48.92
31.17
64.68
37.90
dinov3.seg (Ours)
43.07
17.99
44.72
21.27
54.06
35.11
65.43
42.43
Δ
+4.39
+4.19
+1.89
+4.87
+5.14
+3.95
+0.75
+4.53
Table 2: Seen and unseen class mIoU comparison across datasets. PAS-20 is omitted as all its categories are classified as seen under our similarity-based partition. Δ indicates absolute difference between our method and CAT-Seg.
Figure 5: Qualitative comparison with state-of-the-art. Highlighted regions in bounding boxes illustrate cases where our model achieves more accurate segmentation results.
Config.
A-847
PC-459
A-150
PC-59
PAS-20
Reference Baselines
dinov3.txt ZS
8.26
8.82
18.12
25.94
82.31
dinov3.txt FT
8.86
17.46
28.33
59.64
96.69
Ablation Configurations
Text Ens.
Early Ref.
Focal-Dice
LGA Inf.
I
×
×
×
×
18.70
26.51
41.57
64.05
97.61
Table 3: Ablation study on various design choices. Text Ens. = Text Embedding Ensemble; Early Ref. = Early Refinement; LGA Inf. = LGA-based Inference Strategy.
FT Strategy
A-847
PC-459
A-150
PC-59
PAS-20
QV FT
17.72
26.22
40.62
62.63
97.27
QK FT
17.92
26.61
40.26
62.95
97.32
Full FT (Ours)
18.70
26.51
41.57
64.05
97.61
Table 4: Comparison across finetuning strategies.
LGA
Performance
dinov3.txt
SAM
A-847
PC-459
A-150
PC-59
PAS-20
Avg
×
×
20.06
27.43
42.14
63.94
98.34
50.38
×
✓
20.02
27.49
42.03
64.07
98.37
50.40
✓
×
20.09
27.80
42.19
64.27
97.86
50.44
✓
✓
19.95
27.83
42.10
64.32
97.94
50.43
Table 5: Ablation on Local-Global Aggregation.
Configuration
R2
Global Text Embeddings only
0.0987
Local Text Embeddings only
0.1376
Both (concatenated)
0.1718
Table 6: R2 scores for predicting DINOv3 visual prototypes from text embeddings using Ridge regression.
Class
Embedding Space
Top-5 Nearest Neighbors
Stairway
Visual Prototypes
stairs, bannister, railing, step, grandstand
Global Text Embeddings
stairs , bannister , path, step , railing
Local Text Embeddings
stairs , escalator, bannister , path, railing
Minibike
Visual Prototypes
bicycle, car, van, truck, ashcan
Global Text Embeddings
bicycle , car , road, path, person
Local Text Embeddings
bicycle , chair, truck , car , bench
Table 7: Top-5 nearest neighbors in each embedding space for two representative classes in the A-150 dataset. Neighbors matching the visual space are bolded .
Dataset
# Seen
# Unseen
A-847
72
775
PC-459 1 1 1 PC-459 has 114 absent classes which are not considered in our analysis.
96
249
A-150
56
94
PC-59
56
3
PAS-20
20
0
Table 8: Seen and unseen class splits for different datasets based on visual similarity.
Figure 6: t-SNE visualization of visual prototypes for A-150, with seen classes (green) and unseen classes (red).
Dataset
# Seen
# Unseen
A-847
196
651
PC-459
119
226
A-150
88
62
PC-59
56
3
PAS-20
20
0
Table 9: Seen and unseen class splits for different datasets based on textual similarity.
Figure 7: t-SNE visualization of textual embeddings for A-150, with seen classes (green) and unseen classes (red).
A-847
PC-459
A-150
PC-59
Method
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
SED [ 40 ]
18.98
12.35
35.27
15.99
40.30
27.83
61.43
45.97
MAFT+ [ 13 ]
21.59
13.66
34.80
13.98
41.93
28.39
61.26
26.52
CAT-Seg [ 7 ]
22.60
13.90
38.99
15.73
42.22
31.51
61.43
45.97
dinov3.seg (Ours)
25.57
18.49
40.63
21.04
45.55
37.42
65.26
45.69
Δ
+2.97
+4.58
+1.64
+5.31
+3.33
+5.90
+3.83
-0.28
Table 10: Seen and unseen class (based on textual similarity) mIoU comparison across datasets. Δ indicates absolute improvement over CAT-Seg.
NL
A-847
PC-459
A-150
PC-59
PAS-20
Avg.
1
18.09
26.52
40.19
63.85
97.81
49.29
2
18.70
26.51
41.57
64.05
97.61
49.69
3
18.43
26.28
41.34
64.09
97.75
49.58
Table 11: Effect of number of late refinement blocks.
Figure 8: Accuracy–runtime trade-off for varying numbers of late refinement blocks NL . NL=2 achieves the best average mIoU (49.69) with a moderate inference cost of 0.30 s.
SPE
A-847
PC-459
A-150
PC-59
PAS-20
Avg.
none
18.23
26.15
40.30
63.04
97.76
49.10
SAM-2.1 Hiera-B+
18.33
26.86
41.04
63.77
97.70
49.54
SAM-2.1 Hiera-L
18.39
25.89
40.55
62.84
97.60
49.05
SAM3 PE-L+
18.51
26.60
41.16
63.72
97.84
49.57
SAM ViT-L
18.70
26.51
41.57
64.05
97.61
49.69
Table 12: Comparison of different Semantic Prior Encoders (SPE).
Method
A-847
PC-459
A-150
PC-59
PAS-20
Avg.
Ours with Different VLM Backbones
Ours (CLIP ViT-L)
17.82
26.55
38.48
64.08
97.24
48.83
Ours ( dinov2.txt )
17.25
26.07
39.61
64.46
97.74
49.03
Ours ( dinov3.txt )
20.09
27.80
42.19
64.27
97.86
50.44
Existing CLIP (ViT-L) based Methods
CAT-Seg [ 7 ]
16.0
23.8
37.9
63.3
97.0
47.60
Table 13: Ablation on VLM backbone choice and comparison with state-of-the-art CLIP (ViT-L) based OVSS methods.
λ
γ
A-847
PC-459
A-150
PC-59
PAS-20
Avg.
0.01
2
19.65
28.51
41.82
64.26
97.85
50.42
0.1
2
20.15
27.84
42.10
64.08
97.86
50.40
0.05
2
20.09
27.80
42.19
64.27
97.86
50.44
0.05
1
19.89
28.03
41.89
64.42
97.91
50.43
0.05
3
19.93
27.96
42.10
64.06
97.86
50.38
Table 14: Loss function hyperparameter analysis.
Figure 9: Qualitative results on Inference Strategy. Highlighted regions in boxes show effectiveness of proposed LGA-based strategy over CAT-Seg inference strategy.
Method
A-847
PC-459
A-150
PC-59
PAS-20
Avg.
CAT-Seg w/ dinov3.txt
18.93
26.24
41.17
63.45
97.79
49.51
dinov3.seg
20.09
27.80
42.19
64.27
97.86
50.44
Table 15: Comparison w.r.t CAT-Seg with dinov3.txt .
Model
Parameters (M)
Inf. Time (s)
Inf. GFLOPs
Total
VLM
Other
OVSeg [ 19 ]
532.6
427.9
104.7
1.31
9,810.1
SCAN [ 21 ]
890.3
427.6
462.7
1.09
13,502.9
CAT-Seg [ 7 ]
433.7
427.9
5.8
0.22
1,963.5
dinov3.seg
1,178.2
866.6
311.6
0.37
4,500.4
Table 16: Model complexity comparison. Inference time and GFLOPs are measured on an A100 GPU at 640×640 resolution. GFLOPs are computed using the fvcore library.
Open-vocabulary semantic segmentation (OVSS) leverages textual semantics to segment objects beyond predefined categories. While the self-supervised model DINOv3 provides strong structured visual representations, its lack of native textual alignment hinders its direct application to OVSS. To bridge this gap, we propose DINOde, an ODE-based framework that continuously aligns CLIP text embeddings with the DINO visual manifold. Our approach employs two complementary components: (i) Semantic Text Flow (STF), which evolves text embeddings toward the DINO manifold through a continuous ODE trajectory, and (ii) Global Context Flow (GCF), which progressively refines the holistic image representation carried by DINO's CLS token. To preserve the hyperspherical geometry of the feature space during this evolution, we further introduce Velocity Tangent Projection, which constrains the learned velocity field to the tangent space. By modeling alignment as a continuous trajectory, DINOde avoids the manifold entanglement inherent in discrete MLP projections and yields more robust cross-modal alignment. Extensive experiments demonstrate that DINOde consistently outperforms existing methods and achieves state-of-the-art performance across multiple OVSS benchmarks. The code is available at https://github.com/yoon307/DINOde.
Sung-Hoon Yoon, Hoyong Kwon, Changgyoon Oh +1
Multimodal Intelligence and Perception Lab., DGIST, Republic of Korea · Visual Intelligence Lab., KAIST, Republic of Korea
Remote sensing semantic segmentation is hindered by costly pixel-level annotations, motivating training-free open-vocabulary methods. Recently, the recent release of DINOv3 brings DINO.txt, which equips the standalone DINO backbone with image-text contrastive learning and thus opens up the possibility of open-vocabulary segmentation. We propose DinoSplat-OV, a training-free framework that adapts DINOv3 to remote sensing without fine-tuning or additional pretraining. Targeting the dense distribution, multi-scale nature, and large size of remote sensing imagery, we design two core modules. Its Text-aware Laplacian Propagation module de-noises patch-level predictions by combining textual semantic affinities with local visual similarity, improving regional consistency while preserving boundaries. Its Gaussian Splatting Upsampling module reconstructs pixel-level features through RGB-guided anisotropic aggregation and test-time optimization. A global-anchor sliding-window strategy further supports large-scale imagery. Experiments on UDD5, DOTA, and LoveDA demonstrate competitive or superior performance over existing training-free methods, effectively filling the gap of DINO-series models in training-free open-vocabulary segmentation and providing a viable new path for further advances in this direction.
Changhao Zhao, Haoxiang Li, Yuke Li +2
College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China
Open-Vocabulary Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).
Haoxi Zeng, Qiankun Liu, Yi Bin +5
Tongji University Shanghai, China · Hong Kong Polytechnic University Hong Kong, China · University of Electronic Science and Technology of China Chengdu, China +1