Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
Figures & tables
Figure 2: Overview of RnS when full textual and visual support is available. Having access to a set of pixel-level annotated images, per-image visual class features vci are extracted. These features are then aggregated by class to form visual class features vc , which are combined with textual class features tc , through a mixing coefficient λ , to produce fused class features fcλ . During test-time training, a test-image-relevant subset of visual support features and fused class features , along with their class labels, are used to train a lightweight linear classifier gθ using cross-entropy loss. Each training sample is weighted with a class relevance weight wc ( e.g . w_{\scalebox{0.4}{\blacksquare}} for bg ). At inference, this classifier, trained per test image, is applied to patch-level features xjq to generate segmentation predictions. When SAM is available, patch-level features are replaced by region-level features xrq for improved accuracy.
Figure 3: Full textual and visual support. We compare zero-shot, RnS , kNN- CLIP and FreeDA and their variants without class name information (w/o text) for increasing number of support images per class. SAM 2.1 is used for region proposals. Left: OpenCLIP (ViT-B/16) for region-level predictions. Right: DINOv3.txt (ViT-L/16) for patch-level and region-level predictions.
Figure 4: Partial visual (left) and textual (right) support settings. Results of zero-shot, RnS , kNN- CLIP , FreeDA and their variants without class name information (w/o text). RnS evaluated w/o the pseudo-label loss in ( 12 ). OpenCLIP ViT-B/16 and SAM 2.1 are used. Left: a fraction of classes lack visual examples, while B=3 for the rest. Right: a fraction of classes lack textual class names, and B=1 .
Figure 5: Impact of retrieval on RnS . We replace the retrieved visual support feature set Vr of RnS with a random subset of the visual support feature set V , or different variants of visual support features from the retrieved classes VCr .
Method
B=1
B=5
B=10
RnS
41.59
47.87
49.02
RnS w/o wc
41.20 -0.39
47.43 -0.44
48.54 -0.48
RnS w/o wcΛ={0.8}
36.40 -5.19
46.55 -1.32
48.38 -0.64
RnS w/o text
34.11 -7.48
45.71 -2.16
48.00 -1.02
Table 1: Ablations of RnS . We report average mIoU across the considered datasets for three different numbers of available support images per class ( B=1 , B=5 , B=10 ). Blue numbers denote difference to the number in the same column but first row.
Figure 6: Comparison in a closed vocabulary setting. We compare RnS to the offline baseline competitors. To ensure a fair comparison we tune the learning rate, batch size, and number of iterations using a train-validation split from the available support images. No mask proposals are used. We report average performance on VOC, ADE, and Stuff.
Figure 7: RnS for personalized segmentation. We append examples of a specific instance to the support. Green: personalized instance, red: generic class, and black: background. Support sets shown in the supplementary.
Method
Training set
Annot.
VOC
City
ADE
C-59
Food
CUB
Avg.
SAN † [ 85 ]
COCO
118k
–
–
32.1
57.7
24.5
19.3
–
CAT-Seg [ 12 ]
COCO
118k
82.5
47.0 ∗
37.9
63.3
33.3
22.9
47.8
LPOSS+ [ 67 ]
✗
✗
62.4
37.9
22.3
38.6
26.1
12.0
33.2
CorrCLIP [ 90 ]
✗
✗
76.4
49.9
28.8
47.9
–
–
–
DINOv3.txt [ 65 ] + SAM
✗
✗
31.3
39.3
27.7
36.3
27.2
5.8
27.9
+ RnS B=1
Domain
66
73.2
59.1
37.3
52.7
42.8
34.0
49.9
Table 2: OVS vs . fully supervised segmentation. Fully supervised : best method picked per dataset. ∗ : self-evaluated, † : from CAT-Seg. Domain : using annotations from each training set. Annot: the number of pixel-level semantic segmentation annotations that each method uses. For domain we report the ADE annotations. Full table and details in the supplementary.
Figure 8: Open-vocabulary segmentation (OVS) results. We compare three settings: (i) textual-only support (zero-shot OVS), (ii) a simplified version of RnS using visual-only support, and (iii) the full RnS combining textual+visual support. Text-only support leads to ambiguous predictions (tree branch as bird, and various background hallucinations). Visual-only support helps disambiguate some classes but still confuses contextually similar objects (sofa–chair, tree branch–potted plant). RnS effectively combines both modalities to achieve accurate segmentation.
Figure 9: Out-of-domain vs. in-domain visual support between Cityscapes and ACDC. The left plot reports performance on ACDC as we vary the number of visual support images per class from either Cityscapes (out-of-domain, Cityscapes → ACDC) or ACDC (in-domain, ACDC → ACDC). The right plot reports the analogous results when evaluating on Cityscapes , with support drawn from either ACDC (out-of-domain, ACDC → Cityscapes) or Cityscapes (in-domain, Cityscapes → Cityscapes). In both cases we compare RnS with and without text, and include the corresponding zero-shot baseline.
Figure 10: Backbone comparison. We compare the performance of RnS using three different vision–language backbones (OpenCLIP ViT-B/16, DINOV3.txt ViT-L/16, and SigLIP2 ViT-L/16), along with the corresponding zero-shot baselines for each one of them.
Figure 11: Partial visual (left) and Retrieval quality (right). OpenCLIP (ViT- B/16) + SAM 2.1 for region proposals are used.
Figure 12: Comparison to offline baseline with and w/o hyperparameter tuning . We compare RnS against the offline linear classifier trained on per image visual class features on VOC. We include the curves presented in Figure 6 that use hyperparameter tuning per B and support seed (noted as “optimal”). We report the performance of both methods on two hyperparameter configurations: hyperparameter set1 which corresponds to an optimal set of hyperparameters of the linear classifier for B=1 , and hyperparameter set2 which corresponds to an optimal set of hyperparameters of the linear classifier for B=20 .
Table 3: Partial seen/unseen classes. We report the performance of RnS under varying fractions of unseen classes (classes w/o visual support) on VOC (a) and ADE20K (b). Columns show mean IoU on seen (S), i.e . classes with visual support, unseen (U), i.e . class that lack visual support, and all (A), i.e . the entire class set. We report the zero-shot (no class visually supported) and RnS with full visual support (all classes visually supported) on the same class sets as a reference.
Method
Training set
Annot.
Backbones
VOC
City
ADE
C-59
Food
CUB
Avg.
ODISE [ 83 ]
COCO
118k
Stable Diffusion v1.3, CLIP (ViT-L/14)
84.6
–
29.9
57.3
–
–
–
OVSeg † [ 40 ]
COCO
118k
CLIP (ViT-L/14), Swin-B
–
–
29.6
55.7
16.4
14.0
–
SAN † [ 85 ]
COCO
118k
CLIP (ViT-L/14)
–
–
32.1
57.7
24.5
19.3
–
CAT-Seg [ 12 ]
COCO
118k
CLIP (ViT-L/14)
82.5
47.0 ∗
37.9
63.3
33.3
22.9
47.8
LPOSS+ [ 67 ]
✗
✗
OpenCLIP (ViT-B/16), DINO (ViT-B/16)
62.4
37.9
22.3
38.6
26.1
12.0
33.2
CLIP-DINOiser [ 79 ]
✗
✗
OpenCLIP (ViT-B/16), DINO (ViT-B/16)
62.1
31.7
20.0
35.9
–
–
–
Table 4: OVS vs . fully supervised segmentation. Fully Supervised : best method picked per dataset. ∗ : self-evaluated, † : from CAT-Seg. Mask2Former numbers from DINOv3 [ 65 ] paper. Domain : using annotations from each training set. Annot: the number of pixel-level annotated images that each method uses. For Domain we report the ADE annotations.
Figure 13: Average performance (mIoU) vs . inference time (s). DINOv3.txt, patch-level; B=5 . Avg. on VOC, ADE, Stuff. A single NVIDIA A100 GPU is used.
Method
B=1
B=5
B=10
RnS ( K=1 )
40.02 -1.57
46.83 -1.04
48.06 -0.96
RnS ( K=4 )
41.59
47.87
49.02
RnS ( K=16 )
41.97 +0.38
47.78 -0.09
48.96 -0.06
RnS ( K=32 )
42.02 +0.43
47.59 -0.28
48.84 -0.18
Table 5: Ablations of the k-NN retrieval hyperparameter K in RnS . We report average mIoU across the considered datasets for three different numbers of available support images per class ( B=1 , B=5 , B=10 ). The row with K=4 (highlighted) corresponds to the configuration used in RnS , and blue numbers denote the difference to this row in the same column.
Figure 14: Qualitative comparison of patch-level vs. mask-level segmentation. Each row shows the input image, ground-truth mask, the PCA projection of either patch features or SAM mask features (average VLM patch features on top of SAM mask), and the corresponding predictions of RnS when applied on patches or on region proposals. Both variants use DINOv3.txt features [ 65 ] , and SAM 2.1 is used as the mask proposal generator [ 57 ] .
Figure 15: Qualitative comparisons between zero-shot baseline, FreeDA , kNN- CLIP and RnS with and without class name information. All visually supported methods use one support image per class ( B=1 ). All methods use OpenCLIP ViT-B/16 as VLM features [ 65 ] and SAM 2.1 as region proposal generator [ 57 ] .
Figure 16: Visual support sets for personalized segmentation. RnS used for personalized segmentation using OpenCLIP ViT-B/16 features and SAM 2.1 as region proposer. Initially, visual support includeds images in support before and is later expanded to support after , with support before ⊂ support after . Green: personalized instance, red: generic class, and black: background. Images from PODS [ 68 ] , i-CIR [ 53 ] , and self-collected.
Figure 17: Full textual and visual support per dataset (OpenCLIP, ViT-B/16). We compare zero-shot, RnS , kNN- CLIP and FreeDA and their variants without class name information (w/o text) for increasing number of support images per class. SAM 2.1 is used for region proposals.
Figure 18: Full textual and visual support per dataset (DINOv3.txt, ViT-L/16). We compare zero-shot, RnS , kNN- CLIP and RnS without class name information (w/o text) for increasing number of support images per class. Prediction at the patch level or SAM 2.1 is used for region proposals.
Figure 19: Partial visual support setting per dataset. Results of zero-shot, RnS , kNN- CLIP , and FreeDA , along with ablations of RnS without text and without the pseudo-label loss. OpenCLIP ViT-B/16 and SAM 2.1 are used. A fraction of classes lack visual examples, while B=3 for the remaining classes.
Figure 20: Partial textual support setting per dataset. Results of Zero-shot, RnS , kNN- CLIP , and FreeDA , together with their variants without class name information (w/o text). OpenCLIP ViT-B/16 and SAM 2.1 are used. A fraction of classes lack textual class names, and B=1 for all classes.
Generalized Few-Shot Semantic Segmentation (GFSS) has traditionally been approached as a representation-learning problem, requiring task-specific adaptation to incorporate novel classes from limited support examples. Recent foundation models, however, already exhibit strong open-vocabulary recognition and segmentation capabilities, raising a different question: can GFSS be solved through inference-time coordination of frozen semantic priors rather than parameter adaptation? We answer this question with Open-V, a training-free GFSS framework that combines Segment Anything (SAM3) Promptable Concept Segmentation (PCS) with a K-shot CLIP support centroid through calibrated per-pixel semantic arbitration. OpenV introduces no trainable components and supports arbitrary semantic categories at inference time. Beyond segmentation performance, our study contributes three broader findings. First, we show that support information can be incorporated through inference-time semantic grounding, and that its contribution increases as foundation-model text priors weaken on label-disjoint vocabularies. Second, we identify a reproducibility confound in foundationmodel segmentation, demonstrating that preprocessing and evaluation-space mismatches can silently distort reported performance. Finally, we validate Open-V across PASCAL5i, COCO-20i, and ADE-OW, showing that training-free coordination of foundation-model priors generalizes across both conventional GFSS and open-vocabulary evaluation settings. On PASCAL-5i (1-shot), Open-V attains base/novel/harmonic mIoU of 78.4/77.5/77.9, without GFSS-specific training surpassing the strongest trained baseline by +17.7 HM.
Silas Kwabla Gah, Ebenezer Owusu
Department of Computer Science University of Ghana
Open-vocabulary segmentation labels arbitrary categories from a text query without per-class training, yet on remote sensing imagery it underperforms on categories it handles reliably elsewhere. We find that much of this gap traces to the text query rather than to the segmentation model. Because these models are not specialized for overhead imagery, the class name that serves as the query is often a weak address into the vision-language embedding space. We show that a better name repairs part of the gap, while the remaining failures call for an address that the tested natural-language rephrasings do not provide. We recover that address from a few examples through textual inversion on a frozen model, keeping inference text only. On a representative benchmark this raises the mean intersection over union on the affected categories from 3.9 to 39.4, and across eight remote sensing datasets it improves over few-shot methods that instead inject visual prompts at inference.
Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference. Although modern vision-language models (VLMs) support strong open-vocabulary recognition, their representations learned through global contrastive objectives remain suboptimal for dense prediction, prompting many OVSS methods to depend on limited adaptation or refinement of image-text similarity maps. This, in turn, restricts spatial precision and robustness in complex, cluttered scenes. We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS. Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work. Second, we jointly leverage text embeddings aligned with both the global [CLS] token and local patch-level visual features from ViT-based encoder, effectively combining semantic discrimination with fine-grained spatial locality. Third, unlike prior approaches that rely primarily on post hoc similarity refinement, we perform early refinement of visual representations prior to image-text interaction, followed by late refinement of the resulting image-text correlation features, enabling more accurate and robust dense predictions in cluttered scenes. Finally, we propose a high-resolution local-global inference strategy based on sliding-window aggregation, which preserves spatial detail while maintaining global context. We conduct extensive experiments on five widely adopted OVSS benchmarks to evaluate our approach. The results demonstrate its effectiveness and robustness, consistently outperforming current state-of-the-art methods.
Saikat Dutta, Biplab Banerjee, Hamid Rezatofighi
IITB-Monash Research Academy · IIT Bombay · Monash University