Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?

Tilemachos Aravanis; Vladan Stojnić; Bill Psomas; Nikos Komodakis; Giorgos Tolias

Figure 2: Overview of RnS when full textual and visual support is available. Having access to a set of pixel-level annotated images, per-image visual class features

\mathbf{v}^{i}_{c}

are extracted. These features are then aggregated by class to form visual class features

\mathbf{v}_{c}

, which are combined with textual class features

\mathbf{t}_{c}

, through a mixing coefficient

\lambda

, to produce fused class features

\mathbf{f}_{c\lambda}

. During test-time training, a test-image-relevant subset of visual support features and fused class features , along with their class labels, are used to train a lightweight linear classifier

g_{\theta}

using cross-entropy loss. Each training sample is weighted with a class relevance weight

w_{c}

( e.g .

w_{\scalebox{0.4}{

\blacksquare

}}

for bg ). At inference, this classifier, trained per test image, is applied to patch-level features

\mathbf{x}_{j}^{q}

to generate segmentation predictions. When SAM is available, patch-level features are replaced by region-level features

\mathbf{x}^{\,q}_{r}

for improved accuracy.

Figure 3: Full textual and visual support. We compare zero-shot, RnS , kNN- CLIP and FreeDA and their variants without class name information (w/o text) for increasing number of support images per class. SAM 2.1 is used for region proposals. Left: OpenCLIP (ViT-B/16) for region-level predictions. Right: DINOv3.txt (ViT-L/16) for patch-level and region-level predictions.

Figure 4: Partial visual (left) and textual (right) support settings. Results of zero-shot, RnS , kNN- CLIP , FreeDA and their variants without class name information (w/o text). RnS evaluated w/o the pseudo-label loss in ( 12 ). OpenCLIP ViT-B/16 and SAM 2.1 are used. Left: a fraction of classes lack visual examples, while

B=3

for the rest. Right: a fraction of classes lack textual class names, and

B=1

.

Figure 5: Impact of retrieval on RnS . We replace the retrieved visual support feature set

\mathcal{V}_{r}

of RnS with a random subset of the visual support feature set

\mathcal{V}

, or different variants of visual support features from the retrieved classes

\mathcal{V}_{\mathcal{C}_{r}}

.

Method	$B=1$	$B=5$	$B=10$
RnS	41.59	47.87	49.02
RnS w/o $w_{c}$	41.20 -0.39	47.43 -0.44	48.54 -0.48
RnS w/o $w_{c}\,$ $\Lambda=\{0.8\}$	36.40 -5.19	46.55 -1.32	48.38 -0.64
RnS w/o text	34.11 -7.48	45.71 -2.16	48.00 -1.02

Table 1: Ablations of RnS . We report average mIoU across the considered datasets for three different numbers of available support images per class (

B=1

,

B=5

,

B=10

). Blue numbers denote difference to the number in the same column but first row.

Figure 6: Comparison in a closed vocabulary setting. We compare RnS to the offline baseline competitors. To ensure a fair comparison we tune the learning rate, batch size, and number of iterations using a train-validation split from the available support images. No mask proposals are used. We report average performance on VOC, ADE, and Stuff.

Figure 7: RnS for personalized segmentation. We append examples of a specific instance to the support. Green: personalized instance, red: generic class, and black: background. Support sets shown in the supplementary.

Method	Training set	Annot.	VOC	City	ADE	C-59	Food	CUB	Avg.
SAN † [ 85 ]	COCO	118k	–	–	32.1	57.7	24.5	19.3	–
CAT-Seg [ 12 ]	COCO	118k	82.5	47.0 ∗	37.9	63.3	33.3	22.9	47.8
LPOSS+ [ 67 ]	✗	✗	62.4	37.9	22.3	38.6	26.1	12.0	33.2
CorrCLIP [ 90 ]	✗	✗	76.4	49.9	28.8	47.9	–	–	–
DINOv3.txt [ 65 ] + SAM	✗	✗	31.3	39.3	27.7	36.3	27.2	5.8	27.9
+ RnS $B=1$	Domain	66	73.2	59.1	37.3	52.7	42.8	34.0	49.9

Table 2: OVS vs . fully supervised segmentation. Fully supervised : best method picked per dataset. ∗ : self-evaluated, † : from CAT-Seg. Domain : using annotations from each training set. Annot: the number of pixel-level semantic segmentation annotations that each method uses. For domain we report the ADE annotations. Full table and details in the supplementary.

Figure 8: Open-vocabulary segmentation (OVS) results. We compare three settings: (i) textual-only support (zero-shot OVS), (ii) a simplified version of RnS using visual-only support, and (iii) the full RnS combining textual+visual support. Text-only support leads to ambiguous predictions (tree branch as bird, and various background hallucinations). Visual-only support helps disambiguate some classes but still confuses contextually similar objects (sofa–chair, tree branch–potted plant). RnS effectively combines both modalities to achieve accurate segmentation.

Figure 9: Out-of-domain vs. in-domain visual support between Cityscapes and ACDC. The left plot reports performance on ACDC as we vary the number of visual support images per class from either Cityscapes (out-of-domain, Cityscapes

\rightarrow

ACDC) or ACDC (in-domain, ACDC

\rightarrow

ACDC). The right plot reports the analogous results when evaluating on Cityscapes , with support drawn from either ACDC (out-of-domain, ACDC

\rightarrow

Cityscapes) or Cityscapes (in-domain, Cityscapes

\rightarrow

Cityscapes). In both cases we compare RnS with and without text, and include the corresponding zero-shot baseline.

Figure 10: Backbone comparison. We compare the performance of RnS using three different vision–language backbones (OpenCLIP ViT-B/16, DINOV3.txt ViT-L/16, and SigLIP2 ViT-L/16), along with the corresponding zero-shot baselines for each one of them.

Figure 11: Partial visual (left) and Retrieval quality (right). OpenCLIP (ViT- B/16) + SAM 2.1 for region proposals are used.

Figure 12: Comparison to offline baseline with and w/o hyperparameter tuning . We compare RnS against the offline linear classifier trained on per image visual class features on VOC. We include the curves presented in Figure 6 that use hyperparameter tuning per

B

and support seed (noted as “optimal”). We report the performance of both methods on two hyperparameter configurations: hyperparameter

\mathbf{set~1}

which corresponds to an optimal set of hyperparameters of the linear classifier for

B=1

, and hyperparameter

\mathbf{set~2}

which corresponds to an optimal set of hyperparameters of the linear classifier for

B=20

.

Table 3: Partial seen/unseen classes. We report the performance of RnS under varying fractions of unseen classes (classes w/o visual support) on VOC (a) and ADE20K (b). Columns show mean IoU on seen (S), i.e . classes with visual support, unseen (U), i.e . class that lack visual support, and all (A), i.e . the entire class set. We report the zero-shot (no class visually supported) and RnS with full visual support (all classes visually supported) on the same class sets as a reference.

Method	Training set	Annot.	Backbones	VOC	City	ADE	C-59	Food	CUB	Avg.
ODISE [ 83 ]	COCO	118k	Stable Diffusion v1.3, CLIP (ViT-L/14)	84.6	–	29.9	57.3	–	–	–
OVSeg † [ 40 ]	COCO	118k	CLIP (ViT-L/14), Swin-B	–	–	29.6	55.7	16.4	14.0	–
SAN † [ 85 ]	COCO	118k	CLIP (ViT-L/14)	–	–	32.1	57.7	24.5	19.3	–
CAT-Seg [ 12 ]	COCO	118k	CLIP (ViT-L/14)	82.5	47.0 ∗	37.9	63.3	33.3	22.9	47.8
LPOSS+ [ 67 ]	✗	✗	OpenCLIP (ViT-B/16), DINO (ViT-B/16)	62.4	37.9	22.3	38.6	26.1	12.0	33.2
CLIP-DINOiser [ 79 ]	✗	✗	OpenCLIP (ViT-B/16), DINO (ViT-B/16)	62.1	31.7	20.0	35.9	–	–	–

Table 4: OVS vs . fully supervised segmentation. Fully Supervised : best method picked per dataset. ∗ : self-evaluated, † : from CAT-Seg. Mask2Former numbers from DINOv3 [ 65 ] paper. Domain : using annotations from each training set. Annot: the number of pixel-level annotated images that each method uses. For Domain we report the ADE annotations.

Figure 13: Average performance (mIoU) vs . inference time (s). DINOv3.txt, patch-level;

B{=}5

. Avg. on VOC, ADE, Stuff. A single NVIDIA

A100

GPU is used.

Method	$B=1$	$B=5$	$B=10$
RnS ( $K=1$ )	40.02 -1.57	46.83 -1.04	48.06 -0.96
RnS ( $K=4$ )	41.59	47.87	49.02
RnS ( $K=16$ )	41.97 +0.38	47.78 -0.09	48.96 -0.06
RnS ( $K=32$ )	42.02 +0.43	47.59 -0.28	48.84 -0.18

Table 5: Ablations of the k-NN retrieval hyperparameter

K

in RnS . We report average mIoU across the considered datasets for three different numbers of available support images per class (

B=1

,

B=5

,

B=10

). The row with

K=4

(highlighted) corresponds to the configuration used in RnS , and blue numbers denote the difference to this row in the same column.

Figure 14: Qualitative comparison of patch-level vs. mask-level segmentation. Each row shows the input image, ground-truth mask, the PCA projection of either patch features or SAM mask features (average VLM patch features on top of SAM mask), and the corresponding predictions of RnS when applied on patches or on region proposals. Both variants use DINOv3.txt features [ 65 ] , and SAM 2.1 is used as the mask proposal generator [ 57 ] .

Figure 15: Qualitative comparisons between zero-shot baseline, FreeDA , kNN- CLIP and RnS with and without class name information. All visually supported methods use one support image per class (

B{=}1

). All methods use OpenCLIP ViT-B/16 as VLM features [ 65 ] and SAM 2.1 as region proposal generator [ 57 ] .

Figure 16: Visual support sets for personalized segmentation. RnS used for personalized segmentation using OpenCLIP ViT-B/16 features and SAM 2.1 as region proposer. Initially, visual support includeds images in support before and is later expanded to support after , with support before

\subset

support after . Green: personalized instance, red: generic class, and black: background. Images from PODS [ 68 ] , i-CIR [ 53 ] , and self-collected.

Figure 17: Full textual and visual support per dataset (OpenCLIP, ViT-B/16). We compare zero-shot, RnS , kNN- CLIP and FreeDA and their variants without class name information (w/o text) for increasing number of support images per class. SAM 2.1 is used for region proposals.

Figure 18: Full textual and visual support per dataset (DINOv3.txt, ViT-L/16). We compare zero-shot, RnS , kNN- CLIP and RnS without class name information (w/o text) for increasing number of support images per class. Prediction at the patch level or SAM 2.1 is used for region proposals.

Figure 19: Partial visual support setting per dataset. Results of zero-shot, RnS , kNN- CLIP , and FreeDA , along with ablations of RnS without text and without the pseudo-label loss. OpenCLIP ViT-B/16 and SAM 2.1 are used. A fraction of classes lack visual examples, while

B=3

for the remaining classes.

Figure 20: Partial textual support setting per dataset. Results of Zero-shot, RnS , kNN- CLIP , and FreeDA , together with their variants without class name information (w/o text). OpenCLIP ViT-B/16 and SAM 2.1 are used. A fraction of classes lack textual class names, and

B=1

for all classes.

Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?

Abstract

Figures & tables

Explore similar work

Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration

Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion

dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3