cs.CVMar 30, 2026
SaveINSID3: Training-Free In-Context Segmentation with DINOv3
Organizations: Politecnico di Torino · TU Darmstadt · TU Munich · ELIZA · MCML
Abstract
In-context segmentation (ICS) aims to segment arbitrary concepts, e.g., objects, parts, or personalized instances, given one annotated visual examples. Existing work relies on (i) fine-tuning vision foundation models (VFMs), which improves in-domain results but harms generalization, or (ii) combines multiple frozen VFMs, which preserves generalization but yields architectural complexity and fixed segmentation granularities. We revisit ICS from a minimalist perspective and ask: Can a single self-supervised backbone support both semantic matching and segmentation, without any supervision or auxiliary models? We show that scaled-up dense self-supervised features from DINOv3 exhibit strong spatial structure and semantic correspondence. We introduce INSID3, a training-free approach that segments concepts at varying granularities only from frozen DINOv3 features, given an in-context example. INSID3 achieves state-of-the-art results across one-shot semantic, part, and personalized segmentation, outperforming previous work by +7.5 % mIoU, while using 3x fewer parameters and without any mask or category-level supervision. Code is available at https://github.com/visinf/INSID3 .
Figures & tables
Figure 2: Region-level grouping from DINOv3. Each pair shows an input image (left) and the corresponding clustering map (right) obtained by applying agglomerative clustering to dense DINOv3 features. The resulting clusters delineate coherent object- and part-level regions, providing a structured decomposition of the scene.
Figure 3: Overview of INSID3. We leverage the semantic and spatial structure of DINOv3 to perform in-context segmentation without training or model composition. Dense features from the reference and target images are first debiased to suppress positional bias, improving cross-image matching. The target is then decomposed into coherent regions through agglomerative clustering, providing a structured representation. We retain candidate clusters that match the reference through backward correspondence in the debiased space; a reference prototype derived from the annotated region anchors the seed cluster via cross-image similarity. Finally, we combine cross-image similarity, capturing semantic alignment, with self-similarity, measuring the affinity of each cluster to the seed, to form the final mask from the seed.
Figure 4: Positional bias in DINOv3 features. For both region 4 and keypoint 4 prompts, similarity maps computed with the original DINOv3 features show structured activations aligned with the reference coordinates, independent of semantics. Our debiased features mitigate this behavior. 4 PCA of features from images with low semantic complexity ( e.g . , noise, flat textures) reveals a stable low-dimensional positional subspace underlying this bias.
Table 1: Comparison of INSID3 (mIoU in %, ) on one-shot semantic, part, and personalized segmentation. State-of-the-art methods are grouped into task-specific fine-tuning and training-free approaches. Previous training-free methods rely on SAM, pre-trained with mask-level supervision, whereas INSID3 uses only frozen self-supervised DINOv3 features. Gray indicates the model was trained on the corresponding train split of the dataset; best results bold , 2 nd best underlined . denotes a GF-SAM variant using DINOv3 features.
Figure 5: Comparison of INSID3 with GF-SAM [ 69 ] and SegIC [ 41 ] on one-shot semantic (left) , part (top right) , and personalized (bottom right) segmentation. SegIC performs well in-domain but struggles to generalize across domains and part granularity, reflecting its limited flexibility beyond the training distribution. GF-SAM , relying on the strong segmentation priors of SAM [ 33 ] , produces high-quality masks; however, the decoupled mechanism between correspondence and segmentation often leads to over- or under-segmentation. INSID3 , despite relying solely on self-supervised features, achieves precise localization and competitive mask quality.
| Small | Base | Large | |||||
| original | debias | original | debias | original | debias | ||
| 0.05 | 27.9 | 32.3 | 33.6 | ||||
| 0.10 | 45.7 | 50.0 | 52.0 | ||||
| 0.15 | 55.6 | 59.9 | 62.1 | ||||
| 0.20 | 62.6 | 66.4 | 68.6 | ||||
Table 2: Semantic correspondence on SPair-71k (PCK@ in %, ). Comparison across DINOv3 backbones, w/ and w/o debiasing.
Figure 6: Qualitative examples on SPair-71k with DINOv3-L.
Figure 7: INSID3 hyperparameters. We study the effect of the debiasing rank (left) and the clustering granularity (right) on semantic (COCO-20 i ) and part (PASCAL-Part) segmentation.
| Variant | COCO | PASCAL-Part |
| No clustering | ||
| Thresholding sim. map @ 0.55 (Eq. 2 ) | 44.2 | 35.4 |
| Clustering w/o aggregation | ||
| Coarse clustering ( \tau=\text{0.5} ) | 50.6 | 31.1 |
| Fine clustering ( \tau=\text{0.6} ) | 42.8 | 36.2 |
| Ours : Clustering w/ aggregation ( \tau=\text{0.6} ) | ||
Table 3: Effect of clustering and aggregation. We compare our approach with tuned baselines on COCO-20 i and PASCAL-Part (mIoU %, ). All thresholds (0.55) and clustering granularities ( \tau{=}\text{0.5} , \tau{=}\text{0.6} ) are tuned to optimal values for fair comparison.
Figure 8: Positional bias in DINOv2 vs . DINOv3. We visualize the similarity map between a reference keypoint and all target patches using Eq. 16 . Darker purple regions indicate low similarity, while brighter yellow regions indicate high similarity. DINOv3 exhibits strong coordinate-aligned artifacts, with bright responses appearing at the same absolute spatial location as the reference keypoint. Our debiasing projection suppresses these spurious activations. We observe that DINOv2 shows much weaker positional bias.
| Backbone | Features | [email protected] | [email protected] | [email protected] |
| DINOv2 | original | 34.4 | 50.5 | 66.8 |
| debias | 34.7 (+0.3) | 50.8 (+0.3) | 67.1 (+0.3) | |
| DINOv3 | original | 29.2 | 45.0 | 59.8 |
| debias | 32.3 (+3.1) | 50.0 (+5.0) | 66.4 (+6.6) |
Table 4: Semantic correspondence on SPair-71k. PCK@ (%, ) for DINOv2 and DINOv3 using original and debiased features. Debiasing produces marginal improvements for DINOv2, but yields substantial gains for DINOv3.
| Method | mIoU |
| No debiasing | 54.5 |
| Augmentations, 4 views | 55.7 |
| Augmentations, 12 views | 56.5 |
| Ours (SVD from 1 noise image) | 57.6 |
| SVD from 5 noise images | 57.7 |
| SVD from 10 noise images | 57.7 |
Table 5: Different debiasing strategies. Comparison of training-free strategies for feature debiasing on COCO-20 i using mIoU (in %, ). Augmentations are a viable alternative but require multiple forward passes . In contrast our method adds no overhead, except for a single matrix multiplication, since the debiasing projection is pre-computed offline and stored.
Figure 9: Debiased features preserve semantic structure. We compute PCA bases on COCO training images for (i) the original DINOv3 features and (ii) our debiased features. We then project both representations to principal components and evaluate semantic correspondence ( ) on SPair-71k. Across all , Debias + PCA consistently outperforms PCA-original, indicating that the removed directions primarily encode positional bias. After debiasing, the remaining variance concentrates around semantic structure, yielding better results when compressed via PCA.
| Hyperparameter | Value | |
| (clustering sensitivity) | 0.6 | |
| (aggregation threshold) | 0.2 | |
| (debias rank) | 500 | |
Table 6: Hyperparameter overview. INSID3 uses only three hyperparameters, all fixed across datasets, for both semantic, part, and personalized segmentation.
Table 7: Comparison of INSID3 (mIoU in %, ) on 5-shot semantic and part segmentation. Models are provided with 5 contextual examples and tasked with segmenting the annotated concept in the target image. INSID3 scales effectively to multiple references, achieving robust performance across domains. All hyperparameters are reused from the 1-shot setting without any tuning, highlighting the versatility of our approach. Gray indicates the model was trained on the corresponding train split of the dataset; best results bold , 2 nd best underlined . denotes a GF-SAM variant using DINOv3 features.
| DINOv3 | DINOv2 | Franca | Perception Enc. | Stable Diff. |
Table 8: Backbone analysis for INSID3. We report mIoU (in %, ) on COCO-20 i of INSID3 with different backbone features. DINOv3 features lead to the best segmentation accuracy.
| Method | mIoU |
| Training free : Mask-supervised pre-training | |
| Segment Anything 3 (video-style propagation) | 26.7 |
| Segment Anything 3 (image concatenation) | 52.9 |
| Training free : Unsupervised pre-training | |
| INSID3 (ours) | 57.6 |
Table 9: Comparison with SAM3. We compare INSID3 to Segment Anything 3 (SAM 3) on COCO-20 i using mIoU (in %, ). We evaluate two adaptations of SAM 3: (i) a video-style formulation, where reference and target are treated as consecutive frames and the mask is propagated, and (ii) an image concatenation strategy, where images are combined horizontally and prompted.
| Method | Correct empty predictions (in %) |
| Task-specific fine-tuning : Semantic + mask supervision | |
| SegGPT [ 64 ] | 79 |
| SegIC [ 41 ] | 36 |
| Training free : Mask-supervised pre-training | |
| PerSAM [ 72 ] | n.a. |
| Matcher [ 39 ] | n.a. |
Table 10: Absence of the reference concept. Percentage of correct empty-mask predictions (%, ) when the target image does not contain the prompted object (COCO, 4 000 pairs). n.a. : methods that always return a non-empty mask ( i.e . , in practice).
| Method | Backbone | Resolution | Runtime |
| Task-specific fine-tuning : Semantic + mask supervision | |||
| SegGPT [ 64 ] | ViT | 640 2 | |
| SegIC [ 41 ] | DINOv2 | 896 2 | |
| Training free : Mask-supervised pre-training | |||
| PerSAM [ 72 ] | SAM | 1024 2 | |
| Matcher [ 39 ] | DINOv2 + SAM | 1024 2 | |
Table 11: Computational analysis of INSID3 vs . previous methods in milliseconds (ms, ). Inference time is measured for a single in-context example ( i.e . , example and target image). Runtime is measured on a single RTX 4090.
| Component | Runtime |
| Encoder forward debiasing | |
| Compute similarity cluster scoring | |
| Clustering | 166 ms |
| CRF refinement | 55 ms |
| Total inference time |
Table 12: Detailed computational analysis of INSID3. We report inference time of each component in milliseconds (ms, ). Inference time is measured for a single in-context example. Runtime measured on a single RTX 4090.
| Low resolution Runtime of INSID3 High resolution | ||||||
Table 13: Computational analysis for different resolutions. We report inference time of INSID3 for different resolution in milliseconds (ms, ). Inference time is measured for a single in-context example. Runtime measured on a single RTX 3090.
Figure 10: Qualitatives results with INSID3 . Each pair shows the reference image with its annotated region ( light blue mask ) and the predicted mask on the target ( purple ) . INSID3 handles a wide range of semantic granularities, from full objects to fine-grained parts, and generalizes robustly across diverse domains, including aerial, marine, medical, and everyday scenes.