Self-supervised Vision Transformers (ViTs) like DINO encode rich semantics, enabling training-free segmentation of objects from the background and discovery of their parts via feature clustering. This paper focuses on part discovery: partitioning a foreground object into semantic parts. Existing approaches produce imprecise part boundaries. We trace this to the rigid ViT patch grid used in training DINO, where patches straddling boundaries contain pixels from multiple parts. While DINO's invariance-based training could absorb this pixel-level mixing into clean features, experiments with over 26K foreground patches show it does not: boundary-overlapping patches produce features with higher entropy and lower assignment confidence, blurring predicted part boundaries. Rather than make the massive effort of retraining DINO over geometrically coherent regions, an easier approach is to take boundaries directly from a boundary detector, which supplies the geometric information DINO's features lack. Motivated by this, we introduce Part-DINO, a training-free framework that combines boundaries from low-level image segmentation with DINO's semantic features. We use the boundaries to partition the image into regions, pool DINO features within each region, and group them to discover parts. Without any training, Part-DINO produces structurally coherent part assignments, outperforming training-free baselines across four standard benchmarks (CUB, CelebA, PartImageNet-OOD, PartImageNet-Seg) and remaining competitive with weakly-supervised and unsupervised methods.
Figures & tables
Figure 1: Mixing of pixels from across part boundaries present in boundary-area patches blurs semantic boundaries. (a) Input image. (b) A boundary patch straddling the bird’s head and body pools pixels from both regions, illustrating geometric mixing in pixel space. (c) Across 26K foreground patches, feature entropy rises and the top-2 assignment margin falls as patches approach part boundaries, showing mixing in feature space, which causes ambiguity in inferences of semantic part boundaries as can be seen in (d) where edges from DINO feature similarity (left) blur at part boundaries, while edges from image contrast (right) align with them. Recovery of sharp boundaries from DINO features may thus be expected to benefit from complementary geometric information.
Figure 2: Overview of Part-DINO. Given a set of reference images I , the Region Generation module (Sec. 3.1 , top) extracts a foreground mask of the object Mfg and performs low-level image segmentation to produce atomic regions. In parallel, the Prototype Generation module (Sec. 3.2 , bottom) encodes the reference images through a frozen DINO ViT to obtain dense feature maps. These semantic features are pooled within each atomic region and clustered via K-Means to establish K global part prototypes {c1,c2,…,cK} . At inference (right), the Region Assignment module (Sec. 3.3 ) applies the same region generation pipeline to a test image and assigns each region to its nearest semantic prototype, producing the final part labels without any training.
Method
K=4
K=8
K=16
NME ↓
NMI ↑
ARI ↑
NME ↓
NMI ↑
ARI ↑
NME ↓
NMI ↑
ARI ↑
Weakly-supervised part discovery
Huang and Li (2020)
11.51
29.74
14.04
11.60
35.72
15.90
12.60
43.92
21.01
Van Der Klis et al. (2023)
9.12
37.82
15.26
8.52
50.08
26.96
7.64
56.87
38.05
Choudhury et al. (2021)
9.20
43.50
19.60
-
51.50
28.30
-
-
-
Van Der Klis et al. (2023) + ViT-B/14
7.70
52.59
26.66
6.34
65.01
37.90
5.95
68.63
43.41
Table 1: Quantitative comparisons on CUB dataset across part granularities K ∈{4,8,16} . We report NME ( ↓ ), NMI ( ↑ ), and ARI ( ↑ ) against weakly-supervised, unsupervised, and training-free methods. † indicates replicated with SAM foreground mask
Method
K=4
K=8
K=16
NME ↓
NMI ↑
ARI ↑
NME ↓
NMI ↑
ARI ↑
NME ↓
NMI ↑
ARI ↑
Weakly-supervised part discovery
Huang and Li (2020)
8.75
56.69
34.74
7.96
54.80
34.74
7.62
62.22
41.01
Van Der Klis et al. (2023)
11.11
75.97
69.53
9.82
62.61
51.89
9.46
77.43
70.58
Unsupervised part discovery
GANSeg He et al. (2022b)
12.26
41.71
28.06
6.18
67.28
56.23
-
-
-
Table 2: Quantitative comparison on CelebA across part granularities K ∈{4,8,16} . We report NME ( ↓ ), NMI ( ↑ ), and ARI ( ↑ ) against weakly-supervised, unsupervised, and training-free methods. † indicates replication with SAM foreground masks.
Figure 3: Qualitative comparison across CUB ( K=16 ), CelebA ( K=16 ), PartImageNet-S ( K=50 ), and PartImageNet-O ( K=50 ). MPAE is an unsupervised method that requires dataset-specific training, while Amir et al. (2021) , Amir et al. (2021) + SAM, and our Part-DINO are all training-free.
Table 6
Method
PartImage _O (%)
PartImage _S (%)
K=8
K=25
K=50
K=8
K=25
K=50
NMI ↑
ARI ↑
NMI ↑
ARI ↑
NMI ↑
ARI ↑
NMI ↑
ARI ↑
NMI ↑
ARI ↑
NMI ↑
ARI ↑
Weakly-supervised part discovery
Huang and Li (2020)
5.88
1.53
7.56
1.25
10.19
1.05
-
-
-
-
-
-
Van Der Klis et al. (2023)
27.13
8.76
32.41
10.69
41.49
14.17
10.68
32.43
18.44
40.70
21.98
39.40
Van Der Klis et al. (2023) (ViT-B/14)
19.28
34.72
28.23
25.93
29.48
28.68
16.68
32.01
46.81
40.70
34.13
39.40
Table 5: Quantitative comparison on PartImageNet Out-of-Distribution (PartImageNet O ) and Segmentation (PartImageNet S ) splits across K∈{8,25,50} . Part-DINO shows consistent improvements over the training-free baseline on PartImageNet S across all granularities and metrics, and remains competitive on PartImageNet O . † indicates replicated with SAM foreground mask
Figure 8
Figure 5: Qualitative experiments on region generation. (a) The structural edges from RAMP align with object contours, while DINO’s semantic edges are blurred. (b) Watershed regions are separable (head, body, wing, and tail), while SLIC prioritizes spatial compactness and cuts across them.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Regions/img
NME ↓
NMI ↑
ARI ↑
Amir et al. (2021) †
–
4.87
56.92
35.10
Part-DINO ( smin=10 , default, rerun)
506.7
4.33
64.98
45.83
Part-DINO ( smin=0 , no merge)
525.2
4.36
65.01
43.92
Appendix
Table 7: Effect of the DINO-based micro-region merge on CUB ( K=16 ). With smin=0 , region generation uses no DINO features. Both Part-DINO rows come from the same run. † replicated with SAM foreground mask.
Runtime Breakdown (ms)
Metrics
Method
DINO
SAM
RAMP
Wshed
Merge
Pool
Assign
Total
Avg. Reg.
NME ( ↓ )
NMI ( ↑ )
ARI ( ↑ )
( Amir et al., 2021 ) (patch)
40.0
552.4
0.0
0.0
0.0
6.0
1.1
599.5
1024
4.87
56.92
35.10
Part-DINO ( dmin=15 )
40.2
558.3
32.4
23.8
5.2
1.3
0.4
664.9
23.1
5.20
42.57
25.21
Part-DINO ( dmin=5 )
39.7
562.9
32.8
24.5
6.1
1.3
1.4
672.2
151.1
4.44
61.96
43.13
Part-DINO ( dmin=2 )
39.6
563.1
31.9
29.0
7.7
1.4
3.7
680.2
562.1
4.35
65.45
46.00
Appendix
Table 8: Timing breakdown (ms), average regions, and clustering performance metrics.
Figure 6: Extended Qualitative Analysis (CelebA). Comparison of edge map modalities. Structural boundaries consistently eliminate the semantic bleeding artifacts present in the patch-based baseline.
Figure 7: Quantifying the boundary bottleneck. Across 26,459 foreground patches, all three reliability metrics correlate monotonically with distance to the nearest part boundary: spatial purity rises ρ(d,π)=+0.71 ), while feature entropy falls (( ρ(d,H)=−0.38 ), and top-2 assignment margin grows ( ρ(d,m) = +0.36), confirming that geometric mixing propagates into feature space
Figure 8: Extended Qualitative Analysis (CUB). Comparison of edge map modalities on fine-grained bird categories, highlighting precision improvements in part boundaries.
Figure 9: Extended Qualitative Analysis (PartImageNet). Comparison of structural vs. semantic edge modalities on Out-of-Distribution (OOD) and Segmentation (Seg) examples. Structural boundaries demonstrate robust geometric alignment regardless of semantic category.
Figure 10: Extended Ablation on Region Generation (Part 1). SLIC enforces arbitrary spatial compactness, forcing grid-like boundaries that ignore true anatomical shapes. In contrast, our Watershed approach naturally follows the topographic valleys of the structural gradient, seamlessly wrapping around physical object contours.
Figure 11: Extended Ablation on Region Generation (Part 2). Continued comparison across complex PartImageNet categories, demonstrating the robustness of Watershed boundaries against rigid SLIC superpixels.
In-context segmentation (ICS) aims to segment arbitrary concepts, e.g., objects, parts, or personalized instances, given one annotated visual examples. Existing work relies on (i) fine-tuning vision foundation models (VFMs), which improves in-domain results but harms generalization, or (ii) combines multiple frozen VFMs, which preserves generalization but yields architectural complexity and fixed segmentation granularities. We revisit ICS from a minimalist perspective and ask: Can a single self-supervised backbone support both semantic matching and segmentation, without any supervision or auxiliary models? We show that scaled-up dense self-supervised features from DINOv3 exhibit strong spatial structure and semantic correspondence. We introduce INSID3, a training-free approach that segments concepts at varying granularities only from frozen DINOv3 features, given an in-context example. INSID3 achieves state-of-the-art results across one-shot semantic, part, and personalized segmentation, outperforming previous work by +7.5 % mIoU, while using 3x fewer parameters and without any mask or category-level supervision. Code is available at https://github.com/visinf/INSID3 .
Claudia Cuttano, Gabriele Trivigno, Christoph Reich +3
Politecnico di Torino · TU Darmstadt · TU Munich +2
Self-supervised Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in [CLS] token attention maps of the final layer. However, these maps often contain spurious activations resulting in poor localization of objects. This is because the [CLS] token, trained on an image-level objective, summarizes the entire image instead of focusing on objects. This aggregation dilutes the object-centric information existing in the local, patch-level interactions. We analyze this by computing inter-patch similarity using patch-level attention components (query, key, and value) across all layers. We find that: (1) Object-centric properties are encoded in the similarity maps derived from all three components (q,k,v), unlike prior work that uses only key features or the [CLS] token. (2) This object-centric information is distributed across the network, not just confined to the final layer. Based on these insights, we introduce Object-DINO, a training-free method that extracts this distributed object-centric information. Object-DINO clusters attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects. We demonstrate Object-DINO's effectiveness on two applications: enhancing unsupervised object discovery (+3.6 to +12.4 CorLoc gains) and mitigating object hallucination in Multimodal Large Language Models by providing visual grounding. Our results demonstrate that using this distributed object-centric information improves downstream tasks without additional training.
Samyak Rawlekar, Amitabh Swain, Yujun Cai +3
University of Illinois Urbana-Champaign · University of Queensland · UC Merced
Open-Vocabulary Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).
Haoxi Zeng, Qiankun Liu, Yi Bin +5
Tongji University Shanghai, China · Hong Kong Polytechnic University Hong Kong, China · University of Electronic Science and Technology of China Chengdu, China +1