Visual features extracted from vision models like DINOv2 or Stable Diffusion enable high quality semantic correspondence between images. However, such features exhibit documented limitations that hinder accurate correspondence estimation such as the inability to disambiguate the left from right for categories exhibiting symmetries or repeated parts. Previous attempts to improve these features rely on using supervised data such as annotated keypoint pairs or camera pose information, or make restrictive topological assumptions, such as modelling the object of interest as a 3D sphere. We introduce a new self-supervised approach that allows us to learn features for topologically complex object categories using a simple prior. Instead of relying on an expensive instance-level analysis by synthesis pipeline, our approach makes use of contrastive learning and distribution matching at the global dataset-level to learn the coarse shape and appearance of a category. Combined with additional geometric losses, this enables us to replicate the global and local statistics of visual features which match those that are extracted from real images. Quantitative results are presented across several challenging object categories, demonstrating the effectiveness of our approach.
Figures & tables
Figure 1: Pre-trained self-supervised models like DINOv2 [ 33 ] produce local visual features that encode distinct semantic parts that generalise across different depictions of a category. In (a) and (b) (images from [ 33 ] ), we observe that the features for the horse’s head and tail are similar even if when coming from an image or drawing. In (c) we display the features derived from an abstract 3D collection of spheres that broadly has the same structure as a horse. However, the resulting features are not consistent with those from (a) and (b). In (d) we show features from our proposed self-supervised approach that optimises the shape and appearance of the shape from (c) such that when rendered and passed through DINOv2 they result in consistent visual features.
Figure 2: Overview of our approach which consists of feature learning via canonical shape reconstruction and then feature adapter learning. In the first stage, we start with a collection of images depicting the same object category ( e.g . a horse) and an initial simple 3D shape prior in the form of a graph. We then optimise the shape and appearance of the canonical shape such that image features derived from it match the distribution of those obtained from real images, all without requiring any instance-level supervision. In the second stage, we refine our features using an adapter by exploiting the fact that in the first stage we have generated a mechanism for creating paired weakly supervised training data that maps image features to per-pixel canonical 3D coordinates.
DINOv2 [ 33 ]
87.5
84.3
80.0
DINOv2 [ 33 ] PCA16
83.3
85.5
80.0
SphericalMaps [ 29 ] (sphere only)
64.6
61.5
80.0
SphericalMaps [ 29 ]
91.7
85.5
80.0
Ours Contrastive
81.6
73.5
60.0
Ours Distributive
83.3
72.3
60.0
Table 1: Evaluation on PF-Pascal using [email protected] for cosine similarity nearest neighbour from source to target. We follow other papers and use the image resolution when calculating the threshold rather than the bounding box. For topk, we used k=5 matches.
DINOv2 [ 33 ]
53.4
59.8
51.1
DINOv2 [ 33 ] PCA16
51.8
59.8
50.7
A-CSM [ 24 ]
26.3
32.9
28.6
MagicPony [ 50 ]
42.5
42.9
41.2
Farm3D [ 16 ]
40.2
49.1
36.1
DualPM [ 20 ] (supervised)
66.9
73.2
66.8
Table 2: Keypoint matching on PF-Pascal using [email protected]. A-CSM, MagicPony, Farm3D, and DualPM project their 3D reconstruction vertices onto each image and assign the source feature to the nearest projected vertex which is then matched to the feature it is projected onto in the target image. Like [ 20 , 50 ] , we use the image resolution when calculating the threshold rather than the bounding box. For topk, we used k=5 matches.
Figure 3: Here we display the learned canonical 3D shape for different object categories using the Distributional approach. The first column shows representative training images for each category while the second column shows our learned canonical shape. The next two columns show DINOv2 features extracted from real images while the final two columns show DINOv2 features extracted for our synthetic images. The features are visualised using PCA. Our resulting features are trained to look like those from real images, but the appearance of the 3D shape is not constrained to look realistic.
Figure 4: Qualitative results of applying our adapter to DINOv2 features from real images using shapes learned using the Distributional approach. Columns from left to right, depict results for cows, horses, and sheep, where each row is a different image. For each column, the original DINOv2 features are on the left and the predicted 3D positions on the template representation are on the right. Our results represent the predicted 3D coordinates mapping to a location on the learned template. Unlike the original DINOv2 features, which in many instances are unable to encode left-right disambiguation, our model much more clearly separates such parts.
PFP.
AwA
Initial 3D Prior
7.2
1.8
Lcontr+Lgeoc
37.3
16.0
Ldistr+Lgeoc
31.3
13.8
Lcontr+Lgeoc+Lreg w/ sphere prior
34.9
15.8
Ldistr+Lgeoc+Lreg w/ sphere prior
62.7
34.6
Lreg
26.5
14.0
Table 3: Keypoint matching scores on horses in PF-Pascal (PFP) and AwAPose (AwA) for ablations of the model, evaluated using [email protected]. The poor performance of the initial prior shows the gains of the optimisation while the sphere prior shows having a more complex prior can improve results. Geometric regularisation, is necessary but not sufficient.
Cow
Horse
Sheep
Giraffe
Zebra
DINOv2
48.7
54.7
56.3
69.9
59.0
DINOv2 (pca16)
46.0
53.9
48.6
67.1
59.4
Ours Contrastive
48.5
41.8
43.6
40.9
36.0
Ours Distributive
46.4
36.6
39.9
50.5
25.4
Ours Contrastive (topk)
61.5
64.3
61.3
67.4
61.3
Ours Distributive (topk)
61.4
55.7
60.4
68.0
59.1
Table 4: Matching scores for keypoints with left and right equivalents in AwAPose using [email protected]. In most instances we improve left-right disambiguation compared to DINOv2.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Cow
Horse
Sheep
Giraffe
Zebra
DINOv2
60.5
65.5
61.9
76.3
69.2
DINOv2 (pca16)
56.8
64.7
57.0
73.2
68.1
Ours Contrastive
49.6
51.6
44.5
46.8
46.2
Ours Distributive
48.7
43.3
40.3
56.5
27.0
Ours Contrastive (topk)
64.2
68.5
62.2
71.9
68.9
Ours Distributive (topk)
63.8
63.5
61.5
72.9
66.7
Appendix
Table A1: Keypoint matching scores for five categories in AwAPose including cow, horse, sheep, giraffe and zebra, evaluated using [email protected].
Figure A1 : Results from the PFPascal and AwAPose test sets are presented in the first two columns and their corresponding adapted features in the second two columns. ’+’ indicates the labelled keypoint, while predictions are indicated by a dot where both DINOv2 and our model were successful, a downward arrow where DINOv2 succeeded but ours failed, a star where ours succeeded and DINOv2 failed, and a ’X’ where both models failed. Looking at the first row we see our model has successfully distinguished the four legs of the cow and this has enabled it to differentiate the two front legs. It also now succeeds on the back leg, made difficult due to the ambiguity of the original keypoint. It has also successfully distinguished the two eyes.
Avg
DINOv2 [ 33 ]
86.0
56.3
85.9
90.4
73.4
87.5
72.4
84.3
62.1
51.2
80.0
75.4
DINOv2 [ 33 ] PCA16
85.4
48.4
85.9
88.0
64.1
83.3
60.1
85.5
54.8
36.6
80.0
70.2
SphericalMaps [ 29 ] (sphere only)
66.6
21.9
52.6
58.7
79.9
64.6
68.7
61.5
29.4
19.5
80.0
54.9
SphericalMaps [ 29 ]
90.6
78.1
87.2
94.6
84.2
91.7
84.7
85.5
71.8
65.9
80.0
83.1
Ours Contrastive
51.7
17.2
35.9
59.3
41.3
81.6
52.8
73.5
52.0
22.0
60.0
43.9
Ours Distributive
48.9
18.8
55.1
85.3
73.9
83.3
44.2
72.3
78.5
46.3
60.0
60.6
Appendix
Table A2: Evaluation on PF-Pascal for additional categories, where for each category we calculate [email protected] for cosine similarity nearest neighbour from source to target. We follow other papers and use the image resolution when calculating the threshold rather than the bounding box. For the topk approach we used k=5 matches.
Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.
Artur Jesslen, Olaf Dünkel, Adam Kortylewski
University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany
Partial shape matching is a crucial yet underexplored problem in 3D vision, with significant relevance to real-world scenarios where shapes are often only partially observed. Existing feature descriptors face difficulties in this setting, as traditional representations either struggle with the boundaries of partial shapes or heavily depend on the shape's spatial position. While existing approaches have employed DINO features for partial shape matching, these features are not inherently suited for handling partial observations. In this work, we propose a method to refine DINO features using LoRA-based self-supervised learning, enabling the generation of feature descriptors that are robust to partiality. Our features substantially improve performance on partial shape matching compared to traditional or vision foundation features. Additionally, when integrated into existing partial shape matching pipelines, we achieve state-of-the-art results on partial shape matching and left-right prediction benchmarks.
Viktoria Ehm, Dongliang Cao, Riccardo Marin +4
Technical University of Munich · Munich Center for Machine Learning · University of Bonn
Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, yet the mechanisms underlying this behavior remain unclear. We present a systematic empirical dissection of the DINO family and show that semantic representations arise primarily from enforcing consistency between geometrically distinct global views of the same image instance. This instance-specific global alignment acts as the semantic anchor of DINO-style learning. Across controlled retraining experiments evaluated on semantic correspondence and a diverse suite of 2D and 3D downstream tasks, we find that patch-level masking objectives enhance semantics only when trained jointly with this global alignment, indicating that the iBOT objective refines and densifies existing semantic structure rather than creating it independently. In contrast, local-to-global view alignment does not substantially improve semantic qualities at fixed compute beyond a purely global alignment. Beyond training design, we revisit how semantic representation quality should be evaluated: while classification accuracy is the standard validation score, semantic correspondence provides a complementary axis that more reliably predicts downstream task performance. Together, these findings provide a functional decomposition of DINO-style learning and represent an important step toward understanding how semantic representations emerge in self-supervised vision models.
Basavaraj Sunagad, Artur Jesslen, Adam Kortylewski
CISPA Helmholtz Center for Information Security · University of Freiburg