Self-Supervised Learning of Semantically Consistent Visual Features via a Simple 3D Prior

Sebastian Day; Oisin Mac Aodha

cs.CVOct 12, 2026

Self-Supervised Learning of Semantically Consistent Visual Features via a Simple 3D Prior

Authors: Sebastian Day, Oisin Mac Aodha

Organizations: University of Edinburgh

Abstract

Visual features extracted from vision models like DINOv2 or Stable Diffusion enable high quality semantic correspondence between images. However, such features exhibit documented limitations that hinder accurate correspondence estimation such as the inability to disambiguate the left from right for categories exhibiting symmetries or repeated parts. Previous attempts to improve these features rely on using supervised data such as annotated keypoint pairs or camera pose information, or make restrictive topological assumptions, such as modelling the object of interest as a 3D sphere. We introduce a new self-supervised approach that allows us to learn features for topologically complex object categories using a simple prior. Instead of relying on an expensive instance-level analysis by synthesis pipeline, our approach makes use of contrastive learning and distribution matching at the global dataset-level to learn the coarse shape and appearance of a category. Combined with additional geometric losses, this enables us to replicate the global and local statistics of visual features which match those that are extracted from real images. Quantitative results are presented across several challenging object categories, demonstrating the effectiveness of our approach.

Figures & tables

Figure 1: Pre-trained self-supervised models like DINOv2 [ 33 ] produce local visual features that encode distinct semantic parts that generalise across different depictions of a category. In (a) and (b) (images from [ 33 ] ), we observe that the features for the horse’s head and tail are similar even if when coming from an image or drawing. In (c) we display the features derived from an abstract 3D collection of spheres that broadly has the same structure as a horse. However, the resulting features are not consistent with those from (a) and (b). In (d) we show features from our proposed self-supervised approach that optimises the shape and appearance of the shape from (c) such that when rendered and passed through DINOv2 they result in consistent visual features.

Figure 2: Overview of our approach which consists of feature learning via canonical shape reconstruction and then feature adapter learning. In the first stage, we start with a collection of images depicting the same object category ( e.g . a horse) and an initial simple 3D shape prior in the form of a graph. We then optimise the shape and appearance of the canonical shape such that image features derived from it match the distribution of those obtained from real images, all without requiring any instance-level supervision. In the second stage, we refine our features using an adapter by exploiting the fact that in the first stage we have generated a mechanism for creating paired weakly supervised training data that maps image features to per-pixel canonical 3D coordinates.


DINOv2 [ 33 ]	87.5	84.3	80.0
DINOv2 [ 33 ] PCA16	83.3	85.5	80.0
SphericalMaps [ 29 ] (sphere only)	64.6	61.5	80.0
SphericalMaps [ 29 ]	91.7	85.5	80.0
Ours Contrastive	81.6	73.5	60.0
Ours Distributive	83.3	72.3	60.0

Table 1: Evaluation on PF-Pascal using [email protected] for cosine similarity nearest neighbour from source to target. We follow other papers and use the image resolution when calculating the threshold rather than the bounding box. For topk, we used k=5 matches.


DINOv2 [ 33 ]	53.4	59.8	51.1
DINOv2 [ 33 ] PCA16	51.8	59.8	50.7
A-CSM [ 24 ]	26.3	32.9	28.6
MagicPony [ 50 ]	42.5	42.9	41.2
Farm3D [ 16 ]	40.2	49.1	36.1
DualPM [ 20 ] (supervised)	66.9	73.2	66.8

Table 2: Keypoint matching on PF-Pascal using [email protected]. A-CSM, MagicPony, Farm3D, and DualPM project their 3D reconstruction vertices onto each image and assign the source feature to the nearest projected vertex which is then matched to the feature it is projected onto in the target image. Like [ 20 , 50 ] , we use the image resolution when calculating the threshold rather than the bounding box. For topk, we used k=5 matches.

Panel of Figure 3 — Figure 3: Here we display the learned canonical 3D shape for different object categories using the Distributional approach. The first column shows representative training images for each category while the second column shows our learned canonical shape. The next two columns show DINOv2 features extracted from real images while the final two columns show DINOv2 features extracted for our synthetic images. The features are visualised using PCA. Our resulting features are trained to look like those from real images, but the appearance of the 3D shape is not constrained to look realistic.

Panel of Figure 4 — Figure 4: Qualitative results of applying our adapter to DINOv2 features from real images using shapes learned using the Distributional approach. Columns from left to right, depict results for cows, horses, and sheep, where each row is a different image. For each column, the original DINOv2 features are on the left and the predicted 3D positions on the template representation are on the right. Our results represent the predicted 3D coordinates mapping to a location on the learned template. Unlike the original DINOv2 features, which in many instances are unable to encode left-right disambiguation, our model much more clearly separates such parts.

	PFP.	AwA
Initial 3D Prior	7.2	1.8
$\mathcal{L}_{contr}+\mathcal{L}_{geoc}$	37.3	16.0
$\mathcal{L}_{distr}+\mathcal{L}_{geoc}$	31.3	13.8
$\mathcal{L}_{contr}+\mathcal{L}_{geoc}+\mathcal{L}_{reg}$ w/ sphere prior	34.9	15.8
$\mathcal{L}_{distr}+\mathcal{L}_{geoc}+\mathcal{L}_{reg}$ w/ sphere prior	62.7	34.6
$\mathcal{L}_{reg}$	26.5	14.0

Table 3: Keypoint matching scores on horses in PF-Pascal (PFP) and AwAPose (AwA) for ablations of the model, evaluated using [email protected]. The poor performance of the initial prior shows the gains of the optimisation while the sphere prior shows having a more complex prior can improve results. Geometric regularisation, is necessary but not sufficient.

	Cow	Horse	Sheep	Giraffe	Zebra
DINOv2	48.7	54.7	56.3	69.9	59.0
DINOv2 (pca16)	46.0	53.9	48.6	67.1	59.4
Ours Contrastive	48.5	41.8	43.6	40.9	36.0
Ours Distributive	46.4	36.6	39.9	50.5	25.4
Ours Contrastive (topk)	61.5	64.3	61.3	67.4	61.3
Ours Distributive (topk)	61.4	55.7	60.4	68.0	59.1

Table 4: Matching scores for keypoints with left and right equivalents in AwAPose using [email protected]. In most instances we improve left-right disambiguation compared to DINOv2.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Cow	Horse	Sheep	Giraffe	Zebra
DINOv2	60.5	65.5	61.9	76.3	69.2
DINOv2 (pca16)	56.8	64.7	57.0	73.2	68.1
Ours Contrastive	49.6	51.6	44.5	46.8	46.2
Ours Distributive	48.7	43.3	40.3	56.5	27.0
Ours Contrastive (topk)	64.2	68.5	62.2	71.9	68.9
Ours Distributive (topk)	63.8	63.5	61.5	72.9	66.7

Appendix

Table A1: Keypoint matching scores for five categories in AwAPose including cow, horse, sheep, giraffe and zebra, evaluated using [email protected].

Panel of Figure 10 — Figure A1 : Results from the PFPascal and AwAPose test sets are presented in the first two columns and their corresponding adapted features in the second two columns. ’+’ indicates the labelled keypoint, while predictions are indicated by a dot where both DINOv2 and our model were successful, a downward arrow where DINOv2 succeeded but ours failed, a star where ours succeeded and DINOv2 failed, and a ’X’ where both models failed. Looking at the first row we see our model has successfully distinguished the four legs of the cow and this has enabled it to differentiate the two front legs. It also now succeeds on the back leg, made difficult due to the ambiguity of the original keypoint. It has also successfully distinguished the two eyes.

												Avg
DINOv2 [ 33 ]	86.0	56.3	85.9	90.4	73.4	87.5	72.4	84.3	62.1	51.2	80.0	75.4
DINOv2 [ 33 ] PCA16	85.4	48.4	85.9	88.0	64.1	83.3	60.1	85.5	54.8	36.6	80.0	70.2
SphericalMaps [ 29 ] (sphere only)	66.6	21.9	52.6	58.7	79.9	64.6	68.7	61.5	29.4	19.5	80.0	54.9
SphericalMaps [ 29 ]	90.6	78.1	87.2	94.6	84.2	91.7	84.7	85.5	71.8	65.9	80.0	83.1
Ours Contrastive	51.7	17.2	35.9	59.3	41.3	81.6	52.8	73.5	52.0	22.0	60.0	43.9
Ours Distributive	48.9	18.8	55.1	85.3	73.9	83.3	44.2	72.3	78.5	46.3	60.0	60.6

Appendix

Table A2: Evaluation on PF-Pascal for additional categories, where for each category we calculate [email protected] for cosine similarity nearest neighbour from source to target. We follow other papers and use the image resolution when calculating the threshold rather than the bounding box. For the topk approach we used k=5 matches.

Self-Supervised Visual Representation Learning

Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.

See all 235 papers

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence
May 28, 2026Artur Jesslen, Olaf Dünkel, Adam KortylewskiSemantic Correspondence
University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany
Teaching DINOv3 About Partial 3D Geometry: A Self-Supervised Geometry-Aware Approach
Oct 15, 2026Viktoria Ehm, Dongliang Cao, Riccardo Marin +43D Representation Learning Self-Supervised Visual Representation Learning
Technical University of Munich · Munich Center for Machine Learning · University of Bonn
Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning
Sep 23, 2026Basavaraj Sunagad, Artur Jesslen, Adam KortylewskiMulti-View Learning Self-Distillation
CISPA Helmholtz Center for Information Security · University of Freiburg

May 28, 2026J/K move · Enter open · S save

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Artur Jesslen, Olaf Dünkel, Adam Kortylewski

University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany

Semantic Correspondence

Open paper AskPDF arXiv

Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.

Abstract

Self-Supervised Visual Representation Learning

Explore similar work

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Teaching DINOv3 About Partial 3D Geometry: A Self-Supervised Geometry-Aware Approach

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning