Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.
Figures & tables
Figure 1: Our Fused Gromov-Wasserstein approach combines inter-feature matching with intra-geometric consistency. (Top) Feature matching yields false correspondences (red) when distinct points ( x,xbad ) share similar features to y . (Bottom) 3D Gromov-Wasserstein penalizes distortions to filter invalid matches.
Figure 2: Overview of our pseudo-label generation pipeline. We first compute an initial semantic match to identify high-confidence anchors. These anchors are then used to create a tractable, linear approximation of the otherwise intractable quadratic Gromov-Wasserstein (GW) geometric cost. This approximated geometric cost is fused with the semantic cost, yielding a final fused cost matrix. This cost matrix is then used to solve an UOT problem, producing a transport plan π(T) that serves as our pseudo-label robust to geometric ambiguities.
Figure 3: Impact of the Gromov-Wasserstein term. Wasserstein baseline (blue) vs. Fused Gromov-Wasserstein (orange). Incorporating geometric structural consistency leads to consistent improvements (+2.3 ∼ +7.0%p) across categories.
Method
avg ( ↑ )
ASIC
[ 8 ]
57.9
25.2
68.1
24.7
35.4
28.4
30.9
54.8
21.6
45.0
47.2
39.9
26.2
48.8
14.5
24.5
49.0
24.6
36.9
DINOv2
[ 22 ]
72.7
62.4
85.2
41.4
40.3
52.5
51.5
71.3
36.1
67.2
65.0
67.6
61.1
68.5
30.6
61.9
54.3
24.3
55.7
DIFT
[ 31 ]
63.5
54.5
80.8
34.5
46.2
52.7
48.3
77.7
39.0
76.0
54.9
61.3
53.3
46.0
57.8
57.1
71.1
63.4
57.7
DistillDIFT †
[ 7 ]
70.3
55.4
85.9
36.8
51.9
51.7
50.5
78.0
40.5
73.8
65.3
65.2
54.5
66.0
47.8
59.4
55.6
54.4
59.8
DINOv2 + SD †
[ 40 ]
72.9
63.4
86.4
40.5
52.6
55.4
53.3
78.4
45.2
77.1
64.7
69.4
62.9
68.5
56.8
67.0
65.9
51.8
63.5
Ours
73.5
66.7
89.9
40.8
58.0
55.4
51.1
84.8
52.1
81.0
71.0
75.3
64.1
71.8
62.1
70.2
70.2
64.4
67.9
Table 1: Per-category [email protected] scores (per-keypoint) on SPair-71k ( higher is better ↑ ). Best and second best are highlighted. † denotes results re-evaluated under identical standard evaluation settings from [ 20 ] .
Table 2: Results for different PCK levels (per-keypoint) on SPair-71k and AP-10k. Results for AP-10K are intra-species (I.S.), cross-species (C.S.) and cross-family (C.F.), following [ 39 ] .
Figure 4: Qualitative comparison on challenging cases from SPair-71k. Four challenging scenarios: (a) geometric ambiguity from extreme viewpoint changes and occlusion; (b) extreme pose variation causing self-occlusion in non-rigid objects; (c) large scale changes with feature ambiguity in textureless regions; and (d) fine-grained semantic ambiguity from background clutter and intra-class variations. Green circles indicate correct matches ([email protected]), red circles indicate errors.
Figure 5: Ablation study on pseudo-label generation strategies on SPair-71k. Performance (PCK label @0.1) on the Geometry-aware subset (left) and Overall set (right). (1) Nearest Neighbor, (2) Semantic OT (feature similarity only), (3) Fused OT (semantic and geometric costs with balanced OT), and (4) our Fused UOT.
Figure 6: Qualitative comparison of pseudo-label generation methods. (a) Nearest Neighbor matching produces geometrically inconsistent correspondences with multiple errors ( red lines ). (b) Our method (SoY) generates globally consistent correspondences ( green lines ) by leveraging 3D geometric structure.
Intra-structure distance for Fused OT
PCK label @0.1
Baseline: Semantic OT (w/o GW term)
66.5
(1) w/ 2D distance
65.7
(2) w/ Semantic distance (intra-feature)
66.2
(3) w/ 3D geometric distance (ours)
67.6
Table 3: Ablation on the intra-structure for pseudo-label generation. We compare the 3D geometric structure (ours) against 2D based alternatives to justify the Fused OT strategy (Fig. 5 ).
Table 4: Ablation study of average [email protected] (per-keypoint) on SPair-71k. We compare the zero-shot backbone against adapters trained with (1) Nearest Neighbor (NN) and (2) our proposed FGW labels, along with incremental gains from (3) relaxed cycle consistency (c.c.) and (4) soft target loss. ( ∗ : baseline variants)
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Pseudo-label analysis with DINOv2 backbone. Performance (PCK label @0.1) on the geometry-aware subset (left) and overall set (right). Using only DINOv2 features, we compare four pseudo-label generation strategies: Nearest Neighbor, Semantic OT, Fused OT, and Fused UOT. Both the geometry-aware and overall subsets show a consistent, monotonic improvement across methods, with clear gains from incorporating semantic OT and our geometry-aware matching. This confirms that our pseudo-labeling remains effective even with a weaker backbone.
Figure 2: Pseudo-label hyperparameter ablation on SPair-71k (PCK label @0.1). We study three key hyperparameters of our pseudo-label generator: (Left) number of anchors K , (Middle) feature–geometry trade-off α , and (Right) KL regularization strength ρ in UOT. In all cases, the performance is measured as PCK label @0.1 on the SPair-71k test set.
3D backbone
PCK label @0.1
VGGT [ 33 ]
68.1
DUSt3R [ 35 ]
67.1
Appendix
Table 2: Effect of the 3D foundation backbone on pseudo-label quality on SPair-71k. We report PCK label @0.1 for our pseudo-labels when using either VGGT or DUSt3R to obtain 3D structure.
Figure 3: Failure modes of pseudo-label generation. Top: failure when the object breaks into disconnected parts. Bottom: failure when the features used for pseudo labels are highly noisy and all correspondences collapse to a local region.
Figure 4: Visual comparison of semantic correspondences on SPair-71k [ 20 ] across DistillDIFT [ 7 ] , DINOv2+SD [ 40 ] , and our approach. Correct and incorrect matches are indicated by green lines and red lines , respectively.
Figure 5: Visual comparison of semantic correspondences on SPair-71k [ 20 ] across DistillDIFT [ 7 ] , DINOv2+SD [ 40 ] , and our approach. Correct and incorrect matches are indicated by green lines and red lines , respectively.
University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany