Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

Jiin Im; Sisung Liu; Je Hyeong Hong

cs.CVMar 12, 2026

Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

Authors: Jiin Im, Sisung Liu, Je Hyeong Hong

Organizations: Dept. Electronic Engineering, Hanyang University · Dept. Artificial Intelligence, Hanyang University

Abstract

Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.

Figures & tables

Figure 1: Our Fused Gromov-Wasserstein approach combines inter-feature matching with intra-geometric consistency. (Top) Feature matching yields false correspondences (red) when distinct points ( $x,x_{bad}$ ) share similar features to $y$ . (Bottom) 3D Gromov-Wasserstein penalizes distortions to filter invalid matches.

Figure 2: Overview of our pseudo-label generation pipeline. We first compute an initial semantic match to identify high-confidence anchors. These anchors are then used to create a tractable, linear approximation of the otherwise intractable quadratic Gromov-Wasserstein (GW) geometric cost. This approximated geometric cost is fused with the semantic cost, yielding a final fused cost matrix. This cost matrix is then used to solve an UOT problem, producing a transport plan $\pi^{(T)}$ that serves as our pseudo-label robust to geometric ambiguities.

Figure 3: Impact of the Gromov-Wasserstein term. Wasserstein baseline (blue) vs. Fused Gromov-Wasserstein (orange). Incorporating geometric structural consistency leads to consistent improvements (+2.3 $\sim$ +7.0%p) across categories.

Method																				avg ( $\uparrow$ )
ASIC	[ 8 ]	57.9	25.2	68.1	24.7	35.4	28.4	30.9	54.8	21.6	45.0	47.2	39.9	26.2	48.8	14.5	24.5	49.0	24.6	36.9
DINOv2	[ 22 ]	72.7	62.4	85.2	41.4	40.3	52.5	51.5	71.3	36.1	67.2	65.0	67.6	61.1	68.5	30.6	61.9	54.3	24.3	55.7
DIFT	[ 31 ]	63.5	54.5	80.8	34.5	46.2	52.7	48.3	77.7	39.0	76.0	54.9	61.3	53.3	46.0	57.8	57.1	71.1	63.4	57.7
DistillDIFT †	[ 7 ]	70.3	55.4	85.9	36.8	51.9	51.7	50.5	78.0	40.5	73.8	65.3	65.2	54.5	66.0	47.8	59.4	55.6	54.4	59.8
DINOv2 + SD †	[ 40 ]	72.9	63.4	86.4	40.5	52.6	55.4	53.3	78.4	45.2	77.1	64.7	69.4	62.9	68.5	56.8	67.0	65.9	51.8	63.5
Ours		73.5	66.7	89.9	40.8	58.0	55.4	51.1	84.8	52.1	81.0	71.0	75.3	64.1	71.8	62.1	70.2	70.2	64.4	67.9

Table 1: Per-category [email protected] scores (per-keypoint) on SPair-71k ( higher is better

\uparrow

). Best and second best are highlighted.

\dagger

denotes results re-evaluated under identical standard evaluation settings from [ 20 ] .

		SPair-71k			AP-10k ([email protected])
Models		0.1	0.05	0.01	I.S	C.S.	C.F.
DistillDIFT †	[ 7 ]	59.8	42.7	5.7	65.5	62.8	52.8
DINOv2 + SD †	[ 40 ]	63.5	48.3	8.8	65.5	63.3	51.1
Ours		67.9	50.8	10.0	68.0	65.8	52.9

Table 2: Results for different PCK levels (per-keypoint) on SPair-71k and AP-10k. Results for AP-10K are intra-species (I.S.), cross-species (C.S.) and cross-family (C.F.), following [ 39 ] .

Panel of Figure 4 — Figure 4: Qualitative comparison on challenging cases from SPair-71k. Four challenging scenarios: (a) geometric ambiguity from extreme viewpoint changes and occlusion; (b) extreme pose variation causing self-occlusion in non-rigid objects; (c) large scale changes with feature ambiguity in textureless regions; and (d) fine-grained semantic ambiguity from background clutter and intra-class variations. Green circles indicate correct matches ([email protected]), red circles indicate errors.

Figure 5: Ablation study on pseudo-label generation strategies on SPair-71k. Performance (PCK

{}_{\text{label}}

@0.1) on the Geometry-aware subset (left) and Overall set (right). (1) Nearest Neighbor, (2) Semantic OT (feature similarity only), (3) Fused OT (semantic and geometric costs with balanced OT), and (4) our Fused UOT.

Panel of Figure 6 — Figure 6: Qualitative comparison of pseudo-label generation methods. (a) Nearest Neighbor matching produces geometrically inconsistent correspondences with multiple errors ( red lines ). (b) Our method (SoY) generates globally consistent correspondences ( green lines ) by leveraging 3D geometric structure.

Intra-structure distance for Fused OT			PCK ${}_{\text{label}}$ @0.1
Baseline: Semantic OT (w/o GW term)			66.5
(1) w/ 2D distance			65.7
(2) w/ Semantic distance (intra-feature)			66.2
(3) w/ 3D geometric distance (ours)			67.6

Table 3: Ablation on the intra-structure for pseudo-label generation. We compare the 3D geometric structure (ours) against 2D based alternatives to justify the Fused OT strategy (Fig. 5 ).

Method configuration			[email protected]
Backbone (DINOv2 + SD) (zero-shot)			63.5
Adapter w/ NN labels ∗			64.6
+ relaxed c.c. ∗			64.8
Adapter w/ FGW labels			66.8
+ relaxed c.c.			67.1
+ soft target loss			67.5

Table 4: Ablation study of average [email protected] (per-keypoint) on SPair-71k. We compare the zero-shot backbone against adapters trained with (1) Nearest Neighbor (NN) and (2) our proposed FGW labels, along with incremental gains from (3) relaxed cycle consistency (c.c.) and (4) soft target loss. ( ∗ : baseline variants)

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Pseudo-label analysis with DINOv2 backbone. Performance (PCK

{}_{\text{label}}

@0.1) on the geometry-aware subset (left) and overall set (right). Using only DINOv2 features, we compare four pseudo-label generation strategies: Nearest Neighbor, Semantic OT, Fused OT, and Fused UOT. Both the geometry-aware and overall subsets show a consistent, monotonic improvement across methods, with clear gains from incorporating semantic OT and our geometry-aware matching. This confirms that our pseudo-labeling remains effective even with a weaker backbone.

Figure 2: Pseudo-label hyperparameter ablation on SPair-71k (PCK

{}_{\text{label}}

@0.1). We study three key hyperparameters of our pseudo-label generator: (Left) number of anchors

K

, (Middle) feature–geometry trade-off

\alpha

, and (Right) KL regularization strength

\rho

in UOT. In all cases, the performance is measured as PCK

{}_{\text{label}}

@0.1 on the SPair-71k test set.

3D backbone	PCK ${}_{\text{label}}$ @0.1
VGGT [ 33 ]	68.1
DUSt3R [ 35 ]	67.1

Appendix

Table 2: Effect of the 3D foundation backbone on pseudo-label quality on SPair-71k. We report PCK

{}_{\text{label}}

@0.1 for our pseudo-labels when using either VGGT or DUSt3R to obtain 3D structure.

Panel of Figure 3 — Figure 3: Failure modes of pseudo-label generation. Top: failure when the object breaks into disconnected parts. Bottom: failure when the features used for pseudo labels are highly noisy and all correspondences collapse to a local region.

Panel of Figure 5 — Figure 5: Visual comparison of semantic correspondences on SPair-71k [ 20 ] across DistillDIFT [ 7 ] , DINOv2+SD [ 40 ] , and our approach. Correct and incorrect matches are indicated by green lines and red lines , respectively.

Semantic Correspondence

Semantic correspondence matches equivalent points or parts across images and shapes despite changes in appearance, pose, or viewpoint.

See all 24 papers

May 28, 2026

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.

Artur Jesslen, Olaf Dünkel, Adam Kortylewski

University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany

Semantic Correspondence

Ask PDF

Oct 23, 2026

SemGeo-Gen: Unsupervised Generation of Approximate Cross-Instance Semantic-Geometric Correspondences

We introduce SemGeo-Gen, an unsupervised method for generating approximate cross-instance semantic-geometric supervision from weakly aligned 3D object collections. Our goal is not to recover exact point-level correspondences, which are often ambiguous across different instances and would require expensive, impractical manual supervision at scale. Instead, we automatically produce approximate correspondences that are semantically meaningful, geometrically consistent, and diverse enough to train modern correspondence models. Given only coarse category-level rotation alignment, SemGeo-Gen lifts multi-view DINOv2 features to 3D, aligns object instances using continuous piecewise-affine registration, and prunes candidate matches using semantic consistency. The resulting 3D correspondences can be projected into rendered views, yielding scalable 3D-3D, 3D-2D, and 2D-2D supervision without manual annotation. We validate the generated correspondences against sparse human annotations from KeypointNet, obtaining 93% [email protected] and 97% [email protected]. More importantly, we show that using the generated correspondences for synthetic pretraining improves recent state-of-the-art semantic correspondence models on SPair-71k after fine-tuning on real data, with gains of up to 5 [email protected] points. Together, these results indicate that approximate semantic-geometric supervision generated from 3D assets can improve real-image correspondence learning and serve as a scalable alternative to costly manual annotation. Project page: https://bgu-cs-vil.github.io/SemGeo-Gen/.

Roy Amoyal, Shira Ifergane, Oren Freifeld

The Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel

Semantic Correspondence

Ask PDF

Apr 20, 2026

MARCO: Navigating the Unseen Space of Semantic Correspondence

Recent advances in semantic correspondence rely on dual-encoder architectures, combining DINOv2 with diffusion backbones. While accurate, these billion-parameter models generalize poorly beyond training keypoints, revealing a gap between benchmark performance and real-world usability, where queried points rarely match those seen during training. Building upon DINOv2, we introduce MARCO, a unified model for generalizable correspondence driven by a novel training framework that enhances both fine-grained localization and semantic generalization. By coupling a coarse-to-fine objective that refines spatial precision with a self-distillation framework, which expands sparse supervision beyond annotated regions, our approach transforms a handful of keypoints into dense, semantically coherent correspondences. MARCO sets a new state of the art on SPair-71k, AP-10K, and PF-PASCAL, with gains that amplify at fine-grained localization thresholds (+8.9 [email protected]), strongest generalization to unseen keypoints (+5.1, SPair-U) and categories (+4.7, MP-100), while remaining 3x smaller and 10x faster than diffusion-based approaches. Code is available at https://github.com/visinf/MARCO .

Claudia Cuttano, Gabriele Trivigno, Carlo Masone +1

Politecnico di Torino · TU Darmstadt · ELIZA

Semantic Correspondence

Ask PDF

Abstract

Semantic Correspondence

Explore similar work

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

SemGeo-Gen: Unsupervised Generation of Approximate Cross-Instance Semantic-Geometric Correspondences

MARCO: Navigating the Unseen Space of Semantic Correspondence