We introduce SemGeo-Gen, an unsupervised method for generating approximate cross-instance semantic-geometric supervision from weakly aligned 3D object collections. Our goal is not to recover exact point-level correspondences, which are often ambiguous across different instances and would require expensive, impractical manual supervision at scale. Instead, we automatically produce approximate correspondences that are semantically meaningful, geometrically consistent, and diverse enough to train modern correspondence models. Given only coarse category-level rotation alignment, SemGeo-Gen lifts multi-view DINOv2 features to 3D, aligns object instances using continuous piecewise-affine registration, and prunes candidate matches using semantic consistency. The resulting 3D correspondences can be projected into rendered views, yielding scalable 3D-3D, 3D-2D, and 2D-2D supervision without manual annotation. We validate the generated correspondences against sparse human annotations from KeypointNet, obtaining 93% [email protected] and 97% [email protected]. More importantly, we show that using the generated correspondences for synthetic pretraining improves recent state-of-the-art semantic correspondence models on SPair-71k after fine-tuning on real data, with gains of up to 5 [email protected] points. Together, these results indicate that approximate semantic-geometric supervision generated from 3D assets can improve real-image correspondence learning and serve as a scalable alternative to costly manual annotation. Project page: https://bgu-cs-vil.github.io/SemGeo-Gen/.
Figures & tables
Figure 1: Approximate cross-instance semantic-geometric correspondences between two airplane instances across modalities. Left: 2D–2D correspondences between rendered views. Middle: 3D–2D correspondences between a 3D object and a rendered view. Right: 3D–3D correspondences between object surfaces. Colors in 3D visualize the lifted DINOv2 features reduced to three dimensions.
Figure 2: Overview of our correspondence generation pipeline.
Method
Training regime
aero
bike
boat
bus
car
chair
mbike
train
tv
Mean
Zero-shot / weakly supervised methods
SD [ 27 , 38 ]
Zero-shot
62.8
52.7
31.2
39.1
35.6
32.0
51.0
61.8
52.9
46.57
DINOv2 [ 25 , 38 ]
Zero-shot
73.4
60.2
43.2
46.7
45.1
33.4
60.7
54.2
23.9
48.98
DINOv2+SD [ 38 ]
Zero-shot
73.8
61.0
40.2
47.4
44.1
41.5
61.7
63.5
52.4
53.96
SphericalMaps [ 21 ]
Weakly supervised (SPair-71k)
76.2
60.1
46.5
74.9
68.0
45.1
69.1
73.9
58.1
63.54
Supervised methods
Table 1: Category-wise [email protected] results on selected SPair-71k categories with well-defined semantic-geometric correspondences. P.T. denotes pretraining.
Training regime
Rigid-9
Non-rigid-7
Sym.-2
All-18
Real only
80.48
86.94
72.35
82.09
DenseMatcher [ 40 ] P.T. → Real
80.37
86.54
71.97
81.83
Ours P.T. → Real
84.20
88.23
72.58
84.48
(+3.72)
(+1.29)
(+0.23)
(+2.39)
Table 2: Generalization across all SPair-71k categories. GeoAware-SC pretrained on Rigid-9 synthetic correspondences and fine-tuned on all 18 SPair-71k categories. [email protected] (%); improvements over real-only training are shown in parentheses.
Figure 5
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Qualitative SPair-71k [ 23 ] results for MARCO [ 5 ] . Each image shows a source–target pair with predicted correspondences. We compare MARCO trained on SPair-71k against MARCO pretrained on our generated correspondences and then fine-tuned on SPair-71k.
Figure 6: Qualitative SPair-71k [ 23 ] results for GeoAware-SC [ 39 ] . Each image shows a source–target pair with predicted correspondences. We compare GeoAware-SC trained on SPair-71k against GeoAware-SC pretrained on our generated correspondences and then fine-tuned on SPair-71k.
Figure 7: Qualitative correspondence examples on categories beyond existing correspondence benchmarks. Colors indicate sampled semantic-geometric correspondences established by our method.
Figure 8: Qualitative correspondence results on KeypointNet [ 35 ] We compare our predicted correspondences with human-annotated keypoints on the target object. Predicted correspondences are shown in lighter colors, while human annotations are shown in darker colors. Red lines connect each prediction to its corresponding ground-truth annotation, visualizing the localization difference between them.
Figure 9: KeypointNet [ 35 ] examples with strong annotation agreement. Predicted correspondences are shown in lighter colors, while human annotations are shown in darker colors. The strong overlap between the two indicates that our pipeline recovers correspondences that are highly consistent with human semantic annotations.
Method
@0.05
@0.10
@0.15
@0.20
Back to 3D (1-shot)
0.438
0.632
0.725
0.777
Back to 3D (3-shot)
0.436
0.641
0.740
0.799
SemGeo-Gen
0.648
0.865
0.945
0.977
Appendix
Table 3: Comparison with Back to 3D [ 33 ] on KeypointNet [ 35 ] Mean PCK over 16 categories (100 pairs per category). Back to 3D uses few-shot keypoint supervision; SemGeo-Gen requires none.
Table 4: Registration ablation on KeypointNet. Mean PCK across all 16 categories. PCK thresholds define the allowed normalized distance from the ground-truth keypoint, measured as a fraction of the target bounding-box diagonal. DOF denotes the number of learnable parameters. Time reports the mean registration time per pair on a single A6000 GPU. Best results are shown in bold .
Table 5: Evaluation of annotation quality on KeypointNet correspondences without filters. We report PCK at normalized distance thresholds of 5% , 10% , 15% , and 20% of the object scale, computed per category. (on 100 objects)
Table 7: Ablation on the correspondence-level semantic threshold (with pair-level filtering). Results are averaged over three random seeds and 16 KeypointNet categories. Median GT-used rate denotes the median fraction across categories of ground-truth keypoints that remain evaluable after correspondence-level pruning.
PCK
Stat.
0∘
5∘
10∘
30∘
60∘
90∘
@0.05
Mean
.75
.71
.65
.32
.10
.04
Median
.78
.75
.69
.25
.06
.02
@0.10
Mean
.93
.89
.85
.55
.20
.08
Median
.94
.91
.88
.52
.13
.05
@0.15
Mean
.98
.94
.90
.66
.26
.12
Median
.98
.95
.91
.64
.20
.07
Appendix
Table 8: Sensitivity to rotation misalignment. Mean and median KeypointNet PCK under random-axis rotations applied to either the source or target.
Figure 10: Qualitative analysis of DINOv2 based semantic coverage. Each row shows the original rendering, a PCA visualization of DINOv2 features computed jointly for the object pair, and the resulting semantic coverage mask. Green points have at least one semantically similar point in the paired object, while red points have none.
Figure 11: Sampled correspondences after semantic pruning. Each row shows a different view of the same object pair.
Table 9: 3D–2D correspondence evaluation on KeypointNet. PCK on found correspondences with visible ground-truth keypoints and visible predicted target points. Coverage denotes the fraction of visible ground-truth keypoints for which a valid correspondence is found. The last column reports the original 3D [email protected] over all found keypoints.
Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.
Artur Jesslen, Olaf Dünkel, Adam Kortylewski
University of Freiburg, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · CISPA Helmholtz Center for Information Security, Germany
Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.
Jiin Im, Sisung Liu, Je Hyeong Hong
Dept. Electronic Engineering, Hanyang University · Dept. Artificial Intelligence, Hanyang University
Previous works that leverage video models for image-to-3D scene generation often suffer from geometric distortions and blurry content. Using video generation models to implicitly maintain geometric consistency according to a single-frame input is ineffective. In this paper, we present a two-stage method, named GeoWorld, that renovates the image-to-3D scene generation pipeline by providing full-frame geometry features. The first-stage video generation model, followed by a multi-view geometry model, produces full-frame geometry features, which are then used as a mental draft of geometric conditions to aid the second-stage video-generation model. A geometric loss is proposed to impose real-world geometric constraints, and a geometry adaptation module is introduced to ensure the effective utilization of geometry features. Thanks to full-frame geometric modeling, the two smaller video models in our two-stage method can generate higher-fidelity 3D scenes than SOTA methods, while being even faster, e.g. 7.5× faster than Hunyuan-Voyager. Project page: https://peaes.github.io/GeoWorld.
Yuhao Wan, Lijuan Liu, Jingzhi Zhou +6
1VCIP & AAIS, Nankai University · 2ByteDance Inc. · 3Renmin University of China +2