SemGeo-Gen: Unsupervised Generation of Approximate Cross-Instance Semantic-Geometric Correspondences
Organizations: The Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel
Abstract
We introduce SemGeo-Gen, an unsupervised method for generating approximate cross-instance semantic-geometric supervision from weakly aligned 3D object collections. Our goal is not to recover exact point-level correspondences, which are often ambiguous across different instances and would require expensive, impractical manual supervision at scale. Instead, we automatically produce approximate correspondences that are semantically meaningful, geometrically consistent, and diverse enough to train modern correspondence models. Given only coarse category-level rotation alignment, SemGeo-Gen lifts multi-view DINOv2 features to 3D, aligns object instances using continuous piecewise-affine registration, and prunes candidate matches using semantic consistency. The resulting 3D correspondences can be projected into rendered views, yielding scalable 3D-3D, 3D-2D, and 2D-2D supervision without manual annotation. We validate the generated correspondences against sparse human annotations from KeypointNet, obtaining 93% [email protected] and 97% [email protected]. More importantly, we show that using the generated correspondences for synthetic pretraining improves recent state-of-the-art semantic correspondence models on SPair-71k after fine-tuning on real data, with gains of up to 5 [email protected] points. Together, these results indicate that approximate semantic-geometric supervision generated from 3D assets can improve real-image correspondence learning and serve as a scalable alternative to costly manual annotation. Project page: https://bgu-cs-vil.github.io/SemGeo-Gen/.
Figures & tables
| Method | Training regime | aero | bike | boat | bus | car | chair | mbike | train | tv | Mean |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Zero-shot / weakly supervised methods | |||||||||||
| SD [ 27 , 38 ] | Zero-shot | 62.8 | 52.7 | 31.2 | 39.1 | 35.6 | 32.0 | 51.0 | 61.8 | 52.9 | 46.57 |
| DINOv2 [ 25 , 38 ] | Zero-shot | 73.4 | 60.2 | 43.2 | 46.7 | 45.1 | 33.4 | 60.7 | 54.2 | 23.9 | 48.98 |
| DINOv2+SD [ 38 ] | Zero-shot | 73.8 | 61.0 | 40.2 | 47.4 | 44.1 | 41.5 | 61.7 | 63.5 | 52.4 | 53.96 |
| SphericalMaps [ 21 ] | Weakly supervised (SPair-71k) | 76.2 | 60.1 | 46.5 | 74.9 | 68.0 | 45.1 | 69.1 | 73.9 | 58.1 | 63.54 |
| Supervised methods | |||||||||||
| Training regime | Rigid-9 | Non-rigid-7 | Sym.-2 | All-18 |
|---|---|---|---|---|
| Real only | 80.48 | 86.94 | 72.35 | 82.09 |
| DenseMatcher [ 40 ] P.T. Real | 80.37 | 86.54 | 71.97 | 81.83 |
| Ours P.T. Real | 84.20 | 88.23 | 72.58 | 84.48 |
| (+3.72) | (+1.29) | (+0.23) | (+2.39) |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | @0.05 | @0.10 | @0.15 | @0.20 |
|---|---|---|---|---|
| Back to 3D (1-shot) | 0.438 | 0.632 | 0.725 | 0.777 |
| Back to 3D (3-shot) | 0.436 | 0.641 | 0.740 | 0.799 |
| SemGeo-Gen | 0.648 | 0.865 | 0.945 | 0.977 |
| Method | DOF | Time (s/pair) | [email protected] | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|---|---|
| Affine | 12 | 1.15 | 0.221 | 0.686 | 0.898 | 0.960 | 0.982 |
| TPS ( ) | 660 | 3.71 | 0.226 | 0.739 | 0.919 | 0.966 | 0.983 |
| CPA (ours) | 648 | 1.13 | 0.236 | 0.729 | 0.922 | 0.971 | 0.986 |
| Category | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| Airplane | 0.73 | 0.90 | 0.95 | 0.98 |
| Bathtub | 0.41 | 0.73 | 0.91 | 0.97 |
| Bed | 0.45 | 0.71 | 0.84 | 0.90 |
| Bottle | 0.65 | 0.83 | 0.92 | 0.98 |
| Car | 0.75 | 0.94 | 0.97 | 0.99 |
| Chair | 0.51 | 0.83 | 0.93 | 0.97 |
| Category | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| Airplane | 0.78 | 0.94 | 0.98 | 0.99 |
| Bathtub | 0.68 | 0.86 | 0.94 | 0.97 |
| Bed | 0.63 | 0.85 | 0.94 | 0.98 |
| Bottle | 0.80 | 0.95 | 0.97 | 0.99 |
| Car | 0.77 | 0.95 | 0.98 | 1.00 |
| Chair | 0.71 | 0.94 | 0.98 | 0.99 |
| [email protected] | [email protected] | [email protected] | [email protected] | Median GT-used rate | |
|---|---|---|---|---|---|
| 0.75 | 0.758 | 0.930 | 0.974 | 0.988 | 0.260 |
| 0.575 | 0.749 | 0.926 | 0.973 | 0.989 | 0.576 |
| 0.25 | 0.699 | 0.896 | 0.960 | 0.983 | 0.915 |
| PCK | Stat. | ||||||
|---|---|---|---|---|---|---|---|
| @0.05 | Mean | .75 | .71 | .65 | .32 | .10 | .04 |
| Median | .78 | .75 | .69 | .25 | .06 | .02 | |
| @0.10 | Mean | .93 | .89 | .85 | .55 | .20 | .08 |
| Median | .94 | .91 | .88 | .52 | .13 | .05 | |
| @0.15 | Mean | .98 | .94 | .90 | .66 | .26 | .12 |
| Median | .98 | .95 | .91 | .64 | .20 | .07 |
| Category | [email protected] | [email protected] | [email protected] | [email protected] | Coverage | 3D [email protected] |
|---|---|---|---|---|---|---|
| Airplane | 0.79 | 0.94 | 0.98 | 0.99 | 0.73 | 0.94 |
| Bathtub | 0.81 | 0.95 | 0.99 | 1.00 | 0.26 | 0.88 |
| Bed | 0.81 | 0.96 | 0.99 | 1.00 | 0.45 | 0.91 |
| Bottle | 0.81 | 0.95 | 0.98 | 0.99 | 0.40 | 0.96 |
| Cap | 0.83 | 0.97 | 1.00 | 1.00 | 0.30 | 0.97 |
| Car | 0.76 | 0.95 | 0.98 | 1.00 | 0.88 | 0.95 |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | nose of the airplane | |||
| 1 | cockpit of the airplane | |||
| 2 | tail of the airplane | |||
| 3 | wingtip of the left wing of the airplane | |||
| 4 | wing root of the left wing of the airplane | |||
| 5 | wingtip of the right wing of the airplane |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | top outer corner of the bathtub | |||
| 1 | top outer corner of the bathtub | |||
| 2 | top outer corner of the bathtub | |||
| 3 | top outer corner of the bathtub | |||
| 4 | bottom outer corner of the bathtub | |||
| 5 | bottom outer corner of the bathtub |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | lower end of the front leg of the bed | |||
| 1 | lower end of the headboard leg of the bed | |||
| 2 | lower end of the headboard leg of the bed | |||
| 3 | lower end of the front leg of the bed | |||
| 4 | foot corner of the mattress | |||
| 5 | head corner of the mattress |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | lower rim of the bottle | |||
| 1 | lower rim of the bottle | |||
| 2 | lower rim of the bottle | |||
| 3 | lower rim of the bottle | |||
| 4 | edge between the bottle body and the bottle neck | |||
| 5 | edge between the bottle body and the bottle neck |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | top button of the cap | |||
| 1 | front of the visor of the cap | |||
| 2 | corner of the visor of the cap | |||
| 3 | corner of the visor of the cap | |||
| 4 | center of the back of the cap | |||
| 5 | center of the visor attachment of the cap |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | front right wheel of the car | |||
| 1 | rear right wheel of the car | |||
| 2 | rear left wheel of the car | |||
| 3 | front left wheel of the car | |||
| 4 | corner of the front bumper of the car | |||
| 5 | corner of the rear bumper of the car |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | upper corner of the back of the chair | |||
| 1 | upper corner of the back of the chair | |||
| 2 | lower corner of the back of the chair | |||
| 3 | lower corner of the back of the chair | |||
| 4 | front corner of the seat of the chair | |||
| 5 | front corner of the seat of the chair |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | head of the guitar | |||
| 1 | connection of the body and neck of the guitar | |||
| 2 | corner of the body of the guitar | |||
| 3 | corner of the body of the guitar | |||
| 4 | corner of the body of the guitar | |||
| 5 | corner of the body of the guitar |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | top of the helmet | |||
| 1 | front of the helmet peak | |||
| 2 | back of the helmet peak | |||
| 3 | left side of the helmet peak | |||
| 4 | right side of the helmet peak | |||
| 5 | right bottom of the helmet |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | corner of the laptop body | |||
| 1 | hinge of the laptop | |||
| 2 | hint of the laptop | |||
| 3 | corner of the laptop body | |||
| 4 | corner of the laptop screen | |||
| 5 | corner of the laptop screen |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | headlight of the motorcycle | |||
| 1 | end of the handlebar of the motorcycle | |||
| 2 | end of the handlebar of the motorcycle | |||
| 3 | seat of the motorcycle | |||
| 4 | axle of the front wheel of the motorcycle | |||
| 5 | axle of the back wheel of the motorcycle |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | mug rim | |||
| 1 | mug rim | |||
| 2 | mug rim | |||
| 3 | mug rim | |||
| 4 | mug base rim | |||
| 5 | mug base rim |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | corner of the tail of the skateboard | |||
| 1 | corner of the tail of the skateboard | |||
| 2 | corner of the nose of the skateboard | |||
| 3 | corner of the nose of the skateboard | |||
| 4 | wheel of the skateboard | |||
| 5 | wheel of the skateboard |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | corner of the table top | |||
| 1 | corner of the table top | |||
| 2 | corner of the table top | |||
| 3 | corner of the table top | |||
| 4 | end of the table leg | |||
| 5 | end of the table leg |
| ID | Description | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 0 | bow of a boat | |||
| 1 | lower part of the end of the bow of a boat (starboard) | |||
| 2 | upper part of the end of the bow of a boat (starboard) | |||
| 3 | upper part of the end of the bow of a boat (port) | |||
| 4 | lower part of the end of the bow of a boat (port) | |||
| 5 | lower corner of the boat stern (starboard |
| Category | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| airplane | 0.80 | 0.94 | 0.98 | 0.99 |
| bathtub | 0.72 | 0.89 | 0.96 | 0.99 |
| bed | 0.69 | 0.91 | 0.97 | 0.99 |
| bottle | 0.82 | 0.96 | 0.99 | 0.99 |
| cap | 0.80 | 0.97 | 1.00 | 1.00 |
| car | 0.78 | 0.96 | 0.99 | 1.00 |
| Category | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| airplane | 0.79 | 0.94 | 0.98 | 0.99 |
| bathtub | 0.70 | 0.88 | 0.96 | 0.99 |
| bed | 0.68 | 0.91 | 0.96 | 0.99 |
| bottle | 0.83 | 0.96 | 0.98 | 0.99 |
| cap | 0.80 | 0.97 | 1.00 | 1.00 |
| car | 0.78 | 0.95 | 0.99 | 1.00 |
| Category | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| airplane | 0.78 | 0.94 | 0.98 | 0.99 |
| bathtub | 0.68 | 0.87 | 0.96 | 0.99 |
| bed | 0.67 | 0.90 | 0.96 | 0.98 |
| bottle | 0.82 | 0.95 | 0.98 | 0.99 |
| cap | 0.79 | 0.97 | 1.00 | 1.00 |
| car | 0.78 | 0.95 | 0.98 | 0.99 |
| Category | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| airplane | 0.76 | 0.92 | 0.97 | 0.99 |
| bathtub | 0.53 | 0.79 | 0.93 | 0.98 |
| bed | 0.62 | 0.86 | 0.93 | 0.97 |
| bottle | 0.75 | 0.91 | 0.97 | 0.99 |
| cap | 0.79 | 0.95 | 0.99 | 1.00 |
| car | 0.77 | 0.95 | 0.98 | 0.99 |
Semantic Correspondence
Semantic correspondence matches equivalent points or parts across images and shapes despite changes in appearance, pose, or viewpoint.
See all 24 papers