Point clouds represent 3D object geometry as sets of spatial coordinates. A fundamental task in their analysis is keypoint detection, i.e., identifying a sparse subset of points that captures the most salient features. While supervised methods rely on manual annotations, unsupervised approaches leverage intrinsic geometric properties to localize keypoints without human input. We introduce GEKey, a self-supervised framework that uses the geodesic eccentricity transform, an SE(3)-invariant geometric measure, to identify instance-specific salient points. These points serve as pseudo-labels to train a model that aligns its predictions with the object's intrinsic structure. GEKey is simple, efficient, and robust, preserving intra-class variation by balancing instance-specific and category-level structure. Extensive experiments on ShapeNetCoreV2, KeypointNet, CAPE, ITOP, and DeformingThings4D show state-of-the-art performance in both keypoint detection and alignment, with strong resilience to noise, downsampling, deformations, and real-world conditions. The code and pretrained models are available at https://github.com/Pierreoo/GEKey.
Figures & tables
Figure 2: The geometric analysis of GEKey-PL, outlining the process from geodesic eccentricity to the derivation of pseudo-labels.
Figure 3: Overview of self-supervised GEKey to generalize keypoint prediction. The blue workflow shows a backbone model predicting keypoints Y^ , which are compared, via the bipartite matching loss, to pseudo-labels Y∗ generated by the green pipeline.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
ULCS [ 8 ]
71.0
67.0
75.4
57.5
71.3
68.5
50.1
64.1
52.1
78.4
47.4
60.1
76.9
64.6
Skeleton Merger [ 28 ]
72.1
71.9
72.8
59.5
72.0
69.7
69.3
72.4
92.0
95.3
65.9
79.1
94.2
75.9
SC3K [ 50 ]
87.2
80.0
77.4
56.3
76.1
56.7
96.5
55.0
98.3
85.0
46.3
79.2
92.9
75.9
GEKey
98.5
87.0
61.3
65.0
92.7
90.5
99.7
51.3
99.8
98.2
90.9
91.2
96.3
86.3
Table 1: Keypoint inclusivity on KeypointNet, comparing unsupervised learning-based methods.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
FPS [ 27 ]
97.1
98.3
97.1
93.5
94.0
97.1
97.8
89.7
98.3
94.2
95.2
98.0
97.1
95.9
ISS [ 48 ]
77.0
95.6
96.0
91.2
92.4
93.7
87.6
88.0
85.2
89.1
93.4
95.4
89.9
90.3
GEKey-PL
99.2
98.7
96.7
93.6
95.5
97.6
98.7
88.6
99.6
94.8
96.0
98.5
98.2
96.6
ULCS [ 8 ]
88.6
94.2
80.9
60.8
83.7
83.9
79.8
79.9
76.8
78.9
89.6
83.0
78.8
81.5
Skeleton Merger [ 28 ]
92.6
84.3
91.4
85.0
90.7
85.9
85.7
82.1
77.4
86.1
83.2
91.3
85.3
86.2
SC3K [ 50 ]
96.3
98.2
98.0
94.6
89.8
95.3
97.6
90.5
98.8
94.3
95.2
97.4
97.2
95.6
Table 2: Keypoint coverage on KeypointNet, comparing hand-crafted (top) detectors and unsupervised learning-based (bottom) methods.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
FPS [ 27 ]
67.6
27.9
38.5
17.4
40.4
41.1
64.6
22.9
45.2
37.1
23.0
38.0
48.5
39.4
ISS [ 48 ]
64.0
29.4
38.8
14.3
39.3
38.8
63.7
20.8
38.0
41.2
22.6
32.4
49.7
37.9
GEKey-PL
80.5
45.2
39.6
27.4
31.8
57.1
67.1
23.7
50.0
46.8
26.2
62.6
49.6
46.7
KeypointDeformer [ 13 ]
53.3
40.7
34.0
9.3
53.8
62.0
52.6
13.5
38.1
45.7
12.6
62.0
51.2
40.7
Skeleton Merger [ 28 ]
73.5
59.9
36.5
15.5
53.3
60.0
49.7
16.7
45.0
39.7
23.6
60.9
58.9
45.6
SC3K [ 50 ]
74.9
19.9
38.0
11.3
33.2
39.8
68.0
17.6
47.9
39.7
18.4
46.4
47.5
38.7
Table 3: Keypoint detection on KeypointNet using mIoU metric, comparing hand-crafted (top) detectors and unsupervised learning-based (bottom) methods.
Figure 4: Keypoint detection by GEKey for an example in each category, showing accurate localization at salient regions.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
KeypointDeformer [ 13 ]
73.4
69.5
60.3
56.7
80.2
84.3
62.9
55.3
59.8
61.5
69.8
72.6
73.9
67.7
Skeleton Merger [ 28 ]
77.8
61.3
59.9
53.1
79.5
76.8
63.2
57.0
60.4
57.8
67.2
70.0
72.4
65.9
SC3K [ 50 ]
82.9
64.9
62.7
59.7
75.2
87.0
65.7
58.6
63.0
59.4
75.3
76.0
76.0
69.7
Key-Grid [ 11 ]
84.7
70.6
66.2
58.5
84.5
93.4
69.1
62.5
60.9
63.7
79.4
78.6
74.6
72.8
GEKey
87.6
72.8
59.6
63.5
90.3
97.1
43.4
72.9
58.0
83.8
73.6
91.8
54.1
73.0
Table 4: Keypoint alignment on KeypointNet using DASSC3K metric, comparing unsupervised learning-based methods.
Figure 5: Keypoint alignment by GEKey, with keypoints mapped to consistent regions, including highly varied geometries.
Figure 6: Robustness of GEKey to noise (top) and downsampling (bottom) perturbations, visualized at noise scales of 0.10 and 0.20 and downsampling ratios of 16 and 32.
Dataset
Method
Inclusivity ↑
Coverage ↑
Tcon↑
CAPE [ 22 , 26 ]
ULCS [ 8 ]
47.8
67.6
88.1
SC3K [ 50 ]
79.3
86.5
70.7
SelfGeo [ 49 ]
85.9
91.9
90.4
GEKey
99.5
96.7
93.3
ITOP [ 10 ]
ULCS [ 8 ]
42.1
64.3
76.5
SC3K [ 50 ]
76.4
83.1
42.9
Table 5: Keypoint quality under non-rigid deformations on the CAPE, ITOP, and DeformingThings4D datasets.
Figure 7: Without temporal supervision in training, GEKey detects keypoints that remain stable under non-rigid human deformations in CAPE (top), and transfers to the real-world ITOP dataset without fine-tuning (bottom), despite self-occlusion and sensor noise.
Figure 8: Keypoints detected by GEKey on animal shapes from DeformingThings4D, obtained from geometry alone without temporal supervision. Placements are precise, stable across frames, and consistent across species.
Figure 9: Comparison of Lmatch and LSinkhorn for the airplane and chair categories.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
PointNet++ [ 27 ]
86.2
53.0
52.8
27.7
42.6
66.9
63.9
31.2
53.3
66.4
24.7
71.1
57.2
53.6
PTv1 [ 45 ]
84.4
48.3
46.8
29.6
46.1
65.7
69.0
31.1
52.0
60.6
25.6
69.4
55.5
52.6
PTv2 [ 38 ]
86.5
51.9
46.1
32.1
43.9
65.6
71.6
36.5
53.0
66.2
28.6
70.7
55.8
54.5
Table 6: Keypoint detection on KeypointNet using mIoU metric, comparing different backbone architectures with GEKey.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
PointNet++ [ 27 ]
83.7
70.8
52.2
62.4
92.8
94.0
48.1
79.6
59.8
82.2
55.2
91.6
53.5
71.2
PTv1 [ 45 ]
81.0
69.9
56.1
60.8
90.3
96.9
37.0
76.3
60.7
74.4
63.9
91.6
56.0
70.4
PTv2 [ 38 ]
87.6
72.8
59.6
63.5
90.3
97.1
43.4
72.9
58.0
83.8
73.6
91.8
54.1
73.0
Table 7: Keypoint alignment on KeypointNet using DASSC3K metric, comparing different backbone architectures with GEKey.
Method
Airplane
Bed
Bottle
Cap
Car
Chair
Guitar
Helmet
Knife
Motorcycle
Mug
Table
Vessel
Avg.
FPS [ 27 ]
13.2
10.9
2.8
1.0
2.7
7.4
7.1
2.4
10.7
6.2
2.3
10.7
6.7
6.5
ISS [ 48 ]
1.7
1.3
1.2
1.2
1.5
1.7
1.2
1.4
3.0
1.4
1.1
1.8
1.1
1.5
GEKey-PL
20.0
19.3
2.5
2.5
1.4
13.6
10.8
1.8
14.9
7.8
2.9
21.0
6.3
9.6
GEKey
15.0
12.4
1.0
1.3
0.2
8.7
9.7
1.1
10.9
5.4
1.3
12.0
3.7
6.4
Table 8: Keypoint detection using mIoU metric at a distance threshold of 0.01.
Figure 10: Per-category mIoU (%) curves of GEKey over Euclidean distance thresholds.
Figure 11: Evolution of mIoU-0.10, mIoU-0.01, and DASSC3K over training for the airplane and chair categories, illustrating the negative correlation between detection and alignment.
Airplane
Chair
DAS
mIoU
DAS
mIoU
Skeleton Merger [ 28 ]
77.8
73.5
76.8
60.0
GEKey
73.3
86.5
74.9
65.6
Table 9: Keypoint alignment on KeypointNet using DASSM metric.
Figure 12: Ground-truth keypoints color-coded by semantic ID, with instance-specific circled to highlight intra-class variability: not all instances share the same semantic label subset, constraining the DASSM metric when predictions adapt to per-shape geometry.
k -NN
ecc100(u)
ecc10(u)
ecc(u)−δ(ecc(u))
#Obj
TTot
TTot
mIoU
TTot
mIoU
TTot
Airplane
1022
7.6
79.3
80.5
9.1
80.3
11.6
Bed
146
1.0
11.7
45.2
1.4
42.3
2.4
Bottle
380
2.5
30.2
39.6
3.5
39.1
14.6
Cap
38
0.3
3.0
27.4
0.4
24.9
0.9
Car
1002
7.7
84.8
31.8
10.1
34.8
39.6
Table 10: Processing times (in seconds) for the k -NN graph, geodesic eccentricity, and pseudo-label computation stages of GEKey-PL.
Figure 13: Geodesic distances to the wingtip point on the Delaunay triangulation, alongside geodesic distances on the k -NN graph for comparison.
Figure 14: The local maxima of the geodesic eccentricity transform on the Delaunay triangulation and the k -NN graph.
10 -NN
20 -NN
30 -NN
40 -NN
Cov.
mIoU
Cov.
mIoU
Cov.
mIoU
Cov.
mIoU
Airplane
98.7
79.4
99.2
80.3
99.3
78.3
99.3
77.1
Bed
98.5
37.5
98.7
42.3
98.6
39.6
98.6
38.6
Bottle
96.4
44.4
97.0
39.1
97.1
38.0
97.2
37.5
Cap
93.2
22.0
93.9
24.9
95.0
23.6
95.4
25.1
Car
94.2
42.2
95.4
34.8
95.8
35.5
95.9
36.0
Table 11: Comparison of the k -parameter in the k -NN graph of GEKey-PL on pseudo-label Coverage (Cov., %) and mIoU (%), computed with ecc10(u) .
Figure 16: GEKey exhibits strong synthetic-to-real generalization on real-world examples from the OBJ-BG variant of ScanObjectNN, despite challenges such as background clutter, partial occlusions, sensor noise, and varying point density.
Method
Repeatability (%) ↑
Consistency ( ×10−2 ) ↓
FPS [ 27 ]
11.6
12.0
ISS [ 48 ]
9.0
11.0
GEKey-PL
32.3
4.4
Table 12: Repeatability and consistency error between SMPL-deformed shapes for ten keypoints.
Figure 15: GEKey-PL with stable keypoints across deformations of the SMPL human model.
With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90% [email protected] accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.
Changsheng Lu, Yuxin Chen, Haokun Gui +5
Hong Kong University of Science and Technology, Hong Kong, China · Australian National University, Canberra, Australia · Tencent Inc., China +1
Keypoint-based matching is a fundamental component of modern 3D vision systems, such as Structure-from-Motion (SfM) and SLAM. Most existing learning-based methods are trained on image pairs, a paradigm that fails to explicitly optimize for the long-term trackability of keypoints across sequences under challenging viewpoint and illumination changes. In this paper, we reframe keypoint detection as a sequential decision-making problem. We introduce TraqPoint, a novel, end-to-end Reinforcement Learning (RL) framework designed to optimize the Track-quality (Traq) of keypoints directly on image sequences. Our core innovation is a track-aware reward mechanism that jointly encourages the consistency and distinctiveness of keypoints across multiple views, guided by a policy gradient method. Extensive evaluations on sparse matching benchmarks, including relative pose estimation and 3D reconstruction, demonstrate that TraqPoint significantly outperforms some state-of-the-art (SOTA) keypoint detection and description methods.The code will be available at https://github.com/xiaomi-research/traqpoint.
Yepeng Liu, Hao Li, Liwen Yang +8
School of Computer Science, Wuhan University · Xiaomi EV
Robust local feature detection and description are foundational tasks in computer vision. Existing methods primarily rely on single appearance cues for modeling, leading to unstable keypoints and insufficient descriptor discriminability. In this paper, we propose a multi-cue guided local feature learning framework that leverages semantic and geometric cues to synergistically enhance detection robustness and descriptor discriminability. Specifically, we construct a joint semantic-normal prediction head and a depth stability prediction head atop a lightweight backbone. The former leverages a shared 3D vector field to deeply couple semantic and normal cues, thereby resolving optimization interference from heterogeneous inconsistencies. The latter quantifies the reliability of local regions from a geometric consistency perspective, providing deterministic guidance for robust keypoint selection. Based on these predictions, we introduce the Semantic-Depth Aware Keypoint (SDAK) mechanism for feature detection. By coupling semantic reliability with depth stability, SDAK reweights keypoint responses to suppress spurious features in unreliable regions. For descriptor construction, we design a Unified Triple-Cue Fusion (UTCF) module, which employs a semantic-scheduled gating mechanism to adaptively inject multi-attribute features, improving descriptor discriminability. Extensive experiments on four benchmarks validate the effectiveness of the proposed framework. The source code and pre-trained model will be available at: https://github.com/yiyscut/GESS.git.
Yang Yi, Xieyuanli Chen, Jinpu Zhang +2
Y. Yi, X. Chen, J. Zhang, H. Shen, and D. Hu are with the College of Intelligence Science and Technology, National University of Defense Technology, China.