cs.CVFeb 17, 2026
SaveRaCo: Ranking and Covariance for Practical Learned Keypoints
Organizations: ETH Zurich · Google · Microsoft Mixed Reality & AI Lab
Abstract
This paper introduces RaCo, a lightweight neural network designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks. The model integrates three key components: the repeatable keypoint detector, a differentiable ranker to maximize matches with a limited number of keypoints, and a covariance estimator to quantify spatial uncertainty in metric scale. Trained on perspective image crops only, RaCo operates without the need for covisible image pairs. It achieves strong rotational robustness through extensive data augmentation, even without the use of computationally expensive equivariant network architectures. The method is evaluated on several challenging datasets, where it demonstrates state-of-the-art performance in keypoint repeatability and two-view matching, particularly under large in-plane rotations. Ultimately, RaCo provides an effective and simple strategy to independently estimate keypoint ranking and metric covariance without additional labels, detecting interpretable and repeatable interest points. The code is available at https://github.com/cvg/RaCo.
Figures & tables
Figure 1: Practical interest point detection. RaCo detects repeatable and interpretable corners (left), learned from perspective image crops. A dedicated ranking head (middle) maximizes the downstream accuracy-speed trade-off by ranking matchable points higher. The estimated 2D metric covariances (right) describe the keypoints’ spatial uncertainty in pixels (colored by the angle of the first eigenvector, and whitened where the variance is large).
Figure 2: Overview. Our method consists of three branches: i) A detector head that produces a scoremap with repeatable keypoints, and ii) a covariance head that outputs the 2D spatial uncertainty in pixels, both sharing a lightweight backbone. The iii) ranker module outputs soft keypoint scores which maximize the repeatability at different keypoint budgets.
Figure 3: Keypoint ranking. Inconsistent keypoint ranking between images (left) results in excessive match filtering when the amount of keypoints per image is restricted ( small budget , middle). Our ranking module keeps repeatable points at the top of the list and yields similar ranks for corresponding points (right).
Figure 4: Covariance supervision. We train our covariance estimator by maximizing the log-likelihood of the reprojection error between corresponding keypoints. For corresponding keypoints in view and in view , the reprojection error is modeled as a zero-mean Gaussian: , where is the Jacobian of the homography evaluated at . The resulting covariance estimates (right) are strongly anisotropic (colored by the angle of the covariance’s first eigenvector) and are large in areas with low texture (illustrated by opacity).
| detector | matches @3px | rep. [%] | loc. [px] | AUC | |||
| 1px | 3px | 1px | 3px | ||||
| HPatches [ 4 ] | SIFT [ 38 ] | 282 | 24.8 | 44.0 | 0.99 | 32.7 | 68.8 |
| SuperPoint [ 15 ] | 503 | 22.1 | 54.0 | 1.19 | 41.5 | 74.7 | |
| DISK [ 63 ] | 484 | 23.9 | 49.9 | 1.05 | 27.5 | 63.7 | |
| ALIKED [ 72 ] | 530 | 32.8 | 56.3 | 0.96 | 34.1 | 69.5 | |
| DaD [ 20 ] | 510 | 30.2 | 55.2 | 1.01 | 40.0 | 73.5 | |
Table 1: Homography estimation. We evaluate the repeatability and correspondences of keypoints paired with ground-truth matches on HPatches [ 4 ] and DNIM [ 73 ] . Our method achieves competitive repeatability and pose estimation performance on HPatches, and exhibits superior robustness on image pairs from DNIM with large illumination changes. We color the best and second best results for each metric are colored.
| detector | matches @3px | rep. [%] | loc. [px] | AUC | |||
| 3px | 5px | 5° | 10° | ||||
| MD1800 [ 37 ] | SIFT [ 38 ] | 317 | 34.8 | 39.0 | 1.47 | 66.0 | 78.7 |
| SuperPoint [ 15 ] | 549 | 42.5 | 46.8 | 1.61 | 71.1 | 82.3 | |
| DISK [ 63 ] | 647 | 41.0 | 43.7 | 1.39 | 67.9 | 80.2 | |
| ALIKED [ 72 ] | 579 | 45.9 | 48.6 | 1.30 | 71.2 | 82.5 | |
| DaD [ 20 ] | 588 | 45.9 | 49.9 | 1.51 | 72.4 | 83.3 | |
Table 2: Relative pose estimation. We evaluate two-view matching performance and keypoint repeatability on two datasets: MegaDepth1800 [ 37 , 36 ] and ETH3D-Two-View [ 56 ] . The colors indicate the best and second best results for each metric. On both benchmarks, our method achieves the highest repeatability despite only being trained on homographies.
Figure 5: Rotation evaluation on HPatches [ 4 ] . We plot the repeatability@2px over the rotation angle between image pairs. SIFT [ 40 ] is more robust than any learned keypoint detectors, but our improved rotation augmentations result in state-of-the-art rotational robustness without requiring specialized model architectures.
| detector | repeatability AUC [ ] | run time [ms] | ||
| 1px | 2px | 3px | ||
| SIFT [ 40 ] | 52.3 | 69.6 | 71.9 | 34.8 |
| SuperPoint [ 15 ] | 25.3 | 57.7 | 67.7 | 0 6.8 |
| DISK [ 63 ] | 7.3 | 13.0 | 18.1 | 31.6 |
| ALIKED [ 72 ] | 8.4 | 12.8 | 16.7 | 0 5.1 |
| ALIKED w/ Aug. [ 72 ] | 24.9 | 44.8 | 49.7 | 0 5.1 |
Table 3: Rotation ablation and run time. We report the area under the repeatability curve up to different error thresholds under in-plane rotations up to 360°. The upper section compares our model against state-of-the-art keypoint detectors and the lower section ablates key design decisions like rotation augmentations and equivariant convolutions. We color the best and second best results for each metric. Our method significantly outperforms state-of-the-art models because of strong rotation augmentations, even without expensive equivariant convolutions.
Figure 6: Keypoint ranking evaluation. We evaluate the repeatability at 3px against the number of keypoints per image on HPatches [ 4 ] and MegaDepth1800 [ 37 ] (top). For both SuperPoint [ 15 ] and RaCo, our ranker module is more effective in ordering keypoints than the original detection scores. We also visualize the repeatable points as matches at a budget of 128 ( {\color[rgb]{0,1,0}\bullet} ) and 256 ( {\color[rgb]{0,0,1}\bullet} ) keypoints (bottom). SuperPoint+Ranker (right) almost doubles repeatable points here over SuperPoint [ 15 ] scores (bottom left).
Figure 7: 3D triangulation on ETH3D [ 56 ] . We plot triangulated point cloud accuracy against completeness (within 0.5cm). Each point represents the fraction of the original cloud retained after filtering by spatial uncertainty metrics. Our 2D covariances i) improve both accuracy and completeness even without filtering, and ii) yield superior 3D covariance estimates for more effective filtering.
Figure 8: Covariance Calibration. We plot observed error against the ground truth vs. predicted uncertainty . RaCo is the closest to the ideal calibration , shown as the dashed gray line.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Keypoint detector and covariance estimator model architecture. We use a multi-scale architecture with two heads, one which implements the detector and another which implements the covariance estimator. The network takes as input the RGB image I , normalizes it to and extracts multi-scale features , . The detector head outputs the globally normalized heatmap P . The covariance estimator is a lightweight head which outputs the Cholesky decomposition map L over the whole image.
Figure 10: Ranker model architecture. The ranker is a simple standalone model that comprised of residual blocks. It takes as input the normalized image and outputs the single channel ranker score map R .
Figure 11: Addtional rotation ablations on HPatches [ 4 ] . We plot the repeatability@2px over the rotation angle between image pairs just as in Fig. 5 . We ablate our method against our method with equivariant convolutions [ 67 ] , ours without rotation agumentations, and REKD [ 34 ] , a recent baseline which uses rotationally equivariant convolutions (ReCONVs). Adding equivariant convolutions only adds minor stability at large computational cost. The gap between our model and REKD is large, even without equivariant convolutions, suggesting that the proposed rotation augmentations are effective.
Figure 12: 3D triangulation on ETH3D [ 56 ] . We plot the triangulated point clouds’ accuracy against completeness within 1 cm and 2 cm. Each point corresponds to the fraction of the original point cloud retained after filtering based on different spatial uncertainty metrics.
| F1 Score (%) @ cm | |||
| SuperPoint [ 15 ] | 0.30 | 1.66 | 6.50 |
| ALIKED [ 72 ] | 0.29 | 1.58 | 6.01 |
| DaD [ 20 ] | 0.32 | 1.71 | 6.69 |
| Ours | 0.33 | 1.76 | 6.71 |
Table 4: F1 scores (%) at different thresholds for various methods. The best and second best results for each metric are colored.
Figure 13: RaCo’s outputs on HPatches and DNIM. (left) Keypoints overlayed on the input image (middle left) The detector probability score map (middle right) The ranker score map (right) The weighted map of the angle of the major axis of the estimated covariance ellipsoid described in Appendix D . Our detector learns corner like features that maximize the repeatability objective. The ranker learns an ordering of these keypoints, notice how more prominent corners are assigned a higher ranker score, less prominent corners are ranked lower, followed by edge like features. Textureless regions are assigned the lowest rank as they are the least likely to be matched.
Figure 14: RaCo’s outputs on MegaDepth and ETH3D. (left) Keypoints over the image (middle left) Detector score map (middle right) Ranker score map (right) The weighted map of the angle of the major axis of the estimated covariance ellipsoid described in Appendix D . The estimated covariances are very interpretable, in regions of low texture the rightmost plot is more white as the uncertainty in those regions is higher. On edge, corner and blob like image features, the uncertainty is much lower. Further, the angle of the major axis of the estimated ellipsoid follows the angle of the edge. This is clearly seen in the second row around the metallic helical like structure.
Figure 15: Qualitative examples of two-view matching. The repeatable points ( \color[rgb]{0.75,1,0}\bullet ), unmatched points ( \color[rgb]{1,0,0}\bullet ) and the non-covisible points ( \color[rgb]{0,0,1}\bullet ) are showed for examples from HPatches [ 4 ] and MegaDepth [ 36 ] from the evaluation in Sec. 4.2 . We additionally report the repeatability of each detector. Despite being trained solely on image pairs of perspective image crops, our model generalizes to images with real viewpoint and illumination changes.
Figure 16: Qualitative Analysis of Keypoint Ranking. We provide a qualitative evaluation of our keypoint ranking method by visualizing keypoints and their matches for three examples from HPatches [ 4 ] from the evaluation in Sec. 4.4 . With a fixed budget of 256 keypoints per view, ordering by our ranker scores ( ) significantly increases the number of matches compared to ordering by detector scores ( ). This demonstrates the ranker’s ability to prioritize matchable keypoints ( \color[rgb]{0.75,1,0}\bullet at a budget of 256) while de-prioritizing unmatchable ones ( \color[rgb]{0,0,0}\bullet ). The keypoints that would be matched had the full budget been considered are also shown ( \color[rgb]{0.5,0.5,0.5}\bullet ). We also report repeatability scores at keypoint budgets of 128 and 256.