RaCo: Ranking and Covariance for Practical Learned Keypoints

Abhiram Shenoi; Philipp Lindenberger; Paul-Edouard Sarlin; Marc Pollefeys

cs.CVFeb 17, 2026

RaCo: Ranking and Covariance for Practical Learned Keypoints

Authors: Abhiram Shenoi, Philipp Lindenberger, Paul-Edouard Sarlin, Marc Pollefeys

Organizations: ETH Zurich · Google · Microsoft Mixed Reality & AI Lab

Abstract

This paper introduces RaCo, a lightweight neural network designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks. The model integrates three key components: the repeatable keypoint detector, a differentiable ranker to maximize matches with a limited number of keypoints, and a covariance estimator to quantify spatial uncertainty in metric scale. Trained on perspective image crops only, RaCo operates without the need for covisible image pairs. It achieves strong rotational robustness through extensive data augmentation, even without the use of computationally expensive equivariant network architectures. The method is evaluated on several challenging datasets, where it demonstrates state-of-the-art performance in keypoint repeatability and two-view matching, particularly under large in-plane rotations. Ultimately, RaCo provides an effective and simple strategy to independently estimate keypoint ranking and metric covariance without additional labels, detecting interpretable and repeatable interest points. The code is available at https://github.com/cvg/RaCo.

Figures & tables

Figure 1: Practical interest point detection. RaCo detects repeatable and interpretable corners (left), learned from perspective image crops. A dedicated ranking head (middle) maximizes the downstream accuracy-speed trade-off by ranking matchable points higher. The estimated 2D metric covariances (right) describe the keypoints’ spatial uncertainty in pixels (colored by the angle of the first eigenvector, and whitened where the variance is large).

Figure 2: Overview. Our method consists of three branches: i) A detector head that produces a scoremap with repeatable keypoints, and ii) a covariance head that outputs the 2D spatial uncertainty in pixels, both sharing a lightweight backbone. The iii) ranker module outputs soft keypoint scores which maximize the repeatability at different keypoint budgets.

Figure 3: Keypoint ranking. Inconsistent keypoint ranking between images (left) results in excessive match filtering when the amount of keypoints per image is restricted ( small budget , middle). Our ranking module keeps repeatable points at the top of the list and yields similar ranks for corresponding points (right).

Figure 4: Covariance supervision. We train our covariance estimator by maximizing the log-likelihood of the reprojection error between corresponding keypoints. For corresponding keypoints

\mathbf{x}^{i}_{A}

in view

A

and

\mathbf{x}^{i}_{B}

in view

B

, the reprojection error

\mathbf{e}^{i}_{B\rightarrow A}=\mathbf{H}_{B\rightarrow A}(\mathbf{x}^{i}_{B})-\mathbf{x}^{i}_{A}

is modeled as a zero-mean Gaussian:

\mathbf{e}^{i}_{B\rightarrow A}\sim\mathcal{N}(\mathbf{0},\bm{\Sigma}^{i}_{A}+\mathbf{J}^{i}_{B\rightarrow A}\bm{\Sigma}^{i}_{B}(\mathbf{J}^{i}_{B\rightarrow A})^{\top})

, where

\mathbf{J}^{i}_{B\rightarrow A}

is the Jacobian of the homography evaluated at

\mathbf{x}^{i}_{B}

. The resulting covariance estimates (right) are strongly anisotropic (colored by the angle of the covariance’s first eigenvector) and are large in areas with low texture (illustrated by opacity).

	detector	$\#$ matches @3px	rep. [%]		loc. [px]	AUC $\mathbf{H}$
	detector	$\#$ matches @3px	1px	3px	loc. [px]	1px	3px
HPatches [ 4 ]	SIFT [ 38 ]	282	24.8	44.0	0.99	32.7	68.8
	SuperPoint [ 15 ]	503	22.1	54.0	1.19	41.5	74.7
	DISK [ 63 ]	484	23.9	49.9	1.05	27.5	63.7
	ALIKED [ 72 ]	530	32.8	56.3	0.96	34.1	69.5
	DaD [ 20 ]	510	30.2	55.2	1.01	40.0	73.5

Table 1: Homography estimation. We evaluate the repeatability and correspondences of keypoints paired with ground-truth matches on HPatches [ 4 ] and DNIM [ 73 ] . Our method achieves competitive repeatability and pose estimation performance on HPatches, and exhibits superior robustness on image pairs from DNIM with large illumination changes. We color the best and second best results for each metric are colored.

	detector	$\#$ matches @3px	rep. [%]		loc. [px]	AUC $\mathbf{T}$
	detector	$\#$ matches @3px	3px	5px	loc. [px]	5°	10°
MD1800 [ 37 ]	SIFT [ 38 ]	317	34.8	39.0	1.47	66.0	78.7
	SuperPoint [ 15 ]	549	42.5	46.8	1.61	71.1	82.3
	DISK [ 63 ]	647	41.0	43.7	1.39	67.9	80.2
	ALIKED [ 72 ]	579	45.9	48.6	1.30	71.2	82.5
	DaD [ 20 ]	588	45.9	49.9	1.51	72.4	83.3

Table 2: Relative pose estimation. We evaluate two-view matching performance and keypoint repeatability on two datasets: MegaDepth1800 [ 37 , 36 ] and ETH3D-Two-View [ 56 ] . The colors indicate the best and second best results for each metric. On both benchmarks, our method achieves the highest repeatability despite only being trained on homographies.

Figure 5: Rotation evaluation on HPatches [ 4 ] . We plot the repeatability@2px over the rotation angle between image pairs. SIFT [ 40 ] is more robust than any learned keypoint detectors, but our improved rotation augmentations result in state-of-the-art rotational robustness without requiring specialized model architectures.

detector	repeatability AUC [ $\%$ ]			run time [ms]
detector	1px	2px	3px	run time [ms]
SIFT [ 40 ]	52.3	69.6	71.9	34.8
SuperPoint [ 15 ]	25.3	57.7	67.7	0 6.8
DISK [ 63 ]	7.3	13.0	18.1	31.6
ALIKED [ 72 ]	8.4	12.8	16.7	0 5.1
ALIKED w/ Aug. [ 72 ]	24.9	44.8	49.7	0 5.1

Table 3: Rotation ablation and run time. We report the area under the repeatability curve up to different error thresholds under in-plane rotations up to 360°. The upper section compares our model against state-of-the-art keypoint detectors and the lower section ablates key design decisions like rotation augmentations and equivariant convolutions. We color the best and second best results for each metric. Our method significantly outperforms state-of-the-art models because of strong rotation augmentations, even without expensive equivariant convolutions.

Panel of Figure 6 — Figure 6: Keypoint ranking evaluation. We evaluate the repeatability at 3px against the number of keypoints per image on HPatches [ 4 ] and MegaDepth1800 [ 37 ] (top). For both SuperPoint [ 15 ] and RaCo, our ranker module is more effective in ordering keypoints than the original detection scores. We also visualize the repeatable points as matches at a budget of 128 ( ${\color[rgb]{0,1,0}\bullet}$ ) and 256 ( ${\color[rgb]{0,0,1}\bullet}$ ) keypoints (bottom). SuperPoint+Ranker (right) almost doubles repeatable points here over SuperPoint [ 15 ] scores (bottom left).

Figure 7: 3D triangulation on ETH3D [ 56 ] . We plot triangulated point cloud accuracy against completeness (within 0.5cm). Each point represents the fraction of the original cloud retained after filtering by spatial uncertainty metrics. Our 2D covariances i) improve both accuracy and completeness even without filtering, and ii) yield superior 3D covariance estimates for more effective filtering.

Figure 8: Covariance Calibration. We plot observed error against the ground truth vs. predicted uncertainty

\sqrt{\text{Tr}(\Sigma)}

. RaCo is the closest to the ideal calibration

\beta{=}1

, shown as the dashed gray line.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Keypoint detector and covariance estimator model architecture. We use a multi-scale architecture with two heads, one which implements the detector and another which implements the covariance estimator. The network takes as input the RGB image I , normalizes it to

I_{\text{norm}}

and extracts multi-scale features

\textbf{{F}}_{i}

i\in\{1,2,3,4\}

. The detector head outputs the globally normalized heatmap P . The covariance estimator is a lightweight head which outputs the Cholesky decomposition map L over the whole image.

Figure 10: Ranker model architecture. The ranker is a simple standalone model that comprised of residual blocks. It takes as input the normalized image and outputs the single channel ranker score map R .

Figure 11: Addtional rotation ablations on HPatches [ 4 ] . We plot the repeatability@2px over the rotation angle between image pairs just as in Fig. 5 . We ablate our method against our method with equivariant convolutions [ 67 ] , ours without rotation agumentations, and REKD [ 34 ] , a recent baseline which uses rotationally equivariant convolutions (ReCONVs). Adding equivariant convolutions only adds minor stability at large computational cost. The gap between our model and REKD is large, even without equivariant convolutions, suggesting that the proposed rotation augmentations are effective.

Figure 12: 3D triangulation on ETH3D [ 56 ] . We plot the triangulated point clouds’ accuracy against completeness within 1 cm and 2 cm. Each point corresponds to the fraction of the original point cloud retained after filtering based on different spatial uncertainty metrics.

	F1 Score (%) @ $\tau$ cm
	$\tau=0.5$	$\tau=1.0$	$\tau=2.0$
SuperPoint [ 15 ]	0.30	1.66	6.50
ALIKED [ 72 ]	0.29	1.58	6.01
DaD [ 20 ]	0.32	1.71	6.69
Ours	0.33	1.76	6.71

Appendix

Table 4: F1 scores (%) at different thresholds for various methods. The best and second best results for each metric are colored.

Panel of Figure 13 — Figure 13: RaCo’s outputs on HPatches and DNIM. (left) Keypoints overlayed on the input image (middle left) The detector probability score map (middle right) The ranker score map (right) The weighted map of the angle of the major axis of the estimated covariance ellipsoid described in Appendix D . Our detector learns corner like features that maximize the repeatability objective. The ranker learns an ordering of these keypoints, notice how more prominent corners are assigned a higher ranker score, less prominent corners are ranked lower, followed by edge like features. Textureless regions are assigned the lowest rank as they are the least likely to be matched.

Panel of Figure 14 — Figure 14: RaCo’s outputs on MegaDepth and ETH3D. (left) Keypoints over the image (middle left) Detector score map (middle right) Ranker score map (right) The weighted map of the angle of the major axis of the estimated covariance ellipsoid described in Appendix D . The estimated covariances are very interpretable, in regions of low texture the rightmost plot is more white as the uncertainty in those regions is higher. On edge, corner and blob like image features, the uncertainty is much lower. Further, the angle of the major axis of the estimated ellipsoid follows the angle of the edge. This is clearly seen in the second row around the metallic helical like structure.

Panel of Figure 15 — Figure 15: Qualitative examples of two-view matching. The repeatable points ( $\color[rgb]{0.75,1,0}\bullet$ ), unmatched points ( $\color[rgb]{1,0,0}\bullet$ ) and the non-covisible points ( $\color[rgb]{0,0,1}\bullet$ ) are showed for examples from HPatches [ 4 ] and MegaDepth [ 36 ] from the evaluation in Sec. 4.2 . We additionally report the repeatability of each detector. Despite being trained solely on image pairs of perspective image crops, our model generalizes to images with real viewpoint and illumination changes.

Panel of Figure 16 — Figure 16: Qualitative Analysis of Keypoint Ranking. We provide a qualitative evaluation of our keypoint ranking method by visualizing keypoints and their matches for three examples from HPatches [ 4 ] from the evaluation in Sec. 4.4 . With a fixed budget of 256 keypoints per view, ordering by our ranker scores ( $\mathbf{r}$ ) significantly increases the number of matches compared to ordering by detector scores ( $\mathbf{p}$ ). This demonstrates the ranker’s ability to prioritize matchable keypoints ( $\color[rgb]{0.75,1,0}\bullet$ at a budget of 256) while de-prioritizing unmatchable ones ( $\color[rgb]{0,0,0}\bullet$ ). The keypoints that would be matched had the full budget been considered are also shown ( $\color[rgb]{0.5,0.5,0.5}\bullet$ ). We also report repeatability scores at keypoint budgets of 128 and 256.

Explore similar work

Feb 24, 2026

From Pairs to Sequences: Track-Aware Policy Gradients for Keypoint Detection

Keypoint-based matching is a fundamental component of modern 3D vision systems, such as Structure-from-Motion (SfM) and SLAM. Most existing learning-based methods are trained on image pairs, a paradigm that fails to explicitly optimize for the long-term trackability of keypoints across sequences under challenging viewpoint and illumination changes. In this paper, we reframe keypoint detection as a sequential decision-making problem. We introduce TraqPoint, a novel, end-to-end Reinforcement Learning (RL) framework designed to optimize the Track-quality (Traq) of keypoints directly on image sequences. Our core innovation is a track-aware reward mechanism that jointly encourages the consistency and distinctiveness of keypoints across multiple views, guided by a policy gradient method. Extensive evaluations on sparse matching benchmarks, including relative pose estimation and 3D reconstruction, demonstrate that TraqPoint significantly outperforms some state-of-the-art (SOTA) keypoint detection and description methods.The code will be available at https://github.com/xiaomi-research/traqpoint.

Yepeng Liu, Hao Li, Liwen Yang +8

School of Computer Science, Wuhan University · Xiaomi EV

Keypoint Detection

Ask PDF

Mar 5, 2026

From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications

Keypoint detection underpins many vision tasks, including pose estimation, viewpoint recovery, and 3D reconstruction, yet modern neural models remain vulnerable to small input perturbations. Despite its importance, formal robustness verification for keypoint detectors is largely unexplored due to high-dimensional inputs and continuous coordinate outputs. We propose the first coupled robustness verification framework for heatmap-based keypoint detectors that bounds the joint deviation across all keypoints, capturing their interdependencies and downstream task requirements. Unlike prior decoupled, classification-style approaches that verify each keypoint independently and yield conservative guarantees, our method verifies collective behavior. We formulate verification as a falsification problem using a mixed-integer linear program (MILP) that combines reachable heatmap sets with a polytope encoding joint deviation constraints. Infeasibility certifies robustness, while feasibility provides counterexamples, and we prove the method is sound: if it certifies the model as robust, then the keypoint detection model is guaranteed to be robust. Experiments show that our coupled approach achieves high verified rates and remains effective under strict error thresholds where decoupled methods fail.

Xusheng Luo, Changliu Liu

Carnegie Mellon University, Pittsburgh, PA, USA

Certified Robustness Keypoint Detection

Ask PDF

Aug 20, 2026

RIPE++: Reinforced Keypoint Learning from Positive Pairs Only

Sparse keypoint extraction and matching underpin core tasks in geometric computer vision, including structure-from-motion, visual SLAM, augmented reality, and medical image registration. Learning robust local feature representations, however, typically requires accurate camera poses or depth supervision, which are often unavailable in real-world settings. Reinforcement learning (RL) has recently emerged as a promising alternative, requiring only the information if two images show the same scene or not. However, existing RL formulations such as RIPE rely on coarse binary rewards and carefully constructed negative training pairs, limiting training stability and descriptor discriminability. In this paper, we revisit RL-based keypoint learning and propose a reward that fully exploits the geometric consistency signal, deriving both reward and penalty from a single positive pair without contrasting against negatives. This richer signal provides sufficient supervisory contrast to learn discriminative detectors and descriptors from positive image pairs alone, enabling representation learning under extremely limited supervision. Furthermore, we show that the same RL objective can be extended to the matching stage by adapting LightGlue, raising AUC@5 on MegaDepth1500 from 56.58 to 59.65 and enabling weakly-supervised training of the full sparse matching pipeline from image pairs with partial visual overlap. We validate our approach on established benchmarks, demonstrating competitive results compared to fully-supervised methods. We further show that the method can be even trained on low texture medical video sequences, where camera poses are usually unavailable and standard SfM pipelines often fail. Code and data are available at https://github.com/fraunhoferhhi/RIPEpp .

Johannes Künzel, Peter Eisert, Anna Hilsmann

Fraunhofer Heinrich-Hertz-Institute, HHI, Germany · Humboldt University Berlin, Germany

Representation Learning Reinforcement Learning Image Matching Keypoint Detection Self-Supervised Visual Representation Learning

Ask PDF