Towards Complementary Keypoint Detection via Mixture of Detectors

Xiaoyong Lu; Guobao Xiao; Dong Liang; Songlin Du

cs.CVOct 12, 2026

Towards Complementary Keypoint Detection via Mixture of Detectors

Authors: Xiaoyong Lu, Guobao Xiao, Dong Liang, Songlin Du

Organizations: Southeast University · Tongji University · Nanjing University of Aeronautics and Astronautics

Abstract

Keypoint detection plays a central role in geometric vision pipelines. Existing detectors exhibit complementary strengths across different scene structures and keypoint budgets, and reliance on a single detector often results in suboptimal keypoint distributions. Motivated by this observation, we propose Mixture of Detectors (MoD) for dense matching pipelines, a unified framework for complementary keypoint detection through the adaptive composition of heterogeneous detectors. MoD first distills heterogeneous detectors into a unified model, allowing diverse detection maps to be predicted efficiently within a single forward pass. Detector selection is subsequently formulated as a patch-wise, budget-conditioned decision process, in which a lightweight router assigns the most suitable detector to each local region. To avoid heuristic supervision, the router is trained with task-driven rewards derived from downstream matching quality, jointly capturing both keypoint quantity and precision. Extensive experiments on MegaDepth, ScanNet, and HPatches demonstrate that MoD consistently outperforms strong baselines across a wide range of keypoint budgets, demonstrating its effectiveness in improving keypoint distribution and downstream geometric performance.

Figures & tables

Figure 1: Different detectors exhibit complementary behaviors across varying image structures and keypoint budgets. MoD leverages this by first distilling heterogeneous detectors into a unified model for joint prediction (Stage 1), and then learning a reward-guided router that selects detectors based on local texture and the keypoint budget (Stage 2).

Figure 2: Overview of the proposed method. In stage 1, we distill multiple teacher detectors into student heads. In stage 2, we train a patch-level router with reward-guided supervision to select detectors based on local texture and keypoint budget.

Figure 3: Detector router conditioned on keypoint budget and local texture.

Figure 4: Learned biases of different detectors under varying keypoint budgets. The vertical axis denotes the budgets, and the horizontal axis corresponds to the detectors.

Method	MegaDepth					ScanNet					HPatches
Method	512	1024	2048	4096	8192	512	1024	2048	4096	8192	512	1024	2048	4096	8192
SP	64.6	65.4	65.7	65.4	64.3	28.4	28.7	28.6	28.9	28.6	58.6	59.4	59.1	59.4	58.9
MoD †	64.6	65.4	65.4	65.6	65.0	28.5	28.9	28.6	29.0	29.3	60.4	61.0	60.6	60.8	60.7
ALIKED	65.8	65.7	65.6	65.9	65.9	25.7	26.4	27.2	27.7	27.9	56.8	57.6	57.9	57.4	57.4
MoD †	66.3	66.6	66.4	66.7	65.3	26.3	27.1	27.3	27.8	28.1	59.9	59.9	59.7	59.6	59.6
DeDoDe	63.3	64.7	65.6	65.7	66.4	27.0	27.7	28.2	28.6	28.7	58.1	59.7	61.1	61.9	62.1

Table 1: Pose Estimation and Homography Estimation using RoMa as the matcher. Performance is measured as the AUC@5 ∘ of the pose error and the AUC@3 pixels of the end-point error. MoD † denotes the student detection head corresponding to the teacher detector.

Method	Homography Estimation AUC $\uparrow$
Method	512	1024	2048	4096	8192
w/o budget bias	59.2	61.0	61.4	61.7	61.7
w/o background	59.5	61.1	61.9	61.6	62.4
avg. det. map	59.9	60.3	60.8	61.0	61.2
max det. map	60.0	60.4	61.0	60.6	60.8
global routing	60.2	60.5	61.2	61.2	61.5

Table 2: Ablation Study on HPatches. Performance is measured by AUC@3 pixels of the end-point error.

Figure 5: Reward curves under different keypoint budgets with and without budget bias. MoD without budget bias exhibits conflicting detector preferences across budgets.

Figure 6: Reward distributions for different detection methods (best viewed when zoomed in).

Method	Param. (M)	Time (ms)
SP	0.9	16.3
ALIKED	0.4	17.2
DeDoDe	14.6	65.8
DaD	6.5	27.8
DeepLSD	8.6	45.7
MuGE	72.5	58.7

Table 3: Efficiency Analysis.

Figure 7: Distribution of predicted routing decisions (best viewed when zoomed in).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The detection probability maps predicted by the distilled detection head (best viewed when zoomed in).

Method	Homography Estimation AUC $\uparrow$
Method	512	1024	2048	4096	8192
DeepLSD	43.9	47.9	49.8	50.6	51.1
Canny	34.1	40.2	46.8	50.7	51.3
UAED	41.2	45.4	48.1	49.7	51.5
MuGE	42.7	46.1	49.3	50.5	51.9
SIFT	47.4	50.0	52.3	52.0	52.0

Appendix

Table 4: Homography Estimation on HPatches using RCM+ Lu et al. (2026) as the matcher. Performance is measured as the AUC@3 pixels of the end-point error.

Figure 9: Distribution of predicted routing decisions (best viewed when zoomed in).

Figure 10: Routing decisions under varying keypoint budgets (best viewed when zoomed in).

Figure 11: Failure cases of the router (best viewed when zoomed in).

Figure 12: Keypoint visualization (best viewed when zoomed in).

Explore similar work

GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training
Jul 2, 2026Yejun Zhang, Xinjue Wang, Zihan Wang +2Visual Place Recognition Camera Pose Estimation
Aalto University, Espoo, Finland · Tampere University, Tampere, Finland · University of Oulu, Oulu, Finland
From Pairs to Sequences: Track-Aware Policy Gradients for Keypoint Detection
Feb 24, 2026Yepeng Liu, Hao Li, Liwen Yang +8Keypoint Detection
School of Computer Science, Wuhan University · Xiaomi EV
From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications
Mar 5, 2026Xusheng Luo, Changliu LiuCertified Robustness Keypoint Detection
Carnegie Mellon University, Pittsburgh, PA, USA

Jul 2, 2026J/K move · Enter open · S save

GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training

Yejun Zhang, Xinjue Wang, Zihan Wang, Esa Rahtu+1

Aalto University, Espoo, Finland · Tampere University, Tampere, Finland · University of Oulu, Oulu, Finland

Visual Place Recognition Camera Pose Estimation Geometric Representation Learning Robot Localization

Open paper AskPDF arXiv

Descriptor-free visual localization eliminates high-dimensional descriptor storage, preserves scene privacy, and simplifies map maintenance, yet its accuracy still lags far behind descriptor-based pipelines. We identify this gap to insufficient geometric discriminability in geometry-only matching. Without visual appearance, current methods underutilize local geometry cues, lack the global context among keypoints, and overfit to a single keypoint detector. We further observe that descriptor-free matching naturally enables multi-detector training, as heterogeneous keypoints can be optimized in a shared geometry-only space without aligning descriptor spaces. Building on these insights, we propose GeoMix, a descriptor-free 2D-3D matching framework that strengthens geometric discriminability at three levels. Locally, directional and distance-aware embeddings enrich neighborhood aggregation with fine-grained spatial structure. Globally, learnable context nodes aggregate and redistribute scene-wide information via cross-attention to resolve ambiguities beyond local receptive fields. At the training level, Mix-Training exploits this detector-agnostic geometry space to learn representations across multiple keypoint detectors. Extensive experiments on MegaDepth, Cambridge Landmarks, 7Scenes, and Aachen Day-Night show that GeoMix sets a new state of the art among descriptor-free methods, reducing 75th-percentile rotation error by 89% and translation error by up to 90% over the previous best, while generalizing zero-shot to unseen detectors and narrowing the gap to descriptor-based pipelines. Code is available at this links.

Abstract

Explore similar work

GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training

From Pairs to Sequences: Track-Aware Policy Gradients for Keypoint Detection

From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications