From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications

Xusheng Luo; Changliu Liu

cs.CVMar 5, 2026

From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications

Authors: Xusheng Luo, Changliu Liu

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA

Abstract

Keypoint detection underpins many vision tasks, including pose estimation, viewpoint recovery, and 3D reconstruction, yet modern neural models remain vulnerable to small input perturbations. Despite its importance, formal robustness verification for keypoint detectors is largely unexplored due to high-dimensional inputs and continuous coordinate outputs. We propose the first coupled robustness verification framework for heatmap-based keypoint detectors that bounds the joint deviation across all keypoints, capturing their interdependencies and downstream task requirements. Unlike prior decoupled, classification-style approaches that verify each keypoint independently and yield conservative guarantees, our method verifies collective behavior. We formulate verification as a falsification problem using a mixed-integer linear program (MILP) that combines reachable heatmap sets with a polytope encoding joint deviation constraints. Infeasibility certifies robustness, while feasibility provides counterexamples, and we prove the method is sound: if it certifies the model as robust, then the keypoint detection model is guaranteed to be robust. Experiments show that our coupled approach achieves high verified rates and remains effective under strict error thresholds where decoupled methods fail.

Figures & tables

Figure 1: Overview of the keypoint detection pipeline and the proposed verification framework. A thick red dashed line divides keypoint detection (above) and verification (below). A seed image $\bbX_{0}$ where an airplane is parking at the airport is processed by the model to identify keypoints, which are marked as green dots. The verification framework takes as input the seed image $\bbX_{0}$ and a set of perturbed images $\bbX$ (with local perturbations indicated by red circles) that form the convex hull $\ccalX$ , along with the keypoint error bound $\delta\ccalV$ . By checking the feasibility of the MILP that is derived from the reachable set $\ccalZ$ of the model and the keypoint error bound $\delta\ccalV$ , the method returns whether the model is robust.

Figure 2: An illustration of dynamic indexing for the first keypoint: the green grid indicates the ground-truth keypoint location, while the red grid represents the perturbed keypoint position. The index deviation (1, -1) retrieves the value $Z_{31}$ .

m

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

100.0% (200/200)

99.5% (199/200)

47.5% (95/200)

100.0% (198/198)

99.0% (196/198)

0.0% (0/198)

100.0% (190/190)

71.6% (136/190)

0.0% (0/190)

88.7% (86/97)

10.3% (10/97)

0.0% (0/97)

100.0% (200/200)

99.0% (198/200)

52.5% (105/200)

100.0% (198/198)

0.0% (0/198)

98.4% (187/190)

72.1% (137/190)

0.0% (0/190)

89.7% (87/97)

8.2% (8/97)

0.0% (0/97)

100.0% (200/200)

99.0% (198/200)

53.0% (106/200)

100.0% (198/198)

99.0% (196/198)

0.0% (0/198)

99.4% (189/190)

67.4% (128/190)

0.0% (0/190)

88.7% (86/97)

5.1% (5/97)

0.0% (0/97)

100.0% (200/200)

97.0% (194/200)

53.0% (106/200)

100.0% (198/198)

98.0% (194/198)

0.0% (0/198)

99.5% (189/190)

62.1% (118/190)

0.0% (0/190)

90.7% (88/97)

7.2% (7/97)

0.0% (0/97)

Table 1: Statistical results on verified rate for non-overlapping local perturbations.

m

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

100.0% (196/196)

95.4% (187/196)

50.0% (98/196)

100.0% (195/195)

93.3% (182/195)

0.0% (0/195)

96.3% (183/190)

64.2% (122/190)

0.0% (0/190)

76.3% (74/97)

6.2% (6/97)

0.0% (0/97)

100.0% (192/192)

77.1% (148/192)

41.1% (79/192)

100.0% (190/190)

78.9% (150/190)

0.0% (0/190)

96.7% (175/181)

43.6% (79/181)

0.0% (0/181)

79.1% (72/91)

3.3% (3/91)

0.0% (0/91)

100.0% (157/157)

64.3% (101/157)

33.1% (52/157)

100.0% (157/157)

64.3% (101/157)

0.0% (0/157)

96.0% (142/148)

30.4% (45/148)

0.0% (0/148)

73.9% (51/69)

1.4% (1/69)

0.0% (0/69)

100.0% (126/126)

51.6% (65/126)

31.0% (39/126)

100.0% (126/126)

46.8% (59/126)

0.0% (0/126)

89.0% (105/115)

20.9% (24/115)

0.0% (0/115)

67.9% (36/53)

1.9% (1/53)

0.0% (0/53)

Table 2: Statistical results on verified rate for overlapping local perturbations.


$m$	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$m$	ours	baseline	ours	baseline	ours	baseline	ours	baseline
1	11.8 $\pm$ 7.4	7.6 $\pm$ 6.6	10.8 $\pm$ 5.2	13.0 $\pm$ 9.0	12.6 $\pm$ 16.1	20.4 $\pm$ 6.3	9.1 $\pm$ 12.4	26.4 $\pm$ 6.4
2	17.0 $\pm$ 15.2	12.4 $\pm$ 9.9	18.6 $\pm$ 29.5	19.2 $\pm$ 12.0	16.1 $\pm$ 12.4	26.7 $\pm$ 7.4	12.3 $\pm$ 16.4	33.0 $\pm$ 11.8
3	29.8 $\pm$ 71.0	17.2 $\pm$ 9.6	28.9 $\pm$ 79.2	22.5 $\pm$ 13.5	21.9 $\pm$ 17.9	29.8 $\pm$ 10.7	14.1 $\pm$ 18.1	35.6 $\pm$ 10.8
4	36.0 $\pm$ 73.6	21.5 $\pm$ 12.4	37.2 $\pm$ 77.4	28.0 $\pm$ 15.7	29.0 $\pm$ 23.5	34.7 $\pm$ 12.2	18.5 $\pm$ 23.3	39.2 $\pm$ 14.2

Table 3: Statistical results on verification time for non-overlapping local perturbations (seconds).


$\alpha$	overlapping	pruning	binary	integer	continuous	constraints

1.0	✗	✗	9.42 $\times 10^{4}\pm$ 0.00 $\times 10^{1}$	46.0 $\pm$ 0.0	1.89 $\times 10^{5}\pm$ 5.29 $\times 10^{2}$	6.61 $\times 10^{5}\pm$ 1.06 $\times 10^{3}$
	✗	✓	5.31 $\times 10^{1}\pm$ 4.29 $\times 10^{1}$	46.0 $\pm$ 0.0	8.36 $\times 10^{2}\pm$ 7.48 $\times 10^{2}$	1.95 $\times 10^{3}\pm$ 1.61 $\times 10^{3}$
	✓	✗	9.42 $\times 10^{4}\pm$ 0.00 $\times 10^{1}$	46.0 $\pm$ 0.0	1.92 $\times 10^{5}\pm$ 6.64 $\times 10^{3}$	6.67 $\times 10^{5}\pm$ 1.33 $\times 10^{4}$
✓	✓	8.18 $\times 10^{2}\pm$ 4.11 $\times 10^{3}$	46.0 $\pm$ 0.0	5.07 $\times 10^{3}\pm$ 1.42 $\times 10^{4}$	1.27 $\times 10^{4}\pm$ 4.06 $\times 10^{4}$

Table 4: Statistical results of the MILP size, including the number of binary, integer, and continuous variables, as well as the number of constraints. These results vary with the pose error threshold

\alpha

, the presence of object-airplane overlap, and the use of the pruning strategy. Results obtained with pruning are highlighted in gray for non-overlapping images and in orange for overlapping images.


$m$	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$m$	ours	baseline	ours	baseline	ours	baseline	ours	baseline
1	40.3 $\pm$ 95.9	16.3 $\pm$ 17.8	44.0 $\pm$ 94.0	20.9 $\pm$ 16.6	23.9 $\pm$ 39.8	29.3 $\pm$ 14.2	29.3 $\pm$ 45.9	36.0 $\pm$ 19.5
2	113.0 $\pm$ 155.4	30.8 $\pm$ 27.3	105.9 $\pm$ 147.5	35.1 $\pm$ 23.6	76.9 $\pm$ 114.6	48.4 $\pm$ 37.0	54.8 $\pm$ 69.8	51.1 $\pm$ 40.2
3	170.2 $\pm$ 175.1	48.3 $\pm$ 47.7	166.1 $\pm$ 173.6	56.6 $\pm$ 53.7	99.2 $\pm$ 131.0	65.6 $\pm$ 60.2	74.4 $\pm$ 83.7	64.2 $\pm$ 30.4
4	230.8 $\pm$ 190.0	66.2 $\pm$ 55.8	252.6 $\pm$ 194.0	77.9 $\pm$ 60.4	140.9 $\pm$ 138.5	84.9 $\pm$ 65.1	151.2 $\pm$ 210.3	104.4 $\pm$ 95.3

Table 5: Statistical results on verification time for overlapping local perturbations (seconds).

c

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

0.05

100.0% (200/200)

99.5% (199/200)

55.5% (111/200)

100.0% (198/198)

0.0% (0/198)

100.0% (190/190)

75.3% (143/190)

0.0% (0/190)

94.8% (92/97)

11.3% (11/97)

0.0% (0/97)

0.5

100.0% (200/200)

99.5% (199/200)

55.0% (110/200)

100.0% (198/198)

0.0% (0/198)

99.5% (189/190)

72.1% (137/190)

0.0% (0/190)

91.8% (89/97)

10.3% (10/97)

0.0% (0/97)

100.0% (200/200)

99.5% (199/200)

54.0% (108/200)

100.0% (198/198)

0.0% (0/198)

98.4% (187/190)

64.7% (123/190)

0.0% (0/190)

84.5% (82/97)

8.2% (8/97)

0.0% (0/97)

Table 6: Statistical results on verified rate for contrast.


$c$ %	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$c$ %	ours	baseline	ours	baseline	ours	baseline	ours	baseline
0.05	19.0 $\pm$ 2.8	16.1 $\pm$ 3.4	19.4 $\pm$ 2.7	123.3 $\pm$ 9.6	20.1 $\pm$ 3.6	29.1 $\pm$ 3.7	23.9 $\pm$ 3.5	34.7 $\pm$ 3.2
0.5	22.9 $\pm$ 2.9	19.3 $\pm$ 3.4	23.3 $\pm$ 2.9	26.9 $\pm$ 9.3	24.1 $\pm$ 3.7	33.0 $\pm$ 3.8	28.0 $\pm$ 3.4	38.9 $\pm$ 3.0
1	48.1 $\pm$ 9.4	45.0 $\pm$ 10.3	49.0 $\pm$ 9.7	53.3 $\pm$ 16.0	49.7 $\pm$ 9.8	58.9 $\pm$ 9.9	54.9 $\pm$ 9.8	65.6 $\pm$ 8.8

Table 7: Statistical results on verification time for contrast (seconds).

b

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

100.0% (200/200)

99.5% (199/200)

55.5% (111/200)

100.0% (198/198)

0.0% (0/198)

100.0% (190/190)

72.1% (137/190)

0.0% (0/190)

93.8% (91/97)

10.3% (10/97)

0.0% (0/97)

100.0% (200/200)

99.0% (198/200)

54.5% (109/200)

100.0% (198/198)

0.0% (0/198)

100.0% (190/190)

63.7% (121/190)

0.0% (0/190)

88.6% (86/97)

7.2% (7/97)

0.0% (0/97)

Table 8: Statistical results on verified rate for brightness.


$b$	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$b$	ours	baseline	ours	baseline	ours	baseline	ours	baseline
1	33.4 $\pm$ 5.5	29.0 $\pm$ 5.6	33.9 $\pm$ 5.5	35.2 $\pm$ 9.8	34.3 $\pm$ 5.9	43.4 $\pm$ 6.4	39.6 $\pm$ 11.9	48.9 $\pm$ 5.4
2	65.6 $\pm$ 13.4	58.3 $\pm$ 11.2	67.6 $\pm$ 16.9	65.0 $\pm$ 15.1	66.2 $\pm$ 13.3	75.8 $\pm$ 11.6	72.1 $\pm$ 12.8	77.9 $\pm$ 11.0

Table 9: Statistical results on verification time for brightness (seconds).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: (a) Ground-truth keypoints are shown in green. (b) Deviations from the ground-truth keypoints are illustrated in both the vertical (first element) and horizontal (second element) directions. (c) Decoupled keypoint errors as determined by the method in the baseline ( Luo et al., 2025 ) . (d) Eight scenarios of coupled keypoint errors are evaluated using our approach, where the error of the first keypoint is fixed and the feasible error combinations for the second keypoint are enumerated. For example, the top-left case shows the feasible $\delta h_{2}$ and $\delta w_{2}$ values when $\delta h_{1}=\delta w_{1}=-1$ . The bottom-right case illustrates cases where keypoint errors fall outside the polytope. Specifically, when the first keypoint deviates from the ground truth by 1 pixel in both directions, no feasible solution exists, even if the second keypoint aligns perfectly with the ground truth. Note that (c) and (d) are best interpreted with reference to (b) to evaluate satisfaction of the constraints in ( 10 ).

Panel of Figure 4 — Figure 4: In the first scenario (a)-(c), the MILP is infeasible. Conversely, in the second scenario (d)-(f), the MILP is feasible, resulting in the identification of a counterexample.

Explore similar work

Feb 17, 2026

RaCo: Ranking and Covariance for Practical Learned Keypoints

This paper introduces RaCo, a lightweight neural network designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks. The model integrates three key components: the repeatable keypoint detector, a differentiable ranker to maximize matches with a limited number of keypoints, and a covariance estimator to quantify spatial uncertainty in metric scale. Trained on perspective image crops only, RaCo operates without the need for covisible image pairs. It achieves strong rotational robustness through extensive data augmentation, even without the use of computationally expensive equivariant network architectures. The method is evaluated on several challenging datasets, where it demonstrates state-of-the-art performance in keypoint repeatability and two-view matching, particularly under large in-plane rotations. Ultimately, RaCo provides an effective and simple strategy to independently estimate keypoint ranking and metric covariance without additional labels, detecting interpretable and repeatable interest points. The code is available at https://github.com/cvg/RaCo.

Abhiram Shenoi, Philipp Lindenberger, Paul-Edouard Sarlin +1

ETH Zurich · Google · Microsoft Mixed Reality & AI Lab

Keypoint Detection Local Feature Matching Uncertainty Quantification

Ask PDF

Oct 12, 2026

Towards Complementary Keypoint Detection via Mixture of Detectors

Keypoint detection plays a central role in geometric vision pipelines. Existing detectors exhibit complementary strengths across different scene structures and keypoint budgets, and reliance on a single detector often results in suboptimal keypoint distributions. Motivated by this observation, we propose Mixture of Detectors (MoD) for dense matching pipelines, a unified framework for complementary keypoint detection through the adaptive composition of heterogeneous detectors. MoD first distills heterogeneous detectors into a unified model, allowing diverse detection maps to be predicted efficiently within a single forward pass. Detector selection is subsequently formulated as a patch-wise, budget-conditioned decision process, in which a lightweight router assigns the most suitable detector to each local region. To avoid heuristic supervision, the router is trained with task-driven rewards derived from downstream matching quality, jointly capturing both keypoint quantity and precision. Extensive experiments on MegaDepth, ScanNet, and HPatches demonstrate that MoD consistently outperforms strong baselines across a wide range of keypoint budgets, demonstrating its effectiveness in improving keypoint distribution and downstream geometric performance.

Xiaoyong Lu, Guobao Xiao, Dong Liang +1

Southeast University · Tongji University · Nanjing University of Aeronautics and Astronautics

Keypoint Detection Mixture of Experts Local Feature Matching

Ask PDF

Mar 3, 2026

IoUCert: Robustness Verification for Anchor-based Object Detectors

While formal robustness verification has seen significant success in image classification, scaling these guarantees to object detection remains notoriously difficult due to complex non-linear coordinate transformations and Intersection-over-Union (IoU) metrics. As a fundamental step towards verifying complete detection pipelines, we introduce IoUCert, a novel formal verification framework designed specifically to overcome these core mathematical bottlenecks. By isolating the object localisation task in single-object settings, we propose a coordinate transformation that circumvents precision-degrading relaxations of non-linear box prediction functions. This approach allows us to optimise bounds directly with respect to anchor box offsets, enabling a novel Interval Bound Propagation method that derives optimal IoU bounds. We demonstrate that IoUCert enables, for the first time, the robustness verification of foundational, anchor-based architectures including tractable variants of SSD, YOLOv2, and YOLOv3 against various input perturbations, providing a rigorous theoretical basis for future end-to-end detector verification.

Benedikt Brückner, Alejandro J. Mercado, Yanghao Zhang +2

Safe Intelligence · Imperial College London

Certified Robustness Object Detection

Ask PDF