From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications

Xusheng Luo; Changliu Liu

cs.CVMar 5, 2026

From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications

Authors: Xusheng Luo, Changliu Liu

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA

Abstract

Keypoint detection underpins many vision tasks, including pose estimation, viewpoint recovery, and 3D reconstruction, yet modern neural models remain vulnerable to small input perturbations. Despite its importance, formal robustness verification for keypoint detectors is largely unexplored due to high-dimensional inputs and continuous coordinate outputs. We propose the first coupled robustness verification framework for heatmap-based keypoint detectors that bounds the joint deviation across all keypoints, capturing their interdependencies and downstream task requirements. Unlike prior decoupled, classification-style approaches that verify each keypoint independently and yield conservative guarantees, our method verifies collective behavior. We formulate verification as a falsification problem using a mixed-integer linear program (MILP) that combines reachable heatmap sets with a polytope encoding joint deviation constraints. Infeasibility certifies robustness, while feasibility provides counterexamples, and we prove the method is sound: if it certifies the model as robust, then the keypoint detection model is guaranteed to be robust. Experiments show that our coupled approach achieves high verified rates and remains effective under strict error thresholds where decoupled methods fail.

Figures & tables

Figure 1: Overview of the keypoint detection pipeline and the proposed verification framework. A thick red dashed line divides keypoint detection (above) and verification (below). A seed image $\bbX_{0}$ where an airplane is parking at the airport is processed by the model to identify keypoints, which are marked as green dots. The verification framework takes as input the seed image $\bbX_{0}$ and a set of perturbed images $\bbX$ (with local perturbations indicated by red circles) that form the convex hull $\ccalX$ , along with the keypoint error bound $\delta\ccalV$ . By checking the feasibility of the MILP that is derived from the reachable set $\ccalZ$ of the model and the keypoint error bound $\delta\ccalV$ , the method returns whether the model is robust.

Figure 2: An illustration of dynamic indexing for the first keypoint: the green grid indicates the ground-truth keypoint location, while the red grid represents the perturbed keypoint position. The index deviation (1, -1) retrieves the value $Z_{31}$ .

m

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

100.0% (200/200)

99.5% (199/200)

47.5% (95/200)

100.0% (198/198)

99.0% (196/198)

0.0% (0/198)

100.0% (190/190)

71.6% (136/190)

0.0% (0/190)

88.7% (86/97)

10.3% (10/97)

0.0% (0/97)

100.0% (200/200)

99.0% (198/200)

52.5% (105/200)

100.0% (198/198)

0.0% (0/198)

98.4% (187/190)

72.1% (137/190)

0.0% (0/190)

89.7% (87/97)

8.2% (8/97)

0.0% (0/97)

100.0% (200/200)

99.0% (198/200)

53.0% (106/200)

100.0% (198/198)

99.0% (196/198)

0.0% (0/198)

99.4% (189/190)

67.4% (128/190)

0.0% (0/190)

88.7% (86/97)

5.1% (5/97)

0.0% (0/97)

100.0% (200/200)

97.0% (194/200)

53.0% (106/200)

100.0% (198/198)

98.0% (194/198)

0.0% (0/198)

99.5% (189/190)

62.1% (118/190)

0.0% (0/190)

90.7% (88/97)

7.2% (7/97)

0.0% (0/97)

Table 1: Statistical results on verified rate for non-overlapping local perturbations.

m

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

100.0% (196/196)

95.4% (187/196)

50.0% (98/196)

100.0% (195/195)

93.3% (182/195)

0.0% (0/195)

96.3% (183/190)

64.2% (122/190)

0.0% (0/190)

76.3% (74/97)

6.2% (6/97)

0.0% (0/97)

100.0% (192/192)

77.1% (148/192)

41.1% (79/192)

100.0% (190/190)

78.9% (150/190)

0.0% (0/190)

96.7% (175/181)

43.6% (79/181)

0.0% (0/181)

79.1% (72/91)

3.3% (3/91)

0.0% (0/91)

100.0% (157/157)

64.3% (101/157)

33.1% (52/157)

100.0% (157/157)

64.3% (101/157)

0.0% (0/157)

96.0% (142/148)

30.4% (45/148)

0.0% (0/148)

73.9% (51/69)

1.4% (1/69)

0.0% (0/69)

100.0% (126/126)

51.6% (65/126)

31.0% (39/126)

100.0% (126/126)

46.8% (59/126)

0.0% (0/126)

89.0% (105/115)

20.9% (24/115)

0.0% (0/115)

67.9% (36/53)

1.9% (1/53)

0.0% (0/53)

Table 2: Statistical results on verified rate for overlapping local perturbations.


$m$	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$m$	ours	baseline	ours	baseline	ours	baseline	ours	baseline
1	11.8 $\pm$ 7.4	7.6 $\pm$ 6.6	10.8 $\pm$ 5.2	13.0 $\pm$ 9.0	12.6 $\pm$ 16.1	20.4 $\pm$ 6.3	9.1 $\pm$ 12.4	26.4 $\pm$ 6.4
2	17.0 $\pm$ 15.2	12.4 $\pm$ 9.9	18.6 $\pm$ 29.5	19.2 $\pm$ 12.0	16.1 $\pm$ 12.4	26.7 $\pm$ 7.4	12.3 $\pm$ 16.4	33.0 $\pm$ 11.8
3	29.8 $\pm$ 71.0	17.2 $\pm$ 9.6	28.9 $\pm$ 79.2	22.5 $\pm$ 13.5	21.9 $\pm$ 17.9	29.8 $\pm$ 10.7	14.1 $\pm$ 18.1	35.6 $\pm$ 10.8
4	36.0 $\pm$ 73.6	21.5 $\pm$ 12.4	37.2 $\pm$ 77.4	28.0 $\pm$ 15.7	29.0 $\pm$ 23.5	34.7 $\pm$ 12.2	18.5 $\pm$ 23.3	39.2 $\pm$ 14.2

Table 3: Statistical results on verification time for non-overlapping local perturbations (seconds).


$\alpha$	overlapping	pruning	binary	integer	continuous	constraints

1.0	✗	✗	9.42 $\times 10^{4}\pm$ 0.00 $\times 10^{1}$	46.0 $\pm$ 0.0	1.89 $\times 10^{5}\pm$ 5.29 $\times 10^{2}$	6.61 $\times 10^{5}\pm$ 1.06 $\times 10^{3}$
	✗	✓	5.31 $\times 10^{1}\pm$ 4.29 $\times 10^{1}$	46.0 $\pm$ 0.0	8.36 $\times 10^{2}\pm$ 7.48 $\times 10^{2}$	1.95 $\times 10^{3}\pm$ 1.61 $\times 10^{3}$
	✓	✗	9.42 $\times 10^{4}\pm$ 0.00 $\times 10^{1}$	46.0 $\pm$ 0.0	1.92 $\times 10^{5}\pm$ 6.64 $\times 10^{3}$	6.67 $\times 10^{5}\pm$ 1.33 $\times 10^{4}$
✓	✓	8.18 $\times 10^{2}\pm$ 4.11 $\times 10^{3}$	46.0 $\pm$ 0.0	5.07 $\times 10^{3}\pm$ 1.42 $\times 10^{4}$	1.27 $\times 10^{4}\pm$ 4.06 $\times 10^{4}$

Table 4: Statistical results of the MILP size, including the number of binary, integer, and continuous variables, as well as the number of constraints. These results vary with the pose error threshold

\alpha

, the presence of object-airplane overlap, and the use of the pruning strategy. Results obtained with pruning are highlighted in gray for non-overlapping images and in orange for overlapping images.


$m$	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$m$	ours	baseline	ours	baseline	ours	baseline	ours	baseline
1	40.3 $\pm$ 95.9	16.3 $\pm$ 17.8	44.0 $\pm$ 94.0	20.9 $\pm$ 16.6	23.9 $\pm$ 39.8	29.3 $\pm$ 14.2	29.3 $\pm$ 45.9	36.0 $\pm$ 19.5
2	113.0 $\pm$ 155.4	30.8 $\pm$ 27.3	105.9 $\pm$ 147.5	35.1 $\pm$ 23.6	76.9 $\pm$ 114.6	48.4 $\pm$ 37.0	54.8 $\pm$ 69.8	51.1 $\pm$ 40.2
3	170.2 $\pm$ 175.1	48.3 $\pm$ 47.7	166.1 $\pm$ 173.6	56.6 $\pm$ 53.7	99.2 $\pm$ 131.0	65.6 $\pm$ 60.2	74.4 $\pm$ 83.7	64.2 $\pm$ 30.4
4	230.8 $\pm$ 190.0	66.2 $\pm$ 55.8	252.6 $\pm$ 194.0	77.9 $\pm$ 60.4	140.9 $\pm$ 138.5	84.9 $\pm$ 65.1	151.2 $\pm$ 210.3	104.4 $\pm$ 95.3

Table 5: Statistical results on verification time for overlapping local perturbations (seconds).

c

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

0.05

100.0% (200/200)

99.5% (199/200)

55.5% (111/200)

100.0% (198/198)

0.0% (0/198)

100.0% (190/190)

75.3% (143/190)

0.0% (0/190)

94.8% (92/97)

11.3% (11/97)

0.0% (0/97)

0.5

100.0% (200/200)

99.5% (199/200)

55.0% (110/200)

100.0% (198/198)

0.0% (0/198)

99.5% (189/190)

72.1% (137/190)

0.0% (0/190)

91.8% (89/97)

10.3% (10/97)

0.0% (0/97)

100.0% (200/200)

99.5% (199/200)

54.0% (108/200)

100.0% (198/198)

0.0% (0/198)

98.4% (187/190)

64.7% (123/190)

0.0% (0/190)

84.5% (82/97)

8.2% (8/97)

0.0% (0/97)

Table 6: Statistical results on verified rate for contrast.


$c$ %	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$c$ %	ours	baseline	ours	baseline	ours	baseline	ours	baseline
0.05	19.0 $\pm$ 2.8	16.1 $\pm$ 3.4	19.4 $\pm$ 2.7	123.3 $\pm$ 9.6	20.1 $\pm$ 3.6	29.1 $\pm$ 3.7	23.9 $\pm$ 3.5	34.7 $\pm$ 3.2
0.5	22.9 $\pm$ 2.9	19.3 $\pm$ 3.4	23.3 $\pm$ 2.9	26.9 $\pm$ 9.3	24.1 $\pm$ 3.7	33.0 $\pm$ 3.8	28.0 $\pm$ 3.4	38.9 $\pm$ 3.0
1	48.1 $\pm$ 9.4	45.0 $\pm$ 10.3	49.0 $\pm$ 9.7	53.3 $\pm$ 16.0	49.7 $\pm$ 9.8	58.9 $\pm$ 9.9	54.9 $\pm$ 9.8	65.6 $\pm$ 8.8

Table 7: Statistical results on verification time for contrast (seconds).

b

\alpha=1.0

\alpha=0.5

\alpha=0.2

\alpha=0.1

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

testing

ours

baseline

100.0% (200/200)

99.5% (199/200)

55.5% (111/200)

100.0% (198/198)

0.0% (0/198)

100.0% (190/190)

72.1% (137/190)

0.0% (0/190)

93.8% (91/97)

10.3% (10/97)

0.0% (0/97)

100.0% (200/200)

99.0% (198/200)

54.5% (109/200)

100.0% (198/198)

0.0% (0/198)

100.0% (190/190)

63.7% (121/190)

0.0% (0/190)

88.6% (86/97)

7.2% (7/97)

0.0% (0/97)

Table 8: Statistical results on verified rate for brightness.


$b$	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$		$\alpha=0.1$
$b$	ours	baseline	ours	baseline	ours	baseline	ours	baseline
1	33.4 $\pm$ 5.5	29.0 $\pm$ 5.6	33.9 $\pm$ 5.5	35.2 $\pm$ 9.8	34.3 $\pm$ 5.9	43.4 $\pm$ 6.4	39.6 $\pm$ 11.9	48.9 $\pm$ 5.4
2	65.6 $\pm$ 13.4	58.3 $\pm$ 11.2	67.6 $\pm$ 16.9	65.0 $\pm$ 15.1	66.2 $\pm$ 13.3	75.8 $\pm$ 11.6	72.1 $\pm$ 12.8	77.9 $\pm$ 11.0

Table 9: Statistical results on verification time for brightness (seconds).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: (a) Ground-truth keypoints are shown in green. (b) Deviations from the ground-truth keypoints are illustrated in both the vertical (first element) and horizontal (second element) directions. (c) Decoupled keypoint errors as determined by the method in the baseline ( Luo et al., 2025 ) . (d) Eight scenarios of coupled keypoint errors are evaluated using our approach, where the error of the first keypoint is fixed and the feasible error combinations for the second keypoint are enumerated. For example, the top-left case shows the feasible $\delta h_{2}$ and $\delta w_{2}$ values when $\delta h_{1}=\delta w_{1}=-1$ . The bottom-right case illustrates cases where keypoint errors fall outside the polytope. Specifically, when the first keypoint deviates from the ground truth by 1 pixel in both directions, no feasible solution exists, even if the second keypoint aligns perfectly with the ground truth. Note that (c) and (d) are best interpreted with reference to (b) to evaluate satisfaction of the constraints in ( 10 ).

Panel of Figure 4 — Figure 4: In the first scenario (a)-(c), the MILP is infeasible. Conversely, in the second scenario (d)-(f), the MILP is feasible, resulting in the identification of a counterexample.

Explore similar work

RaCo: Ranking and Covariance for Practical Learned Keypoints
Feb 17, 2026Abhiram Shenoi, Philipp Lindenberger, Paul-Edouard Sarlin +1Keypoint Detection Local Feature Matching
ETH Zurich · Google · Microsoft Mixed Reality & AI Lab
Towards Complementary Keypoint Detection via Mixture of Detectors
Oct 12, 2026Xiaoyong Lu, Guobao Xiao, Dong Liang +1Keypoint Detection Mixture of Experts
Southeast University · Tongji University · Nanjing University of Aeronautics and Astronautics
IoUCert: Robustness Verification for Anchor-based Object Detectors
Mar 3, 2026Benedikt Brückner, Alejandro J. Mercado, Yanghao Zhang +2Certified Robustness Object Detection
Safe Intelligence · Imperial College London

Feb 17, 2026J/K move · Enter open · S save

RaCo: Ranking and Covariance for Practical Learned Keypoints

Abhiram Shenoi, Philipp Lindenberger, Paul-Edouard Sarlin, Marc Pollefeys

ETH Zurich · Google · Microsoft Mixed Reality & AI Lab

Keypoint Detection Local Feature Matching Uncertainty Quantification

Open paper AskPDF arXiv

This paper introduces RaCo, a lightweight neural network designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks. The model integrates three key components: the repeatable keypoint detector, a differentiable ranker to maximize matches with a limited number of keypoints, and a covariance estimator to quantify spatial uncertainty in metric scale. Trained on perspective image crops only, RaCo operates without the need for covisible image pairs. It achieves strong rotational robustness through extensive data augmentation, even without the use of computationally expensive equivariant network architectures. The method is evaluated on several challenging datasets, where it demonstrates state-of-the-art performance in keypoint repeatability and two-view matching, particularly under large in-plane rotations. Ultimately, RaCo provides an effective and simple strategy to independently estimate keypoint ranking and metric covariance without additional labels, detecting interpretable and repeatable interest points. The code is available at https://github.com/cvg/RaCo.

Abstract

Explore similar work

RaCo: Ranking and Covariance for Practical Learned Keypoints

Towards Complementary Keypoint Detection via Mixture of Detectors

IoUCert: Robustness Verification for Anchor-based Object Detectors