Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

Sreejani Chatterjee; Venkatesh Mullur; Abhinav Gandhi; Berk Calli

Figure 1: Image (a) shows the original images for different robot configurations covering the workspace with ArUco markers attached, and image (b) presents the corresponding binary masks highlighting these markers. Both (a) and (b) are provided as input to a modified LaMa inpainting model, which generates the corresponding reconstructed images shown in (c). The reconstructed images, along with the original ArUco marker centers, are then used to generate a labeled training dataset, as shown in image (d), where the green bounding boxes are drawn with the same orientation of the ArUco markers and the red dots are the marker centers. This dataset is used to train a modified keypointrcnn_resnet50_fpn model for real-time keypoint prediction. Image (e) illustrates the model’s prediction for an arbitrary configuration. (f) depicts the start and goal configurations, along with the resulting control trajectory generated using the predicted keypoints. (g–l) depicts the same control experiment as (f) with occlusion in the environment. (g), (i), (k) show occluded start, intermediate, and goal frames; (h), (j), (l) are their inpainted counterparts. (m) depicts predicted control trajectory under occlusion, corresponding to (f), using inpainted keypoints.

Figure 2: The data collection phase for out-of-plane motion. Image (a) shows the original images for different robot configurations covering the workspace in 3D with ArUco markers attached, and image (b) presents the corresponding binary masks highlighting these markers. Both (a) and (b) are provided as input to a modified LaMa inpainting model, which generates the corresponding reconstructed images shown in (c). The reconstructed images, along with the original ArUco marker centers, are then used to generate a labeled training dataset, as shown in image (d), where the green bounding boxes are drawn with the same orientation of the ArUco markers and the red dots are the marker centers. This dataset is used to train a modified keypointrcnn_resnet50_fpn model for real-time keypoint prediction. Image (e) illustrates the model’s prediction for an arbitrary out-of-plane configuration. (f) depicts the start and goal configurations, along with the resulting control trajectory generated using the predicted keypoints.

Figure 3: Overall data-collection pipeline.

Figure 4: Overall pipeline for adaptive visual servoing using predicted keypoints from inpainted images.

Figure 5: Comparison of inpainting results across multiple models. Each row shows an input image, its ground truth label, and outputs from AttU-Net, ResU-Net, U2NET, and CoModGAN. Column headers are shown only once for clarity.

Figure 6: Overview of the Attention U-Net based GAN model.

Figure 7: Samples from the dataset used to train the Attention U-Net GAN model. The first row shows occlusions created by overlaying patches of YCB objects onto the robot images. The second row depicts occlusions simulated using randomly shaped and colored patches. The third row presents occlusions generated from real-world scene fragments, enhancing the model’s robustness to diverse visual clutter.

Filters	Mean Error (px)	RMSE (px)	Failure Rate %
Kalman	$18.41$	$39.34$	$24.94$
Extended Kalman	$18.41$	$39.34$	$24.94$
Unscented Kalman	$18.32$	$39.37$	$24.66$

Table 1: Filtering-only baseline without inpainting. Keypoint estimation error under occlusion for the three temporal filters, averaged across the five sequences. These rows apply temporal filtering alone to the occluded detections and serve as a baseline rather than the proposed pipeline, which we evaluate in Table 4 .

Filters	Mean Error (px)	RMSE (px)	Failure Rate %
Only UKF	$18.32$	$39.37$	$24.66$
Inpainting with UKF	$3.38$	$7.29$	$6.06$

Table 2: Inpainting with Filter evaluation results averaged across five experiments

Figure 8: The images with ArUco marker are the ground truth and the red dot on the corresponding image is the keypoint predicted by Keypoint R-CNN model trained with data generated using our present method.

Figure 9: Smooth keypoint detection during continuous robot motion for

4

keypoints configuration.

Figure 10: The images with ArUco marker are the ground truth and the red dot on the corresponding image is the keypoint predicted by Keypoint R-CNN model trained with data generated using our present method.

Figure 11: Smooth keypoint detection during continuous robot motion for

5

keypoints configuration.

Figure 12: The images with ArUco marker are the ground truth and the red dot on the corresponding image is the keypoint predicted by Keypoint R-CNN model trained with data generated using our present method.

Figure 13: Smooth keypoint detection during continuous robot motion in 3D with

8

keypoints configuration.

Figure 14: Example video frames from two experiments illustrating keypoint prediction and correction under occlusion. Each set of three rows corresponds to a single experiment. The first row shows real-time keypoint detection on occluded images using Keypoint R-CNN (blue circles), with missing keypoints corrected using the Unscented Kalman Filter (red circles). Ground truth keypoints obtained under full visibility are overlaid in green for reference. The second row uses inpainted images as input to the Keypoint R-CNN, followed by UKF correction of missing keypoints. The third row shows the corresponding inpainted reconstructions used in the second row. The comparison highlights the effect of inpainting on improving keypoint prediction accuracy and visual stability under occlusion.

Experiment	Only Inpainting (%)	After UKF Correction (%)
exp_01	99.526	100.000
exp_02	99.341	99.615
exp_03	88.767	89.425
exp_04	94.619	97.310
exp_05	77.229	83.976

Table 3: Keypoint detection accuracy before and after UKF correction.

Pipeline	Mean Error (px)	RMSE (px)	Failure Rate %	Detection %
Visible-only	$1.48$	$2.42$	$26.97$	$73.28$
Filtering-only (UKF)	$11.92$	$24.50$	$22.48$	$100.00$
Inpainting + UKF (ours)	$\mathbf{2.82}$	$\mathbf{5.35}$	$\mathbf{5.09}$	$\mathbf{100.00}$
Occlusion-trained	$3.38$	$5.44$	$3.02$	$100.00$
Occlusion-trained + UKF	$3.21$	$5.02$	$2.74$	$100.00$

Table 4: Keypoint estimation under occlusion, averaged across the five occluded sequences and scored on the

5

control keypoints. The first three pipelines share one detector trained on clean images, while the last two use a detector retrained on occluded images.

Stage	Latency (ms)
Inpainting (Attention U-Net)	$21.2$
Keypoint detection (Keypoint R-CNN)	$63.7$
Outlier gate	$0.1$
UKF update	$2.0$
End-to-end, with inpainting	$87.1$
End-to-end, full visibility	$65.9$

Table 5: Per-frame latency of each pipeline stage on an NVIDIA GeForce RTX 3080 laptop GPU, averaged over

485

frames pooled across the five occluded sequences.

Pipeline	Mean Error (px)	RMSE (px)	Failure Rate %	Detection %
Raw detection	$5.33$	$8.92$	$3.91$	$98.04$
Detection + UKF	$5.42$	$7.69$	$3.00$	$100.00$
Detection + gate + UKF (ours)	$\mathbf{5.05}$	$\mathbf{5.46}$	$\mathbf{1.93}$	$\mathbf{100.00}$

Table 6: Keypoint detection on a cluttered laboratory background with ArUco marker centers as ground truth, averaged across five sequences. The detector is the deployed model, trained only on the black-background data, with no retraining for this scene.

Figure 15: Qualitative keypoint detection on the cluttered laboratory background. Four sequences are shown, each as a pair with the ArUco-marker frame on the left providing the ground-truth keypoint positions and the corresponding marker-free frame on the right. On the right frame of each pair, the green circle is the ArUco ground truth, the red circle is the raw Keypoint R-CNN detection, and the blue circle is the final estimate after the outlier gate and the UKF. In the top-left pair the raw detector misses one keypoint, which therefore has no red circle, and the gate together with the UKF recovers it so the blue estimate stays on the keypoint. In the top-right and bottom-left pairs the raw detector localizes every keypoint correctly, so all three circles coincide. In the bottom-right pair a raw red detection lands far from its marker, an outlier caused by the background clutter, and the gate rejects it so the blue estimate stays on the true keypoint. The detector is the deployed model trained only on the black-background data, with no retraining for this scene.

Pipeline	Mean Error (px)	RMSE (px)	Failure Rate %	Detection %
Detection + UKF (no inpainting)	$26.55$	$53.05$	$30.90$	$100.00$
Inpainting + UKF (ours)	$26.00$	$49.06$	$30.83$	$100.00$
Occlusion-trained + UKF	$22.66$	$32.23$	$71.46$	$100.00$
Mixed-trained + UKF	$21.09$	$30.05$	$70.02$	$100.00$

Table 7: Keypoint estimation under combined partial occlusion and background clutter, averaged across five sequences and scored on the five control keypoints against the clean-detector reference. The first two pipelines share the clean detector with the outlier gate and the UKF and differ only in whether inpainting is applied. All four are evaluated against a background none of them was trained on.

Figure 16: Qualitative behavior under combined partial occlusion and background clutter, for three sequences (rows). The columns are, from left to right, the proposed pipeline (clean detector with inpainting, the outlier gate, and the UKF), the inpainted reconstruction that pipeline produces, the occlusion-trained detector, and the mixed-trained detector. In the three overlay columns the green circle is the ground truth, obtained from the clean detector on the unoccluded frame, and the blue circle is the method’s prediction. The reconstruction column shows that, against a background the generator never saw, the inpainting stage repaints the robot only inside the suppressed region and does not restore the occluded structure, so the predictions for the occluded keypoints drift from the ground truth for all three methods. The keypoints that remain visible are still localized correctly.

Performance metrics	Prior Method	Proposed algorithm
System rise time (s)	$6.16\pm 2.10$	$5.85\pm 2.9$
System settling time (s)	$11.29\pm 2.42$	$8.31\pm 2.9$
End effector rise time (s)	$5.37\pm 1.66$	$5.44\pm 2.65$
End effector settling time (s)	$10.05\pm 3.48$	$7.72\pm 3.1$
Overshoot (%)	$2.9\pm 2.73$	$3.43\pm 3.28$

Table 8: Performance comparison summary

Performance metrics	Baseline keypoint detection (No occlusion)	Small occlusion	Large occlusion
System rise time (s)	$10.68\pm 5.82$	$13.94\pm 3.72$	$14.18\pm 6.22$
System settling time (s)	$15.04\pm 7.19$	$19.66\pm 5.85$	$24.08\pm 11.49$
End effector rise time (s)	$8.56\pm 5.87$	$9.02\pm 5.17$	$12.83\pm 6.79$
End effector settling time (s)	$14.02\pm 6.83$	$16.66\pm 8.12$	$18.98\pm 7.08$
Overshoot (%)	$2.33\pm 1.03$	$4.11\pm 2.08$	$11.37\pm 9.88$

Table 9: Performance results for the control experiments with

5

keypoints and

3

joints, with and without occlusion

Figure 17: Image feature error norm (top) and individual image feature errors (bottom) for

3

keypoints.

Figure 18: Comparison of trajectory smoothness between the proposed and prior methods. The low-noise profiles indicate that our method achieves robustness comparable to the existing approach.

Figure 19: Image feature error norm (top) and individual image feature errors (bottom) for

5

keypoints without occlusion in the scene.

Figure 20: Image feature error norm (top) and individual image feature errors (bottom) for

5

keypoints with occlusion in the scene.

Figure 21: Control experiment results comparing baseline keypoint detection with keypoint detection under varying levels of occlusion. Each row corresponds to the same start and goal configuration. The first image in each row shows the baseline without occlusion, the second includes a smaller occlusion, and the third includes a larger occlusion. The robot configuration with red keypoints represents the goal. While the controller successfully converges in all cases, the presence of occlusion leads to visibly noisier trajectories.

Performance metrics	Proposed algorithm
System rise time (s)	$17.69\pm 10.77$
System settling time (s)	$34.39\pm 17.57$
End effector rise time (s)	$14.19\pm 8.62$
End effector settling time (s)	$27.12\pm 15.6$
Overshoot (%)	$3.21\pm 2.32$

Table 10: Performance summary for the control experiments in out-of-plane motion

Figure 22: Image feature error norm (top) and individual image feature errors (bottom) for control experiments with 3D motion.

Figure 23: Control experiment results for out-of-plane motions using

6

keypoints and

4

joints. The robot configurations with the red keypoints represent the goal. As observed, the controller achieves smooth and noiseless trajectories in all cases.

Figure 24: Image feature error norm (top) and individual image feature errors (bottom) for control experiments with 3D motion in clutter.

Figure 25: Control experiment results for out-of-plane motions using

6

keypoints and

4

joints, with clutter background. In each panel the faded robot denotes the start configuration and the solid robot the goal configuration, and the colored curves trace the image-space trajectory of each keypoint from start to goal. As observed, the controller achieves smooth and noiseless trajectories in all three cases.

Performance metrics	Proposed algorithm
System rise time (s)	$13.94\pm 8.36$
System settling time (s)	$37.84\pm 20.10$
End effector rise time (s)	$9.94\pm 4.02$
End effector settling time (s)	$29.80\pm 17.13$
Overshoot (%)	$6.38\pm 2.87$

Table 11: Performance summary for the out-of-plane control experiments conducted in the cluttered laboratory background, averaged over five runs.

Run	Steps	$\boldsymbol{\kappa(\hat{J})}$ median	Within $\boldsymbol{90^{\circ}}$	Cosine
1	224	10.9	91.0%	0.88
2	175	14.4	100.0%	0.94
3	293	9.2	100.0%	0.89
4	687	11.2	99.8%	0.74
5	505	10.5	97.7%	0.72
Mean		11.2	97.7%	0.83

Table 12: Direct evaluation of the online interaction-matrix estimate

\hat{J}

across five out-of-plane control runs.

\kappa(\hat{J})

is the condition number of the estimate. The direction agreement is the share of steps in which the predicted feature-motion direction lies within ninety degrees of the observed direction, and the cosine is the median over the run, both at a ten-step horizon.

	Proposed algorithm
System rise time (s)	$6.82\pm 0.11$
System settling time (s)	$9.12\pm 0.23$
End effector rise time (s)	$6.82\pm 0.11$
End effector settling time (s)	$9.0\pm 0.1$
Overshoot (%)	$1.096\pm 0.27$

Table 13: Summary of repeatability tests -

2

Joints and

3

Keypoints

	Proposed algorithm
System rise time (s)	$9.68\pm 0.35$
System settling time (s)	$19.72\pm 0.19$
End effector rise time (s)	$7.42\pm 0.41$
End effector settling time (s)	$19.72\pm 0.19$
Overshoot (%)	$3.99\pm 0.68$

Table 14: Summary of repeatability tests -

3

Joints,

5

Keypoints, No occlusion

	Proposed algorithm
System rise time (s)	$31.54\pm 2.36$
System settling time (s)	$40.76\pm 3.63$
End effector rise time (s)	$21.44\pm 1.68$
End effector settling time (s)	$33.66\pm 3.72$
Overshoot (%)	$1.74\pm 0.59$

Table 15: Summary of repeatability tests -

4

Joints,

6

Keypoints, 3D motion

	Proposed algorithm
System rise time (s)	$13.2\pm 0.41$
System settling time (s)	$19.78\pm 1.05$
End effector rise time (s)	$13.2\pm 0.41$
End effector settling time (s)	$16.10\pm 0.26$
Overshoot (%)	$5.48\pm 0.34$

Table 16: Summary of repeatability tests -

3

Joints,

5

Keypoints, With occlusion

Figure 26: Conditioning of the online interaction-matrix estimate over the control runs. The condition number of

\hat{J}

stays in a narrow well-conditioned band and is flat over each run, including at convergence.

Figure 27: Predicted versus observed feature-motion direction across the control runs. The cosine between the predicted and the observed feature-motion direction stays above the ninety-degree loop-stability bound (dashed) for nearly all steps, so the estimate keeps the commanded velocity in a descent direction throughout.

Figure 28: A real-time cup stacking experiment using adaptive visual servoing under occlusion. The robotic manipulator stacks cups while tracking body keypoints (red-on-green circles), which are used as visual features for adaptive control. These keypoints are detected from reconstructed images generated via inpainting. The rows labeled “Real-time experiment” show the original occluded images, while the “Reconstructed view” rows display the inpainted images used for keypoint detection. This approach enables accurate and continuous control despite partial visual occlusions during the task.

Figure 29: Example of pose ambiguity introduced by occlusion. All four images show the same configuration for the first two joints, but the occluded end-effector is reconstructed differently in each case, resulting in multiple plausible but incorrect poses for the last joint.

Figure 30: The first image contains a large occlusion completely covering the end effector. The inpainting model reconstructs two end effector regions due to pose ambiguity as seen in second image. As a result, the initial keypoint prediction (red circles) is missed. Despite this, the UKF, leveraging distance-based thresholding, is able to generate a reasonably accurate estimate of the keypoint (green circles)

Figure 31: The first image contains a bright white occlusion that significantly affects reconstruction quality, as seen in the second image. As a result, the initial keypoint prediction (red circle) is severely mislocalized, falling outside the correction range of the UKF (green circle).

Figure 32: (a) Initial image. (b) Goal image with desired keypoints (red). (c–d) Intermediate images during control: although the goal requires motion away from the camera, the image-space arrangement of predicted keypoints becomes nearly indistinguishable from the goal, creating a depth-direction ambiguity that initially drives the arm toward the camera.

Figure 33: Extension of the markerless data generation pipeline to the soft origami arm. (a) Sample images of the arm with red markers. (b) Corresponding binary masks of the markers. (c) Reconstructed images with the markers removed by inpainting. (d) Sample annotations of the marker centers and bounding boxes used to build the Keypoint R-CNN training set. (e) and (f) show two configurations with keypoints predicted in real time by the trained Keypoint R-CNN on the origami arm.

Figure 34: Keypoint tracking on the two-module origami arm across two experiments. In the top row, the trained Keypoint R-CNN detects all keypoints along the body from frame (a) to (d), shown as blue circles on the red markers. In the bottom row, frame (a) shows correct detection. In frame (b) the detector misses keypoints kp_1 and kp_4, which the temporal tracking module reconstructs. In frames (c) and (d), kp_1 stays undetected by the network and is held by temporal tracking, while kp_4 is detected but severely mislocalized in frame (c) and then corrected by the refinement.

Figure 35: Keypoint tracking on the three-module origami arm during spatial motion. Frames (a) to (c) show different arm configurations, where blue circles denote the keypoints predicted in real time by the trained Keypoint R-CNN and the temporal tracking pipeline.

Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

Abstract

Figures & tables

Explore similar work

MarUco: A Markerless 6D Pose Estimation Framework for Closed-Loop Control of Surgical Continuum Manipulators

IKKA: Inversion Classification via Critical Anomalies for Robust Visual Servoing

Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training