We present a novel visual servoing framework for controlling a robotic manipulator in configuration space using only natural visual features. To train our data-driven keypoint detector, we attach ArUco markers along the robot body, use their centers as keypoint labels, and apply image inpainting to remove the markers and reconstruct the occluded regions. This produces automatically labeled, markerless robot images without requiring accurate camera calibration or robot models. At runtime, a second inpainting model reconstructs robot regions that are partially occluded, enabling continuous keypoint detection. An Unscented Kalman Filter (UKF) further improves temporal consistency and robustness of the keypoint estimates. We demonstrate successful model-free, vision-based control using natural robot features under both full visibility and partial occlusion. To show broader applicability, we also extend the perception pipeline to two-module and three-module soft origami arms and qualitatively evaluate keypoint detection and temporal tracking on these platforms.
Figures & tables
Figure 1: Image (a) shows the original images for different robot configurations covering the workspace with ArUco markers attached, and image (b) presents the corresponding binary masks highlighting these markers. Both (a) and (b) are provided as input to a modified LaMa inpainting model, which generates the corresponding reconstructed images shown in (c). The reconstructed images, along with the original ArUco marker centers, are then used to generate a labeled training dataset, as shown in image (d), where the green bounding boxes are drawn with the same orientation of the ArUco markers and the red dots are the marker centers. This dataset is used to train a modified keypointrcnn_resnet50_fpn model for real-time keypoint prediction. Image (e) illustrates the model’s prediction for an arbitrary configuration. (f) depicts the start and goal configurations, along with the resulting control trajectory generated using the predicted keypoints. (g–l) depicts the same control experiment as (f) with occlusion in the environment. (g), (i), (k) show occluded start, intermediate, and goal frames; (h), (j), (l) are their inpainted counterparts. (m) depicts predicted control trajectory under occlusion, corresponding to (f), using inpainted keypoints.
Figure 2: The data collection phase for out-of-plane motion. Image (a) shows the original images for different robot configurations covering the workspace in 3D with ArUco markers attached, and image (b) presents the corresponding binary masks highlighting these markers. Both (a) and (b) are provided as input to a modified LaMa inpainting model, which generates the corresponding reconstructed images shown in (c). The reconstructed images, along with the original ArUco marker centers, are then used to generate a labeled training dataset, as shown in image (d), where the green bounding boxes are drawn with the same orientation of the ArUco markers and the red dots are the marker centers. This dataset is used to train a modified keypointrcnn_resnet50_fpn model for real-time keypoint prediction. Image (e) illustrates the model’s prediction for an arbitrary out-of-plane configuration. (f) depicts the start and goal configurations, along with the resulting control trajectory generated using the predicted keypoints.
Figure 3: Overall data-collection pipeline.
Figure 4: Overall pipeline for adaptive visual servoing using predicted keypoints from inpainted images.
Figure 5: Comparison of inpainting results across multiple models. Each row shows an input image, its ground truth label, and outputs from AttU-Net, ResU-Net, U2NET, and CoModGAN. Column headers are shown only once for clarity.
Figure 6: Overview of the Attention U-Net based GAN model.
Figure 7: Samples from the dataset used to train the Attention U-Net GAN model. The first row shows occlusions created by overlaying patches of YCB objects onto the robot images. The second row depicts occlusions simulated using randomly shaped and colored patches. The third row presents occlusions generated from real-world scene fragments, enhancing the model’s robustness to diverse visual clutter.
Filters
Mean Error (px)
RMSE (px)
Failure Rate %
Kalman
18.41
39.34
24.94
Extended Kalman
18.41
39.34
24.94
Unscented Kalman
18.32
39.37
24.66
Table 1: Filtering-only baseline without inpainting. Keypoint estimation error under occlusion for the three temporal filters, averaged across the five sequences. These rows apply temporal filtering alone to the occluded detections and serve as a baseline rather than the proposed pipeline, which we evaluate in Table 4 .
Filters
Mean Error (px)
RMSE (px)
Failure Rate %
Only UKF
18.32
39.37
24.66
Inpainting with UKF
3.38
7.29
6.06
Table 2: Inpainting with Filter evaluation results averaged across five experiments
Figure 8: The images with ArUco marker are the ground truth and the red dot on the corresponding image is the keypoint predicted by Keypoint R-CNN model trained with data generated using our present method.
Figure 9: Smooth keypoint detection during continuous robot motion for 4 keypoints configuration.
Figure 10: The images with ArUco marker are the ground truth and the red dot on the corresponding image is the keypoint predicted by Keypoint R-CNN model trained with data generated using our present method.
Figure 11: Smooth keypoint detection during continuous robot motion for 5 keypoints configuration.
Figure 12: The images with ArUco marker are the ground truth and the red dot on the corresponding image is the keypoint predicted by Keypoint R-CNN model trained with data generated using our present method.
Figure 13: Smooth keypoint detection during continuous robot motion in 3D with 8 keypoints configuration.
Figure 14: Example video frames from two experiments illustrating keypoint prediction and correction under occlusion. Each set of three rows corresponds to a single experiment. The first row shows real-time keypoint detection on occluded images using Keypoint R-CNN (blue circles), with missing keypoints corrected using the Unscented Kalman Filter (red circles). Ground truth keypoints obtained under full visibility are overlaid in green for reference. The second row uses inpainted images as input to the Keypoint R-CNN, followed by UKF correction of missing keypoints. The third row shows the corresponding inpainted reconstructions used in the second row. The comparison highlights the effect of inpainting on improving keypoint prediction accuracy and visual stability under occlusion.
Experiment
Only Inpainting (%)
After UKF Correction (%)
exp_01
99.526
100.000
exp_02
99.341
99.615
exp_03
88.767
89.425
exp_04
94.619
97.310
exp_05
77.229
83.976
Table 3: Keypoint detection accuracy before and after UKF correction.
Pipeline
Mean Error (px)
RMSE (px)
Failure Rate %
Detection %
Visible-only
1.48
2.42
26.97
73.28
Filtering-only (UKF)
11.92
24.50
22.48
100.00
Inpainting + UKF (ours)
2.82
5.35
5.09
100.00
Occlusion-trained
3.38
5.44
3.02
100.00
Occlusion-trained + UKF
3.21
5.02
2.74
100.00
Table 4: Keypoint estimation under occlusion, averaged across the five occluded sequences and scored on the 5 control keypoints. The first three pipelines share one detector trained on clean images, while the last two use a detector retrained on occluded images.
Stage
Latency (ms)
Inpainting (Attention U-Net)
21.2
Keypoint detection (Keypoint R-CNN)
63.7
Outlier gate
0.1
UKF update
2.0
End-to-end, with inpainting
87.1
End-to-end, full visibility
65.9
Table 5: Per-frame latency of each pipeline stage on an NVIDIA GeForce RTX 3080 laptop GPU, averaged over 485 frames pooled across the five occluded sequences.
Pipeline
Mean Error (px)
RMSE (px)
Failure Rate %
Detection %
Raw detection
5.33
8.92
3.91
98.04
Detection + UKF
5.42
7.69
3.00
100.00
Detection + gate + UKF (ours)
5.05
5.46
1.93
100.00
Table 6: Keypoint detection on a cluttered laboratory background with ArUco marker centers as ground truth, averaged across five sequences. The detector is the deployed model, trained only on the black-background data, with no retraining for this scene.
Figure 15: Qualitative keypoint detection on the cluttered laboratory background. Four sequences are shown, each as a pair with the ArUco-marker frame on the left providing the ground-truth keypoint positions and the corresponding marker-free frame on the right. On the right frame of each pair, the green circle is the ArUco ground truth, the red circle is the raw Keypoint R-CNN detection, and the blue circle is the final estimate after the outlier gate and the UKF. In the top-left pair the raw detector misses one keypoint, which therefore has no red circle, and the gate together with the UKF recovers it so the blue estimate stays on the keypoint. In the top-right and bottom-left pairs the raw detector localizes every keypoint correctly, so all three circles coincide. In the bottom-right pair a raw red detection lands far from its marker, an outlier caused by the background clutter, and the gate rejects it so the blue estimate stays on the true keypoint. The detector is the deployed model trained only on the black-background data, with no retraining for this scene.
Pipeline
Mean Error (px)
RMSE (px)
Failure Rate %
Detection %
Detection + UKF (no inpainting)
26.55
53.05
30.90
100.00
Inpainting + UKF (ours)
26.00
49.06
30.83
100.00
Occlusion-trained + UKF
22.66
32.23
71.46
100.00
Mixed-trained + UKF
21.09
30.05
70.02
100.00
Table 7: Keypoint estimation under combined partial occlusion and background clutter, averaged across five sequences and scored on the five control keypoints against the clean-detector reference. The first two pipelines share the clean detector with the outlier gate and the UKF and differ only in whether inpainting is applied. All four are evaluated against a background none of them was trained on.
Figure 16: Qualitative behavior under combined partial occlusion and background clutter, for three sequences (rows). The columns are, from left to right, the proposed pipeline (clean detector with inpainting, the outlier gate, and the UKF), the inpainted reconstruction that pipeline produces, the occlusion-trained detector, and the mixed-trained detector. In the three overlay columns the green circle is the ground truth, obtained from the clean detector on the unoccluded frame, and the blue circle is the method’s prediction. The reconstruction column shows that, against a background the generator never saw, the inpainting stage repaints the robot only inside the suppressed region and does not restore the occluded structure, so the predictions for the occluded keypoints drift from the ground truth for all three methods. The keypoints that remain visible are still localized correctly.
Performance metrics
Prior Method
Proposed algorithm
System rise time (s)
6.16±2.10
5.85±2.9
System settling time (s)
11.29±2.42
8.31±2.9
End effector rise time (s)
5.37±1.66
5.44±2.65
End effector settling time (s)
10.05±3.48
7.72±3.1
Overshoot (%)
2.9±2.73
3.43±3.28
Table 8: Performance comparison summary
Performance metrics
Baseline keypoint detection (No occlusion)
Small occlusion
Large occlusion
System rise time (s)
10.68±5.82
13.94±3.72
14.18±6.22
System settling time (s)
15.04±7.19
19.66±5.85
24.08±11.49
End effector rise time (s)
8.56±5.87
9.02±5.17
12.83±6.79
End effector settling time (s)
14.02±6.83
16.66±8.12
18.98±7.08
Overshoot (%)
2.33±1.03
4.11±2.08
11.37±9.88
Table 9: Performance results for the control experiments with 5 keypoints and 3 joints, with and without occlusion
Figure 17: Image feature error norm (top) and individual image feature errors (bottom) for 3 keypoints.
Figure 18: Comparison of trajectory smoothness between the proposed and prior methods. The low-noise profiles indicate that our method achieves robustness comparable to the existing approach.
Figure 19: Image feature error norm (top) and individual image feature errors (bottom) for 5 keypoints without occlusion in the scene.
Figure 20: Image feature error norm (top) and individual image feature errors (bottom) for 5 keypoints with occlusion in the scene.
Figure 21: Control experiment results comparing baseline keypoint detection with keypoint detection under varying levels of occlusion. Each row corresponds to the same start and goal configuration. The first image in each row shows the baseline without occlusion, the second includes a smaller occlusion, and the third includes a larger occlusion. The robot configuration with red keypoints represents the goal. While the controller successfully converges in all cases, the presence of occlusion leads to visibly noisier trajectories.
Performance metrics
Proposed algorithm
System rise time (s)
17.69±10.77
System settling time (s)
34.39±17.57
End effector rise time (s)
14.19±8.62
End effector settling time (s)
27.12±15.6
Overshoot (%)
3.21±2.32
Table 10: Performance summary for the control experiments in out-of-plane motion
Figure 22: Image feature error norm (top) and individual image feature errors (bottom) for control experiments with 3D motion.
Figure 23: Control experiment results for out-of-plane motions using 6 keypoints and 4 joints. The robot configurations with the red keypoints represent the goal. As observed, the controller achieves smooth and noiseless trajectories in all cases.
Figure 24: Image feature error norm (top) and individual image feature errors (bottom) for control experiments with 3D motion in clutter.
Figure 25: Control experiment results for out-of-plane motions using 6 keypoints and 4 joints, with clutter background. In each panel the faded robot denotes the start configuration and the solid robot the goal configuration, and the colored curves trace the image-space trajectory of each keypoint from start to goal. As observed, the controller achieves smooth and noiseless trajectories in all three cases.
Performance metrics
Proposed algorithm
System rise time (s)
13.94±8.36
System settling time (s)
37.84±20.10
End effector rise time (s)
9.94±4.02
End effector settling time (s)
29.80±17.13
Overshoot (%)
6.38±2.87
Table 11: Performance summary for the out-of-plane control experiments conducted in the cluttered laboratory background, averaged over five runs.
Run
Steps
κ(J^) median
Within 90∘
Cosine
1
224
10.9
91.0%
0.88
2
175
14.4
100.0%
0.94
3
293
9.2
100.0%
0.89
4
687
11.2
99.8%
0.74
5
505
10.5
97.7%
0.72
Mean
11.2
97.7%
0.83
Table 12: Direct evaluation of the online interaction-matrix estimate J^ across five out-of-plane control runs. κ(J^) is the condition number of the estimate. The direction agreement is the share of steps in which the predicted feature-motion direction lies within ninety degrees of the observed direction, and the cosine is the median over the run, both at a ten-step horizon.
Proposed algorithm
System rise time (s)
6.82±0.11
System settling time (s)
9.12±0.23
End effector rise time (s)
6.82±0.11
End effector settling time (s)
9.0±0.1
Overshoot (%)
1.096±0.27
Table 13: Summary of repeatability tests - 2 Joints and 3 Keypoints
Proposed algorithm
System rise time (s)
9.68±0.35
System settling time (s)
19.72±0.19
End effector rise time (s)
7.42±0.41
End effector settling time (s)
19.72±0.19
Overshoot (%)
3.99±0.68
Table 14: Summary of repeatability tests - 3 Joints, 5 Keypoints, No occlusion
Proposed algorithm
System rise time (s)
31.54±2.36
System settling time (s)
40.76±3.63
End effector rise time (s)
21.44±1.68
End effector settling time (s)
33.66±3.72
Overshoot (%)
1.74±0.59
Table 15: Summary of repeatability tests - 4 Joints, 6 Keypoints, 3D motion
Proposed algorithm
System rise time (s)
13.2±0.41
System settling time (s)
19.78±1.05
End effector rise time (s)
13.2±0.41
End effector settling time (s)
16.10±0.26
Overshoot (%)
5.48±0.34
Table 16: Summary of repeatability tests - 3 Joints, 5 Keypoints, With occlusion
Figure 26: Conditioning of the online interaction-matrix estimate over the control runs. The condition number of J^ stays in a narrow well-conditioned band and is flat over each run, including at convergence.
Figure 27: Predicted versus observed feature-motion direction across the control runs. The cosine between the predicted and the observed feature-motion direction stays above the ninety-degree loop-stability bound (dashed) for nearly all steps, so the estimate keeps the commanded velocity in a descent direction throughout.
Figure 28: A real-time cup stacking experiment using adaptive visual servoing under occlusion. The robotic manipulator stacks cups while tracking body keypoints (red-on-green circles), which are used as visual features for adaptive control. These keypoints are detected from reconstructed images generated via inpainting. The rows labeled “Real-time experiment” show the original occluded images, while the “Reconstructed view” rows display the inpainted images used for keypoint detection. This approach enables accurate and continuous control despite partial visual occlusions during the task.
Figure 29: Example of pose ambiguity introduced by occlusion. All four images show the same configuration for the first two joints, but the occluded end-effector is reconstructed differently in each case, resulting in multiple plausible but incorrect poses for the last joint.
Figure 30: The first image contains a large occlusion completely covering the end effector. The inpainting model reconstructs two end effector regions due to pose ambiguity as seen in second image. As a result, the initial keypoint prediction (red circles) is missed. Despite this, the UKF, leveraging distance-based thresholding, is able to generate a reasonably accurate estimate of the keypoint (green circles)
Figure 31: The first image contains a bright white occlusion that significantly affects reconstruction quality, as seen in the second image. As a result, the initial keypoint prediction (red circle) is severely mislocalized, falling outside the correction range of the UKF (green circle).
Figure 32: (a) Initial image. (b) Goal image with desired keypoints (red). (c–d) Intermediate images during control: although the goal requires motion away from the camera, the image-space arrangement of predicted keypoints becomes nearly indistinguishable from the goal, creating a depth-direction ambiguity that initially drives the arm toward the camera.
Figure 33: Extension of the markerless data generation pipeline to the soft origami arm. (a) Sample images of the arm with red markers. (b) Corresponding binary masks of the markers. (c) Reconstructed images with the markers removed by inpainting. (d) Sample annotations of the marker centers and bounding boxes used to build the Keypoint R-CNN training set. (e) and (f) show two configurations with keypoints predicted in real time by the trained Keypoint R-CNN on the origami arm.
Figure 34: Keypoint tracking on the two-module origami arm across two experiments. In the top row, the trained Keypoint R-CNN detects all keypoints along the body from frame (a) to (d), shown as blue circles on the red markers. In the bottom row, frame (a) shows correct detection. In frame (b) the detector misses keypoints kp_1 and kp_4, which the temporal tracking module reconstructs. In frames (c) and (d), kp_1 stays undetected by the network and is held by temporal tracking, while kp_4 is detected but severely mislocalized in frame (c) and then corrected by the refinement.
Figure 35: Keypoint tracking on the three-module origami arm during spatial motion. Frames (a) to (c) show different arm configurations, where blue circles denote the keypoints predicted in real time by the trained Keypoint R-CNN and the temporal tracking pipeline.
Flexible endoscopic continuum manipulators offer high dexterity and access to complex anatomy, but nonlinear hysteresis limits feedforward control accuracy. Closed-loop control can compensate for these errors but requires accurate six-degree-of-freedom (6D) end-effector pose feedback. We present MarUco, a markerless 6D pose estimation framework for closed-loop control using only stereo vision during operation. A photorealistic pseudo-rigid-body simulation pipeline generates large-scale annotated training data without manual labeling. A multifeature fusion network integrates masks, keypoints, heatmaps, and bounding boxes from both stereo views to estimate an initial pose, followed by a learned single-pass render-and-compare module that refines the pose without iterative optimization. Kinematics-free hand-eye calibration estimates camera-to-robot-base extrinsics, and self-supervised adaptation uses unlabeled real stereo pairs to mitigate sim-to-real pose error. Across 1,000 real samples, MarUco achieves translation and rotation errors of 0.78 ± 0.50 mm and 3.07 ± 1.25°, respectively, with a total processing time of 52.4 ms per stereo pair. In closed-loop experiments over eight reference paths, MarUco achieves a mean terminal translation error of 1.8 mm, an 88% reduction relative to uncompensated open-loop control. To the best of our knowledge, this is the first markerless position-based visual servoing framework for continuum manipulators.
Junhyun Park, Chunggil An, Myeongbo Park +3
Department of Robotics and Mechatronics Engineering, DGIST, Daegu 42988, Republic of Korea · AI Research Lab, DEEPNOID, Seoul, Republic of Korea
We introduce IKKA (Inversion Classification via Critical Anomalies), a topologically motivated weighting framework for robust visual servoing under distribution shift. Unlike conventional outlier handling, IKKA treats maverick points as structurally informative observations: points where small perturbations can induce qualitatively different control responses or class assignments. The method combines local extremality, boundary transversality, and multi-scale persistence into a single anomaly weight, W(x) = E(x) x T(x) x M(x), which modulates control updates near ambiguous decision regions. We instantiate IKKA in a CPU-only embedded visual-servoing pipeline on Raspberry Pi 4 and evaluate it across 230 reproducible runs under nominal and stress conditions. In stress scenarios involving dim illumination and transient occlusion, IKKA reduces the 95th-percentile lateral error by 24% relative to a hybrid baseline (0.124 to 0.094) while increasing throughput from 20.0 to 24.8 Hz. Non-parametric analysis confirms a large effect size (Cliff's delta = 0.79).
Efficient and robust tracking of surgical robotic instruments is important for robot-assisted minimally invasive surgery, yet remains challenging due to the complexity of surgical scenes and the unconventional geometry of surgical instruments. Keypoint-based approaches are efficient, but their performance depends on reliable feature detection. Improving these detectors with real-world supervision is difficult because accurate real-world annotations are costly to obtain at scale. To address this limitation, we introduce a tracker-guided self-training framework that adapts a model pretrained on synthetic images to unlabeled real-world videos. Given measured robot joint states, an uncertainty-aware EKF recursively corrects the instrument pose and the observable joint angles by comparing projected model features with detected keypoints, shaft boundaries, and mask-derived cues. An RTS smoother subsequently refines the resulting trajectory, which is projected into pseudo-labels for fine-tuning the feature detector without laborious pose annotations. Experiments on real-world videos demonstrate consistent improvements from self-training across all evaluated keypoint metrics, and the resulting model outperforms prior approaches in both accuracy and runtime. The code and data will be released upon publication.
Hanyang Hu, Zekai Liang, Florian Richter +1
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA.