Learning From a Steady Hand: A Weakly Supervised Agent for Robot Assistance under Microscopy
Authors: Huanyu Tian, Martin Huber, Lingyun Zeng, Zhe Han, Wayne Bennett, Giuseppe Silvestri, Gerardo Mendizabal-Ruiz, Tom Vercauteren, +2 more
Organizations: H. Tian, M. Huber, Z. Han, L. Zeng, T. Vercauteren, and C. Bergeles are with the School of Biomedical Engineering & Imaging Sciences, King’s College London, UK. · W. Bennett, G. Silvestri, G. Mendizabal-Ruiz, C. Bergeles, and A. Chavez-Badiola are with Conceivable Life Sciences, New York City, US and London, UK. · A. Chavez-Badiola is with Hope IVF Mexico, Mexico.
This paper rethinks steady-hand robotic manipulation by using a weakly supervised framework that fuses calibration-aware perception with admittance control. Unlike conventional automation that relies on labor-intensive 2D labeling, our framework leverages reusable warm-up trajectories to extract implicit spatial information, thereby achieving calibration-aware, depth-resolved perception without the need for external fiducials or frame-by-frame manual depth annotation. By explicitly characterizing residuals from observation and calibration models, the system establishes a task-space error budget from recorded warm-ups. Under the evaluated laboratory conditions, the recorded trajectories yielded an empirical lateral error level of approximately 49μm and a conservative axial error estimate of approximately 291 μm during large in-plane moves. In a within-subject user study (N=8), the learned agent reduces overall NASA-TLX workload by 77.1% relative to the simple steady-hand assistance baseline. Together, these results demonstrate successful closed-loop micromanipulation and reduced subjective workload under the evaluated laboratory conditions, without external fiducials or frame-by-frame manual depth annotation, supporting the proposed approach as a framework for microscope-guided robotic assistance.
Figures & tables
Figure 1: The data collection relies on “warm-up” episodes: short trajectories generated by the operator dragging the tool under steady-hand guidance. These maneuvers are standardized into two tasks: lateral sweeps for FOV coverage (the red line), and axial scans for controlled depth-axis motion (the orange line). The agent leverages warm-up trajectories to train the 3D tip observer and estimate hand-eye fusion parameters. Leveraging the dataset-learned calibration and tip recognition, the agent servos laterally to the operator-designated target (e.g., for aspiration) and regulates axial depth for station-keeping.
Features
Visual Servoing
Stereo- tactic
Steady-Hand
Perception
Markerless
✓
✗
✓
Occlusion robust
✗
✓
✓
Calibration-free
✓
✗
Few warm-ups
Control
Macro-to-micro
✗
✓
✓
Table I: Comparison of Features across Different Micromanipulation Paradigms.
Figure 2: To circumvent dense manual labeling, we utilize “steady-hand” warm-up clips. By manually prompting SAM 2 on only a few anchor frames (sparse annotation), the system propagates tip masks bidirectionally to generate dense, real-time 3D tip estimation targets.
Figure 3: We deploy a two-stage detector for real-time 3D tip estimation. The tip estimation block (TEB) returns the keypoint observations and the depth estimation block (DEB) returns the normalized depth from tip to the focal plane and an auxiliary classification label.
Figure 4: Dataset acquisition setup. (a) Upright microscope. (b-c) Inverted microscope with illumination control and a linear module for camera motion. (d-e) Steady-hand data collection under both setups.
Table III: Asynchrony-induced error (Reproj − Chamfer) and pose error (Bi-Chamfer, symmetric mean), computed from the trimmed statistics (top 10% removed). Euler angles are the optimized extrinsic rotations (rx,ry,rz) in radians.
(A) Tool Tip Detection at τdet=0.35 (Testing samples 8,604 @ 192 × 144)
Model
Median Err.
FPR (%)
FNR (%)
Conf. (mean)
PCK@6 (%)
PCK@10 (%)
PCK@15 (%)
EfficientViT (Orig.)
2.83
3.70
10.86
0.478
91.61
93.90
94.60
MobileVIT (Spatial Attn.)
2.24
6.94
6.92
0.619
94.42
95.07
95.31
EfficientViT (Spatial Attn.)
1.41
7.03
6.50
0.665
95.08
95.95
96.22
MobileNetV4
2.24
8.80
7.22
0.565
94.01
94.95
95.07
EfficientNet
2.36
11.57
8.63
0.553
93.72
94.48
94.72
Table IV: Unified comparison of (A) tool tip detection, (B) depth estimation and classification, and (C) adversarial robustness.
Figure 5: Robotic micromanipulation setup. A Meca500 arm with a loadcell operates under an inverted microscope. The system integrates manual macro-guidance and automated visual servoing.
Figure 6: Shared control workflow: (a-c) Manual macro-guidance brings the tool into the FOV; (d-f) Automated visual servoing refines position; (g-i) Microscope view during lateral alignment; (j-l) View during depth regulation.
Figure 7: Depth regulation performance. Top: Final positioning error vs. initial displacement (Mean ± Std over 7 trials). Bottom: Convergence time. Our method (blue) shows a linear descent trend, indicating robust coverage over a larger depth range compared to the non-linear baseline (orange).
Figure 8: 3D validation routine visualized in normalized task space. The lateral axis is normalized to [−2,2] and the depth axis to [0,1] . Clusters represent the steady-state landing positions at Center (Blue) and Edge (Orange) targets. The reference depth ( 0 mm, normalized to 0 ) is defined by the center of the bottom-most blue cluster (the initial low-depth starting point).
Figure 9: Lateral accuracy robustness. Error ellipses derived from paired Target-Actual clusters. We compare targets reached under Low-Depth (defocused) and High-Depth (focused) conditions. The comparable ellipse sizes ( σ≈10 px) indicate that lateral visual servoing remains precise even under optical blur.
Figure 10: Active axial constraint. Large lateral motions (top row) are achieved with minimal axial disturbance (bottom row), as the controller actively regulates ∣ez∣<30μm , validating the kinematic decoupling.
Figure 11: NASA-TLX workload distributions ( N=8 ). The proposed Agent method achieves the lowest scores across all subscales. Notably, the lower Performance (PE) score under the Agent condition indicates better perceived performance (Agent ≈8 vs. Manual ≈66 ).
Robotic manipulation often requires object specific programming, manual data annotation, or calibrated perception pipelines, which limits rapid deployment in practical settings. Learning from demonstration offers a more direct alternative, but collecting demonstrations can still demand human teleoperation or kinesthetic teaching. This paper presents a self supervised visual manipulation method in which a robot automatically generates demonstrations around a target pose and learns relative pose corrections directly from wrist mounted RGB images. The proposed pipeline uses ROS~2 and Isaac Sim to collect labeled image-pose pairs without requiring explicit camera to robot extrinsic calibration. Separate datasets are generated for planar refinement and coarse three dimensional approach, and a convolutional network is trained to regress relative translation and rotation from single frame RGB observations. During execution, a coarse to fine controller first approaches the object using models trained with height variation and then refines the final alignment using planar data. The method is evaluated both in simulation and on a real UR5e collaborative robot equipped with a gripper and a monocular camera. In simulation, the refinement stage reduces the final planar dispersion from 9.69 mm to 5.38 mm. In real world experiments, the system performs end to end grasp attempts on three physical objects and reaches success rates of 66.6% and 63.6% for two objects without object rotation, while still maintaining partial robustness under rotated conditions. These results show that automatically generated demonstrations can support practical visual manipulation with limited setup effort, while also exposing remaining challenges in depth prediction and object dependent generalization.
Andres Rivas, Anselmo R. Cukla, Rodrigo S. Guerra +2
Robotics and AI Lab, Technological University of Uruguay, Rivera, Uruguay · Center of Technology, Federal University of Santa Maria, Santa Maria, Brazil · Computer Science Center, Federal University of Rio Grande, Rio Grande, Brazil
Teleoperating a robotic manipulator in industrial environments demands precision that camera-based interfaces alone struggle to deliver. The operator must align the end-effector with a target in clutter, under limited depth perception, and without colliding with the surrounding structures. This paper presents a shared-autonomy framework that assists the operator throughout this process. A single RGB-D camera captures the operator's arm motion and hand gestures without wearables, fiducials, or a calibration stage. The intended target is specified by a free-form text prompt, grounded by a vision-language model in the robot's gripper camera, and tracked across its onboard cameras by a promptable video-segmentation model, resulting in a grasp frame continuously separated from the obstacle map. Every commanded motion is executed by a GPU-accelerated model-predictive controller that enforces self- and environment-collision avoidance against an online volumetric reconstruction, while a potential field corrects the operator's reference toward the grounded target during the final approach. An autonomous mode can be gesture-triggered to complete the grasp on the same target without a separate perception pipeline. The framework is validated on a quadruped mobile manipulator. The interface achieves a positional RMSE of 59 mm relative to motion-capture ground truth, and the controller keeps the arm at least 18 cm from obstacles while the operator deliberately commands the arm into them by 6 cm. In an industrial valve manipulation and a pick-and-place task, the full framework succeeded in all trials, while ablating either the collision or the assistance module produced failures through complementary mechanisms, and autonomous execution succeeded in four of five trials per task.
Murilo Vinicius da Silva, Ricardo V. Godoy, Juliano Negri +3
Department of Mechanical Engineering, University of São Paulo, São Carlos, Brazil · Instituto Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein, São Paulo, Brazil · Graduate School of Information Sciences, Tohoku University, Sendai, Japan.
Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)