Learning From a Steady Hand: A Weakly Supervised Agent for Robot Assistance under Microscopy
Authors: Huanyu Tian, Martin Huber, Lingyun Zeng, Zhe Han, Wayne Bennett, Giuseppe Silvestri, Gerardo Mendizabal-Ruiz, Tom Vercauteren, +2 more
Organizations: H. Tian, M. Huber, Z. Han, L. Zeng, T. Vercauteren, and C. Bergeles are with the School of Biomedical Engineering & Imaging Sciences, King’s College London, UK. · W. Bennett, G. Silvestri, G. Mendizabal-Ruiz, C. Bergeles, and A. Chavez-Badiola are with Conceivable Life Sciences, New York City, US and London, UK. · A. Chavez-Badiola is with Hope IVF Mexico, Mexico.
This paper rethinks steady-hand robotic manipulation by using a weakly supervised framework that fuses calibration-aware perception with admittance control. Unlike conventional automation that relies on labor-intensive 2D labeling, our framework leverages reusable warm-up trajectories to extract implicit spatial information, thereby achieving calibration-aware, depth-resolved perception without the need for external fiducials or frame-by-frame manual depth annotation. By explicitly characterizing residuals from observation and calibration models, the system establishes a task-space error budget from recorded warm-ups. Under the evaluated laboratory conditions, the recorded trajectories yielded an empirical lateral error level of approximately 49μm and a conservative axial error estimate of approximately 291 μm during large in-plane moves. In a within-subject user study (N=8), the learned agent reduces overall NASA-TLX workload by 77.1% relative to the simple steady-hand assistance baseline. Together, these results demonstrate successful closed-loop micromanipulation and reduced subjective workload under the evaluated laboratory conditions, without external fiducials or frame-by-frame manual depth annotation, supporting the proposed approach as a framework for microscope-guided robotic assistance.
Figures & tables
Figure 1: The data collection relies on “warm-up” episodes: short trajectories generated by the operator dragging the tool under steady-hand guidance. These maneuvers are standardized into two tasks: lateral sweeps for FOV coverage (the red line), and axial scans for controlled depth-axis motion (the orange line). The agent leverages warm-up trajectories to train the 3D tip observer and estimate hand-eye fusion parameters. Leveraging the dataset-learned calibration and tip recognition, the agent servos laterally to the operator-designated target (e.g., for aspiration) and regulates axial depth for station-keeping.
Features
Visual Servoing
Stereo- tactic
Steady-Hand
Perception
Markerless
✓
✗
✓
Occlusion robust
✗
✓
✓
Calibration-free
✓
✗
Few warm-ups
Control
Macro-to-micro
✗
✓
✓
Table I: Comparison of Features across Different Micromanipulation Paradigms.
Figure 2: To circumvent dense manual labeling, we utilize “steady-hand” warm-up clips. By manually prompting SAM 2 on only a few anchor frames (sparse annotation), the system propagates tip masks bidirectionally to generate dense, real-time 3D tip estimation targets.
Figure 3: We deploy a two-stage detector for real-time 3D tip estimation. The tip estimation block (TEB) returns the keypoint observations and the depth estimation block (DEB) returns the normalized depth from tip to the focal plane and an auxiliary classification label.
Figure 4: Dataset acquisition setup. (a) Upright microscope. (b-c) Inverted microscope with illumination control and a linear module for camera motion. (d-e) Steady-hand data collection under both setups.
Table III: Asynchrony-induced error (Reproj − Chamfer) and pose error (Bi-Chamfer, symmetric mean), computed from the trimmed statistics (top 10% removed). Euler angles are the optimized extrinsic rotations (rx,ry,rz) in radians.
(A) Tool Tip Detection at τdet=0.35 (Testing samples 8,604 @ 192 × 144)
Model
Median Err.
FPR (%)
FNR (%)
Conf. (mean)
PCK@6 (%)
PCK@10 (%)
PCK@15 (%)
EfficientViT (Orig.)
2.83
3.70
10.86
0.478
91.61
93.90
94.60
MobileVIT (Spatial Attn.)
2.24
6.94
6.92
0.619
94.42
95.07
95.31
EfficientViT (Spatial Attn.)
1.41
7.03
6.50
0.665
95.08
95.95
96.22
MobileNetV4
2.24
8.80
7.22
0.565
94.01
94.95
95.07
EfficientNet
2.36
11.57
8.63
0.553
93.72
94.48
94.72
Table IV: Unified comparison of (A) tool tip detection, (B) depth estimation and classification, and (C) adversarial robustness.
Figure 5: Robotic micromanipulation setup. A Meca500 arm with a loadcell operates under an inverted microscope. The system integrates manual macro-guidance and automated visual servoing.
Figure 6: Shared control workflow: (a-c) Manual macro-guidance brings the tool into the FOV; (d-f) Automated visual servoing refines position; (g-i) Microscope view during lateral alignment; (j-l) View during depth regulation.
Figure 7: Depth regulation performance. Top: Final positioning error vs. initial displacement (Mean ± Std over 7 trials). Bottom: Convergence time. Our method (blue) shows a linear descent trend, indicating robust coverage over a larger depth range compared to the non-linear baseline (orange).
Figure 8: 3D validation routine visualized in normalized task space. The lateral axis is normalized to [−2,2] and the depth axis to [0,1] . Clusters represent the steady-state landing positions at Center (Blue) and Edge (Orange) targets. The reference depth ( 0 mm, normalized to 0 ) is defined by the center of the bottom-most blue cluster (the initial low-depth starting point).
Figure 9: Lateral accuracy robustness. Error ellipses derived from paired Target-Actual clusters. We compare targets reached under Low-Depth (defocused) and High-Depth (focused) conditions. The comparable ellipse sizes ( σ≈10 px) indicate that lateral visual servoing remains precise even under optical blur.
Figure 10: Active axial constraint. Large lateral motions (top row) are achieved with minimal axial disturbance (bottom row), as the controller actively regulates ∣ez∣<30μm , validating the kinematic decoupling.
Figure 11: NASA-TLX workload distributions ( N=8 ). The proposed Agent method achieves the lowest scores across all subscales. Notably, the lower Performance (PE) score under the Agent condition indicates better perceived performance (Agent ≈8 vs. Manual ≈66 ).
Robotics and AI Lab, Technological University of Uruguay, Rivera, Uruguay · Center of Technology, Federal University of Santa Maria, Santa Maria, Brazil · Computer Science Center, Federal University of Rio Grande, Rio Grande, Brazil
Department of Mechanical Engineering, University of São Paulo, São Carlos, Brazil · Instituto Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein, São Paulo, Brazil · Graduate School of Information Sciences, Tohoku University, Sendai, Japan.