SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Zhengdong Hong

cs.RONov 2, 2026

SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Authors: Zhengdong Hong

Organizations: Zhejiang University

Abstract

Markerless hand-eye calibration under continuous motion requires jointly resolving camera geometry and the time offset between images and joint states. We present SyncHeC (Synchronized Hand-Eye Calibration), a dense differentiable-rendering framework that estimates both from a single robot trajectory. Continuous-time forward kinematics makes latency differentiable, metric depth complements mask contours, and an offline planner combines spatial sensitivity with temporal excitation. Alternating updates support Eye-to-Hand, Eye-in-Hand, and multi-camera calibration. On a 7-DoF arm, one $6.5$ ,s motion yields $0.21$ ,cm translation error against a marker-board reference, $0.34$ ,cm touch-test error, and $2.7$ ,ms latency error relative to an $87$ ,ms reference with $95\%$ uncertainty of $\pm1.5$ ,ms. Against CM-Calib on matched recordings, translation and nominal latency errors improve by $3.7\times$ and $2.3\times$ . Experiments span seven arms and four depth sensors. Compensating the recovered offset during policy training and deployment increases success by $8.1$ percentage points across six families and six tasks ( $1{,}800$ matched pairs; paired $95\%$ confidence interval $[6.3,9.9]$ points).

Figures & tables

Figure 1: Robotic hand-eye coordination has two coupled facets. Left: spatial pose calibration recovers the camera pose in the robot frame, in both Eye-to-Hand and Eye-in-Hand setups. Right: temporal calibration recovers the sensor-to-control latency $\Delta t$ : a frame time-stamped at $t_{0}{+}\Delta t$ actually observes the joint state at $t_{0}$ . We solve both jointly.

Method	Marker-free	EiH	Latency	Hetero.	Dyn.
Marker-based ( Tsai and Lenz, 1989 )	✗	✓	✗	✓	✗
Pose networks ( Lee et al., 2020 ; Valassakis, Dreczkowski, and Johns, 2022 )	✓	✓	✗	✗	✗
EasyHeC ( Chen et al., 2023 )	✓	✗	✗	✓	✗
EasyHeC++ ( Hong, Zheng, and Chen, 2024 )	✓	✓	✗	✓	✗
CM-Calib ( Wu et al., 2020 )	✗	✓	✓	✓	✓
SyncHeC	✓	✓	✓	✓	✓

Table 1: Capability comparison of representative hand-eye calibration paradigms. Marker-free , EiH (Eye-in-Hand support), Latency (estimates

\Delta t

), Hetero. (heterogeneous arms), Dyn. (calibrates during continuous motion).

Figure 2: Overview. Stage 1 imports the URDF into a simulator, sizes an adaptive sampling sphere from the first real depth frame, selects an initial pose by a 2D–3D fitness, and refines it by differentiable rendering to obtain $\mathbf{T}^{0}$ . Stage 2 executes one offline-planned continuous trajectory, parameterizes the high-frequency joint stream as continuous-time forward kinematics, and jointly optimizes the extrinsics and latency $\Delta t$ from a 2D–3D differentiable-rendering loss.

Panel of Figure 3 — Figure 3: Complementary geometric constraints. Solid and dashed shapes show reference and axially shifted geometry. (a) Mask contours constrain axial translation through perspective, with weak sensitivity in distant or narrow projections. (b) Metric depth adds a direct surface-distance residual.

Eye-to-Hand
Method	#v	$E_{r}$ ( ∘ ) $\downarrow$	$E_{t}$ (cm) $\downarrow$
Marker-based ( Tsai and Lenz, 1989 )	20	0.870	2.000
DREAM ( Lee et al., 2020 )	5	0.704	0.303
EasyHeC ( Chen et al., 2023 )	5	0.081	0.206
EasyHeC++ ( Hong, Zheng, and Chen, 2024 )	5	0.045	0.135
SyncHeC ( $\Delta t{=}0$ )	5	0.038	0.105

Table 2: Simulation calibration accuracy. Top: Eye-to-Hand. Bottom: Eye-in-Hand. #v: number of views. Baseline values are reproduced from the EasyHeC++ evaluation ( Hong, Zheng, and Chen, 2024 ) .

Method	EtH	EiH
DREAM ( Lee et al., 2020 )	1.50	—
Zhang et al. (2023)	—	1.35
Valassakis, Dreczkowski, and Johns (2022)	—	4.30
EasyHeC ( Chen et al., 2023 )	0.40	—
EasyHeC++ ( Hong, Zheng, and Chen, 2024 )	0.30	0.31
Hydra ( Huber et al., 2025 )	0.35	0.28

Table 3: Real static touch-test error (cm), averaged over ten poses. Hydra and SyncHeC are evaluated here; the other results are reproduced from Hong, Zheng, and Chen (2024) . Protocols are specified in Appendix D.3 .

Eye-to-Hand
Method	$E_{t}$ (cm) $\downarrow$	$E_{r}$ ( ∘ ) $\downarrow$	$E_{\text{lat}}$ (ms) $\downarrow$
SL-Calib ‡	1.15	0.92	11.5
CM-Calib ( Wu et al., 2020 )	0.78 $\pm$ 0.08	0.55	6.2 $\pm$ 0.8
EasyHeC † ( Chen et al., 2023 )	4.85	3.42	—
EasyHeC++ † ( Hong, Zheng, and Chen, 2024 )	3.92	2.75	—
SyncHeC	0.21 $\pm$ 0.04	0.15 $\pm$ 0.05	2.7 $\pm$ 0.6

Table 4: Continuous-motion calibration against the

87

ms reference (

U_{95}=1.5

ms). CM-Calib and ours use five matched recordings; SDs are shown for CM translation/latency and all our metrics.

Figure 4: Heterogeneous multi-camera co-calibration. Left: the rig, with the tripod-mounted Eye-to-Hand and wrist-mounted Eye-in-Hand cameras labelled. Right: frames from each camera over one continuous trajectory.

Platform	$E_{t}$ (cm)	$E_{r}$ ( ∘ )	$\Delta t^{*}$	$\Delta t_{\text{ref}}$
xArm7	0.25	0.18	85.6	87.0
Franka Panda	0.31	0.22	86.8	88.5
RM75-B	0.28	0.19	86.1	87.5
Z1 Pro	0.68	0.45	84.2	89.0
EC66	0.36	0.25	86.5	88.0
ER7	0.24	0.16	87.3	88.5

Table 5: Representative runs across arms (top) and depth sensors (bottom). xArm7/D435i is repeated as the shared reference, giving ten distinct configurations. Each configuration has five motions; repeat summaries appear in the text. Offsets are in ms.

Variant	EtH			EiH
Variant	$E_{t}$	$E_{r}$	$E_{\rm lat}$	$E_{t}$	$E_{r}$	$E_{\rm lat}$
Stage-2 depth off	1.18	0.85	3.5	1.35	0.96	8.2
Spatial planning only	0.86	0.65	16.4	0.95	0.72	18.2
Full (Ours)	0.21	0.15	2.7	0.24	0.18	5.4

Table 6: Dynamic ablations with the Stage-1 anchor fixed.

E_{t}

is in cm,

E_{r}

in degrees, and

E_{\rm lat}

in ms. The full row repeats the five-run means in Table 4 .

Policy	Stack	Cup	Ban.	Kettle	Drawer	Towel
ACT ( Zhao et al., 2023 )	$6\!\to\!8$	$44\!\to\!50$	$40\!\to\!44$	$42\!\to\!46$	$38\!\to\!42$	$34\!\to\!40$
DP ( Chi et al., 2025 )	$12\!\to\!16$	$58\!\to\!64$	$48\!\to\!56$	$64\!\to\!68$	$46\!\to\!54$	$36\!\to\!52$
$\pi_{0}$ ( Black et al., 2025 )	$16\!\to\!22$	$54\!\to\!66$	$50\!\to\!62$	$46\!\to\!62$	$48\!\to\!60$	$62\!\to\!74$
DP3 ( Ze et al., 2024 )	$2\!\to\!4$	$12\!\to\!16$	$42\!\to\!48$	$40\!\to\!48$	$28\!\to\!32$	$54\!\to\!66$
ManiFlow ( Yan et al., 2025 )	$10\!\to\!14$	$52\!\to\!60$	$46\!\to\!54$	$36\!\to\!50$	$46\!\to\!54$	$60\!\to\!64$
WM †	$8\!\to\!14$	$40\!\to\!54$	$38\!\to\!50$	$34\!\to\!48$	$40\!\to\!52$	$30\!\to\!40$

Table 7: Success rate (%) without

\to

with recovered-offset compensation. Each cell uses

50

matched pairs on the six tasks: stack three blocks; place cup, banana, or kettle; open drawer; fold towel. Across

1800

pairs, the gain is

8.11

points (paired

95\%

[6.3,9.9]

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Temporal updates	0	225	450	675	900
Offset (ms)	0	43.8	78.1	84.0	84.3

Appendix

Table 8: Representative EtH optimization trace. The count is the cumulative number of temporal optimizer updates.

Component	Function
Calibration objective	optimize.CalibrationProblem
Joint interpolation	kinematics.q_sync
Temporal derivative	kinematics.temporal_jacobian
EtH / EiH transforms	kinematics.LinkPoseModel
Mask / depth losses	render.losses
Initialization fitness	stage1.joint_fitness

Appendix

Table 9: Core mathematical components in SyncHeC.

Trajectory	EtH	EiH
Spatial excitation only ( $\beta=0$ )	0.03	0.02
Spatial and temporal excitation	0.21	0.18

Appendix

Table 10: Minimum singular value of the normalized, stacked data Jacobian.

Component	ms
Exposure phase, including RGB–depth skew	0.25
Clock mapping	0.10
Physical motion-onset localization	0.20
Repeated-mean $95\%$ half-width	0.95
Combined $95\%$ uncertainty $U_{95}$	1.50

Appendix

Table 11: Uncertainty budget for the nominal

87

ms reference. The first three entries are residual bias bounds; the fourth is a confidence half-width for the repeated mean.

Task	Regime	Randomization of the initial configuration	Success criterion
Stack three blocks	long-horizon	three $20{\times}40$ cm cells, $\pm 5$ cm jitter	all three stacked, tower stable at timeout
Place cup	pose-constrained	$60{\times}40$ cm board, $360^{\circ}$ yaw	cup upright inside the target region
Place banana on plate	pose-constrained	plate on midline $+10$ cm; banana $10$ – $15$ cm left, $360^{\circ}$ yaw	banana resting on the plate
Place kettle on stove	pose-constrained	$20{\times}20$ cm region, $\pm 60^{\circ}$	kettle upright on the stove plate
Open drawer	articulated	L-shaped region ( $70{\times}20$ $+$ $20{\times}20$ cm), rotated $90^{\circ}$	drawer pulled to $\geq 50\%$ extension
Fold towel	deformable	$70{\times}50$ cm region, $\pm 30^{\circ}$	towel folded, free edge within tolerance

Appendix

Table 12: The six real-robot tasks: regime, randomization of the initial object configuration, and success criterion. Yaw ranges are about the vertical axis.

	Compensated
Uncompensated	Success	Failure
Success	617	64
Failure	210	909

Appendix

Table 13: Aggregate paired outcomes over

1800

initial configurations.

Hand-Eye Calibration

Hand-eye calibration estimates the rigid transform between a robot and its camera or sensor for accurate perception-guided motion.

See all 15 papers

Sep 29, 2026

DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

Accurate hand-eye calibration is crucial for precision manipulation. Traditional methods rely on markers, with their precision dependent on marker accuracy and observability. In contrast, markerless methods, such as learning-based approaches, use deep neural networks to directly extract keypoints or features from images, enabling the computation of hand-eye transformation with a single image and without the need for physical markers. Recently, differentiable rendering-based methods for hand-eye calibration have leveraged physical models to render binary masks and compare them with observations, enabling hand-eye calibration without fiducial markers in the calibration stage and providing interpretable optimization. While the state-of-the-art differentiable rendering methods achieve remarkable accuracy, the use of binary masks can result in the loss of internal profile details, reducing precision. Additionally, these methods can also suffer from unstable optimization and local minima. In this study, we propose a novel RGB-based differentiable rendering framework that provides richer geometric and appearance cues by incorporating color and mask geometric features, thereby improving calibration accuracy and optimization stability. Additionally, we propose a mask-guided image-to-image translation method to ensure explicit preservation of color and geometric consistency throughout the translation. Our approach is validated through both simulation and real-world experiments, with results demonstrating strong accuracy and robustness and clear improvements over existing differentiable rendering methods. Our method achieves a grasping success rate of 88.9% and insertion success rate of 57.4% on the UR5e real-world experiment, outperforming the state-of-the-art differentiable rendering hand-eye calibration method EasyHeC by 46.3 and 48.1 percentage points, respectively.

Xiaotian Zhang, Yusheng Wang, Naoya Kagawa +4

Department of Precision Engineering, School of Engineering, The University of Tokyo, Tokyo, Japan · Research into Artifacts, Center for Engineering (RACE), School of Engineering, The University of Tokyo, Tokyo, Japan · FA Products Business Unit, DENSO WAVE INCORPORATED, Aichi, Japan

Differentiable Rendering Hand-Eye Calibration Camera Calibration

Ask PDF

Apr 29, 2025

Hydra: Marker-Free RGB-D Hand-Eye Calibration

This work presents an RGB-D imaging-based approach to marker-free hand-eye calibration using a novel implementation of the iterative closest point (ICP) algorithm with a robust point-to-plane (PTP) objective formulated on a Lie algebra. Its applicability is demonstrated through comprehensive experiments using three well known serial manipulators and two RGB-D cameras. With only three randomly chosen robot configurations, our approach achieves approximately 90% successful calibrations, demonstrating 2-3x higher convergence rates to the global optimum compared to both marker-based and marker-free baselines. We also report 2 orders of magnitude faster convergence time (0.8 +/- 0.4 s) for 9 robot configurations over other marker-free methods. Our method exhibits significantly improved accuracy (5 mm in task space) over classical approaches (7 mm in task space) whilst being marker-free. The benchmarking dataset and code are open sourced under Apache 2.0 License, and a ROS 2 integration with robot abstraction is provided to facilitate deployment.

Martin Huber, Huanyu Tian, Christopher E. Mower +4

School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK · Noah’s Ark Lab, Huawei, London, UK · independent contributor

Hand-Eye Calibration Robotics Robot Calibration

Ask PDF

Nov 2, 2026

AnchorHEC: Targetless Hand-Eye Calibration of Wrist and Fixed Cameras

Recent robot systems combine wrist and fixed cameras, requiring camera-to-robot transforms for both. Most hand-eye calibration methods rely on targets or robot-body observations. Scene-based methods using 3D reconstruction require neither but address only wrist cameras. We introduce AnchorHEC, a targetless hand-eye calibration framework for wrist and fixed cameras in a single procedure using RGB images, forward kinematics (FK), and known camera intrinsics. We initialize both camera types using FK and a shared 3D foundation model reconstruction. FK-constrained wrist-image bundle adjustment reduces the impact of reconstructed pose and scale errors. If this refinement fails or retains too few inliers, we relax the constraints on potentially inaccurate FK poses and lens parameters. We use the refined wrist transform and FK to triangulate metric points and estimate fixed-camera poses. Across three backbones, calibration errors in simulation decrease by 86.4% in translation and 71.6% in rotation, averaged over both camera types, relative to reconstruction-based baselines. We also validate AnchorHEC on a physical robot. Project page: https://hwanggb.github.io/AnchorHEC/

Guebin Hwang, Taeyeop Lee, Yeonseo Lee +2

All authors are with the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea.

Hand-Eye Calibration Camera Pose Estimation

Ask PDF

Abstract

Hand-Eye Calibration

Explore similar work

DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

Hydra: Marker-Free RGB-D Hand-Eye Calibration

AnchorHEC: Targetless Hand-Eye Calibration of Wrist and Fixed Cameras