SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Zhengdong Hong

cs.RONov 2, 2026

SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Authors: Zhengdong Hong

Organizations: Zhejiang University

Abstract

Markerless hand-eye calibration under continuous motion requires jointly resolving camera geometry and the time offset between images and joint states. We present SyncHeC (Synchronized Hand-Eye Calibration), a dense differentiable-rendering framework that estimates both from a single robot trajectory. Continuous-time forward kinematics makes latency differentiable, metric depth complements mask contours, and an offline planner combines spatial sensitivity with temporal excitation. Alternating updates support Eye-to-Hand, Eye-in-Hand, and multi-camera calibration. On a 7-DoF arm, one $6.5$ ,s motion yields $0.21$ ,cm translation error against a marker-board reference, $0.34$ ,cm touch-test error, and $2.7$ ,ms latency error relative to an $87$ ,ms reference with $95\%$ uncertainty of $\pm1.5$ ,ms. Against CM-Calib on matched recordings, translation and nominal latency errors improve by $3.7\times$ and $2.3\times$ . Experiments span seven arms and four depth sensors. Compensating the recovered offset during policy training and deployment increases success by $8.1$ percentage points across six families and six tasks ( $1{,}800$ matched pairs; paired $95\%$ confidence interval $[6.3,9.9]$ points).

Figures & tables

Figure 1: Robotic hand-eye coordination has two coupled facets. Left: spatial pose calibration recovers the camera pose in the robot frame, in both Eye-to-Hand and Eye-in-Hand setups. Right: temporal calibration recovers the sensor-to-control latency $\Delta t$ : a frame time-stamped at $t_{0}{+}\Delta t$ actually observes the joint state at $t_{0}$ . We solve both jointly.

Method	Marker-free	EiH	Latency	Hetero.	Dyn.
Marker-based ( Tsai and Lenz, 1989 )	✗	✓	✗	✓	✗
Pose networks ( Lee et al., 2020 ; Valassakis, Dreczkowski, and Johns, 2022 )	✓	✓	✗	✗	✗
EasyHeC ( Chen et al., 2023 )	✓	✗	✗	✓	✗
EasyHeC++ ( Hong, Zheng, and Chen, 2024 )	✓	✓	✗	✓	✗
CM-Calib ( Wu et al., 2020 )	✗	✓	✓	✓	✓
SyncHeC	✓	✓	✓	✓	✓

Table 1: Capability comparison of representative hand-eye calibration paradigms. Marker-free , EiH (Eye-in-Hand support), Latency (estimates

\Delta t

), Hetero. (heterogeneous arms), Dyn. (calibrates during continuous motion).

Figure 2: Overview. Stage 1 imports the URDF into a simulator, sizes an adaptive sampling sphere from the first real depth frame, selects an initial pose by a 2D–3D fitness, and refines it by differentiable rendering to obtain $\mathbf{T}^{0}$ . Stage 2 executes one offline-planned continuous trajectory, parameterizes the high-frequency joint stream as continuous-time forward kinematics, and jointly optimizes the extrinsics and latency $\Delta t$ from a 2D–3D differentiable-rendering loss.

Panel of Figure 3 — Figure 3: Complementary geometric constraints. Solid and dashed shapes show reference and axially shifted geometry. (a) Mask contours constrain axial translation through perspective, with weak sensitivity in distant or narrow projections. (b) Metric depth adds a direct surface-distance residual.

Eye-to-Hand
Method	#v	$E_{r}$ ( ∘ ) $\downarrow$	$E_{t}$ (cm) $\downarrow$
Marker-based ( Tsai and Lenz, 1989 )	20	0.870	2.000
DREAM ( Lee et al., 2020 )	5	0.704	0.303
EasyHeC ( Chen et al., 2023 )	5	0.081	0.206
EasyHeC++ ( Hong, Zheng, and Chen, 2024 )	5	0.045	0.135
SyncHeC ( $\Delta t{=}0$ )	5	0.038	0.105

Table 2: Simulation calibration accuracy. Top: Eye-to-Hand. Bottom: Eye-in-Hand. #v: number of views. Baseline values are reproduced from the EasyHeC++ evaluation ( Hong, Zheng, and Chen, 2024 ) .

Method	EtH	EiH
DREAM ( Lee et al., 2020 )	1.50	—
Zhang et al. (2023)	—	1.35
Valassakis, Dreczkowski, and Johns (2022)	—	4.30
EasyHeC ( Chen et al., 2023 )	0.40	—
EasyHeC++ ( Hong, Zheng, and Chen, 2024 )	0.30	0.31
Hydra ( Huber et al., 2025 )	0.35	0.28

Table 3: Real static touch-test error (cm), averaged over ten poses. Hydra and SyncHeC are evaluated here; the other results are reproduced from Hong, Zheng, and Chen (2024) . Protocols are specified in Appendix D.3 .

Eye-to-Hand
Method	$E_{t}$ (cm) $\downarrow$	$E_{r}$ ( ∘ ) $\downarrow$	$E_{\text{lat}}$ (ms) $\downarrow$
SL-Calib ‡	1.15	0.92	11.5
CM-Calib ( Wu et al., 2020 )	0.78 $\pm$ 0.08	0.55	6.2 $\pm$ 0.8
EasyHeC † ( Chen et al., 2023 )	4.85	3.42	—
EasyHeC++ † ( Hong, Zheng, and Chen, 2024 )	3.92	2.75	—
SyncHeC	0.21 $\pm$ 0.04	0.15 $\pm$ 0.05	2.7 $\pm$ 0.6

Table 4: Continuous-motion calibration against the

87

ms reference (

U_{95}=1.5

ms). CM-Calib and ours use five matched recordings; SDs are shown for CM translation/latency and all our metrics.

Figure 4: Heterogeneous multi-camera co-calibration. Left: the rig, with the tripod-mounted Eye-to-Hand and wrist-mounted Eye-in-Hand cameras labelled. Right: frames from each camera over one continuous trajectory.

Platform	$E_{t}$ (cm)	$E_{r}$ ( ∘ )	$\Delta t^{*}$	$\Delta t_{\text{ref}}$
xArm7	0.25	0.18	85.6	87.0
Franka Panda	0.31	0.22	86.8	88.5
RM75-B	0.28	0.19	86.1	87.5
Z1 Pro	0.68	0.45	84.2	89.0
EC66	0.36	0.25	86.5	88.0
ER7	0.24	0.16	87.3	88.5

Table 5: Representative runs across arms (top) and depth sensors (bottom). xArm7/D435i is repeated as the shared reference, giving ten distinct configurations. Each configuration has five motions; repeat summaries appear in the text. Offsets are in ms.

Variant	EtH			EiH
Variant	$E_{t}$	$E_{r}$	$E_{\rm lat}$	$E_{t}$	$E_{r}$	$E_{\rm lat}$
Stage-2 depth off	1.18	0.85	3.5	1.35	0.96	8.2
Spatial planning only	0.86	0.65	16.4	0.95	0.72	18.2
Full (Ours)	0.21	0.15	2.7	0.24	0.18	5.4

Table 6: Dynamic ablations with the Stage-1 anchor fixed.

E_{t}

is in cm,

E_{r}

in degrees, and

E_{\rm lat}

in ms. The full row repeats the five-run means in Table 4 .

Policy	Stack	Cup	Ban.	Kettle	Drawer	Towel
ACT ( Zhao et al., 2023 )	$6\!\to\!8$	$44\!\to\!50$	$40\!\to\!44$	$42\!\to\!46$	$38\!\to\!42$	$34\!\to\!40$
DP ( Chi et al., 2025 )	$12\!\to\!16$	$58\!\to\!64$	$48\!\to\!56$	$64\!\to\!68$	$46\!\to\!54$	$36\!\to\!52$
$\pi_{0}$ ( Black et al., 2025 )	$16\!\to\!22$	$54\!\to\!66$	$50\!\to\!62$	$46\!\to\!62$	$48\!\to\!60$	$62\!\to\!74$
DP3 ( Ze et al., 2024 )	$2\!\to\!4$	$12\!\to\!16$	$42\!\to\!48$	$40\!\to\!48$	$28\!\to\!32$	$54\!\to\!66$
ManiFlow ( Yan et al., 2025 )	$10\!\to\!14$	$52\!\to\!60$	$46\!\to\!54$	$36\!\to\!50$	$46\!\to\!54$	$60\!\to\!64$
WM †	$8\!\to\!14$	$40\!\to\!54$	$38\!\to\!50$	$34\!\to\!48$	$40\!\to\!52$	$30\!\to\!40$

Table 7: Success rate (%) without

\to

with recovered-offset compensation. Each cell uses

50

matched pairs on the six tasks: stack three blocks; place cup, banana, or kettle; open drawer; fold towel. Across

1800

pairs, the gain is

8.11

points (paired

95\%

[6.3,9.9]

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Temporal updates	0	225	450	675	900
Offset (ms)	0	43.8	78.1	84.0	84.3

Appendix

Table 8: Representative EtH optimization trace. The count is the cumulative number of temporal optimizer updates.

Component	Function
Calibration objective	optimize.CalibrationProblem
Joint interpolation	kinematics.q_sync
Temporal derivative	kinematics.temporal_jacobian
EtH / EiH transforms	kinematics.LinkPoseModel
Mask / depth losses	render.losses
Initialization fitness	stage1.joint_fitness

Appendix

Table 9: Core mathematical components in SyncHeC.

Trajectory	EtH	EiH
Spatial excitation only ( $\beta=0$ )	0.03	0.02
Spatial and temporal excitation	0.21	0.18

Appendix

Table 10: Minimum singular value of the normalized, stacked data Jacobian.

Component	ms
Exposure phase, including RGB–depth skew	0.25
Clock mapping	0.10
Physical motion-onset localization	0.20
Repeated-mean $95\%$ half-width	0.95
Combined $95\%$ uncertainty $U_{95}$	1.50

Appendix

Table 11: Uncertainty budget for the nominal

87

ms reference. The first three entries are residual bias bounds; the fourth is a confidence half-width for the repeated mean.

Task	Regime	Randomization of the initial configuration	Success criterion
Stack three blocks	long-horizon	three $20{\times}40$ cm cells, $\pm 5$ cm jitter	all three stacked, tower stable at timeout
Place cup	pose-constrained	$60{\times}40$ cm board, $360^{\circ}$ yaw	cup upright inside the target region
Place banana on plate	pose-constrained	plate on midline $+10$ cm; banana $10$ – $15$ cm left, $360^{\circ}$ yaw	banana resting on the plate
Place kettle on stove	pose-constrained	$20{\times}20$ cm region, $\pm 60^{\circ}$	kettle upright on the stove plate
Open drawer	articulated	L-shaped region ( $70{\times}20$ $+$ $20{\times}20$ cm), rotated $90^{\circ}$	drawer pulled to $\geq 50\%$ extension
Fold towel	deformable	$70{\times}50$ cm region, $\pm 30^{\circ}$	towel folded, free edge within tolerance

Appendix

Table 12: The six real-robot tasks: regime, randomization of the initial object configuration, and success criterion. Yaw ranges are about the vertical axis.

	Compensated
Uncompensated	Success	Failure
Success	617	64
Failure	210	909

Appendix

Table 13: Aggregate paired outcomes over

1800

initial configurations.

Hand-Eye Calibration

Hand-eye calibration estimates the rigid transform between a robot and its camera or sensor for accurate perception-guided motion.

See all 15 papers

DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients
Sep 29, 2026Xiaotian Zhang, Yusheng Wang, Naoya Kagawa +4Differentiable Rendering Hand-Eye Calibration
Department of Precision Engineering, School of Engineering, The University of Tokyo, Tokyo, Japan · Research into Artifacts, Center for Engineering (RACE), School of Engineering, The University of Tokyo, Tokyo, Japan · FA Products Business Unit, DENSO WAVE INCORPORATED, Aichi, Japan
Hydra: Marker-Free RGB-D Hand-Eye Calibration
Apr 29, 2025Martin Huber, Huanyu Tian, Christopher E. Mower +4Hand-Eye Calibration Robotics
School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK · Noah’s Ark Lab, Huawei, London, UK · independent contributor
AnchorHEC: Targetless Hand-Eye Calibration of Wrist and Fixed Cameras
Nov 2, 2026Guebin Hwang, Taeyeop Lee, Yeonseo Lee +2Hand-Eye Calibration Camera Pose Estimation
All authors are with the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea.

Sep 29, 2026J/K move · Enter open · S save

DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

Xiaotian Zhang, Yusheng Wang, Naoya Kagawa, Noritaka Takamura+3

Department of Precision Engineering, School of Engineering, The University of Tokyo, Tokyo, Japan · Research into Artifacts, Center for Engineering (RACE), School of Engineering, The University of Tokyo, Tokyo, Japan · FA Products Business Unit, DENSO WAVE INCORPORATED, Aichi, Japan

Differentiable Rendering Hand-Eye Calibration Camera Calibration

Open paper AskPDF arXiv

Accurate hand-eye calibration is crucial for precision manipulation. Traditional methods rely on markers, with their precision dependent on marker accuracy and observability. In contrast, markerless methods, such as learning-based approaches, use deep neural networks to directly extract keypoints or features from images, enabling the computation of hand-eye transformation with a single image and without the need for physical markers. Recently, differentiable rendering-based methods for hand-eye calibration have leveraged physical models to render binary masks and compare them with observations, enabling hand-eye calibration without fiducial markers in the calibration stage and providing interpretable optimization. While the state-of-the-art differentiable rendering methods achieve remarkable accuracy, the use of binary masks can result in the loss of internal profile details, reducing precision. Additionally, these methods can also suffer from unstable optimization and local minima. In this study, we propose a novel RGB-based differentiable rendering framework that provides richer geometric and appearance cues by incorporating color and mask geometric features, thereby improving calibration accuracy and optimization stability. Additionally, we propose a mask-guided image-to-image translation method to ensure explicit preservation of color and geometric consistency throughout the translation. Our approach is validated through both simulation and real-world experiments, with results demonstrating strong accuracy and robustness and clear improvements over existing differentiable rendering methods. Our method achieves a grasping success rate of 88.9% and insertion success rate of 57.4% on the UR5e real-world experiment, outperforming the state-of-the-art differentiable rendering hand-eye calibration method EasyHeC by 46.3 and 48.1 percentage points, respectively.

Abstract

Hand-Eye Calibration

Explore similar work

DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

Hydra: Marker-Free RGB-D Hand-Eye Calibration

AnchorHEC: Targetless Hand-Eye Calibration of Wrist and Fixed Cameras