Tele-Catch: Adaptive Teleoperation for Dexterous Dynamic 3D Object Catching

Weiguang Zhao; Junting Dong; Rui Zhang; Kailin Li; Qin Zhao; Kaizhu Huang

cs.ROMar 30, 2026

Tele-Catch: Adaptive Teleoperation for Dexterous Dynamic 3D Object Catching

Authors: Weiguang Zhao, Junting Dong, Rui Zhang, Kailin Li, Qin Zhao, Kaizhu Huang

Organizations: University of Liverpool · Xi’an Jiaotong-Liverpool University · Shanghai AI Lab · Zhejiang University ·

Abstract

Teleoperation is a key paradigm for transferring human dexterity to robots, yet most prior work targets objects that are initially static, such as grasping or manipulation. Dynamic object catch, where objects move before contact, remains underexplored. Pure teleoperation in this task often fails due to timing, pose, and force errors, highlighting the need for shared autonomy that combines human input with autonomous policies. To this end, we present Tele-Catch, a systematic framework for dexterous hand teleoperation in dynamic object catching. At its core, we design DAIM, a dynamics-aware adaptive integration mechanism that realizes shared autonomy by fusing glove-based teleoperation signals into the diffusion policy denoising process. It adaptively modulates control based on the interaction object state. To improve policy robustness, we introduce DP-U3R, which integrates unsupervised geometric representations from point cloud observations into diffusion policy learning, enabling geometry-aware decision making. Extensive experiments demonstrate that Tele-Catch significantly improves accuracy and robustness in dynamic catching tasks, while also exhibiting consistent gains across distinct dexterous hand embodiments and previously unseen object categories.

Figures & tables

Figure 1: Teleoperation in Dexterous Manipulation. (a) Retargeting maps human hand motions to robotic hands for teleoperation. (b) Teleoperation with static object interaction. (c) Teleoperation with dynamic object interaction (our focus). Four representative cases illustrate the key challenges in teleoperation with dynamic objects: (1) grasping too early, (2) incorrect grasp pose, (3) excessive grasping force, and (4) retargeting errors between human and robotic hands, all of which eventually cause the object to slip or fall.

Method	Venue	Shared-Aut.	Deployment	Dynamic-Cat.	Visual	Unseen-Obj.	Hand-Types	DOF
DSA [ 25 ]	TRO’16	✗	Sim.+Real	✓	2D	✗	Single	16
Bi-DexHands [ 2 ]	TPAMI’23	✗	Sim.	✓	3D	✗	Single	16
DP3 [ 35 ]	RSS’24	✗	Sim.+Real	✗	3D	✗	Single	16
DexterityGen [ 33 ]	RSS’25	Fixed	Sim.+Real	✗	✗	✓	Single	16
DexHandDiff [ 23 ]	CVPR’25	✗	Sim.	✗	✗	✗	Single	24
Tele-Catch (Ours)	–	Adaptive	Sim.+Real	✓	3D	✓	Multi	16/24

Table 1: Comparison with main related work. Shared-Aut. is Shared-Autonomy. And Obj. stands for Objects. Dynamic-Cat. denotes Dynamic-Catching Task. Sim. and Real denote deployment in simulation and on real robots, respectively.

Figure 2: Our Method. For clarity, we present our framework by dividing it into four components: (a) Train RL Policy (b) Data Collection (c) Train DP-U3R (d) Teleoperation. Recon. stands for Reconstruction and Seq. is Sequence.

Methods	Ball	Bear	Cookie	Cup	Fish	Ipod	Showerhead	Teapot	Truck	Vase	Mean $\uparrow$
Tele-Pure	20.0	46.7	80.0	26.7	6.7	13.3	60.0	33.3	46.7	20.0	35.3
Tele-Catch	46.7	53.3	80.0	46.7	20.0	13.3	80.0	86.7	73.3	46.7	54.7

Table 2: Success rate (SR) comparison on all classes. Tele-Pure abbreviates Pure Teleoperation.

	Ball	Bear	Cookie	Cup	Fish	Ipod	Showerhead	Teapot	Truck	Vase
DP	0.152	0.145	0.158	0.139	0.136	0.155	0.156	0.156	0.161	0.140
DP3	0.150	0.143	0.157	0.133	0.136	0.155	0.159	0.152	0.159	0.136
DP-U3R	0.150	0.142	0.155	0.129	0.136	0.154	0.154	0.152	0.158	0.134

Table 3: Action noise MSE results across different objects.

	Open-Loop	RL	Tele-Pure	Tele-Catch(DP [ 7 ] ) #	Tele-Catch(DP3 [ 35 ] )	Tele-Catch(DP-U3R)
SR $\uparrow$	46. 7	33.3	46.7	60.0	66.7	73.3

Table 4: Success rate (SR) comparison on Truck. # stands for our baseline.

Figure 5: Qualitative analysis. The top row shows the teleoperation glove guidance, the middle row depicts the corresponding robotic hand responses in simulation, and the bottom row illustrates the synchronized point-cloud observations. Together, these visualizations demonstrate how Tele-Catch integrates human teleoperation with geometry-aware policy control to achieve stable dynamic object catching over time.

Objects	ShadowHand	LinkerHand-L20	Seen	Unseen	Tele-Pure	Tele-Catch
Teapot	✓		✓		33.3	86.7
Teapot		✓	✓		40.0	73.3
Eraser	✓			✓	53.3	66.7
Telephone	✓			✓	46.7	53.3
Bottle	✓			✓	20.0	33.3

Table 5: Generalization analysis.

Figure 8: Sensitivity analysis. The orange ellipse highlights the variation of the ring finger.

Component	Symbol	Coefficient	Coefficient Value
Hand–object distance reward	$R_{\text{dist}}$	$\lambda_{\text{dist}}$	10.0
Object orientation reward	$R_{\text{rot}}$	$\lambda_{\text{rot}}$	0.1
Fingertip–object distance reward	$R_{\text{ftip-dist}}$	$\lambda_{\text{ftip-dist}}$	10.0
Fingertip linear velocity penalty	$P_{\text{ftip-linvel}}$	$\lambda_{\text{lin}}$	0.3
Fingertip angular velocity penalty	$P_{\text{ftip-angvel}}$	$\lambda_{\text{ang}}$	0.03
Action penalty	$P_{\text{action}}$	$\lambda_{\text{act}}$	$2e^{-4}$

Table 6: Reward components and coefficients used in our RL training.

Category	Hyperparameters
General	Random seed = 22, Observation clipping = 5.0, Action clipping = 1.0
Policy Network	Actor hidden layers = [1024, 1024, 512]
	Critic hidden layers = [1024, 1024, 512], Activation = ELU
Training Setup	Maximum iterations = 6500, Schedule = adaptive, Save interval = 1000
Optimization	Rollout steps = 8, Epochs per update = 5, Minibatches per epoch = 4
	Max gradient norm = 1, Learning rate = $3\times 10^{-4}$

Table 7: PPO hyperparameter setting.

Category	Key Parameters
Environment	numEnvs = 8192, envSpacing = 0.75, episodeLength = 50
Control	dofSpeedScale = 20, controlFrequency = 60 Hz
Noise (init/reset)	posNoise = 0.01, dofPosRandom = 0.2
Task Success	successTolerance = 0.1
Randomization	hand/object mass [0.5, 1.5], friction [0.7, 1.3], object scale [0.95, 1.05]
Physics (PhysX)	solver = TGS, numPosIter = 8, contactOffset = 0.002, bounceThreshold = 0.2

Table 8: Isaac Gym simulation parameter setting.

Robot Teleoperation

Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.

See all 206 papers

Towards Human-level Dexterous Teleoperation
Jul 13, 2026Puhao Li, Zeyuan Chen, Yingying Wu +9RL for Robotics Long-Horizon Robotic Manipulation
1Tsinghua University · 2State Key Lab of General Artificial Intelligence, BIGAI · 3Peking University
DexCast: Learning Real-World Dexterous Control from Human Videos
Oct 10, 2026Zihan Gao, Yuhong Zhang, Mengzhe Wang +7Robotic Manipulation Robot Imitation Learning
Tsinghua University · TsingTec · Shenzhen University +2
DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control
Jul 7, 2026Yuanchuan Lai, Qing Gao, Ziyan Liang +3Bimanual Robotic Manipulation Robot Teleoperation
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK

Jul 13, 2026J/K move · Enter open · S save

Towards Human-level Dexterous Teleoperation

Puhao Li, Zeyuan Chen, Yingying Wu, Pengkun Wei+8

1Tsinghua University · 2State Key Lab of General Artificial Intelligence, BIGAI · 3Peking University

RL for Robotics Long-Horizon Robotic Manipulation In-Hand Manipulation Contact-Rich Robotic Manipulation Robot Teleoperation

Open paper AskPDF arXiv

Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.

Abstract

Robot Teleoperation

Explore similar work

Towards Human-level Dexterous Teleoperation

DexCast: Learning Real-World Dexterous Control from Human Videos

DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control