Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy

Shuo Sha; Yixuan Wang; Binghao Huang; Antonio Loquercio; Yunzhu Li

cs.ROMar 17, 2026

Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy

Authors: Shuo Sha, Yixuan Wang, Binghao Huang, Antonio Loquercio, Yunzhu Li

Organizations: Columbia University · University of Pennsylvania

Abstract

Fine-grained, contact-rich teleoperation remains slow, error-prone, and unreliable in real-world manipulation tasks, even for experienced operators. Shared autonomy offers a promising way to improve performance by combining human intent with automated assistance, but learning effective assistance in simulation requires a faithful model of human behavior, which is difficult to obtain in practice. We propose a real-to-sim-to-real shared autonomy framework that augments human teleoperation with learned corrective behaviors, using a simple yet effective k-nearest-neighbor (kNN) human surrogate to model operator actions in simulation. The surrogate is fit from less than five minutes of real-world teleoperation data and enables stable training of a residual copilot policy with model-free reinforcement learning. The resulting copilot is deployed to assist human operators in real-world fine-grained manipulation tasks. Through simulation experiments and a user study with sixteen participants on industry-relevant tasks, including nut threading, gear meshing, and peg insertion, we show that our system improves task success for novice operators and execution efficiency for experienced operators compared to direct teleoperation and shared-autonomy baselines that rely on expert priors or behavioral-cloning pilots. In addition, copilot-assisted teleoperation produces higher-quality demonstrations for downstream imitation learning.

Figures & tables

Fig. 2 : Method overview of residual copilot learning with a real-to-sim-to-real pipeline. Left: less than five minutes of real-world teleoperation interaction data is collected under admittance control. Center: in simulation, a lightweight kNN-based human surrogate constructed from real-world data generates base actions conditioned on the environment state, while a residual copilot policy is trained with reinforcement learning to produce corrective residual actions. Right: at deployment, the residual copilot assists the human operator in zero-shot, conditioning on the estimated environment state and human base actions. The residual formulation preserves human intent while improving alignment, contact regulation, and execution robustness.

Fig. 3 : Human experiment results across three contact-rich assembly tasks. Rows correspond to Gear Meshing, Nut Threading, and Peg Insertion. Left: success rate and mean completion time across successful episodes (solid line; shaded region denotes standard deviation; dashed line indicates per-method mean). Right: subjective workload (NASA-TLX subscales; lower is better) and user satisfaction (higher is better). Across tasks, the Residual Copilot improves success rate and can effectively reduce completion time, while lowering reported workload and increasing user satisfaction compared to direct teleoperation and a residual baseline.

Fig. 4 : User study qualitative results. Representative trajectory temporal overlays comparing unassisted teleoperation and our Residual Copilot on three high-precision tasks. Top row (effectiveness): our copilot enables reliable task completion in cases where teleoperation fails due to force, rotation, or grasp errors. Bottom row (efficiency): when both succeed, our copilot consistently reduces completion time and exemplifies smooth motions. Together, these results demonstrate that our method provides both reliable and efficient assistance in challenging manipulation scenarios.

Copilot	Eval. Pilot
	Laggy Pilot			Noisy Pilot			Expert Pilot			BC Pilot			kNN Pilot
	Gear	Peg	Nut	Gear	Peg	Nut	Gear	Peg	Nut	Gear	Peg	Nut	Gear	Peg	Nut
No Copilot	0.97	0.94	0.49	0.92	0.91	0.24	0.99	0.99	0.61	0.84	0.71	0.03	0.87	0.85	0.16
GD Expert	0.96	0.89	0.24	0.95	0.91	0.25	0.94	0.90	0.28	0.94	0.89	0.34	0.95	0.90	0.27
GD BC	0.63	0.56	0.00	0.65	0.56	0.00	0.62	0.53	0.01	0.63	0.52	0.01	0.66	0.55	0.00
Residual BC	0.96	0.44	0.00	0.91	0.44	0.00	0.98	0.44	0.00	0.70	0.40	0.00	0.86	0.46	0.02

TABLE I : Cross-pilot generalization and robustness in simulation. Rows denote copilots trained with different training pilots; columns evaluate each trained copilot under distinct evaluation pilots across three tasks. Each cell reports task progression (over 1000 simulation trials); see Appendix for metric details. Gray and blue diagonal entries indicate in-distribution evaluation, i.e., a copilot evaluated with the pilot it was trained on; off-diagonal entries measure generalization to unseen pilots. The Residual Copilot maintains consistently high performance across evaluation pilots, demonstrating robustness to pilot mismatch and perturbations such as lag and action noise.

	Matched attempts		Matched successes
Method	Grasp	Insert	Grasp	Insert
Teleop	7/20	1/20	6/20	0/20
Residual (Ours)	18/20	11/20	19/20	9/20

TABLE II : Downstream policy progression from copilot vs. teleoperation data (Gear Meshing). Diffusion policies are trained on demonstrations collected under Residual Copilot assistance or direct teleoperation, using matched attempts and matched successes to control for data budget. Results show copilot assistance improves learnability and consistency of the collected demonstrations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Controller	Parameter	Real	Sim	Unit
Task-space admittance controller	$K_{x}$	$1000$	$200$	N/m
	$K_{r}$	$10$	$100$	Nm/rad
	$M_{x}$	$0.1$	$0.125$	kg
	$M_{r}$	$1\times 10^{-3}$	$0.015$	kg $\cdot$ m 2
	$D_{x}$	$0$	$5$	N $\cdot$ s/m
	$D_{r}$	$0$	$1.2$	N $\cdot$ m $\cdot$ s/rad

Appendix

TABLE III : Controller parameters. Real-world and simulation settings for the task-space admittance and joint-space PID controllers.

Group	Symbol(s)	Distribution / value
Ctrl.	$K_{x},K_{r},M_{x},M_{r}$	$\mathcal{U}(0.95,\,1.05)\odot(\cdot)$
Pose	$\Delta p_{\mathrm{fix}},\Delta p_{\mathrm{held}}$	$\mathcal{N}(0,\,0.002^{2}I)$
Init.	$\Delta p_{0},\Delta\theta_{0}$	$\mathcal{N}(0,\,0.02^{2}I)$ , $\mathcal{N}(0,\,0.035^{2}I)$
Base	$L$	$L\sim\mathcal{U}\{5,\dots,15\}$
	$\varepsilon_{t}$	$\mathcal{U}([-0.6,0.6]^{d})$
	$\beta,p_{\mathrm{on}}$	$\beta=0.8,\;p_{\mathrm{on}}=0.5$

Appendix

TABLE IV : Domain randomization (DMR). Per-episode sampling unless noted; base-action DMR is sampled per timestep. For base-action randomization,

L

denotes the length of kNN-retrieved action chunks,

\varepsilon_{t}

is i.i.d. residual noise, and

(\beta,p_{\mathrm{on}})

parameterize a smooth stochastic gate controlling the activation frequency and temporal correlation of the noise.

Category	Name	Term	Scale
$\mathcal{R}_{\text{general}}$	Regularization	$-\lVert a_{\text{res}}\rVert$	$0.1$
	Tilt Penalty	$-\theta_{\text{tilt}}$	$1.0$
	Force Penalty	$-\phi(F)$	$0.2$
	Axis Align	$\mathbf{1}[\lVert p_{xy}-p^{*}_{xy}\rVert<\epsilon]$	$0.05$
	Smoothness	$-\lVert a_{\text{res}}^{prev}-a_{\text{res}}\rVert$	0.1
	Termination	$-\mathbf{1}[\text{fail / timeout}]$	$50.0$

Appendix

TABLE V: Reward terms.

a_{\text{res}}

and

a_{\text{res}}^{prev}

denote current and previous normalized residual actions,

\theta_{\text{tilt}}

the end-effector tilt angle from upright,

\phi(F)

a clipped contact-force penalty, and

p_{xy}

the held-object planar position. The success reward is issued once per episode upon first satisfying the task-specific success condition. The termination condition is dropping the held object early on.

Hyperparameter	Value
Architecture
Backbone	LSTM (2 layers, 1024 units)
MLP	[512, 128, 64], ELU
Actor–critic	shared trunk
Policy head	Gaussian (learnable $\log\sigma$ )
PPO / Optimization

Appendix

TABLE VI : RL hyperparameters. Actor–critic architecture and PPO settings used for training.

Category	State DP	Vision DP
Inputs / Outputs
Observation	$s_{t}\in\mathbb{R}^{20}$	RGB $(200{\times}200)$ + $s_{t}\in\mathbb{R}^{8}$
Action	$a_{t}\in\mathbb{R}^{8}$	$a_{t}\in\mathbb{R}^{8}$
Horizon ( $H$ )	16	16
Action steps	8	8
Diffusion model

Appendix

TABLE VII : Diffusion policy hyperparameters. State- and vision-based variants share the diffusion process and training setup, differing only in input modality, encoder, and training scale.

Fig. 6 : State-based DP training data distributions. Spatial coverage of 3D fingertip positions, downsampled to 100k points for visualization. The top row shows training data for BC Pilot , obtained by augmenting real-world teleoperation demonstrations; the bottom row shows training data for Expert Pilot , collected from successful rollouts of the residual copilot policies. Columns correspond to the three tasks: Gear Meshing, Nut Threading, and Peg Insertion. The top row (left to right) contains 442k, 786k, and 427k transitions, while the bottom row contains 220k, 758k, and 195k transitions.

Fig. 7 : Vision-based DP training data distributions. Spatial coverage visualization of initial states in the training set used to train vision-based diffusion policies. Columns show episodes collected via direct vs. copilot-assisted teleoperation. Rows correspond to the two controlled IV-C settings (same number of successful episodes and same number of teleoperation attempts), isolating how data quality and collection efficiency affect downstream policy learning.

Explore similar work

SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLA
Jun 14, 2026Crystal Zhou, Jehan Yang, Douglas J. Weber +1Robot Teleoperation Robotic Manipulation
Carnegie Mellon University Pittsburgh, PA, USA
Adaptor: Advancing Assistive Teleoperation with Few-Shot Learning and Cross-Operator Generalization
Apr 10, 2026Yu Liu, Yihang Yin, Tianlv Huang +8Robot Teleoperation Robot Learning
School of Artificial Intelligence, Jilin University. · IO-AI TECH.
BlenDAgger: Blended Shared Control for Interactive Imitation Learning
Sep 29, 2026Cailyn Smith, Geoffrey Sun, Henny Admoni +1Robotic Control Robotic Manipulation
The Robotics Institute, School of Computer Science Carnegie Mellon University Pittsburgh, USA · School of Computer Science Carnegie Mellon University Pittsburgh, USA

Jun 14, 2026J/K move · Enter open · S save

SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLA

Crystal Zhou, Jehan Yang, Douglas J. Weber, Zackory Erickson

Carnegie Mellon University Pittsburgh, PA, USA

Robot Teleoperation Robotic Manipulation Vision-Language-Action Models Shared Autonomy Human-Robot Interaction

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models perform well in generalist robot manipulation but remain brittle to out-of-distribution perturbations, while full human teleoperation is cognitively demanding. To bridge this gap, we introduce Shared Autonomy for Policy Steering (SAPS), a lightweight framework that blends real-time human teleoperation commands with pretrained VLA actions. SAPS operates purely at the action level, requiring no policy retraining, auxiliary dynamics models, or architectural modifications. We evaluate three arbitration strategies across simulated (LIBERO, CALVIN) and real-world evaluations. In a 14-person user study, SAPS achieved up to 77.4% hardware success, compared with 8.3% for full autonomy and 34.5% for teleoperation. Furthermore, active human intervention was required only 45% of the time, reducing NASA-TLX workload by 20.8% in simulation and 28.2% on hardware. A case study with an expert user with quadriplegia similarly demonstrated matched or improved success while reducing completion time. These results highlight SAPS as a highly effective, model-agnostic approach for reliable real-world deployment and assistive teleoperation. A website with additional details is located at shared-autonomy-policy.github.io.

Abstract

Explore similar work

SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLA

Adaptor: Advancing Assistive Teleoperation with Few-Shot Learning and Cross-Operator Generalization

BlenDAgger: Blended Shared Control for Interactive Imitation Learning