DiSCo: Diffusion Sequence Copilots for Shared Autonomy

Andy Wang; Xu Yan; Brandon McMahan; Michael Zhou; Yuyang Yuan; Johannes Y. Lee; Ali Shreif; Matthew Li

cs.HCMar 24, 2026

DiSCo: Diffusion Sequence Copilots for Shared Autonomy

Authors: Andy Wang, Xu Yan, Brandon McMahan, Michael Zhou, Yuyang Yuan, Johannes Y. Lee, Ali Shreif, Matthew Li, +4 more

Organizations: University of California, Los Angeles, Los Angeles, USA

Abstract

Shared autonomy combines human user and AI copilot actions to control complex systems such as robotic arms. When a task is challenging, requires high dimensional control, or is subject to corruption, shared autonomy can significantly increase task performance by using a trained copilot to effectively correct user actions in a manner consistent with the user's goals. To significantly improve the performance of shared autonomy, we introduce Diffusion Sequence Copilots (DiSCo): a method of shared autonomy with diffusion policy that plans action sequences consistent with past user actions. DiSCo seeds and inpaints the diffusion process with user-provided actions with hyperparameters to balance conformity to expert actions, alignment with user intent, and perceived responsiveness. We demonstrate that DiSCo substantially improves task performance in simulated driving and robotic arm tasks. Project website: https://sites.google.com/view/disco-shared-autonomy/

Figures & tables

Figure 1. Overview. a , Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\mathbf{O}_{t}$ , to generate an action sequence $\mathbf{A}_{t}^{0}$ . b , DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\mathbf{A}}_{t}^{0}$ , that reflects user intent. c, d, e , We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice & Drawers). Overview. \textbf{a}, Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\O_t$, to generate an action sequence $\A_t^0$. \textbf{b}, DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\A}_t^0$ that reflects user intent. \textbf{c, d, e}, We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice \& Drawers). — Figure 1. Overview. a , Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\mathbf{O}_{t}$ , to generate an action sequence $\mathbf{A}_{t}^{0}$ . b , DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\mathbf{A}}_{t}^{0}$ , that reflects user intent. c, d, e , We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice & Drawers). Overview. a, Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\O_t$ , to generate an action sequence $\A_t^0$ . b, DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\A}_t^0$ that reflects user intent. **c, d, e**, We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice & Drawers).

Figure 2. DiSCo overview. a , In autonomous diffusion policy, standard Gaussian noise $\mathbf{A}^{K}_{t}$ is reverse diffused to an action sequence, $\mathbf{A}_{t}^{0}$ . b , In DiSCo, a user action sequence, $\mathbf{U}_{t}^{0}$ , is forward diffused by $k_{sw}$ steps to a noisy sequence $\mathbf{U}_{t}^{k_{sw}}$ . The noised user inputs $\mathbf{U}_{t}^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\mathbf{A}}^{0}_{t}$ represents the reverse diffused policy action sequence, now inpainted with user actions. c , Illustration of the observation horizon $O$ , inference delay $D$ , re-plan interval $R$ , and total prediction horizon $P$ . d , The seeding strategies for User-centered Anchoring and Hybrid Anchoring. e , The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip}=k_{sw}$ , and $\tilde{\mathbf{a}}_{0}^{0}=\mathbf{u}_{0}^{0}$ , $\tilde{\mathbf{a}}_{1}^{0}=\mathbf{u}_{1}^{0}$ , etc. DiSCo overview. \textbf{a}, In autonomous diffusion policy, standard Gaussian noise $\A^K_t$ is reverse diffused to an action sequence, $\A_t^0$. \textbf{b}, In DiSCo, a user action sequence, $\U_t^0$, is forward diffused by $k_{sw}$ steps to a noisy sequence $\U_t^{k_{sw}}$. The noised user inputs $\U_t^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\A}^0_t$ represents the reverse diffused policy action sequence, now inpainted with user actions. \textbf{c}, Illustration of the observation horizon $O$, inference delay $D$, re-plan interval $R$, and total prediction horizon $P$. \textbf{d}, The seeding strategies for User-centered Anchoring and Hybrid Anchoring. \textbf{e}, The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip} = k_{sw}$, and $\tilde{\a}_0^0 = \u_0^0$, $\tilde{\a}_1^0 = \u_1^0$, etc. — Figure 2. DiSCo overview. a , In autonomous diffusion policy, standard Gaussian noise $\mathbf{A}^{K}_{t}$ is reverse diffused to an action sequence, $\mathbf{A}_{t}^{0}$ . b , In DiSCo, a user action sequence, $\mathbf{U}_{t}^{0}$ , is forward diffused by $k_{sw}$ steps to a noisy sequence $\mathbf{U}_{t}^{k_{sw}}$ . The noised user inputs $\mathbf{U}_{t}^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\mathbf{A}}^{0}_{t}$ represents the reverse diffused policy action sequence, now inpainted with user actions. c , Illustration of the observation horizon $O$ , inference delay $D$ , re-plan interval $R$ , and total prediction horizon $P$ . d , The seeding strategies for User-centered Anchoring and Hybrid Anchoring. e , The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip}=k_{sw}$ , and $\tilde{\mathbf{a}}_{0}^{0}=\mathbf{u}_{0}^{0}$ , $\tilde{\mathbf{a}}_{1}^{0}=\mathbf{u}_{1}^{0}$ , etc. DiSCo overview. a, In autonomous diffusion policy, standard Gaussian noise $\A^K_t$ is reverse diffused to an action sequence, $\A_t^0$ . b, In DiSCo, a user action sequence, $\U_t^0$ , is forward diffused by $k_{sw}$ steps to a noisy sequence $\U_t^{k_{sw}}$ . The noised user inputs $\U_t^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\A}^0_t$ represents the reverse diffused policy action sequence, now inpainted with user actions. c, Illustration of the observation horizon $O$ , inference delay $D$ , re-plan interval $R$ , and total prediction horizon $P$ . d, The seeding strategies for User-centered Anchoring and Hybrid Anchoring. e, The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip} = k_{sw}$ , and $\tilde{\a}_0^0 = \u_0^0$ , $\tilde{\a}_1^0 = \u_1^0$ , etc.

Figure 3. 4-Goal MetaDrive hyperparameter sweep and trajectories. a , Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. b , Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$ , were able to correctly achieve the goal (green). c, d , Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$ . 4-Goal MetaDrive hyperparameter sweep and trajectories. \textbf{a}, Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. \textbf{b}, Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$, were able to correctly achieve the goal (green). \textbf{c, d}, Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$. — Figure 3. 4-Goal MetaDrive hyperparameter sweep and trajectories. a , Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. b , Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$ , were able to correctly achieve the goal (green). c, d , Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$ . 4-Goal MetaDrive hyperparameter sweep and trajectories. a, Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. b, Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$ , were able to correctly achieve the goal (green). **c, d**, Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$ .

Figure 4. Block Choice hyperparameter sweep and trajectories. a, b , Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. c , Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ( $\gamma=0.3$ ), and DiSCo ( $\gamma=0.3$ , $\rho=0.1$ , $\beta=0.3$ ) control. Green lines correspond to trajectories of correct trials, while red lines correspond to trajectories of wrong trials. Block Choice hyperparameter sweep and trajectories. \textbf{a, b}, Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. \textbf{c}, Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ($\gamma= 0.3$), and DiSCo ($\gamma= 0.3$, $\rho=0.1$, $\beta=0.3$) control. Green lines correspond to trajectories of correct trials while red lines correspond to these of wrong trials. — Figure 4. Block Choice hyperparameter sweep and trajectories. a, b , Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. c , Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ( $\gamma=0.3$ ), and DiSCo ( $\gamma=0.3$ , $\rho=0.1$ , $\beta=0.3$ ) control. Green lines correspond to trajectories of correct trials, while red lines correspond to trajectories of wrong trials. Block Choice hyperparameter sweep and trajectories. **a, b**, Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. c, Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ( $\gamma= 0.3$ ), and DiSCo ( $\gamma= 0.3$ , $\rho=0.1$ , $\beta=0.3$ ) control. Green lines correspond to trajectories of correct trials while red lines correspond to these of wrong trials.

Policy	Success Rate	Crash Rate
No Copilot	0.002	0.998
State-Based Copilot ( $\gamma=0.3$ )	0.200	0.120
DiSCo ( $\gamma=0.3$ , $\rho=0.0$ )	0.381	0.386
DiSCo ( $\gamma=0.3$ , $\rho=0.5$ )	0.471	0.500
DiSCo ( $\gamma=1.0$ , $\rho=0.9$ )	0.398	0.596

Table 1. Comparison of performance metrics under different copilot configurations. Each condition was evaluated over 50 episodes 10 times and we found all DiSCo conditions had significantly higher success rate than others (

p<0.01

, Wilcoxon rank-sum test).

Measure	$H$	$p$	$p<0.01$
Success	12.764	$5.2\times 10^{-3}$	Yes
Time (s)	133.321	$1.0\times 10^{-28}$	Yes
Collisions	346.107	$1.0\times 10^{-74}$	Yes
Wrong	55.216	$6.2\times 10^{-12}$	Yes
Ease	139.892	$4.0\times 10^{-30}$	Yes
Control	171.022	$7.7\times 10^{-37}$	Yes

Table 2. Kruskal–Wallis test results for Block Choice dataset (

\alpha=0.01

). Statistics across

n=10

participants.

Policy	Success $\uparrow$	Wrong $\downarrow$	Time (s) $\downarrow$	Collisions $\downarrow$	Ease $\uparrow$	Control $\uparrow$	Autonomy $\uparrow$
No Copilot	0.45	0.00	17.82	4.24	2.80	3.14	3.26
State-Based Copilot	0.09	0.23	19.97	0.25	1.72	1.90	1.94
DiSCo	0.80	0.00	14.90	0.15	3.62	3.78	4.04

Table 3. Comparison of performance on Block Choice across all policy conditions.

Task	Policy	Mental Demand ( $\downarrow$ )	Physical Demand ( $\downarrow$ )	Effort ( $\downarrow$ )	Performance ( $\uparrow$ )	Frustration ( $\uparrow$ )
	No Copilot	3.23	3.17	3.65	4.78	5.12
Block Choice	State-Based Copilot	4.28	4.65	4.75	2.25	3.93
	DiSCo	2.75	3.05	3.32	5.82	5.85
	No Copilot	3.56	2.93	4.02	3.93	4.07
Drawers	State-Based Copilot	4.53	4.32	4.90	1.70	2.22
	DiSCo	3.28	3.00	3.41	3.95	4.05

Table 4. Comparison of user perceived experience on Block Choice and Drawers tasks. Arrows indicate the desirable direction:

\downarrow

means lower is better, and

\uparrow

means higher is better.

Policy	Success	Correct Drawer	Correct Object	Time (s)	Collisions	Ease	Control	Effect
No Copilot	0.28	0.89	0.67	93.03	9.11	2.66	2.82	3.16
State-Based Copilot	0.00	0.52	0.27	98.91	1.00	1.68	1.66	1.77
DiSCo	0.42	0.92	0.82	88.64	0.74	3.08	2.94	2.99

Table 5. Comparison of performance on Drawers across all policy conditions.

Measure	$H$	$p$	$p<0.01$
Correct Drawer	15.620	$4.1\times 10^{-4}$	Yes
Correct Object	15.062	$5.4\times 10^{-4}$	Yes
Success	16.870	$2.2\times 10^{-4}$	Yes
Time (s)	46.039	$1.0\times 10^{-10}$	Yes
Collisions	131.441	$2.9\times 10^{-29}$	Yes
Ease	67.279	$2.5\times 10^{-15}$	Yes

Table 6. Kruskal–Wallis test results for Drawers dataset (

\alpha=0.01

). Statistics across

n=10

participants.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	ID	Age	Sex	Exp.	Ethnicity
Block	1	24	M	C	Asian
Block	2	20	M	C	Asian
Block	3	20	F	C	Asian
Block	4	23	F	C	Asian
Drawer	5	19	M	C	Asian
Drawer	6	20	F	C	Asian

Appendix

Table 7. Participant demographics. Exp. denotes self-reported robotics experience: (A) Master: collected data and trained successful robot policies; (B) Intermediate: controlled robots for tasks (e.g., data collection or user studies); (C) Novice: no experience controlling robotic arms.

Explore similar work

May 19, 2026

Proximal State Nudging: Reducing Skill Atrophy from AI Assistance

Skill atrophy, the gradual decline of human capability under AI assistance, poses a safety risk in shared-control of semi-autonomous systems, where operators may be unable to distinguish their own inputs from autonomous corrections. We propose Proximal State Nudging (PSN), a shared autonomy algorithm that jointly optimizes for skill development and task performance by nudging users toward states estimated to be most learnable. We first show that PSN outperforms existing shared autonomy baselines in balancing student improvement in unassisted reward with overall shared performance, using simulated students in the classic LunarLander environment. We then present, to the best of our knowledge, the first human subject studies of a planner incorporating learning-compatible shared autonomy: across two driving tasks in the CARLA simulator (High Performance Racing and Parallel Parking, n = 60), PSN produces up to 7x larger gains in unassisted skill than standard blended shared autonomy, while incurring 50% fewer collisions than unassisted self-practice.

Megha Srivastava, Jonathan Ouyang, Eric Zhou +6

Stanford University, Stanford, CA, USA. · University of California Los Angeles, Los Angeles, CA, USA. · Toyota Research Institute, Los Altos, CA, USA.

Human-in-the-Loop AI Shared Autonomy

Ask PDF

May 20, 2026

HITL-D: Human In The Loop Diffusion Assisted Shared Control

Autonomous manipulation systems have achieved remarkable capabilities, yet the integration of human expertise with diffusion-based policies in shared control remains relatively unexplored. In this paper, we propose Human-In-The-Loop Diffusion (HITL-D), a shared control framework that enhances user performance in multi-step, insertion, and fine manipulation tasks. HITL-D leverages a novel combination of diffusion-based policies and human control to provide autonomous end effector orientation updates conditioned on a scene point cloud and the Cartesian position of the end effector. This approach reduces the number of joystick control axes required, thereby lowering mental workload. In a multi-task user study with 12 participants, HITL-D reduced average task completion times by 40%, decreased perceived workload by 37%, and improved Likert-scale ratings for independence, intuitiveness, and confidence compared to traditional teleoperation methods. These results demonstrate that HITL-D effectively integrates human expertise with autonomous assistance, improving both objective and subjective aspects of teleoperation.

Riley Zilka, Sergey Khlynovskiy, Allie Wang +1

Department of Computing Science, University of Alberta, Edmonton Alberta, Canada, T6G 2E8

Robot Teleoperation Diffusion Policy Robotic Manipulation Shared Autonomy

Ask PDF

Sep 26, 2026

Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter

A common approach for shared autonomy blends human inputs with autonomous assistance based on the human's likely goal. However, most existing approaches assume that a static set of possible goals is known a priori, which limits the use of such methods in unstructured assistive settings. We instead investigate how to enable shared autonomy with open-ended and dynamically changing goals. We formulate goal-oriented shared autonomy as a particle filter in which particles represent candidate human goals. Unlike conventional approaches with a fixed goal set, our transition model dynamically proposes new candidate goals as the interaction evolves, and human actions update the belief over these goals in real time. We instantiate this framework with foundation models (e.g., vision grounding and large language models) that propose context-relevant semantic goals, generate goal-conditioned assistance from low-level skill primitives, and refine those skills from human corrections. We assess our approach through a user study where 12 participants perform a variety of tabletop manipulation tasks with our method and state-of-the-art shared autonomy baselines. The results show that our particle filter-based approach reduces the amount of time users spend teleoperating the system and improves user satisfaction. User study videos: https://youtu.be/Ii26XuRqm9c

Mengxue Fu, Ethan Xu, Sam Iyer-Singh +3

Collaborative Robotics Lab (Collab), Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061 · RT2 Lab, Dept. of Computer Science, UW-Madison

Shared Autonomy Assistive Robotics Particle Filtering

Ask PDF