DiSCo: Diffusion Sequence Copilots for Shared Autonomy

Andy Wang; Xu Yan; Brandon McMahan; Michael Zhou; Yuyang Yuan; Johannes Y. Lee; Ali Shreif; Matthew Li

cs.HCMar 24, 2026

DiSCo: Diffusion Sequence Copilots for Shared Autonomy

Authors: Andy Wang, Xu Yan, Brandon McMahan, Michael Zhou, Yuyang Yuan, Johannes Y. Lee, Ali Shreif, Matthew Li, +4 more

Organizations: University of California, Los Angeles, Los Angeles, USA

Abstract

Shared autonomy combines human user and AI copilot actions to control complex systems such as robotic arms. When a task is challenging, requires high dimensional control, or is subject to corruption, shared autonomy can significantly increase task performance by using a trained copilot to effectively correct user actions in a manner consistent with the user's goals. To significantly improve the performance of shared autonomy, we introduce Diffusion Sequence Copilots (DiSCo): a method of shared autonomy with diffusion policy that plans action sequences consistent with past user actions. DiSCo seeds and inpaints the diffusion process with user-provided actions with hyperparameters to balance conformity to expert actions, alignment with user intent, and perceived responsiveness. We demonstrate that DiSCo substantially improves task performance in simulated driving and robotic arm tasks. Project website: https://sites.google.com/view/disco-shared-autonomy/

Figures & tables

Figure 1. Overview. a , Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\mathbf{O}_{t}$ , to generate an action sequence $\mathbf{A}_{t}^{0}$ . b , DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\mathbf{A}}_{t}^{0}$ , that reflects user intent. c, d, e , We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice & Drawers). Overview. \textbf{a}, Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\O_t$, to generate an action sequence $\A_t^0$. \textbf{b}, DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\A}_t^0$ that reflects user intent. \textbf{c, d, e}, We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice \& Drawers). — Figure 1. Overview. a , Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\mathbf{O}_{t}$ , to generate an action sequence $\mathbf{A}_{t}^{0}$ . b , DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\mathbf{A}}_{t}^{0}$ , that reflects user intent. c, d, e , We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice & Drawers). Overview. a, Diffusion Policy runs a reverse diffusion process on unit Gaussian noise, conditioned on vision observations $\O_t$ , to generate an action sequence $\A_t^0$ . b, DiSCo diffuses user action sequences with inpainting to generate an action sequence, $\tilde{\A}_t^0$ that reflects user intent. **c, d, e**, We evaluate DiSCo on one simulated task (MetaDrive) and two human-in-the-loop robotic tasks (Block Choice & Drawers).

Figure 2. DiSCo overview. a , In autonomous diffusion policy, standard Gaussian noise $\mathbf{A}^{K}_{t}$ is reverse diffused to an action sequence, $\mathbf{A}_{t}^{0}$ . b , In DiSCo, a user action sequence, $\mathbf{U}_{t}^{0}$ , is forward diffused by $k_{sw}$ steps to a noisy sequence $\mathbf{U}_{t}^{k_{sw}}$ . The noised user inputs $\mathbf{U}_{t}^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\mathbf{A}}^{0}_{t}$ represents the reverse diffused policy action sequence, now inpainted with user actions. c , Illustration of the observation horizon $O$ , inference delay $D$ , re-plan interval $R$ , and total prediction horizon $P$ . d , The seeding strategies for User-centered Anchoring and Hybrid Anchoring. e , The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip}=k_{sw}$ , and $\tilde{\mathbf{a}}_{0}^{0}=\mathbf{u}_{0}^{0}$ , $\tilde{\mathbf{a}}_{1}^{0}=\mathbf{u}_{1}^{0}$ , etc. DiSCo overview. \textbf{a}, In autonomous diffusion policy, standard Gaussian noise $\A^K_t$ is reverse diffused to an action sequence, $\A_t^0$. \textbf{b}, In DiSCo, a user action sequence, $\U_t^0$, is forward diffused by $k_{sw}$ steps to a noisy sequence $\U_t^{k_{sw}}$. The noised user inputs $\U_t^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\A}^0_t$ represents the reverse diffused policy action sequence, now inpainted with user actions. \textbf{c}, Illustration of the observation horizon $O$, inference delay $D$, re-plan interval $R$, and total prediction horizon $P$. \textbf{d}, The seeding strategies for User-centered Anchoring and Hybrid Anchoring. \textbf{e}, The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip} = k_{sw}$, and $\tilde{\a}_0^0 = \u_0^0$, $\tilde{\a}_1^0 = \u_1^0$, etc. — Figure 2. DiSCo overview. a , In autonomous diffusion policy, standard Gaussian noise $\mathbf{A}^{K}_{t}$ is reverse diffused to an action sequence, $\mathbf{A}_{t}^{0}$ . b , In DiSCo, a user action sequence, $\mathbf{U}_{t}^{0}$ , is forward diffused by $k_{sw}$ steps to a noisy sequence $\mathbf{U}_{t}^{k_{sw}}$ . The noised user inputs $\mathbf{U}_{t}^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\mathbf{A}}^{0}_{t}$ represents the reverse diffused policy action sequence, now inpainted with user actions. c , Illustration of the observation horizon $O$ , inference delay $D$ , re-plan interval $R$ , and total prediction horizon $P$ . d , The seeding strategies for User-centered Anchoring and Hybrid Anchoring. e , The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip}=k_{sw}$ , and $\tilde{\mathbf{a}}_{0}^{0}=\mathbf{u}_{0}^{0}$ , $\tilde{\mathbf{a}}_{1}^{0}=\mathbf{u}_{1}^{0}$ , etc. DiSCo overview. a, In autonomous diffusion policy, standard Gaussian noise $\A^K_t$ is reverse diffused to an action sequence, $\A_t^0$ . b, In DiSCo, a user action sequence, $\U_t^0$ , is forward diffused by $k_{sw}$ steps to a noisy sequence $\U_t^{k_{sw}}$ . The noised user inputs $\U_t^{k_{sw}}$ seed the reverse diffusion process and inpaint the reverse process with user actions. The output $\tilde{\A}^0_t$ represents the reverse diffused policy action sequence, now inpainted with user actions. c, Illustration of the observation horizon $O$ , inference delay $D$ , re-plan interval $R$ , and total prediction horizon $P$ . d, The seeding strategies for User-centered Anchoring and Hybrid Anchoring. e, The inpainting strategy for shared autonomy. After forward diffusion for $k_{sw}$ steps, we run $k_{ip}$ reverse diffusion steps where the the user input sequence is inpainted. For the remaining $k_{sw}-k_{ip}$ steps, we run reverse diffusion with no inpainting. Total inpainting occurs when $k_{ip} = k_{sw}$ , and $\tilde{\a}_0^0 = \u_0^0$ , $\tilde{\a}_1^0 = \u_1^0$ , etc.

Figure 3. 4-Goal MetaDrive hyperparameter sweep and trajectories. a , Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. b , Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$ , were able to correctly achieve the goal (green). c, d , Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$ . 4-Goal MetaDrive hyperparameter sweep and trajectories. \textbf{a}, Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. \textbf{b}, Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$, were able to correctly achieve the goal (green). \textbf{c, d}, Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$. — Figure 3. 4-Goal MetaDrive hyperparameter sweep and trajectories. a , Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. b , Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$ , were able to correctly achieve the goal (green). c, d , Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$ . 4-Goal MetaDrive hyperparameter sweep and trajectories. a, Hyperparameter sweep of $\gamma$ and $\rho$ for 4-Goal MetaDrive using the Corrupted Policy. b, Example trajectories where the correct goal was to turn left. The No Copilot condition repeatedly crashed (black). The State-Based Copilot condition had non-crash trajectories that reached the wrong goal (red). DiSCo, at varying levels of $\gamma$ and $\rho$ , were able to correctly achieve the goal (green). **c, d**, Same as (a, b) but for the "Broken Pedal" policy where the pedal is zeroed with probability $0.85$ .

Figure 4. Block Choice hyperparameter sweep and trajectories. a, b , Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. c , Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ( $\gamma=0.3$ ), and DiSCo ( $\gamma=0.3$ , $\rho=0.1$ , $\beta=0.3$ ) control. Green lines correspond to trajectories of correct trials, while red lines correspond to trajectories of wrong trials. Block Choice hyperparameter sweep and trajectories. \textbf{a, b}, Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. \textbf{c}, Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ($\gamma= 0.3$), and DiSCo ($\gamma= 0.3$, $\rho=0.1$, $\beta=0.3$) control. Green lines correspond to trajectories of correct trials while red lines correspond to these of wrong trials. — Figure 4. Block Choice hyperparameter sweep and trajectories. a, b , Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. c , Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ( $\gamma=0.3$ ), and DiSCo ( $\gamma=0.3$ , $\rho=0.1$ , $\beta=0.3$ ) control. Green lines correspond to trajectories of correct trials, while red lines correspond to trajectories of wrong trials. Block Choice hyperparameter sweep and trajectories. **a, b**, Hyperparameter sweeps for User-Centered Anchoring and Hybrid Anchoring seeding strategies. c, Sample trajectories of block choice for User 2 under No Copilot, State-Based Copilot ( $\gamma= 0.3$ ), and DiSCo ( $\gamma= 0.3$ , $\rho=0.1$ , $\beta=0.3$ ) control. Green lines correspond to trajectories of correct trials while red lines correspond to these of wrong trials.

Policy	Success Rate	Crash Rate
No Copilot	0.002	0.998
State-Based Copilot ( $\gamma=0.3$ )	0.200	0.120
DiSCo ( $\gamma=0.3$ , $\rho=0.0$ )	0.381	0.386
DiSCo ( $\gamma=0.3$ , $\rho=0.5$ )	0.471	0.500
DiSCo ( $\gamma=1.0$ , $\rho=0.9$ )	0.398	0.596

Table 1. Comparison of performance metrics under different copilot configurations. Each condition was evaluated over 50 episodes 10 times and we found all DiSCo conditions had significantly higher success rate than others (

p<0.01

, Wilcoxon rank-sum test).

Measure	$H$	$p$	$p<0.01$
Success	12.764	$5.2\times 10^{-3}$	Yes
Time (s)	133.321	$1.0\times 10^{-28}$	Yes
Collisions	346.107	$1.0\times 10^{-74}$	Yes
Wrong	55.216	$6.2\times 10^{-12}$	Yes
Ease	139.892	$4.0\times 10^{-30}$	Yes
Control	171.022	$7.7\times 10^{-37}$	Yes

Table 2. Kruskal–Wallis test results for Block Choice dataset (

\alpha=0.01

). Statistics across

n=10

participants.

Policy	Success $\uparrow$	Wrong $\downarrow$	Time (s) $\downarrow$	Collisions $\downarrow$	Ease $\uparrow$	Control $\uparrow$	Autonomy $\uparrow$
No Copilot	0.45	0.00	17.82	4.24	2.80	3.14	3.26
State-Based Copilot	0.09	0.23	19.97	0.25	1.72	1.90	1.94
DiSCo	0.80	0.00	14.90	0.15	3.62	3.78	4.04

Table 3. Comparison of performance on Block Choice across all policy conditions.

Task	Policy	Mental Demand ( $\downarrow$ )	Physical Demand ( $\downarrow$ )	Effort ( $\downarrow$ )	Performance ( $\uparrow$ )	Frustration ( $\uparrow$ )
	No Copilot	3.23	3.17	3.65	4.78	5.12
Block Choice	State-Based Copilot	4.28	4.65	4.75	2.25	3.93
	DiSCo	2.75	3.05	3.32	5.82	5.85
	No Copilot	3.56	2.93	4.02	3.93	4.07
Drawers	State-Based Copilot	4.53	4.32	4.90	1.70	2.22
	DiSCo	3.28	3.00	3.41	3.95	4.05

Table 4. Comparison of user perceived experience on Block Choice and Drawers tasks. Arrows indicate the desirable direction:

\downarrow

means lower is better, and

\uparrow

means higher is better.

Policy	Success	Correct Drawer	Correct Object	Time (s)	Collisions	Ease	Control	Effect
No Copilot	0.28	0.89	0.67	93.03	9.11	2.66	2.82	3.16
State-Based Copilot	0.00	0.52	0.27	98.91	1.00	1.68	1.66	1.77
DiSCo	0.42	0.92	0.82	88.64	0.74	3.08	2.94	2.99

Table 5. Comparison of performance on Drawers across all policy conditions.

Measure	$H$	$p$	$p<0.01$
Correct Drawer	15.620	$4.1\times 10^{-4}$	Yes
Correct Object	15.062	$5.4\times 10^{-4}$	Yes
Success	16.870	$2.2\times 10^{-4}$	Yes
Time (s)	46.039	$1.0\times 10^{-10}$	Yes
Collisions	131.441	$2.9\times 10^{-29}$	Yes
Ease	67.279	$2.5\times 10^{-15}$	Yes

Table 6. Kruskal–Wallis test results for Drawers dataset (

\alpha=0.01

). Statistics across

n=10

participants.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	ID	Age	Sex	Exp.	Ethnicity
Block	1	24	M	C	Asian
Block	2	20	M	C	Asian
Block	3	20	F	C	Asian
Block	4	23	F	C	Asian
Drawer	5	19	M	C	Asian
Drawer	6	20	F	C	Asian

Appendix

Table 7. Participant demographics. Exp. denotes self-reported robotics experience: (A) Master: collected data and trained successful robot policies; (B) Intermediate: controlled robots for tasks (e.g., data collection or user studies); (C) Novice: no experience controlling robotic arms.

Explore similar work

Proximal State Nudging: Reducing Skill Atrophy from AI Assistance
May 19, 2026Megha Srivastava, Jonathan Ouyang, Eric Zhou +6Human-in-the-Loop AI Shared Autonomy
Stanford University, Stanford, CA, USA. · University of California Los Angeles, Los Angeles, CA, USA. · Toyota Research Institute, Los Altos, CA, USA.
HITL-D: Human In The Loop Diffusion Assisted Shared Control
May 20, 2026Riley Zilka, Sergey Khlynovskiy, Allie Wang +1Robot Teleoperation Diffusion Policy
Department of Computing Science, University of Alberta, Edmonton Alberta, Canada, T6G 2E8
Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter
Sep 26, 2026Mengxue Fu, Ethan Xu, Sam Iyer-Singh +3Shared Autonomy Assistive Robotics
Collaborative Robotics Lab (Collab), Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061 · RT2 Lab, Dept. of Computer Science, UW-Madison

May 19, 2026J/K move · Enter open · S save

Proximal State Nudging: Reducing Skill Atrophy from AI Assistance

Megha Srivastava, Jonathan Ouyang, Eric Zhou, Andrew Silva+5

Stanford University, Stanford, CA, USA. · University of California Los Angeles, Los Angeles, CA, USA. · Toyota Research Institute, Los Altos, CA, USA.

Human-in-the-Loop AI Shared Autonomy

Open paper AskPDF arXiv

Skill atrophy, the gradual decline of human capability under AI assistance, poses a safety risk in shared-control of semi-autonomous systems, where operators may be unable to distinguish their own inputs from autonomous corrections. We propose Proximal State Nudging (PSN), a shared autonomy algorithm that jointly optimizes for skill development and task performance by nudging users toward states estimated to be most learnable. We first show that PSN outperforms existing shared autonomy baselines in balancing student improvement in unassisted reward with overall shared performance, using simulated students in the classic LunarLander environment. We then present, to the best of our knowledge, the first human subject studies of a planner incorporating learning-compatible shared autonomy: across two driving tasks in the CARLA simulator (High Performance Racing and Parallel Parking, n = 60), PSN produces up to 7x larger gains in unassisted skill than standard blended shared autonomy, while incurring 50% fewer collisions than unassisted self-practice.

Abstract

Explore similar work

Proximal State Nudging: Reducing Skill Atrophy from AI Assistance

HITL-D: Human In The Loop Diffusion Assisted Shared Control

Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter