Dream to Assist: Intent-Conditioned World Models for Shared Vehicle Control

Jonathan DeCastro; Thomas Balch; Emily Sumner; Deepak Gopinath; Laporsha Dees; Patricio Reyes Gomez; Elliot Weiss; Megan Applegate-Kenton

cs.ROOct 26, 2026

Dream to Assist: Intent-Conditioned World Models for Shared Vehicle Control

Authors: Jonathan DeCastro, Thomas Balch, Emily Sumner, Deepak Gopinath, Laporsha Dees, Patricio Reyes Gomez, Elliot Weiss, Megan Applegate-Kenton,

Organizations: Toyota Research Institute, Cambridge, MA 02139, USA · Toyota Research Institute, Los Altos, CA 94022, USA · Massachusetts Institute of Technology, Cambridge, MA 02139, USA

Abstract

Assistive agents in shared-control settings face a dual challenge: acting proactively to align with the human's intentions while remaining legible enough for the human to comprehend and fluently respond in real time. We present Dream2Assist, a model-based reinforcement learning framework for tightly-coupled shared decision making that addresses both sides of this challenge. Our approach extends a recurrent state-space world model with intent inference, enabling an assistive policy trained via fictitious co-play to anticipate and support discrete human objectives without overriding human agency. Beyond the computational framework, we describe the full deployment pipeline needed to translate a world-model-based policy into a safe, human-centered system in a resource-efficient way. Crucially, we address the sim-to-real-human gap via policy down-selection and reward restructuring within an iterative human-machine interface design. We validate the framework through three human-subjects studies and a preliminary on-vehicle deployment spanning two qualitatively distinct domains: high-speed racing (N=30) and urban driving with richer intent structure (N=10). Our results demonstrate that intent conditioning yields measurable improvements in torque alignment and collision rate. We surface key design insights and practices for deploying learning-based assistive agents in safety-critical, real-time human-robot teaming.

Figures & tables

Figure 1: Dream2Assist RSSM: We blend human actions linearly with the actions from an unfrozen assistive agent. For both human and assistive agents, a multi-head RSSM architecture is then used to predict the observations, reward, and human intent (assistive only), trained to maximize their log-likelihoods against samples taken from the environment. The intent head is trained to match frozen human intents, which are fixed a priori. The assistive agent’s rewards are shaped based on the optimal policy $\pi^{*}_{\hat{y}^{A}}$ and optimal predicted reward $r^{*}_{\hat{y}^{A}}$ for inferred intent $\hat{y}^{A}$ .

Figure 2: Racing (left) and urban (right) domains.

Figure 3: The GRIP physical test platform used for immersive VR-based mixed sim/real testing.

Figure 4: Overall workflow diagram. Notice the combination of iterative refinement and selection of policy choices, as well as the iterative piloting of the overall policy and interface behavior.

Study	$\boldsymbol{N}$	Environment (map)	Comparison	Purpose
Eval 1 : Racing characterization	24	CARLA (Thunderhill)	Dream2Assist -assisted vs. unassisted driving	Characterize shared control interaction across track geometries; identify failure modes for targeted refinement
Eval 2 : Racing refinement	6	CARLA (Thunderhill)	Refined vs. earlier configuration (within-subjects)	Validate that targeted refinements (data augmentation, output smoothing, reward restructuring) improved the interaction
Eval 3 : Urban validation	10	Unreal Engine (CrazyTown)	Intent-conditioned vs. no-intent (double-blind A/B)	Controlled test isolating the contribution of intent conditioning with the complete Dream2Assist framework
Physical platform (GRIP)	3	GRIP + VR Unreal (CrazyTown)	Intent-conditioned vs. no-intent	Feasibility demonstration of policy transfer to higher-fidelity motion platform; no retraining

Table 1: Overview of human-subjects studies. Each study used a specific system and interface configuration detailed in Table 2 .

	Policy Training Configuration						Runtime / Interface Configuration
Study	Human data augmentation	FCP output smoothing	Intent class rebalancing	Longitudinal	Reward + Intents	Safety layer	Trajectory visualization	Haptics mode	Output smoothing
Eval 1: Racing characterization ( $N{=}24$ )	✗	✗	✗	✗	Baseline task rewards (Appendix B ); default pass/progress scalars. $\mathcal{Y}=\{\text{pass},\text{stay}\}$ .	✗	Line by proximity to opponent; color by accel/decel	Steering setpoint of human+policy; direct pedal blending	✗
Eval 2: Racing refinement ( $N{=}6$ )	✓ (20%)	✓	✗	✗	Augmented stay-behind and pass rewards (( 12 )–( 14 )). $\mathcal{Y}=\{\text{pass},\text{stay}\}$ .	✗	Line by proximity to opponent; color by accel/decel	Steering setpoint of human+policy; direct pedal blending	✗
Eval 3: Urban validation ( $N{=}10$ )	✗	✓	✓	✗	Refined collision, boundary, speed, proximity features, left/right passing rewards (Appendix B ). $\mathcal{Y}=\{\text{left},\text{right}\}$ .	✓ (traj. viz. only)	Line by policy actions (alignment); color by accel/decel; windowed smoothing	Steering setpoint of human+policy; direct pedal blending	✓
GRIP platform ( $N{=}3$ )	✗	✓	✓	✓	As urban + longitudinal behaviors (stop/go rewards in Appendix B ). $\mathcal{Y}=\{\text{left},\text{right}\}\times\{\text{go},\text{slow},\text{stop}\}$ .	✓ (traj. viz. only)	As urban validation	As urban; minimal parameter adjustment for platform hardware	✓

Table 2: Configuration reference for each study (see Table 1 for study overview). Sections III and IV describe all Dream2Assist components in full generality; this table records which were active and how they were configured in each deployment. ✓ = active; ✗ = not used. In Eval 1 , D2A refers to the configuration shown in the first row. D2A and D2A -Refined are used exclusively in Eval 2 . D2A -Full , D2A -NoIntent , and No-policy are used exclusively in Eval 3 and GRIP.

	Str.	Turn	All	vs. Neutral
Question	$M\pm SE$	$M\pm SE$	$M\pm SE$	$t(23)$	$p$
AI knew my intent to pass $\uparrow$	$5.49\pm.09$	$5.09\pm.13$	$5.33\pm.08$	$10.63$	${<}\,.001$ ***
AI helped me perform the maneuver $\uparrow$	$4.01\pm.13$	$3.16\pm.14$	$3.67\pm.10$	$0.44$	$.666$
AI’s action was safe $\uparrow$	$4.55\pm.11$	$3.80\pm.14$	$4.25\pm.09$	$3.60$	$.002$ **
Better off without AI $\downarrow$	$4.82\pm.11$	$5.12\pm.12$	$4.94\pm.08$	$3.57$	$.002$ **

Table 3: Post-drive survey ratings for Eval 1 (

N=24

), Likert 1–7 (Neutral = 4). One-sample

t

-tests against neutral (two-tailed,

df=23

\uparrow

higher is better;

\downarrow

reverse-coded. ***

p<.001

; **

p<.01

. Straight vs. turn breakdowns: intent recognition higher on straights (

p=.001

); helpfulness above neutral on straights (

p=.054

) but below on turns (

p=.025

); safety above neutral on straights (

p<.001

) but not turns (

p=.057

Outcome	Track	No-policy	D2A	$\Delta$
		$M\pm\mathit{SE}$	$M\pm\mathit{SE}$
Overtake Rate $\uparrow$	Straight	$0.810\pm.034$	$\mathbf{0.899\pm.027}$	$+8.9\%$
	Turn	$\mathbf{0.547\pm.058}$	$0.274\pm.046$	$-27.3\%$
Spinout Rate $\downarrow$	Straight	$0.146\pm.030$	$\mathbf{0.132\pm.030}$	$-1.4\%$
	Turn	$\mathbf{0.293\pm.053}$	$0.326\pm.048$	$+3.3\%$
Off-Track Rate $\downarrow$	Straight	$0.073\pm.022$	$\mathbf{0.070\pm.023}$	$-0.3\%$

Table 4: Behavioral Outcomes by Track Type: D2A vs. No-policy (

N=24

). Values are proportions pooled over trials; the paired tests in the text use per-participant proportions.

Outcome	No-policy	D2A	$\Delta$
	$M\pm\mathit{SE}$	$M\pm\mathit{SE}$
Overtake Rate $\uparrow$	$\mathbf{0.721\pm.031}$	$0.630\pm.032$	$-9.0\%$
Spinout Rate $\downarrow$	$\mathbf{0.195\pm.027}$	$0.209\pm.027$	$+1.3\%$
Off-Track Rate $\downarrow$	$\mathbf{0.088\pm.019}$	$0.104\pm.020$	$+1.6\%$
Collision Rate $\downarrow$	$\mathbf{0.014\pm.008}$	$0.026\pm.011$	$+1.2\%$
Right-Side Pass	$0.376\pm.037$	$0.465\pm.038$	$+8.9\%$

Table 5: Overall Behavioral Outcomes: D2A vs. No-policy (

N=24

). Values are proportions pooled over trials; the paired tests in the text use per-participant proportions. Note that overall this initial policy inhibited the driver.

Metric	No-policy	D2A	$\Delta$
	$M\pm\mathit{SE}$	$M\pm\mathit{SE}$
Mean Overtake Speed (m/s) $\uparrow$	$\mathbf{26.39\pm.29}$	$26.27\pm.28$	$-0.13$
Min Opp. Clearance (m) $\uparrow$	$1.62\pm.10$	$\mathbf{1.77\pm.48}$	$+0.14$

Table 6: Overtake Quality Metrics: D2A vs. No-policy (Overtaking Trials Only). Values are means

\pm

SE over overtaking trials.

Metric	Track	No-policy	D2A	$\Delta$
		$M\pm\mathit{SE}$	$M\pm\mathit{SE}$
Mean O/T Speed (m/s) $\uparrow$	Straight	$\mathbf{27.89\pm.22}$	$27.44\pm.22$	$-0.46$
	Turn	$\mathbf{22.67\pm.51}$	$21.06\pm.36$	$-1.62$
Min Opp. Clearance (m) $\uparrow$	Straight	$1.50\pm.11$	$\mathbf{1.97\pm.59}$	$+0.47$
	Turn	$\mathbf{1.99\pm.21}$	$1.01\pm.22$	$-0.98$

Table 7: Overtake Quality Metrics by Track Type: D2A vs. No-policy (Overtaking Trials Only)

Figure 5: Aligned minus misaligned AI steering per participant under D2A and D2A -Refined (

n=5

; P106 excluded for missing steering data). Higher values mean more steering in the driver’s intended direction. Thin lines are participants, the thick line the mean; paired

t

-test

p=.28

Figure 6: AI steering around overtake onset (

t=0

), all participants and segments ( D2A : 52 passing events; D2A -Refined : 50). Top: steering magnitude. Bottom: signed steering. Lines show mean

\pm

SEM; individual events dimmed.

Figure 7: Policy preferences between the initial vs. updated policy faceted by position where the overtake occurred.

Figure 8: Averaged torque traces for participant 03 in the oncoming car event. About 2.3 to 1.5 s before closest approach (

t=0

), D2A -Full steers away from the oncoming car, in line with the driver’s avoidance, while D2A -NoIntent steers toward it (bottom trace). In the final second the pattern reverses: D2A -Full turns toward the car and D2A -NoIntent is near neutral.

Panel of Figure 9 — Figure 9: (a) Overall torque alignment counts, (b) torque magnitude, as defined in Eq. ( 9 ), and (c) aligned torque, as defined in Eq. ( 10 ) for the oncoming car event, where each point represents one participant. Higher values indicate a higher average torque in alignment with the driver’s intent to pass left or right around the oncoming car. Lines connect each participant’s two conditions. Panel (a) counts all 12 sessions: P04, drawn hollow, has no D2A -NoIntent trials, and P12, excluded from the subjective analysis, is included. D2A -Full had more alignment events for 8 participants, tied for 2 and fewer for 1. Of the 11 participants with both conditions, 7 received higher overall torque under D2A -Full (b) and 8 experienced more intent-aligned torque (c).

Metric	D2A	D2A -Refined	$\Delta$	$p$
	$M\pm SE$	$M\pm SE$
Likability $\uparrow$	$2.95\pm.42$	$3.31\pm.39$	$+.36$	$.108$
Helpfulness $\uparrow$	$2.80\pm.33$	$2.85\pm.20$	$+.05$	$.887$

Table 8: Subjective ratings: D2A vs. D2A -Refined (

N=6

). Likert 1–7. Paired

t

-tests (

df=5

). This pilot targeted directional trends rather than statistical significance. Five out of six participants rated D2A -Refined to be higher on likability.

Metric	D2A -Full	D2A -NoIntent	$\Delta$	$p$
	$M\pm SE$	$M\pm SE$
Directionality $\uparrow$	$4.92\pm.32$	$4.65\pm.29$	$+.27$	$.528$
Timing $\uparrow$	$5.10\pm.17$	$5.05\pm.30$	$+.05$	$.891$
Adapting Dir. $\uparrow$	$4.90\pm.36$	$4.55\pm.33$	$+.35$	$.368$
Fight Dir. $\downarrow$	$3.62\pm.48$	$3.62\pm.27$	$+.00$	$1.00$
Safety $\uparrow$	$4.38\pm.40$	$4.45\pm.23$	$-.07$	$.870$

Table 9: Paired comparison of subjective ratings: D2A -Full vs. D2A -NoIntent (

N=10

). Likert 1–7; Overall Rating 1–10. Two-tailed paired

t

-tests (

df=9

); none reached significance.

\uparrow

higher is better;

\downarrow

lower is better (less resistance from AI). Participants qualitatively distinguished the two policies despite non-significant ratings (Sec. 6.4 ).

Participant	D2A -Full	D2A -NoIntent	$\Delta$
P01	5.75	4.00	$+1.75$
P02	4.75	5.75	$-1.00$
P03	7.50	4.75	$+2.75$
P05	7.25	6.50	$+0.75$
P06	5.50	6.50	$-1.00$
P07	5.75	8.25	$-2.50$

Table 10: Per-Participant Overall Ratings: D2A -Full vs. D2A -NoIntent . Overall Rating on a 1-10 scale; values averaged across trials.

Panel of Figure 10 — Figure 10: Two examples from GRIP experiments: an oncoming car scenario where lateral assistance can be observed (top) and a roundabout-approach scenario where longitudinal assistance is observed (bottom).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Architecture	Output
Encoder $q_{\phi}^{\kappa}$	MLP: 2 layers $\times$ 512 units	Input to posterior
Recurrent model $f_{\phi}^{\kappa}$	GRU w/ LayerNorm, 512 hidden	$h_{t}^{\kappa}\in\mathbb{R}^{512}$
Posterior $q_{\phi}^{\kappa}(z_{t}\|h_{t},x_{t})$	MLP: 1 layer $\times$ 512 units	$z_{t}^{\kappa}$ (Cat. $32\times 32$ )
Prior $p_{\phi}^{\kappa}(\hat{z}_{t}\|h_{t})$	MLP: 1 layer $\times$ 512 units	$\hat{z}_{t}^{\kappa}$ (Cat. $32\times 32$ )
Decoder $p_{\phi}^{\kappa}(\hat{x}_{t}\|s_{t})$	MLP: 2 layers $\times$ 512 units	$\hat{x}_{t}^{\kappa}$ (symlog MSE)
Reward head $p_{\phi}^{\kappa}(\hat{r}_{t}\|s_{t})$	MLP: 2 layers $\times$ 512 units	symlog disc.

Appendix

Table 11: RSSM component architecture. All MLP layers use SiLU activations and LayerNorm. “Cat.” denotes a categorical distribution parameterized by logits.

	Racing (CARLA)	Urban / GRIP
Num. agents in env	2	up to 10
Max. opponents in obs	1	3
Action repeat	2	2
Episode time limit	1000 steps	1000 steps
Observation components (ego-centric)
Ego state	position, velocity, heading (rel. to track)

Appendix

Table 12: Observation and action spaces by domain. All observations are expressed in the ego vehicle’s body frame.

Hyperparameter	Value
World model
Batch size	16
Sequence length	64
Train ratio	512
Learning rate	$1\times 10^{-4}$
Optimizer	Adam ( $\epsilon=10^{-8}$ )

Appendix

Table 13: Training hyperparameters. All values are shared between the racing and urban configurations unless marked.

Parameter	Racing	Urban
Collision penalties
Inner collision	$-5{,}000$	$-10{,}000$
Outer collision	—	$-10{,}000$
Boundary violation	$-100$	$-10{,}000$
Intent rewards (human training)
Pass/left reward scale	10	$\alpha_{\text{lat}}^{+}$

Appendix

Table 14: Key reward scalars by domain and training role. “Human” denotes training of FCP sub-populations; “AI” denotes training of the assistive policy.

Reward term	Type	Racing	Urban	GRIP
Collision Eq. ( 17 )	$r^{task}$	✓	✓	✓
Boundary Eq. ( 18 )–( 19 )	$r^{task}$	✓	✓	✓
Smoothness Eq. ( 20 )–( 21 )	$r^{task}$	✓	✓	✓
Speed / progress Eq. ( 22 )–( 23 )	$r^{task}$	✓	✓	✓
Finish-line bonus	$r^{task}$	✓	✗	✗
Pass / stay Eq. ( 12 )–( 14 )	$r^{y}$	✓	✗	✗

Appendix

Table 15: Mapping of reward terms to the general framework (Eqs. ( 1 ) and ( 5 )) and to each domain. ✓ = active; ✗ = not used.

Name	Description	Domain	Platform
Policy Training
Real human data augmentation	20% of replay buffer replaced with relabeled real human driving data to reduce sim-to-real gap.	Racing Urban	Sim
FCP population diversity	Synthetic humans sampled at varying training checkpoints (suboptimal to optimal) to expose the assistant to diverse behaviors.	Racing Urban	Sim
Action smoothing	Causal moving-average (window $=10$ ) applied to synthetic human and AI outputs to suppress physically unrealistic jitter.	Racing Urban	Sim GRIP
Intent class re-balancing	Training reweights intent classes to prevent mode collapse toward a dominant intent.	Urban	Sim
Pass/stay reward restructuring	Bounded per-intent indicator reward plus speed progress term, capped to prevent single-mode collapse (Eq. ( 12 )–( 14 )).	Racing	Sim

Appendix

Table 16: Design variables and human factors considered in Dream2Assist , organized by category. Domain and platform indicate where each factor was actively applied. Factors marked (excluded) were considered but rejected during piloting.

Figure 11: Interfaces tested during iterative piloting for the racing domain, showing variations in trajectory line color, length, and style.

Figure 12: Final trajectory visualization used in Eval 3 (Urban Validation). Line length scales with assistance magnitude; green indicates acceleration, red indicates deceleration.

Figure 13: Driving flow for Eval 1 (Racing Characterization). Each block consists of two follow laps and two overtake-attempt laps, with difficulty increasing via a staircase design.

Figure 14: Per-participant overtake rate by track type, without and with the policy, in Eval 1 (

n=23

participants with both conditions). Thin lines are participants, colored by the direction of change; the thick line is the mean.

p

-values are paired

t

-tests.

Figure 15: Driving flow for Eval 3 (Urban Validation).

Panel of Figure 16 — Figure 16: Averaged torque traces across all participants and all (a) straight and (b) turn trials for Eval 2 . On turns, D2A applies larger steering 4–3 s before overtake onset ( $t=0$ ) than D2A -Refined ; on straights the two are similar. Traces show raw signed steering, not steering relative to the driver’s intent.

Figure 17: Event-locked traces for the roundabout-approach scenario on GRIP, aligned to the roundabout-gate crossing (

t=0

). Top: vehicle speed. Bottom: distance to the crossing vehicle. Lines show mean

\pm

SEM with individual trials dimmed;

n

= trials per condition.

Panel of Figure 18 — Figure 18: Averaged torque traces for selected participants in the oncoming car event in Eval 3 . Top: torque magnitude. Bottom: signed torque, positive away from the oncoming car (aligned with the driver’s avoidance). From 3 to 1.5 s before closest approach, D2A -Full is less misaligned than D2A -NoIntent for all seven participants shown; in the final second, D2A -Full is the more misaligned of the two for five of them (P01, P02, P03, P05, P12).

Shared Autonomy

Shared autonomy combines human input with autonomous control to improve safety, task performance, and ease of interaction.

See all 37 papers

DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation
Oct 3, 2026Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez +1Language-Conditioned Robot Manipulation Robot Imitation Learning
Knowledge Technology Group, Department of Informatics, University of Hamburg, 20146 Hamburg, Germany · Mathematics Department, Faculty of Science, Aswan University, 81528 Aswan, Egypt · School of Electronics and Computer Science, University of Southampton, Southampton SO17 1BJ, UK
Dreaming Of Others: Latent Teammate Modeling In World Models For Multi-Agent Reinforcement Learning
May 29, 2026Tomas Leroy-StoneMulti-Agent Coordination Theory of Mind
Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter
Sep 26, 2026Mengxue Fu, Ethan Xu, Sam Iyer-Singh +3Shared Autonomy Assistive Robotics
Collaborative Robotics Lab (Collab), Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061 · RT2 Lab, Dept. of Computer Science, UW-Madison

Oct 3, 2026J/K move · Enter open · S save

DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation

Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter

Knowledge Technology Group, Department of Informatics, University of Hamburg, 20146 Hamburg, Germany · Mathematics Department, Faculty of Science, Aswan University, 81528 Aswan, Egypt · School of Electronics and Computer Science, University of Southampton, Southampton SO17 1BJ, UK

Language-Conditioned Robot Manipulation Robot Imitation Learning Imitation Learning Embodied World Models

Open paper AskPDF arXiv

We introduce DreamFormer, a model-based agent that acquires language-conditioned, multi-task skills by imitating expert demonstrations within the latent imagination of a learned world model. DreamFormer first learns a task-agnostic Transformer world model from unstructured play data, then acquires task-specific behaviors by optimizing an intrinsic reward that aligns agent-generated rollouts with expert demonstrations in latent space. Since the policy is trained on-policy inside imagination, it is exposed to its own errors during training, mitigating the covariate shift inherent to offline behavioral cloning. To make long-horizon imagination affordable, DreamFormer encodes a high-resolution multi-view robotic observation into a single input token, avoiding both spatial downsampling and the multi-token representations used by prior Transformer world models. On the long-horizon CALVIN benchmark, DreamFormer outperforms LUMOS, the comparable model-based agent, on single-environment evaluation (2.52 vs 2.34 average tasks completed per chain of five) while keeping imagination rollouts tractable. Against HULC, the behavior cloning baseline, it nearly doubles performance on zero-shot transfer to an unseen environment (1.30 vs 0.67), indicating that dynamics learned by the world model transfer more readily than a directly cloned policy. This is consistent with the role attributed to internal models in biological agents, where a model of environment dynamics supports behavior in situations not previously encountered.

Abstract

Shared Autonomy

Explore similar work

DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation

Dreaming Of Others: Latent Teammate Modeling In World Models For Multi-Agent Reinforcement Learning

Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter