Tasks such as quadrotor dynamic-gate racing require agents to anticipate external motion while maintaining precise high-frequency control. World-model-based reinforcement learning provides a natural solution by learning latent dynamics for future imagination, yet it often entangles environmental evolution and the robot's own dynamics in a single learned rollout, where model bias can affect policy optimization efficiency. We propose DiffDreamer, a differentiable reinforcement learning framework that tightly couples a learned world model with differentiable simulation for actor optimization. In one optimization step, the learned world model propagates latent states that capture external dynamics and history-dependent features, while the differentiable simulator propagates the robot's explicit physical state. We further decompose the actor objective into a critic-driven pathway for non-differentiable task rewards and a simulator-driven pathway for differentiable motion rewards. Experiments on aggressive tracking and dynamic-gate racing show that DiffDreamer improves training convergence over the model-free, model-based, and differentiable-RL baselines, and final performance over these baselines and classical planning and control methods. Additional experiments assess reward decomposition and the odometry-conditioned relative-state representation through ablations. We also provide proof-of-concept real-world flight experiments for validation.
Figures & tables
Fig. 1: A schematic overview of DiffDreamer for dynamic-gate racing. The quadrotor must predict the future motion of dynamic gates while executing precise flight control. DiffDreamer trains the actor through a tightly coupled dual-pathway process: a learned world model imagines future environmental states (e.g., target motion), while a differentiable simulation predicts the quadrotor’s physical dynamics and provides analytical gradients. The policy takes both implicit latent states and explicit physical states as input to generate control commands. The quadrotor is digitally reconstructed and calibrated through system identification for zero-shot transfer to the real world.
Fig. 2: Coupling external-motion prediction with internal control. Left: Catching a ball requires imagining its trajectory and planning body motion. Right: DiffDreamer combines latent world-model features with explicit physical states for policy optimization. It imagines future external motion through a learned world model and propagates the robot’s physical state through differentiable simulation.
Fig. 3: Real-world experiments using DiffDreamer for aggressive tracking and dynamic-gate racing. Left: flight trajectories and selected consecutive frames showing tight target tracking. Right: dynamic-gate racing, where the quadrotor anticipates the future positions of moving gates.
Fig. 4: The experimental results. (A) The predicted trajectory of the target over a 2-second horizon in aggressive tracking. The scattered points denote imagined trajectory samples. (B) Heatmaps illustrating the spatial distribution of the target within the quadrotor’s body frame. More intense colors correspond to higher spatial density. The region bounded by the blue rays represents the camera’s actual field of view (FOV). (C) The predicted trajectories of the next two gates over a 2-second horizon in dynamic-gate racing. (D) The average number of successfully passed gates during dynamic-gate racing.
Fig. 5: Training reward curves for aggressive tracking (top) and dynamic-gate racing (bottom). Solid lines and shaded regions represent the mean and standard deviation across five independent random seeds, respectively.
Fig. 6: Ablation studies. Left and center: Odometry/representation ablation: reconstruction-error densities and target distributions in the quadrotor’s XY/XZ planes. Relative and absolute world models are compared with Elastic Tracker (labeled [1]) [ 13 ] ; blue rays bound the camera field of view. Right: Reward ablation: gates passed within 512 steps versus maximum gate speed (m/s), using the full reward or removing the discrete gate-passing or differentiable motion component.
Predictive world models enable robots to plan by imagining the outcomes of their actions, but their value for control hinges on generating many rollouts quickly. This creates a bottleneck for diffusion-based world models: multistep sampling makes each rollout expensive, limiting large-scale action search at inference time. We introduce DriftWorld, an action-conditioned world model based on drifting generative models. Rather than denoising iteratively at inference, DriftWorld learns an action-conditioned drift during training, allowing it to generate future frames from the current observation and a candidate action sequence in a single forward pass at 30+ fps, which is 17x faster on average than diffusion based baselines. We evaluate DriftWorld on standard vision-based robotic manipulation benchmarks, including Bridge-V2, RT-1, Language Table, Push-T, and Robomimic. By producing rollouts that are both accurate and fast, DriftWorld achieves state-of-the-art decision-making performance with far less inference time than diffusion-based world model baselines. Beyond online control, DriftWorld can also serve as an offline simulator for ranking real-world robot policies, with rollout-based scores correlating with ground truth at up to 0.99. These results show that drifting models are a strong fit for robot world modeling, where fast, high-quality imagination directly supports planning and policy evaluation.
Susie Lu, Haonan Chen, Weirui Ye +1
Massachusetts Institute of Technology · Harvard University
Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications. When planning the trajectory, existing approaches both on optimization and learning typically regard quadrotor as a point-mass model, giving path or velocity commands then tracking the commands by outer-loop controller. However, at high speeds, planned trajectories sometimes become dynamically infeasible in actual flight, which beyond the capacity of controller. In this paper, we propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning via differentiable simulation. The high-fidelity simulation in training after parameter identification significantly reduces all the gaps between training, simulation and real world. Analytical process by differentiable simulation provides accurate gradient to ensure efficiently training the low-level policy without expert guidance. The policy employs a lightweight and the most simple inference pipeline that runs without explicit mapping, backbone networks, primitives, recurrent structures, or backend controllers, nor curriculum or privileged guidance. By inferring low-level command directly to the hardware controller, the method enables full flight envelope control and avoids the dynamic-infeasible issue.Experimental results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines across multiple benchmarks. The policy also exhibits strong generalization, successfully deploying zero-shot in unseen, outdoor environments while reaching speeds of up to 7.5m/s as well as stably flying in the super-dense forest. This work is released at https://github.com/Fanxing-LI/avoidance.
Fanxing Li, Shengyang Wang, Yuxiang Huang +5
Shanghai Jiao Tong University · Kunshan Duke University
First-order reinforcement learning with differentiable simulation is promising for quadrotor control, but practical progress remains fragmented across task-specific settings. To support more systematic development and evaluation, we present a unified differentiable framework for multi-task quadrotor control. The framework is wrapped, extensible, and equipped with deployment-oriented dynamics, providing a common interface across four representative tasks: hovering, tracking, landing, and racing. We also present the suite of first-order learning algorithms, where we identify two practical bottlenecks of standard first-order training: limited state coverage caused by horizon initialization and gradient bias caused by partially non-differentiable rewards. To address these issues, we propose Amended Backpropagation Through Time (ABPT), which combines differentiable rollout optimization, a value-based auxiliary objective, and visited-state initialization to improve training robustness. Experimental results show that ABPT yields the clearest gains in tasks with partially non-differentiable rewards, while remaining competitive in fully differentiable settings. We further provide proof-of-concept real-world deployments showing initial transferability of policies learned in the proposed framework beyond simulation.