Learning Event-based Obstacle Avoidance via Differentiable Simulation

Chenlu Shou; Yunfan Ren; Jiaxu Xing; Davide Scaramuzza

cs.ROOct 28, 2026

Learning Event-based Obstacle Avoidance via Differentiable Simulation

Authors: Chenlu Shou, Yunfan Ren, Jiaxu Xing, Davide Scaramuzza

Organizations: Robotics and Perception Group, University of Zurich, Switzerland

Abstract

Reliable obstacle avoidance is essential for safe quadrotor flight in unknown environments, yet conventional frame-based cameras remain challenged by motion blur, limited dynamic range, and latency under fast motion. Event cameras, with sub-millisecond latency, high dynamic range, and negligible motion blur, offer a promising alternative, but leveraging them for closed-loop control remains difficult. Existing approaches either adopt cascaded architectures that first estimate depth and then map it to control commands, introducing error propagation, or achieve end-to-end control through costly offline pretraining before online fine-tuning can even begin. We address these challenges with a CUDA-based event simulator that exploits GPU parallelism to render events efficiently within the training loop, integrated into a differentiable simulation pipeline that enables direct gradient backpropagation through the differentiable physics model, improving sample efficiency. This allows us to train an end-to-end policy that maps event frames directly to control commands, without intermediate depth estimation. We validate our approach through zero-shot real-world deployment on a quadrotor, achieving robust obstacle avoidance in both odometry-assisted and odometry-free settings, including under low-light conditions where conventional cameras typically fail.

Figures & tables

Fig. 1 : Real-world flight trajectories demonstrating event-based obstacle avoidance in an odometry-free setting (top) and under challenging low-light conditions (bottom). The inset in each panel shows the onboard event-camera view at a representative time step.

Fig. 2 : System Framework Overview. The simulator renders a grayscale frame from each quadrotor state, and two consecutive frames are used to compute the event frame. The event frame is fed into a differentiable policy consisting of a CNN and a GRU, which predicts an action. The differentiable point-mass dynamics model then propagates the current state forward using this action. A loss is computed based on the resulting state and action, and the corresponding gradients are propagated backward through the entire differentiable pipeline to update the policy.

Panel of Figure 3 — Fig. 3 : Illustration of the simulation environment along with example rendered outputs. (a) shows the simulation environment, (b)-(c) the rendered grayscale frame, and (d) the resulting event frame synthesized by comparing consecutive grayscale frames.

Panel of Figure 4 — Fig. 4 : Offline evaluation of the velocity estimator on two representative real-world sequences from the UZH-FPV Drone Racing Dataset [ 29 ] . For each sequence, the left plots show the predicted versus ground-truth velocity over time, and the right plot shows the corresponding flight trajectory.

Panel of Figure 6 — Fig. 7 : Flight trajectory across consecutive event camera timestamps. (a) shows the full scene, with the drone’s position at each timestamp marked. For each timestamp $t_{1}$ – $t_{5}$ in (b)–(f), the top row shows the raw event frames and the bottom row shows the corresponding downsampled event frames.

Condition	Successful	Total Trials
Odometry-assisted	7	8
Odometry-free	3	3

TABLE I : Real-world flight outcomes under different odometry conditions.

Panel of Figure 8 — Fig. 8 : Representative comparison between our event-based policy and the depth-based baselines reported in [ 8 ] . (a) Success rate across average flight speed. (b) Top-down view of the evaluation environment and the drone’s trajectory.

Fig. 9 : Simulation throughput and memory vs. number of parallel environments

N

. (a) Throughput in environment steps per second. Our method’s advantage over ESIM widens with increasing

N

, until the GPU’s streaming multiprocessors are saturated. (b) Process memory footprint. Our method’s GPU memory grows only marginally with

N

, whereas ESIM’s host memory grows with

N

due to its per-environment simulator state.

Collision Avoidance

Collision avoidance develops methods for predicting and preventing unsafe encounters among autonomous agents, vehicles, robots, and obstacles.

See all 93 papers

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation
Apr 12, 2026Fanxing Li, Shengyang Wang, Yuxiang Huang +5Quadrotor Control Vision-Based Robot Control
Shanghai Jiao Tong University · Kunshan Duke University
Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environments
Mar 8, 2026Nico Messikommer, Jiaxu Xing, Leonard Bauersfeld +3Robot Imitation Learning Robotic Control
Robotics and Perception Group, University of Zurich, Switzerland.
Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation
Oct 27, 2026Dequan Ou, Austin Lopez, Quentin Missinne +2Robotics Simulation Event-Based Vision
All authors are with the Micro Air Vehicle Lab of the Faculty of Aerospace Engineering, Delft University of Technology, 2629 HS Delft, The Netherlands

Apr 12, 2026J/K move · Enter open · S save

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

Fanxing Li, Shengyang Wang, Yuxiang Huang, Fangyu Sun+4

Shanghai Jiao Tong University · Kunshan Duke University

Quadrotor Control Vision-Based Robot Control Robot Collision Avoidance RL for Robotics

Open paper AskPDF arXiv

Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications. When planning the trajectory, existing approaches both on optimization and learning typically regard quadrotor as a point-mass model, giving path or velocity commands then tracking the commands by outer-loop controller. However, at high speeds, planned trajectories sometimes become dynamically infeasible in actual flight, which beyond the capacity of controller. In this paper, we propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning via differentiable simulation. The high-fidelity simulation in training after parameter identification significantly reduces all the gaps between training, simulation and real world. Analytical process by differentiable simulation provides accurate gradient to ensure efficiently training the low-level policy without expert guidance. The policy employs a lightweight and the most simple inference pipeline that runs without explicit mapping, backbone networks, primitives, recurrent structures, or backend controllers, nor curriculum or privileged guidance. By inferring low-level command directly to the hardware controller, the method enables full flight envelope control and avoids the dynamic-infeasible issue.Experimental results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines across multiple benchmarks. The policy also exhibits strong generalization, successfully deploying zero-shot in unseen, outdoor environments while reaching speeds of up to 7.5m/s as well as stably flying in the super-dense forest. This work is released at https://github.com/Fanxing-LI/avoidance.

Abstract

Collision Avoidance

Explore similar work

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environments

Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation