Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation
Authors: Dequan Ou, Austin Lopez, Quentin Missinne, Christophe De Wagter, Guido C. H. E. de Croon
Organizations: All authors are with the Micro Air Vehicle Lab of the Faculty of Aerospace Engineering, Delft University of Technology, 2629 HS Delft, The Netherlands
Event cameras offer low latency, high dynamic range, and milliwatt-level power consumption, making them ideal for agile robots such as tiny drones navigating through cluttered environments. However, existing simulators generally render photorealistic images before converting them into events, making large-scale reinforcement learning prohibitively slow. We propose an analytic event simulator that exploits what event cameras fundamentally measure - moving contrast edges - to synthesize event streams directly from scene geometry and ego-motion without visual rendering. Our simulator generates a 64x64 event observation in 17 microseconds on a single GPU - over 400x faster than a state-of-the-art rendering pipeline - enabling end-to-end reinforcement learning for goal-directed obstacle avoidance to converge in under 30 minutes - three orders of magnitude faster than training on rendered events. Policies transfer zero-shot to physically based event simulations (ray-traced Isaac Sim). Our policy reaches 83% success in cluttered courses unseen during training (mid-difficulty, traversability probability approx. 0.57), at 2.4 m/s average goal velocity, and retains 70% in a simulated texture-dense indoor hall. Furthermore, we quantized the policy to a 1.2 MB int8 network running fully onboard a 56-gram Crazyflie equipped with a Prophesee GenX320 event camera and an OpenMV N6 microcontroller board. Without any fine-tuning, external processing, or positioning, the policy achieves an average success rate of 82% (5/5, 5/6, 4/6 flights) across three real-world obstacle courses of increasing difficulty. Code: https://github.com/tudelft/rendering-free-event-nav
Figures & tables
Fig. 1 : (a) A real flight through our test arena: the 56 g Crazyflie (inset) navigates the clutter fully onboard using a GenX320 event camera and an STM32N6 microcontroller. (b–d) Event frames from the three worlds (blue ON, red OFF): (b) the analytic proxy used for training and (c) the Isaac Sim+EVIS [ 1 ] rendered validation, both showing a replica of this arena from a matched viewpoint; (d) the real sensor view in flight.
Fig. 2 : The rendering-free event-proxy pipeline: one obstacle (orange) is traced through every stage. (a) A scene is only 3D edge points on depth discontinuities (trunks brown, walls gray, props green; the red triangle is the camera, the arrow its velocity). (b) Visible points project to the image plane and move under ego-motion; arrow color is polarity. (c) The two firing mechanisms: per-cell event mass follows a saturating trigger (the block-level aggregate of binary pixel crossings; its knee—the displacement at which a cell saturates—is fitted to real sensor recordings), and polarity is a hashed contrast class times the covering/revealing side of the moving edge. (d) Signed accumulation yields the 64×64 DIFF3D frame the policy observes; the traced obstacle’s streak is circled.
parallel envs
512
rollout length
128
epochs / minibatches
2 / 8
discount γ
0.99
GAE λ
0.95
clip ϵ
0.2
learning rate (Adam)
3×10−4
entropy coef.
5×10−3
value coef.
0.5
reward scale
0.05
total steps
40 M
wall-clock
34 min
TABLE I : Training configuration (identical for all policies).
Fig. 3 : Simulation evaluation under the shared action interface. Top and middle rows: the two appearance conditions with identical obstacle layouts—the shadow scene (bare surfaces, directional light) and the texture scene (court markings, panels, window bands under diffuse light)—as ray-traced views (left) and simulated 64×64 DIFF3D observations (blue ON, red OFF); the texture scene adds background events the shadow scene lacks. Bottom: all 30 evaluation episodes of our policy in each condition (every episode a distinct procedurally generated layout; blue = reached, red × = failure, dashed circle = goal), 25/30 and 21/30.
active px
ON share
p50/p90
blob
blk. var.
proxy (training)
15%
37%
2 / 6
1.19
216
Isaac+EVIS, shadow
25%
40%
5 / 14
1.59
1801
Isaac+EVIS, texture
54%
40%
3 / 13
1.61
2882
real GenX320
66%
39%
3 / 14
1.54
6947
TABLE II : Event-frame statistics of the policy’s operational worlds: proxy training scenes (champion flights, domain randomization active), the two Isaac evaluation conditions, and a real in-flight GenX320 recording. Net-count deviations from the 128 background; block variance over 4×4 -pixel sums; trajectories differ across rows.
Fig. 4 : The 56 g flight platform. Top: exploded view—Crazyflie 2.1 base (24 g), LiPo battery (9 g), OpenMV N6 board (9 g), Prophesee GenX320 event camera (3 g), printed camera mount (3 g), and flow deck (2 g) for velocity estimation. Bottom: assembled, front and side. The 56 g total includes these components, cables, and a boost converter.
sensor sim
closed loop
wall-clock to
( 64×64 frame)
(policy steps/s)
trained policy
Rendered (Isaac+EVIS)
6.9 ms
∼ 2.5
∼ 200 days †
Analytic proxy (ours)
17 μ s ‡
19–21k
34 min
† extrapolated from the measured GPU-day (195k steps).
‡ full observation path (bare event splat 4 μ s).
TABLE III : Measured event-generation and training throughput (RTX 4090, identical scenes).
input
SR (shadow)
SR (texture)
AGV
PPO, privileged depth
depth
100%
97%
2.19
PPO, analytic proxy (ours) a
events
83%
70%
2.38
Flow balancing (reactive)
events
13%
17%
1.10
PPO, rendered, 1 GPU-day
events
40%
53%
1.78
a All methods share the deployed action interface
( v∈[0,3.5] m/s at 13.3 Hz); scenes: Sec. IV-A .
TABLE IV : Baseline comparison in the Isaac protocol ( n=30 , identical seeds, props). AvoidBench metrics: success rate, average goal velocity (m/s, shadow scene), path optimality (excess over the A*-shortest path; successes only).
Fig. 5 : All real flights, overhead composites. Top: environment 1 (poles and boxes), 5/5 reached. Middle: environment 2 (extended course with block towers, plants, and screens), 5/6. Bottom: environment 3 (compact arena with tight passages), 4/6; crashes marked. All flights fully onboard—GenX320 events, int8 policy on the N6, flow-based velocity—with no external processing or positioning.
Event cameras are increasingly adopted in embodied perception for their microsecond temporal resolution, high dynamic range, and resilience to motion blur. However, training event-based models for robotics requires large-scale, action-conditioned data with dense physical annotations that are difficult to collect in the real world. We introduce EVIS, an open-source physics-grounded event simulator integrated into NVIDIA Isaac Sim that generates events from linear-HDR radiance from a closed-loop robot training episode. Rather than relying on learning-based video interpolation or expensive dense rendering, EVIS exploits renderer-provided motion vectors and depth maps through bi-directional warping and depth-based splatting. This enables high throughput, real-time, and high-fidelity event generation. We evaluate EVIS across runtime efficiency, sim-to-real transfer, and zero-shot model compatibility. A rotation-speed estimator trained solely on EVIS events achieves 2.75 rad/s MAE on real sensor data. Pretrained models for reconstruction, matching, and tracking perform competitively on EVIS events without any fine-tuning. EVIS sustains real-time generation across GPU-parallel environments on a single GPU. Code repository: https://github.com/spikelab-jhu/isaac-sim-event-camera-plugin.
Event cameras offer high temporal resolution and low latency, making them ideal sensors for high-speed robotic applications where conventional cameras suffer from motion blur. However, their widespread adoption in robot learning is severely bottlenecked by the computational cost of simulating high-frequency event data during online training. In this work, we present Approximate Imitation Learning, a novel framework that fundamentally resolves this bottleneck, reducing policy training time for complex, agile drone flight from 52.44 hours to just 1.86 hours - a 28x computational speedup. Our key insight is to separate representation learning from policy search. We first leverage a large-scale offline dataset to learn a task-specific representation space. Subsequently, the policy is fine-tuned through online interactions that rely solely on lightweight state information, completely eliminating the need to render events during the active policy search phase. This training paradigm drastically reduces development overhead and enables event-based control policies to scale to complex environments. Furthermore, our approach eliminates the reliance on standard cameras or intermediate representations during deployment, mapping events directly to control commands. In simulation, our method matches or exceeds the performance of standard imitation learning baselines that require full online event rendering. Finally, we successfully validate the framework in the real world, demonstrating that a policy trained via this ultra-efficient paradigm enables a quadrotor to fly through highly cluttered environments at remarkable speeds of up to 9.8 m/s.
Nico Messikommer, Jiaxu Xing, Leonard Bauersfeld +3
Robotics and Perception Group, University of Zurich, Switzerland.
Event-based vision is becoming an increasingly important sensing paradigm for robotics, yet its adoption remains limited by sensor availability and the lack of integrated simulation tools for modern robotics platforms. This paper presents EsaacSim, a multimodal event camera add-on for NVIDIA Isaac Sim that enables online simulation of configurable event cameras with grayscale and Bayer RGGB event generation. The framework supports multiple event camera resolutions and provides synchronized RGB, APS, event, depth, and IMU outputs through native ROS2 interfaces. A motion-guided frame-gap synthesis strategy further increases the effective temporal resolution while preserving compatibility with the Isaac Sim rendering pipeline. Experimental evaluation demonstrates synchronized multimodal simulation across representative robotic scenes and efficient online performance over five event camera resolutions at effective event rates from 240 to 960Hz. Event stream generation requires 6.98--27.28ms for grayscale events and 7.58--29.16ms for Bayer RGGB events while using less than 400MB of additional GPU memory on an NVIDIA RTX~4060 GPU. These results show that EsaacSim enables supports online multimodal event-camera simulation for robotics research and synthetic data generation. We release an early version of the simulator and report its current architecture and performance.
Leibniz Universität Hannover, L3S Research Center, Hanover, Germany · Ekumen, Buenos Aires, Argentina · University of Southern Denmark, Odense, Denmark +1