cs.ROFeb 16, 2026
SaveNeurosim: A Fast Simulator for Neuromorphic Robot Perception
Organizations: GRASP Laboratory, University of Pennsylvania
Abstract
Neurosim is a fast, real-time, high-performance library for simulating sensors such as dynamic vision sensors, RGB cameras, depth sensors, and inertial sensors. It can also simulate agile dynamics of multi-rotor vehicles in complex and dynamic environments. Neurosim can achieve frame rates as high as ~2700 FPS on a desktop GPU. Neurosim integrates with a ZeroMQ-based communication library called Cortex to facilitate seamless integration with machine learning and robotics workflows. Cortex provides a high-throughput, low-latency message-passing system for Python and C++ applications, with native support for NumPy arrays and PyTorch tensors. This paper discusses the design philosophy behind Neurosim and Cortex. It demonstrates how they can be used to (i) train neuromorphic perception and control algorithms, e.g., using self-supervised learning on time-synchronized multi-modal data, and (ii) test real-time implementations of these algorithms in closed-loop. Neurosim and Cortex are available at https://github.com/grasp-lyrl/neurosim .
Figures & tables
Figure 1: Event cameras capture the scene without temporal aliasing, with high dynamic range, and consume very little power. We illustrate the temporal non-aliasing property of event cameras in the left image. Unlike standard RGB cameras, event cameras are asynchronous sensors that respond to scene intensity changes and produce a continuous stream of information at high temporal and spatial precision.
Figure 2: Overview of the design of Neurosim. Neurosim is designed to be modular, high-performance, and easy to use for a variety of applications in embodied perception on multirotors. As shown in (A) , it consists of four main components: (1) A data stack in the form of 3D scene assets and the specifications for sensors and multirotors, (2) A real-time rendering engine for high-fidelity vision sensor simulation, (3) A fast and accurate multirotor dynamics model for aerodynamics and physics, and (4) A communication interface to interact with the simulator – receive sensor data and send control commands. In (B) , we illustrate how we can use the communication interface to connect Neurosim to a visualizer – here Rerun [ 16 ] – and draw the simulated data in real-time. It displays the rendered RGB image, depth map, IMU readings, 6-DoF pose, navigation mesh, and the event stream. For visualization, events are accumulated over short time windows (20 ) and colored by polarity.
Figure 3: Neurosim event simulator backend is optimized to prevent bottlenecking of the full simulation pipeline. We benchmark Neurosim’s event camera simulator against existing simulators: a CUDA-based simulator [ 15 ] , a GPU-based PyTorch implementation, and a CPU-based implementation [ 19 ] . A. We measure the average latency to simulate events from a single intensity image update – the time taken to process a new intensity frame, trigger events based on the contrast threshold model, update the internal pixel states, and output the event stream. Neurosim achieves over 31 kHz for VGA and 23 kHz for HD frame sizes. It is roughly 8–13 faster than other GPU-based implementations, and 55–121 faster than CPU-based implementations. B. Neurosim using the custom event simulator backend achieves 2300 FPS for a full simulation step (including rendering, dynamics, and event simulation) with VGA vision sensors. Other event simulation backends bottleneck the full simulation pipeline, achieving only 200–1200 FPS. C. Neurosim’s event simulator only takes up 8 % of the total simulation step time compared to 40–90 % for other event simulators. This high performance is attributed to the warp-synchronous CUDA kernel design that minimizes atomic operations during event aggregation and allows our event generation model to run in a single kernel launch.
Figure 4: Cortex provides high-throughput, low-latency, and scalable communication in Python. We benchmark different aspects of Cortex’s performance on a modern desktop CPU (AMD Ryzen 9 7950X). A. We use NumPy float (32 ) arrays of different sizes ranging from 10 elements (40 ) to 1080 1920 RGB images (23.7 ) as our message payloads. We measure the max achieved successful publish-subscribe rate (in ) for each payload size over a 15 interval. Cortex achieves 100+ for small ( 39 ) messages and maintains sufficiently high rates (250 ) for large 1080p RGB images. B. We measure the achieved throughput (in ) for the same payload sizes. Cortex achieves up to 7 throughput for large messages, saturating the underlying hardware limits. For smaller messages, the throughput is limited by per-message overheads – the messaging rate caps at 100 . C. We test the scalability of Cortex by varying the number of subscribers from 1 to 32, each receiving 40 messages from a single publisher. The achieved message rate per subscriber remains constant at around 100 as the number of subscribers increases up to 8, after which it degrades due to CPU contention from many busy-waiting subscribers on the same machine. D. We measure the achieved throughput (in ) for the same setup as in C. As deduced from the message rates, the throughput scales roughly linearly with the number of subscribers until CPU contention effects kick in beyond 8 subscribers.
Figure 5: Examples of a quadrotor tracking randomly generated MinSnap trajectories in a variety of indoor scenes. In each example, from left to right, we plot the color image, depth image (near to far is colored as dark blue to red), and events aggregated with polarity over 20 , on the top. 3D trajectory with current pose, navigation mesh with current position, and IMU readings are plotted at the bottom. The navigation mesh marks free space and non-reachable locations at the current flying height with gray and white, respectively. These scenes demonstrate fast flights across multiple floors, corridors, and interconnected rooms, with angular and linear velocities often exceeding 4 and 2 , respectively.
Figure 6: Neurosim can be used for closed-loop controller testing, online data generation for learning and on-policy reinforcement learning. (A) shows a latency and frequency analysis of Neurosim in typical indoor flying scenes for closed-loop control. The simulator runs at 10 , performing dynamics updates. The controller and simulator nodes work separately and communicate asynchronously using Cortex. We vary the controller rate (same as the sensor rendering rate) from 100 to 5 to measure the effects of communication-simulation latencies and bandwidth saturation. We use a VGA (640 480) RGB camera as our sensor, that is, each sensor observation is 0.9 in size. (i) shows the average end-to-end latency experienced by the controller – the time taken from sending a control command to receiving the next sensor observation after the command has been applied. This latency captures the communication overhead as well as the simulator processing time. As the controller and target sensor rate increase, the communication latency increases slightly due to increased IPC bandwidth contention. However, the overall latency remains below 0.7 even at 5 controller rates. We also plot the ideal latency at a particular control rate, that is, the time taken to take control in real-time. Any control rate achieving a controller-to-observation latency less than the ideal latency is guaranteed to receive the latest sensor observations at the control rate. (ii) shows the sensor observation rate received by the controller vs the desired sensor rate. The controller is able to receive the VGA color sensor readings at the target rate up to 2.3 , after which it plateaus. This plateau is the current limit of the simulator’s rendering speed on a single Nvidia 4090 GPU, combined with the communication overhead of Cortex. (B) shows qualitative examples of learning event-based monocular depth. (i) shows an example MinSnap trajectory sampled in a complex indoor environment, avoiding obstacles. For visualization, the trajectory is
More in this topic
Robotics Simulation
Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.
See all 320 papers