cs.ROFeb 16, 2026

Neurosim: A Fast Simulator for Neuromorphic Robot Perception

Authors: Richeek Das, Pratik Chaudhari

Organizations: GRASP Laboratory, University of Pennsylvania

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Neurosim is a fast, real-time, high-performance library for simulating sensors such as dynamic vision sensors, RGB cameras, depth sensors, and inertial sensors. It can also simulate agile dynamics of multi-rotor vehicles in complex and dynamic environments. Neurosim can achieve frame rates as high as ~2700 FPS on a desktop GPU. Neurosim integrates with a ZeroMQ-based communication library called Cortex to facilitate seamless integration with machine learning and robotics workflows. Cortex provides a high-throughput, low-latency message-passing system for Python and C++ applications, with native support for NumPy arrays and PyTorch tensors. This paper discusses the design philosophy behind Neurosim and Cortex. It demonstrates how they can be used to (i) train neuromorphic perception and control algorithms, e.g., using self-supervised learning on time-synchronized multi-modal data, and (ii) test real-time implementations of these algorithms in closed-loop. Neurosim and Cortex are available at https://github.com/grasp-lyrl/neurosim .

Figures & tables

Figure 1: Event cameras capture the scene without temporal aliasing, with high dynamic range, and consume very little power. We illustrate the temporal non-aliasing property of event cameras in the left image. Unlike standard RGB cameras, event cameras are asynchronous sensors that respond to scene intensity changes and produce a continuous stream of information at high temporal and spatial precision.
Figure 2: Overview of the design of Neurosim. Neurosim is designed to be modular, high-performance, and easy to use for a variety of applications in embodied perception on multirotors. As shown in (A) , it consists of four main components: (1) A data stack in the form of 3D scene assets and the specifications for sensors and multirotors, (2) A real-time rendering engine for high-fidelity vision sensor simulation, (3) A fast and accurate multirotor dynamics model for aerodynamics and physics, and (4) A communication interface to interact with the simulator – receive sensor data and send control commands. In (B) , we illustrate how we can use the communication interface to connect Neurosim to a visualizer – here Rerun [ 16 ] – and draw the simulated data in real-time. It displays the rendered RGB image, depth map, IMU readings, 6-DoF pose, navigation mesh, and the event stream. For visualization, events are accumulated over short time windows (20 ms\mathrm{ms} ) and colored by polarity.
Figure 3: Neurosim event simulator backend is optimized to prevent bottlenecking of the full simulation pipeline. We benchmark Neurosim’s event camera simulator against existing simulators: a CUDA-based simulator [ 15 ] , a GPU-based PyTorch implementation, and a CPU-based implementation [ 19 ] . A. We measure the average latency to simulate events from a single intensity image update – the time taken to process a new intensity frame, trigger events based on the contrast threshold model, update the internal pixel states, and output the event stream. Neurosim achieves over 31 kHz for VGA and 23 kHz for HD frame sizes. It is roughly 8–13 ×\times faster than other GPU-based implementations, and 55–121 ×\times faster than CPU-based implementations. B. Neurosim using the custom event simulator backend achieves ∼\sim 2300 FPS for a full simulation step (including rendering, dynamics, and event simulation) with VGA vision sensors. Other event simulation backends bottleneck the full simulation pipeline, achieving only 200–1200 FPS. C. Neurosim’s event simulator only takes up ∼\sim 8 % of the total simulation step time compared to 40–90 % for other event simulators. This high performance is attributed to the warp-synchronous CUDA kernel design that minimizes atomic operations during event aggregation and allows our event generation model to run in a single kernel launch.
Figure 4: Cortex provides high-throughput, low-latency, and scalable communication in Python. We benchmark different aspects of Cortex’s performance on a modern desktop CPU (AMD Ryzen 9 7950X). A. We use NumPy float (32 bit\mathrm{bit} ) arrays of different sizes ranging from 10 elements (40 B\mathrm{B} ) to 1080 ×\times 1920 RGB images (23.7 MB\mathrm{MB} ) as our message payloads. We measure the max achieved successful publish-subscribe rate (in Hz\mathrm{Hz} ) for each payload size over a 15 s\mathrm{s} interval. Cortex achieves 100+ kHz\mathrm{kHz} for small ( << 39 kB\mathrm{kB} ) messages and maintains sufficiently high rates (250 Hz\mathrm{Hz} ) for large 1080p RGB images. B. We measure the achieved throughput (in MB s−1\mathrm{MB}\text{\,}{\mathrm{s}}^{-1} ) for the same payload sizes. Cortex achieves up to 7 GB s−1\mathrm{GB}\text{\,}{\mathrm{s}}^{-1} throughput for large messages, saturating the underlying hardware limits. For smaller messages, the throughput is limited by per-message overheads – the messaging rate caps at ∼\sim 100 kHz\mathrm{kHz} . C. We test the scalability of Cortex by varying the number of subscribers from 1 to 32, each receiving 40 B\mathrm{B} messages from a single publisher. The achieved message rate per subscriber remains constant at around 100 kHz\mathrm{kHz} as the number of subscribers increases up to 8, after which it degrades due to CPU contention from many busy-waiting subscribers on the same machine. D. We measure the achieved throughput (in MB s−1\mathrm{MB}\text{\,}{\mathrm{s}}^{-1} ) for the same setup as in C. As deduced from the message rates, the throughput scales roughly linearly with the number of subscribers until CPU contention effects kick in beyond 8 subscribers.
Figure 5: Examples of a quadrotor tracking randomly generated MinSnap trajectories in a variety of indoor scenes. In each example, from left to right, we plot the color image, depth image (near to far is colored as dark blue to red), and events aggregated with polarity over 20 ms\mathrm{ms} , on the top. 3D trajectory with current pose, navigation mesh with current position, and IMU readings are plotted at the bottom. The navigation mesh marks free space and non-reachable locations at the current flying height with gray and white, respectively. These scenes demonstrate fast flights across multiple floors, corridors, and interconnected rooms, with angular and linear velocities often exceeding 4 rad s−1\mathrm{rad}\text{\,}{\mathrm{s}}^{-1} and 2 m s−1\mathrm{m}\text{\,}{\mathrm{s}}^{-1} , respectively.
Figure 6: Neurosim can be used for closed-loop controller testing, online data generation for learning and on-policy reinforcement learning. (A) shows a latency and frequency analysis of Neurosim in typical indoor flying scenes for closed-loop control. The simulator runs at 10 kHz\mathrm{kHz} , performing dynamics updates. The controller and simulator nodes work separately and communicate asynchronously using Cortex. We vary the controller rate (same as the sensor rendering rate) from 100 Hz\mathrm{Hz} to 5 kHz\mathrm{kHz} to measure the effects of communication-simulation latencies and bandwidth saturation. We use a VGA (640 ×\times 480) RGB camera as our sensor, that is, each sensor observation is ∼\sim 0.9 MB\mathrm{MB} in size. (i) shows the average end-to-end latency experienced by the controller – the time taken from sending a control command to receiving the next sensor observation after the command has been applied. This latency captures the communication overhead as well as the simulator processing time. As the controller and target sensor rate increase, the communication latency increases slightly due to increased IPC bandwidth contention. However, the overall latency remains below 0.7 ms\mathrm{ms} even at 5 kHz\mathrm{kHz} controller rates. We also plot the ideal latency at a particular control rate, that is, the time taken to take control in real-time. Any control rate achieving a controller-to-observation latency less than the ideal latency is guaranteed to receive the latest sensor observations at the control rate. (ii) shows the sensor observation rate received by the controller vs the desired sensor rate. The controller is able to receive the VGA color sensor readings at the target rate up to 2.3 kHz\mathrm{kHz} , after which it plateaus. This plateau is the current limit of the simulator’s rendering speed on a single Nvidia 4090 GPU, combined with the communication overhead of Cortex. (B) shows qualitative examples of learning event-based monocular depth. (i) shows an example MinSnap trajectory sampled in a complex indoor environment, avoiding obstacles. For visualization, the trajectory is
More in this topic

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 319 papers

Explore similar work

CardsList
Aug 25, 2026

CRESSim-Neo: A Batched GPU Simulation Engine for Surgical Robotics and Robot Learning

We introduce CRESSim-Neo, a batched GPU simulation engine for surgical robotics and robot learning. CRESSim-Neo combines position-based simulation of rigid bodies, deformable tissues, fluids, and strands with batched rendering, surgery-specific sensing, and a GPU-resident data pipeline. The engine supports applications including tissue manipulation, fluid suction, suturing, cable-driven robots, and ultrasound image synthesis. Direct access to physics and rendering buffers enables GPU-resident robot learning and zero-copy PyTorch integration using DLPack. We demonstrate CRESSim-Neo across rigid-body, deformable-body, and fluid simulation tasks, including vision-based and surgical robot-learning scenarios. On an NVIDIA RTX 4090, the engine achieves up to 2.03 million environment steps per second for 8192 parallel CartPole environments, and scales to batched surgical scenarios involving tissue deformation, fluid interaction, and ultrasound sensing. Overall, CRESSim-Neo provides a unified and scalable platform for surgical simulation, synthetic data generation, and surgical robot learning.

Yafei Ou, Ahnaf Naheen, Tleukhan Mussin +3

Department of Electrical and Computer Engineering; 2Department of Biomedical Engineering, University of Alberta, Edmonton, Alberta, Canada.

Robotics SimulationSynthetic Data Generation for RoboticsPhysics-Based SimulationSurgical RoboticsDeformable Object Simulation
AskPDF
Sep 1, 2026

Real-Time Neuromorphic Spectrum Intelligence Simulator

We present the Real-Time Neuromorphic Spectrum Intelligence Simulator (RT-NuSIS), a modular framework to study spiking neural network (SNN) and memristor-inspired agents for dynamic spectrum access under constrained energy budgets and adversarial conditions. RT-NuSIS couples leaky integrate-and-fire neuronal dynamics, memristive synaptic models, physics-informed energy-harvesting models (triboelectric and RF), and adversary models including jamming and Byzantine behavior. We formalize the simulator mathematically, prove boundedness, present a mean-field adversary threshold, analyze per-step complexity, and provide a reproducible benchmark harness for energy-per-inference, latency, and robustness metrics. The codebase is modular, deterministic by seed, and designed for large-scale event-driven simulations.

Navaneetha Krishnan Kamalakannan

SIMATS Engineering Saveetha University

Neural Network RobustnessSpiking Neural NetworksNeuromorphic ComputingWireless Resource Allocation
AskPDF
Aug 9, 2026

SuperNeuroMAT: An Efficient Matrix-based Simulator for Spiking Neural Networks

Spiking neural networks (SNNs) offer a promising pathway to energy-efficient AI and brain-inspired computing. However, their widespread adoption is hindered by a lack of fast, accessible, and versatile simulation frameworks. In this paper, we introduce SuperNeuroMAT, an open-source, scalable, and highly efficient Python-based SNN simulator. We devise a novel matrix-based approach to model the leaky integrate-and-fire (LIF) neuron dynamics and natively support dense and sparse execution modes. This enables fast simulation of approximately 10,000 neurons in dense mode and 100,000 neurons in sparse mode on standard laptops and desktops without requiring specialized hardware. We demonstrate that SuperNeuroMAT consistently outperforms four established SNN simulators---NEST, Brian2, BindsNET, and snnTorch---on two performance metrics (execution speed and peak resident memory) and across various network sizes and connection probabilities. Furthermore, we demonstrate SuperNeuroMAT's applicability across a diverse set of problems. SuperNeuroMAT can efficiently handle conventional machine learning benchmarks such as the Digits and citation network datasets as well as neuromorphic event-based vision tasks such as N-CARS and ASL-DVS. Moreover, it can be extended beyond machine learning workloads and facilitate general-purpose workloads. We validated this by implementing the neuromorphic shortest path algorithm and two arithmetic primitives (addition and multiplication). SuperNeuroMAT can be installed via the Python Package Index (PyPI), thereby lowering the barrier to entry into the field of neuromorphic computing and accelerating the broader development of neuromorphic algorithms.

Prasanna Date, Kevin Zhu, Shruti Kulkarni +12

Oak Ridge National Laboratory, Oak Ridge, 37830, Tennessee, United States. · George Mason University, Fairfax, 22030, Virginia, United States. · University of Tennessee, Knoxville, Knoxville, 37996, Tennessee, United States. +1

LIF NeuronsSpiking Neural NetworksNeuromorphic Computing
AskPDF