Robotic perception increasingly relies on heterogeneous sensor suites combining cameras with emerging modalities (event, SPAD, iToF/dToF, and thermal). Yet existing simulators idealize these sensors, hiding critical modality-specific failure modes that limit real-world deployment. We present VisionSIM, a unified, physically grounded simulation platform that elevates sensor realism to a first-class concern by modeling the raw physical measurement process for diverse conventional and emerging modalities. We expose fine-grained control over physical sensor parameters, reproducing failure modes overlooked by general-purpose simulators: pileup and glare for LiDAR; phase wrapping, saturation, and motion artifacts for iToF; refractory drops for event cameras; and history-dependent emission for thermal cameras. This enables principled cross-modality comparison on identical scenes and trajectories. We release the platform and a large indoor dataset with synchronized raw measurements and rich ground-truth annotations.
Figures & tables
Modality
Raw Measurement
Recoverable Scene Properties
Sim.
HW
Outlook
RGB cameras*
Visible radiance
Appearance, texture, coarse geometry cues
Stereo / multi-view*
Multi-view radiance, parallax
Metric 3D geometry, surface layout
✹ —
LiDAR (SPAD)*
Photon arrival times; transients
Low-light depth, long-range geometry
LiDAR (iToF)*
Correlation measurements
Depth, dynamic structure
Radar (mmWave)
RF phase, Doppler, reflections
Coarse geometry, velocity, micro-motion
Thermal (LWIR)*
Thermal emission (8–14 μ m)
Temperature fields, heat signatures
Table 1: Computational imaging modality taxonomy. Each modality is characterized by its raw physical measurement and the recoverable scene properties . The Sim. and HW columns indicate simulation and hardware maturity, respectively. Outlook glyphs indicate likely near- to mid-term trajectories in availability and adoption. *Currently supported by VisionSIM.
Figure 1: Multi-modal, high-fidelity and controllable sensor emulation. Robotic simulation pipelines often focus on scene content, overlooking physically accurate sensor modeling. VisionSIM aims to bridge this gap by emulating diverse sensor types and their failure modes. (Top) A single scene is rendered once with dense ground truth — color, depth, optical flow, surface normals, segmentation, point map, direct and indirect illumination, and a temperature map. (Middle) The same scene is then replayed through a breadth of sensor models. (Bottom) Each model exposes and enables sweeping across the sensor’s physical parameters (e.g., thermal time constant, correlation function, event thresholds), thereby reproducing modality-specific failure modes.
Table 2: Failure modes and parameters modeled by VisionSIM. For each supported modality, we list the non-idealities and failure modes VisionSIM reproduces and the principal user-controllable parameters. Full forward models for all modalities are given in Appendix S.3 .
Figure 2: Indirect Time-of-Flight (iToF) simulation and failure cases: ( Top ) Ground truth RGB scene, ground truth depth map, and ideal correlation measurements (taps 1, 2, and 3) for a conventional sinusoidal coding scheme. ( Bottom ) Common iToF failure cases and their effects on depth reconstruction, including: incorrect modulation frequency (causing phase wrapping), sensor saturation (clipping correlation measurements), ambient light corruption (reducing signal-to-noise ratio, especially for farther objects), motion blur (multi-tap artifacts from scene dynamics), and sensor noise and quantization artifacts (high frequency depth errors).
Figure 3: Transient Thermal Simulation: Thermal video of ceramic, steel, and aluminum cups over the 4 seconds following a hot-tea fill. The distinct heating rates reflect each material’s thermal diffusivity ( α=2,18.8,97mm2/s , respectively).
Figure 4: Application Case-Studies: ( Top ) A digital twin of a commercial LiDAR, where sensor parameters such as dead time and impulse response function are estimated from real-world data to correct for artifacts like range-walk. ( Middle ) Sensor modality migrations: VisionSIM can uplift existing datasets (e.g., XVFI [ 84 ] ) or high-speed renders to emulate new sensor modalities like single-photon or event cameras. ( Bottom ) Sensor suite evaluation: VisionSIM enables rapid evaluation of different sensor configurations for perception tasks.
Figure 5: Simulation Fidelity. Comparison of simulated vs. captured scenes. The top two rows are simulated end-to-end using a digital twin of the real scene, while the bottom two emulate the corresponding sensors from a co-located conventional camera capture (inset).
Annotations
Sensors
Dataset
Frames
Depth
Flow
Sem.
Pt. Maps
Camera
SPAD
Event
dToF
iToF
Therm.
IMU
nuScenes [ 10 ]
1.4M
-
-
-
-
-
SPICE-HL3 [ 74 ]
1.2M
-
-
-
-
-
-
-
-
CroMo [ 97 ]
80k
-
-
-
-
-
-
-
MPD [ 37 ]
103k
-
-
-
-
-
-
-
-
CARLA-Loc [ 33 ]
66k *
-
-
-
-
Table 3: Landscape of computer vision datasets. Existing datasets focus primarily on specific modalities (e.g., event cameras or LiDAR) or lack advanced sensor emulation (e.g., SPADs or thermal), VisionSIM provides a comprehensive suite of multi-modal data including HDR stereo views, optical flow, point maps, and a wide variety of physically-based sensor emulations in a single unified dataset.
Suppl. Fig. 1: Scalable Rendering and Interpolation: ( A ) Client/server rendering architecture: multiple rendering services can be spun up on local or remote machines, each exposing a high-level rendering API via remote procedure calls (RPCs) and broadcasting their addresses over UDP for automated discovery via a registry server. This enables multiple clients to be orchestrated in tandem to distribute rendering jobs and speed up processing. ( B ) Interpolation quality/speed trade-off: we render a scene at different frame rates and interpolate using RIFE [ 36 ] or simple frame duplication until we reach 6400 fps. A low rendering rate can introduce severe artifacts, shown here quantitatively (left) as a high perceptual similarity score (lower is better) and qualitatively (right), while a fast rate wastes compute for marginal quality gains. ( C ) Create a pool of two render workers and distribute work amongst them. For each of the three blender files, we enable the rendering of normals (L5), slow down the animations by a factor of ten (L6), and then render everything. By instead freeing the camera from any constraints we can move it manually and generate any arbitrary view. Here we spawn and connect to a single render client and fly the camera in a circle around the origin while always looking towards the origin. We also change the resolution (L3), render out depth buffers (L4), and disable any animations (L5).
Suppl. Fig. 2: Convergence of the keyframe-multiplier search on the classroom scene. Each histogram shows the distribution of per-pixel optical flow magnitudes measured at a different iteration, with the target flow of 1 px. The 95th-percentile estimate V^95 is shown as the dashed red line for each iteration. The first iteration overshoots, as many smaller flows are discovered to be around {\raise 0.73193pt\hbox{\scriptstyle\sim}}0.2 px, while the second iteration finds the optimal keyframe multiplier.
Suppl. Fig. 3: VisionSIM enables physically accurate 3D sensor simulations: Conventional simulators create idealistic depth maps ( B ) from scene data ( A ). They fail to bridge the sim-to-real gap, ignoring sensor physics by emulating low temporal resolution ( C ) and low spatial resolution ( D ) via naive quantization. VisionSIM enables physically accurate emulation of sensor properties such as per-pixel FOVs ( E ); high spatial resolution with low sensor noise ( F ); dead time effects ( G ) and hardware constraints, including low spatial ( H ) and temporal ( I ) resolution.
Suppl. Fig. 4: Error Maps for digital twin evaluation; inset: RMSE between the measured depth map and the simulated depth map.
Suppl. Fig. 5: Example scenes from the VisionSIM dataset. A collection of diverse indoor scenes featuring different lighting conditions, materials, and ground truth annotations.
We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.
Henry Che, Xuanchi Ren, Sanja Fidler +3
NVIDIA · University of Illinois, Urbana-Champaign · University of Toronto
Event-based vision is becoming an increasingly important sensing paradigm for robotics, yet its adoption remains limited by sensor availability and the lack of integrated simulation tools for modern robotics platforms. This paper presents EsaacSim, a multimodal event camera add-on for NVIDIA Isaac Sim that enables online simulation of configurable event cameras with grayscale and Bayer RGGB event generation. The framework supports multiple event camera resolutions and provides synchronized RGB, APS, event, depth, and IMU outputs through native ROS2 interfaces. A motion-guided frame-gap synthesis strategy further increases the effective temporal resolution while preserving compatibility with the Isaac Sim rendering pipeline. Experimental evaluation demonstrates synchronized multimodal simulation across representative robotic scenes and efficient online performance over five event camera resolutions at effective event rates from 240 to 960Hz. Event stream generation requires 6.98--27.28ms for grayscale events and 7.58--29.16ms for Bayer RGGB events while using less than 400MB of additional GPU memory on an NVIDIA RTX~4060 GPU. These results show that EsaacSim enables supports online multimodal event-camera simulation for robotics research and synthetic data generation. We release an early version of the simulator and report its current architecture and performance.
Leibniz Universität Hannover, L3S Research Center, Hanover, Germany · Ekumen, Buenos Aires, Argentina · University of Southern Denmark, Odense, Denmark +1
Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on brittle workflow glue across visual perception tools and simulators: manual tuning of visual foundation models, mesh cleanup, coordinate frame alignments, etc. We introduce Agentic Real2Sim, a framework for generalized physical world modeling with vision-language agents that converts a real-world recording of object-robot interaction into a simulatable episodic twin, and connects the resulting twin to downstream policy fine-tuning and evaluation. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining a comparable conversion success rate. The framework further supports custom scene conversion, fine-tuning of a pretrained policy with data generated from converted episodes, and works effectively as a surrogate for real-world policy evaluation. The project site, including code is available at https://agentic-real2sim.github.io.
Guanxiong Chen, Qianjun Xia, Jiawei Peng +24
University of British Columbia. · National University of Singapore. · Johns Hopkins University. +4