VisionSIM: Beyond Idealized Sensors - A Multi-Modal Simulator for Robot Perception

Sacha Jungerman; Kaustubh Sadekar; Avery Gump; Sriram Narayanan; Shantanu Gupta; Bhuyashi Deka; Srinivasa Narasimhan; Atul Ingle

cs.RONov 5, 2026

VisionSIM: Beyond Idealized Sensors - A Multi-Modal Simulator for Robot Perception

Authors: Sacha Jungerman, Kaustubh Sadekar, Avery Gump, Sriram Narayanan, Shantanu Gupta, Bhuyashi Deka, Srinivasa Narasimhan, Atul Ingle,

Organizations: University of Wisconsin – Madison · Portland State University · Carnegie Mellon University

Abstract

Robotic perception increasingly relies on heterogeneous sensor suites combining cameras with emerging modalities (event, SPAD, iToF/dToF, and thermal). Yet existing simulators idealize these sensors, hiding critical modality-specific failure modes that limit real-world deployment. We present VisionSIM, a unified, physically grounded simulation platform that elevates sensor realism to a first-class concern by modeling the raw physical measurement process for diverse conventional and emerging modalities. We expose fine-grained control over physical sensor parameters, reproducing failure modes overlooked by general-purpose simulators: pileup and glare for LiDAR; phase wrapping, saturation, and motion artifacts for iToF; refractory drops for event cameras; and history-dependent emission for thermal cameras. This enables principled cross-modality comparison on identical scenes and trajectories. We release the platform and a large indoor dataset with synchronized raw measurements and rich ground-truth annotations.

Figures & tables

Modality		Raw Measurement	Recoverable Scene Properties
	RGB cameras*	Visible radiance	Appearance, texture, coarse geometry cues
	Stereo / multi-view*	Multi-view radiance, parallax	Metric 3D geometry, surface layout
✹ —	LiDAR (SPAD)*	Photon arrival times; transients	Low-light depth, long-range geometry
	LiDAR (iToF)*	Correlation measurements	Depth, dynamic structure
	Radar (mmWave)	RF phase, Doppler, reflections	Coarse geometry, velocity, micro-motion
	Thermal (LWIR)*	Thermal emission (8–14 $\mu$ m)	Temperature fields, heat signatures

Table 1: Computational imaging modality taxonomy. Each modality is characterized by its raw physical measurement and the recoverable scene properties . The Sim. and HW columns indicate simulation and hardware maturity, respectively. Outlook glyphs indicate likely near- to mid-term trajectories in availability and adoption. *Currently supported by VisionSIM.

Figure 1: Multi-modal, high-fidelity and controllable sensor emulation. Robotic simulation pipelines often focus on scene content, overlooking physically accurate sensor modeling. VisionSIM aims to bridge this gap by emulating diverse sensor types and their failure modes. (Top) A single scene is rendered once with dense ground truth — color, depth, optical flow, surface normals, segmentation, point map, direct and indirect illumination, and a temperature map. (Middle) The same scene is then replayed through a breadth of sensor models. (Bottom) Each model exposes and enables sweeping across the sensor’s physical parameters (e.g., thermal time constant, correlation function, event thresholds), thereby reproducing modality-specific failure modes.

Modality	Modeled Non-idealities & Failure Modes	Key Exposed Parameters
RGB cameras	Shot/read noise, motion blur, full-well saturation, quantization	Exposure, gain, full-well capacity, bit depth, ISP artifacts
High-speed cameras	Elevated read noise, low signal, motion blur	Exposure, gain, read noise
Stereo / multi-view	Disparity error, inherited camera non-idealities	Baseline, intrinsics, per-camera parameters
✹ — LiDAR (SPAD, dToF)	Pileup, range walk, count saturation, ambient background, glare	IRF, dead time, quantum efficiency, ambient rate, TDC resolution
LiDAR (iToF)	Phase wrapping, saturation, ambient corruption, multi-tap & intra-tap motion, noise, quantization	Coding function, modulation frequency, taps $K$ , full-well capacity, read noise, ADC bits
Passive single-photon	Shot-noise, quantum efficiency, dark counts	Exposure, dark count rate, fill factor

Table 2: Failure modes and parameters modeled by VisionSIM. For each supported modality, we list the non-idealities and failure modes VisionSIM reproduces and the principal user-controllable parameters. Full forward models for all modalities are given in Appendix S.3 .

Figure 2: Indirect Time-of-Flight (iToF) simulation and failure cases: ( Top ) Ground truth RGB scene, ground truth depth map, and ideal correlation measurements (taps 1, 2, and 3) for a conventional sinusoidal coding scheme. ( Bottom ) Common iToF failure cases and their effects on depth reconstruction, including: incorrect modulation frequency (causing phase wrapping), sensor saturation (clipping correlation measurements), ambient light corruption (reducing signal-to-noise ratio, especially for farther objects), motion blur (multi-tap artifacts from scene dynamics), and sensor noise and quantization artifacts (high frequency depth errors).

Figure 3: Transient Thermal Simulation: Thermal video of ceramic, steel, and aluminum cups over the 4 seconds following a hot-tea fill. The distinct heating rates reflect each material’s thermal diffusivity ( $\alpha=2,\ 18.8,\ 97\ \mathrm{mm^{2}/s}$ , respectively).

Figure 4: Application Case-Studies: ( Top ) A digital twin of a commercial LiDAR, where sensor parameters such as dead time and impulse response function are estimated from real-world data to correct for artifacts like range-walk. ( Middle ) Sensor modality migrations: VisionSIM can uplift existing datasets (e.g., XVFI [ 84 ] ) or high-speed renders to emulate new sensor modalities like single-photon or event cameras. ( Bottom ) Sensor suite evaluation: VisionSIM enables rapid evaluation of different sensor configurations for perception tasks.

Figure 5: Simulation Fidelity. Comparison of simulated vs. captured scenes. The top two rows are simulated end-to-end using a digital twin of the real scene, while the bottom two emulate the corresponding sensors from a co-located conventional camera capture (inset).

		Annotations				Sensors
Dataset	Frames	Depth	Flow	Sem.	Pt. Maps	Camera	SPAD	Event	dToF	iToF	Therm.	IMU
nuScenes [ 10 ]	1.4M		-				-	-		-	-
SPICE-HL3 [ 74 ]	1.2M	-	-	-	-			-	-	-	-
CroMo [ 97 ]	80k		-	-	-		-	-	-		-
MPD [ 37 ]	103k	-	-		-		-	-	-	-		-
CARLA-Loc [ 33 ]	66k *		-				-			-	-

Table 3: Landscape of computer vision datasets. Existing datasets focus primarily on specific modalities (e.g., event cameras or LiDAR) or lack advanced sensor emulation (e.g., SPADs or thermal), VisionSIM provides a comprehensive suite of multi-modal data including HDR stereo views, optical flow, point maps, and a wide variety of physically-based sensor emulations in a single unified dataset.

Suppl. Fig. 1: Scalable Rendering and Interpolation: ( A ) Client/server rendering architecture: multiple rendering services can be spun up on local or remote machines, each exposing a high-level rendering API via remote procedure calls (RPCs) and broadcasting their addresses over UDP for automated discovery via a registry server. This enables multiple clients to be orchestrated in tandem to distribute rendering jobs and speed up processing. ( B ) Interpolation quality/speed trade-off: we render a scene at different frame rates and interpolate using RIFE [ 36 ] or simple frame duplication until we reach $6400$ fps. A low rendering rate can introduce severe artifacts, shown here quantitatively (left) as a high perceptual similarity score (lower is better) and qualitatively (right), while a fast rate wastes compute for marginal quality gains. ( C ) Create a pool of two render workers and distribute work amongst them. For each of the three blender files, we enable the rendering of normals (L5), slow down the animations by a factor of ten (L6), and then render everything. By instead freeing the camera from any constraints we can move it manually and generate any arbitrary view. Here we spawn and connect to a single render client and fly the camera in a circle around the origin while always looking towards the origin. We also change the resolution (L3), render out depth buffers (L4), and disable any animations (L5).

Suppl. Fig. 2: Convergence of the keyframe-multiplier search on the classroom scene. Each histogram shows the distribution of per-pixel optical flow magnitudes measured at a different iteration, with the target flow of

1

px. The 95th-percentile estimate

\hat{\mathcal{V}}_{95}

is shown as the dashed red line for each iteration. The first iteration overshoots, as many smaller flows are discovered to be around

{\raise 0.73193pt\hbox{

\scriptstyle\sim

}}0.2

px, while the second iteration finds the optimal keyframe multiplier.

Suppl. Fig. 3: VisionSIM enables physically accurate 3D sensor simulations: Conventional simulators create idealistic depth maps ( B ) from scene data ( A ). They fail to bridge the sim-to-real gap, ignoring sensor physics by emulating low temporal resolution ( C ) and low spatial resolution ( D ) via naive quantization. VisionSIM enables physically accurate emulation of sensor properties such as per-pixel FOVs ( E ); high spatial resolution with low sensor noise ( F ); dead time effects ( G ) and hardware constraints, including low spatial ( H ) and temporal ( I ) resolution.

Suppl. Fig. 4: Error Maps for digital twin evaluation; inset: RMSE between the measured depth map and the simulated depth map.

Suppl. Fig. 5: Example scenes from the VisionSIM dataset. A collection of diverse indoor scenes featuring different lighting conditions, materials, and ground truth annotations.

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 330 papers

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors
Oct 13, 2026Henry Che, Xuanchi Ren, Sanja Fidler +3World Models World Model Learning
NVIDIA · University of Illinois, Urbana-Champaign · University of Toronto
EsaacSim: A Multimodal Event Camera Add-on for NVIDIA Isaac Sim
Aug 9, 2026Ignacio Bugueno-Cordova, Malte Kuhlmann, Nicolás Navarro-Guerrero +2Event-Based Vision Robotics Simulation
Leibniz Universität Hannover, L3S Research Center, Hanover, Germany · Ekumen, Buenos Aires, Argentina · University of Southern Denmark, Odense, Denmark +1
Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
Jul 21, 2026Guanxiong Chen, Qianjun Xia, Jiawei Peng +24World Models for Robotics Robotics Simulation
University of British Columbia. · National University of Singapore. · Johns Hopkins University. +4

Oct 13, 2026J/K move · Enter open · S save

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Henry Che, Xuanchi Ren, Sanja Fidler, Shenlong Wang+2

NVIDIA · University of Illinois, Urbana-Champaign · University of Toronto

World Models World Model Learning Point Cloud Generation Autonomous Driving Simulation

Open paper AskPDF arXiv

We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.

Abstract

Robotics Simulation

Explore similar work

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

EsaacSim: A Multimodal Event Camera Add-on for NVIDIA Isaac Sim

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents