Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Henry Che; Xuanchi Ren; Sanja Fidler; Shenlong Wang; Laura Leal-Taixé; Shengyu Huang

cs.CVOct 13, 2026

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Authors: Henry Che, Xuanchi Ren, Sanja Fidler, Shenlong Wang, Laura Leal-Taixé, Shengyu Huang

Organizations: NVIDIA · University of Illinois, Urbana-Champaign · University of Toronto

Abstract

We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.

Figures & tables

Figure 1: LidarVerse unifies generative and reconstruction-grounded LiDAR sensor simulation. By changing the available scene evidence, one distilled causal checkpoint generates new worlds, forecasts future point clouds, or resimulates observed scenes along new sensor trajectories with real-time, long-horizon inference.

Figure 2: Architecture : LidarVerse is trained in three stages that address representation, task unification, and efficient deployment. (a) A fully attention-based LiDAR tokenizer learns a common latent space while preserving spatial and temporal structure (Sec. 3.1 ). (b) A bidirectional teacher learns a unified interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text (Sec. 3.2 ). (c) Causal distillation and a rolling persistent spatial cache enable real-time, long-horizon inference with the same conditioning interface (Sec. 3.3 ).

Figure 3: Unified inference : the same causal checkpoint performs LiDAR generation, point-cloud forecasting, and generalizable LiDAR resimulation through different condition combinations. All tasks additionally receive the requested target sensor poses; gray labels denote omitted conditions.

Configuration	Bidirectional training	Causal initialization	Self forcing distillation
Global batch size	32	32	32
Training clip length	9 $\rightarrow$ 21 frames	21 frames	21 frames
Model optimizer	AdamW $\epsilon=10^{-8}$ $\mathrm{lr}=10^{-5}$ $\beta=(0.9,0.999)$ $\mathrm{wd}=0.01$	AdamW $\epsilon=10^{-8}$ $\mathrm{lr}=10^{-5}$ $\beta=(0.9,0.999)$ $\mathrm{wd}=0.01$	AdamW $\epsilon=10^{-8}$ $\mathrm{lr}=2\times 10^{-6}$ $\beta=(0.9,0.999)$ $\mathrm{wd}=0.01$
Fake-score optimizer	–	–	AdamW $\epsilon=10^{-8}$ $\mathrm{lr}=4\times 10^{-7}$ $\beta=(0.9,0.999)$ $\mathrm{wd}=0.01$
RF timestep shift	5	5	5
Distillation teacher CFG	–	–	3.0

Table 1: Training configurations for our three-stage pipeline.

Method	ST Comp. rate	Reconstruction			Generation			TTCE $\downarrow$		CTC $\downarrow$		ICP Avg. $\downarrow$	TFLOPs /frame $\downarrow$
Method	ST Comp. rate	CD $\downarrow$	$L_{1}$ $\downarrow$	AbsRel $\downarrow$	JSD $\downarrow$	MMD $\downarrow$	FUD $\downarrow$	3	4	3	4	ICP Avg. $\downarrow$	TFLOPs /frame $\downarrow$
ST compression rate: 256x
Cosmos CV ( NVIDIA et al., 2025b )	8x8x4	0.439	0.790	0.031	0.061	6.41	46.26	1.773	2.561	0.322	0.357	0.089	0.058
Wan2.1 ( Team Wan et al., 2025 )	8x8x4	0.297	0.395	0.017	0.057	5.72	37.52	1.983	2.832	0.337	0.381	0.077	0.911
Ours	16x16x1	0.175	0.143	0.008	0.061	6.87	32.84	1.729	2.492	0.311	0.357	0.072	0.400
ST compression rate: 32x / 64x

Table 2: Tokenizer comparison using the same DiT. ST denotes spatiotemporal compression. Reconstruction and generation use nine-frame sequences. MMD values are multiplied by 10 5 .

Figure 4: HD-Map-conditioned generation with the bidirectional model. Sample 0 depicts a high-speed highway, Sample 1 a dense intersection with pedestrians, and Sample 2 a moving truck approaching from behind. Each sequence is conditioned only on the HD-Map.

Figure 5: Unconditional generation with the bidirectional model. LidarVerse produces high-quality and diverse unconditional samples. Sample 0 depicts a highway scene, Sample 1 dense traffic, and Sample 2 an intersection. All conditioning inputs are set to null.

Figure 7: Weather-controlled generation with the text-conditioned bidirectional teacher. We modify the original sunny-scene prompt to request rainy weather.

Figure 8: Autoregressive LiDAR generation : LidarVerse generates realistic, temporally consistent LiDAR sequences over long horizons, whereas the baselines collapse after 100 frames.

Method	Sampling steps	$\text{JSD}\downarrow$	$\text{MMD}\downarrow$	FUD $\downarrow$	TTCE $\downarrow$		CTC $\downarrow$				ICP Avg. $\downarrow$	FPS $\uparrow$
Method	Sampling steps	$\text{JSD}\downarrow$	$\text{MMD}\downarrow$	FUD $\downarrow$	3	4	1	2	3	4	ICP Avg. $\downarrow$	FPS $\uparrow$
HD-Map-conditioned generation
LiDARCrafter ( Liang et al., 2026 )	256	0.065	14.13	97.24	1.263	1.989	0.278	0.360	0.441	0.519	0.120	0.127
LaGen ∗ ( Zhou et al., 2025b )	35	0.075	19.19	50.29	1.584	2.296	0.205	0.268	0.330	0.390	0.071	0.402
Sensor2Sensor ∗ ( Wang et al., 2026 )	35	0.047	5.210	37.96	1.522	2.236	0.225	0.287	0.347	0.404	0.081	0.421
Ours	4	0.046	4.390	25.01	1.491	2.255	0.210	0.253	0.293	0.331	0.068	11.18

Table 3: Autoregressive LiDAR generation. We evaluate the causal LidarVerse checkpoint. MMD values are multiplied by 10 5 . ∗ indicates a reimplementation using our tokenizer and a matched DiT.

Figure 10: Point-cloud forecasting : LidarVerse preserves dynamic agents and synthesizes plausible geometry as the future viewpoint reveals regions not observed in the input.

Method	1 s			2 s			3 s			Aggregated
Method	$L_{1}$ $\downarrow$	AbsRel $\downarrow$	CD $\downarrow$	$L_{1}$ $\downarrow$	AbsRel $\downarrow$	CD $\downarrow$	$L_{1}$ $\downarrow$	AbsRel $\downarrow$	CD $\downarrow$	$L_{1}$ $\downarrow$	AbsRel $\downarrow$	CD $\downarrow$
Raycasting ( Khurana et al., 2023 )	2.461	0.171	1.584	3.161	0.217	3.343	3.768	0.251	6.049	2.807	0.193	2.739
4D-Occ ( Khurana et al., 2023 )	2.146	0.113	2.173	2.602	0.139	3.230	3.052	0.170	4.575	2.369	0.126	2.924
UnO ∗ ( Agro et al., 2024 )	2.040	0.112	1.000	2.300	0.130	1.117	2.654	0.165	1.440	2.208	0.126	1.070
Ours	1.482	0.090	0.485	2.104	0.126	0.805	2.554	0.156	1.328	1.768	0.108	0.676

Table 4: Point-cloud forecasting. ∗ indicates a reimplementation to the best of our ability.

Figure 11: Generalizable LiDAR resimulation : the causal LidarVerse checkpoint achieves strong resimulation fidelity in both the interpolation setting (frames 3 and 7) and the extrapolation setting (frames 17 and 19). It requires no per-scene optimization and renders in real time.

Rank	Sequence ID	Rank	Sequence ID
1	2506799708748258165_6455_000_6475_000	11	16767575238225610271_5185_000_5205_000
2	5372281728627437618_2005_000_2025_000	12	13573359675885893802_1985_970_2005_970
3	6491418762940479413_6520_000_6540_000	13	271338158136329280_2541_070_2561_070
4	15396462829361334065_4265_000_4285_000	14	10203656353524179475_7625_000_7645_000
5	18305329035161925340_4466_730_4486_730	15	10335539493577748957_1372_870_1392_870
6	13415985003725220451_6163_000_6183_000	16	1457696187335927618_595_027_615_027

Table 5: Top-20 travel-distance validation sequences. Sequences are ranked by total ego path length over the full log.

Method	All-202						Top-20 Travel Distance						Optim. time $\downarrow$	Real-time render
	Interpolation		Extrapolation		Aggregate		Interpolation		Extrapolation		Aggregate
	$L_{1}\downarrow$	CD $\downarrow$	$L_{1}\downarrow$	CD $\downarrow$	$L_{1}\downarrow$	CD $\downarrow$	$L_{1}\downarrow$	CD $\downarrow$	$L_{1}\downarrow$	CD $\downarrow$	$L_{1}\downarrow$	CD $\downarrow$
LiDAR4D ( Zheng et al., 2024 )	1.129	0.188	1.881	0.460	1.364	0.273	1.999	0.998	3.284	1.769	2.401	1.239	89.8	$\times$
LiDAR-RT ( Zhou et al., 2025a )	1.089	0.182	1.593	0.418	1.247	0.256	1.905	0.444	3.094	1.292	2.277	0.709	76.3	✓
SplatAD ( Hess et al., 2025 )	1.799	0.265	2.325	0.408	1.964	0.309	3.125	0.357	4.654	0.947	3.603	0.542	15.0	✓
Ours (bidir)	1.124	0.169	1.376	0.270	1.202	0.200	1.399	0.184	1.838	0.354	1.536	0.237	0.00	$\times$

Table 6: Generalizable LiDAR resimulation. The causal student is our default real-time model; the bidirectional teacher is included as a quality reference. Per-scene optimization time is measured in minutes.

Method	Interp.	Extrap.	Agg.
LiDAR4D	0.077	0.094	0.082
LiDAR-RT	0.083	0.100	0.089
SplatAD	0.115	0.134	0.121
Ours	0.078	0.091	0.082

Table 7: Generalizable LiDAR resimulation intensity RMSE on the top-20 travel-distance subset. Ours denotes the intensity-fine-tuned bidirectional teacher. Lower is better.

Figure 12: Cross-dataset reconstruction. One tokenizer handles 32-, 64-, and 128-beam LiDAR configurations and generalizes to PandaSet, which is not included in training.

Metric	Model	nuScenes	Waymo	NVIDIA PAI
$L_{1}\downarrow$	Specialist	0.087	0.056	0.286
$L_{1}\downarrow$	Universal	0.065	0.037	0.087
CD $\downarrow$	Specialist	0.557	0.718	1.240
CD $\downarrow$	Universal	0.568	0.710	1.135

Table 8: Universal vs. specialist tokenizer reconstruction. Reconstruction is evaluated on matched 21-frame sequences. Metrics are measured in meters; lower is better.

Figure 13: Cross-dataset generation. By changing the sensor embedding, our fine-tuned bidirectional teacher generates distinct LiDAR scanning patterns from the same Waymo HD-Map input.

Figure 14: Cross-dataset first-frame-conditioned generation with intensity. The fine-tuned bidirectional teacher produces high-quality cross-dataset generation with intensity in a single model, enabled by the universal tokenizer. Each sequence is conditioned on its first ground-truth LiDAR frame.

Method	Generation			Resimulation
Method	FUD $\downarrow$	MMD $\downarrow$	ICP Avg. $\downarrow$	$L_{1}$ $\downarrow$	CD $\downarrow$
Specialist	14.67	6.950	0.047	1.380	0.235
Unified	16.05	6.843	0.044	1.536	0.237

Table 10: Unified versus specialists. MMD values are multiplied by 10 5 .

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 366 papers

Nov 26, 2025

LaGen: Towards Autoregressive LiDAR Scene Generation

Generative world models for autonomous driving (AD) are of great value in applications such as data augmentation, closed-loop simulation, and safety-critical scenario evaluation. Unlike the widely studied image modality, in this work we explore generative world models for LiDAR data. Existing generation methods for LiDAR predominantly focus on single frame generation or lack the capacity for interactive simulation, while existing prediction approaches require multiple frames of historical input and can only deterministically predict multiple frames at once. Both paradigms fail to support long-horizon interactive generation. To this end, we introduce LaGen, which, to the best of our knowledge is the first autoregressive framework capable of generating long-horizon LiDAR scenes in a frame-by-frame, interactive manner. LaGen is able to take a single-frame input as a starting point and effectively utilize bounding box information as conditions to generate high-fidelity 4D scene. In addition, we introduce a scene decoupling estimation module to enhance the model's interactive generation capability for object-level content, as well as a noise modulation module to mitigate error accumulation during long-horizon generation. We extensively evaluate LaGen's performance in controlled data generation and long-horizon scene generation on the nuScenes dataset. The experimental results demonstrate that LaGen achieves state-of-the-art performance, especially on later frames. The code is publicly available at: https://github.com/szzhou88/LaGen.

Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang +7

University of Science and Technology of China, Hefei, Anhui 230026, China · Shanghai Innovation Institute, Shanghai 200231, China · Fudan University, Shanghai 200433, China +1

3D Scene Generation Autoregressive Generation World Models for Autonomous Driving Conditional Generative Modeling 4D Scene Generation Autonomous Driving Simulation+2

Ask PDF

May 8, 2026

GEM: Generating LiDAR World Model via Deformable Mamba

World models, which simulate environmental dynamics and generate sensor observations, are gaining increasing attention in autonomous driving. However, progress in LiDAR-based world models has lagged behind those built on camera videos or occupancy data, primarily due to two core challenges: the inherent disorder of LiDAR point clouds and the difficulty of distinguishing dynamic objects from static structures. To address these issues, we propose GEM: a Generative LiDAR world model that leverages deformable mamba architecture, significantly improving fidelity and imaginative capability. Specifically, leveraging the structural similarity between sequential laser scanning and Mamba's processing mechanism, we first tokenize LiDAR sweeps into compact representations via a custom LiDAR scene tokenizer. After unsupervised disentanglement of tokenized features via a dynamic-static separator, a tri-path deformable Mamba is introduced to perform selective scanning and adaptive gating fusion over the disentangled features, leading to enhanced spatial-temporal understanding of the world evolution. Optionally, a planner and a BEV layout controller can be integrated to explore the model's capability for autonomous rollout and its potential to generate ``what-if" scenarios. Extensive experiments show that GEM achieves state-of-the-art performances across diverse benchmarks and evaluation settings, demonstrating its superiority and effectiveness. Project page: https://github.com/wuyang98/GEM.

Yang Wu, Zhaojiang Liu, Qiang Meng +5

SJTU · FDU · NTU +1

Mamba World Model Learning World Models Autonomous Driving World Models for Autonomous Driving

Ask PDF

Jun 1, 2026

Not All Points Are Equal: Uncertainty-Aware 4D LiDAR Scene Synthesis

Constructing faithful 4D worlds from LiDAR-acquired sequences is crucial for embodied AI, yet current generative frameworks apply uniform modeling capacity across all spatial regions. This ignores that perceptual difficulty varies dramatically within a single scan: distant surfaces, occluded boundaries, and small-scale objects carry far higher uncertainty than well-observed structures. We present U4D, a new framework that explicitly leverages spatial uncertainty to guide LiDAR scene generation in a "hard-to-easy" schedule. U4D derives per-point uncertainty maps via Shannon Entropy from a pretrained segmentor, then applies an unconditional diffusion stage to synthesize high-entropy areas with precise geometry, followed by a conditional completion stage that fills in the remaining regions using these structures as priors. A MoST (Mixture of Spatio-Temporal) block further maintains cross-frame coherence by dynamically balancing spatial detail and temporal continuity. Extensive experiments on nuScenes and SemanticKITTI demonstrate state-of-the-art scene fidelity, temporal consistency, and downstream performance.

Xiang Xu, Alan Liang, Youquan Liu +5

1NUAA · 2NUS · 3FDU +4

3D Scene Generation Generative Modeling 4D Scene Understanding Diffusion Models 4D Scene Generation

Ask PDF

Abstract

World Models

Explore similar work

LaGen: Towards Autoregressive LiDAR Scene Generation

GEM: Generating LiDAR World Model via Deformable Mamba

Not All Points Are Equal: Uncertainty-Aware 4D LiDAR Scene Synthesis