LO-World: Unified BEV Latent Modeling for Joint 4D Occupancy and LiDAR Forecasting
Authors: Zhimin Liao, Ping Wei, Kai Mao, Shuaijia Chen
Organizations: National Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
LiDAR point clouds and 3D semantic occupancy represent two distinct modalities capturing fine-grained metric geometry and dense volumetric semantics, respectively, holding natural potential for mutual complementarity. However, existing world models typically forecast these modalities independently or cascade them heuristically, missing the opportunity for reciprocal cross-modal regularization. To address this, we propose LO-World, a unified 4D world model that forecasts both modalities within a shared BEV latent space. This single-stream formulation enables dense semantics and metric geometry to mutually reinforce each other, while eliminating multi-pipeline redundancy. Specifically, LO-World couples an efficient hybrid range-voxel autoencoder with a flow-matching generator that caches historical context to reduce iterative sampling latency. Extensive experiments demonstrate that LO-World outperforms prior state-of-the-art baselines, improving occupancy forecasting IoU by 10.9% and reducing LiDAR Chamfer Distance by 30.1%. Meanwhile, it sustains a real-time throughput exceeding 25~FPS on a single RTX 4090. Our code is available at https://github.com/lzzzzzm/LO-World.
Figures & tables
Figure 1: (a) Unified forecasting. LO-World jointly encodes occupancy and LiDAR into a shared BEV latent, forecasting both modalities through a single generative process. (b) Performance comparison. Under comparable control conditions, LO-World achieves a 10.9% relative improvement in 4D IoU and a 30.1% reduction in Chamfer Distance over prior state-of-the-art baselines.
Figure 2: Overall architecture of LO-World. Our framework consists of two components: (1) LO-VAE, which jointly tokenizes occupancy and LiDAR into a unified BEV latent space; and (2) LO-Former, a flow-matching-based generation framework that efficiently generates future unified BEV latents from Gaussian noise.
Method
Input/Condition
mIoU (%) ↑
IoU (%) ↑
FPS ↑
1s
2s
3s
Avg.
1s
2s
3s
Avg.
OccWorld-O [ 42 ]
O /Traj.
25.78
15.14
10.51
17.14
34.63
25.07
20.18
26.63
18.00
UniScene [ 13 ]
O/Traj.&Box&Map
35.37
29.59
25.08
30.01
38.34
32.70
29.09
33.38
-
I2 -World [ 19 ]
O/Traj.
47.62
38.58
32.98
39.73
54.29
49.43
45.69
49.80
37.04
GenieDrive [ 37 ]
O/Traj.
50.47
41.47
35.83
42.59
56.87
51.46
47.08
51.80
41.38
LO-World-T ∗ (Ours)
O/Traj.
52.92
44.54
38.37
45.28
57.64
53.10
49.67
53.47
48.25
Table 1: 4D occupancy forecasting performance on the Occ3D-nus validation set. “C”, “O”, and “L” denote camera, occupancy, and LiDAR modalities, respectively. ∗ denotes the single-modality variant of our method.
Method
Input/Condition
Forecasting Quality( m2 ) ↓
Generation Quality
FPS-E2E ↑
CD1s
CD2s
CD3s
Avg.
FPD ↓
B-MMD ↓
B-JSD ↓
HERMES [ 44 ]
C/Traj.
0.78
0.95
1.17
0.97
105.96
21.04
4.01
0.51
OpenDWM-DiT [ 5 ]
L/Traj.&Box&Map
1.96
2.78
3.21
2.65
381.91
5.73
19.90
0.10
LaGen [ 43 ]
L/Traj.&Box&Map
0.61
0.81
1.08
0.83
-
-
3.97
-
UniScene [ 13 ]
O/Traj.&Box&Map
-
-
-
-
976.47
13.61
31.55
0.19
LO-World-T ∗ (Ours)
L/Traj.
0.46
0.81
1.18
0.82
8.84
2.68
4.54
28.16
Table 2: 4D LiDAR forecasting performance on the nuScenes validation set. “C”, “O”, and “L” denote camera, occupancy, and LiDAR, respectively. B-MMD and B-JSD are computed from BEV point-count histograms. Values are multiplied by 104 and 102 , respectively. ∗ denotes the single-modality variant of our method.
Method
Occ. mIoU ( % ) ↑
LiDAR ( m2 ) ↓
4s
5s
6s
CD4.0s
CD5.5s
I2 -World [ 19 ]
28.59
25.32
22.55
-
-
GenieDrive [ 37 ]
31.16
27.17
23.66
-
-
LaGen [ 43 ]
-
-
-
1.32
1.73
LO-World-T
34.01
30.70
27.48
1.49
2.12
LO-World-TBM
48.16
45.74
43.52
1.10
1.52
Table 3: Long-horizon evaluation on nuScenes val set.
Method
Occupancy ( % ) ↑
LiDAR ( m2 ) ↓
Avg.mIoU
Avg.IoU
CD1s
CD3s
OpenDWM † [ 5 ]
-
-
2.27
3.35
I2 -World [ 19 ]
36.38
52.36
-
-
LO-World-T
41.31
58.30
0.91
1.77
LO-World-T †
46.00
61.08
0.51
1.28
Table 4: Performance comparison on the Waymo dataset. † denotes models fine-tuned on Waymo, while unmarked models are evaluated in a zero-shot setting without target domain fine-tuning.
Method
Frames
Condition
FVD ↓
mIoU ↑
mAP ↑
MagicDrive [ 7 ]
60
BEV + Box
217.94
18.27
11.86
UniScene [ 13 ]
32
2 Hz GT Occ.
610.15
18.59
6.24
GenieDrive [ 37 ]
81
2 Hz GT Occ.
92.78
31.03
18.89
Ours
81
12 Hz Pred. Occ.
60.41
34.83
27.15
Oracle
81
12 Hz GT Occ.
40.78
36.47
30.24
Table 5: Downstream video generation on the nuScenes validation set. “Oracle” denotes the ground-truth-conditioned reference using 12 Hz occupancy.
Method
Avg. IoU ( % ) ↑
Avg. CD ( m2 ) ↓
FPD ↓
JC ↑
FPS-E2E ↑
(a)
54.08
0.84
9.01
0.52
17.76
(b)
54.71
0.80
8.94
0.55
10.24
(c)
57.46
0.70
5.16
0.77
26.26
w/o Hb
50.01
0.92
75.23
0.65
26.30
w/o Hb ⋄
55.87
0.75
32.59
0.67
6.72
w/o Sem
–
0.97
56.12
0.60
26.27
Table 6: Ablation study on the unified multimodal design. “Hb” indicates the hybrid range-voxel representation for LiDAR, “Sem” specifies whether semantic labels are incorporated into the occupancy volume, and ⋄ denotes the high-resolution voxelization setting.
Method
Avg. mIoU ↑
Avg. CD ↓
FPS-E2E ↑
Architecture Ablation
Full-Model
46.01
0.70
26.26
w/o Context
37.19
1.58
34.34
w/o Context ‡
43.60
0.74
18.43
Layers Ablation (Context, Flow)
(4, 4)
42.81
0.90
31.71
Table 7: Ablation studies on LO-Former. ‡ indicates the depth-matched variant with 12 flow blocks and no context blocks.
Figure 3: Qualitative forecasting on nuScenes. (a) LiDAR and occupancy forecasts at 2, 4, and 6 s versus OpenDWM and I2 -World , respectively. (b) Paired forecasts under straight, right-turn, and left-turn ego-vehicle trajectories.
Setting
Value
Ego-frame bounds (m)
[−40,40)×[−40,40)×[−1,5.4)
Occupancy voxel size
0.4×0.4×0.4 m
Occupancy grid (Nx,Ny,Nz)
200×200×16
Occupancy labels
18 (free-space ID 17)
Range-image size (Hr,Wr)
32×1024
Vertical projection bounds
−30∘,+10∘
Table A1: Spatial and temporal settings for the six-frame nuScenes task. Spatial bounds follow (x,y,z) , and the ego exclusion region uses sensor-frame (x,y) coordinates. Observed frames include the current frame.
Setting
Value
LO-VAE
Occupancy embedding width
16
Stem width per modality
64
Encoder stage widths
64, 128, 256
Spatial downsampling factor
4
Latent size
50×50×16
Table A2: Architecture and sampling settings for LO-VAE and LO-Former. Latent sizes list spatial dimensions followed by channel width. Encoder widths follow stage order.
Setting
LO-VAE
LO-Former
2-frame
6-frame
12-frame
Learning rate
10−4
5×10−4
10−4
10−5
Total batch size
16
8
8
8
Training epochs
40
72
20
8
AdamW β2
0.95
0.999
0.999
0.999
Gradient clipping norm
1
15
15
15
Table A3: Training settings for LO-VAE and the successive LO-World-T forecasting stages on nuScenes. The 12-frame stage fine-tunes the six-frame model. Batch sizes are totals across eight GPUs. Warmup ratios are relative to the listed learning rates (LR).
Figure A1 : Examples with higher and lower JC. Each row pairs occupancy and LiDAR views from one example. For each modality, we show ground truth (GT), prediction, and their superimposition. The associated JC score is shown at the right.
Time (s)
LO-World-T
LO-World-TBM
mIoU ↑ (%)
IoU ↑ (%)
CD ↓ ( m2 )
mIoU ↑ (%)
IoU ↑ (%)
CD ↓ ( m2 )
0.5
60.02
64.58
0.3164
63.32
64.95
0.3066
1.0
53.56
61.72
0.4206
59.65
62.49
0.3896
1.5
48.89
59.31
0.5440
57.01
60.46
0.4829
2.0
45.15
57.14
0.6833
54.86
58.60
0.5817
2.5
42.00
55.02
0.7906
52.93
56.77
0.6526
Table A4: Twelve-frame forecasting on nuScenes at 0.5 s intervals. The divider after 3 s separates the two six-frame chunks.
Steps
Occupancy-only
Joint
LiDAR-only
mIoU ↑ (%)
IoU ↑ (%)
mIoU ↑ (%)
IoU ↑ (%)
CD ↓ ( m2 )
CD ↓ ( m2 )
1
45.85
53.60
47.20
57.52
0.83
1.01
2
45.72
53.52
47.19
57.50
0.83
0.98
3
45.47
53.49
46.77
57.48
0.75
0.89
4
45.28
53.47
46.01
57.46
0.70
0.82
8
42.51
51.46
43.80
55.33
0.68
0.81
Table A5: Sampling-step ablation for unimodal and joint LO-World-T.
Module
Parameters (M)
Time (ms)
LO-VAE
16.7
125.8
LO-Former
43.2
102.7
Total
59.9
228.5
Table A6 : Parameter count and execution time of each module.
Figure A2 : Occupancy, LiDAR, and RGB views for three scenes.
Figure A3 : Two failure cases under reverse ego-motion trajectories. The cases are shown side by side, with occupancy and LiDAR predictions paired in adjacent columns. Successive frames run from top to bottom.
World models, which simulate environmental dynamics and generate sensor observations, are gaining increasing attention in autonomous driving. However, progress in LiDAR-based world models has lagged behind those built on camera videos or occupancy data, primarily due to two core challenges: the inherent disorder of LiDAR point clouds and the difficulty of distinguishing dynamic objects from static structures. To address these issues, we propose GEM: a Generative LiDAR world model that leverages deformable mamba architecture, significantly improving fidelity and imaginative capability. Specifically, leveraging the structural similarity between sequential laser scanning and Mamba's processing mechanism, we first tokenize LiDAR sweeps into compact representations via a custom LiDAR scene tokenizer. After unsupervised disentanglement of tokenized features via a dynamic-static separator, a tri-path deformable Mamba is introduced to perform selective scanning and adaptive gating fusion over the disentangled features, leading to enhanced spatial-temporal understanding of the world evolution. Optionally, a planner and a BEV layout controller can be integrated to explore the model's capability for autonomous rollout and its potential to generate ``what-if" scenarios. Extensive experiments show that GEM achieves state-of-the-art performances across diverse benchmarks and evaluation settings, demonstrating its superiority and effectiveness. Project page: https://github.com/wuyang98/GEM.
We present TriO, a multi-modal unsupervised world model that predicts 4D occupancy, obstacle segmentation, flow and LiDAR. In contrast to prior work, TriO utilizes three distinct sensor modalities (camera, LiDAR, and RADAR) as both inputs and sources of self-supervision, eliminating the need for additional human annotations. Thanks to its novel supervision, the model is able to segment any occupancy from the drivable surface, overcoming the limitations of existing open-set methods in handling long-tail objects. TriO achieves state-of-the-art results in multiple 3D and 4D tasks, including occupancy, flow, and LiDAR prediction, as well as zero-shot road obstacle segmentation across multiple datasets such as Argoverse 2, and Spotting the Unexpected.
Quinlan Sykora, Sourav Biswas, Christopher Diehl +3
Real-time 4D occupancy forecasting requires reliably extrapolating both ego motion and surrounding actor motion under strict latency budgets. Recent occupancy world models improve long-horizon fidelity through generative decoders and 3D volumetric modeling, yet their computational cost makes real-time deployment challenging. Light-weight BEV forecasters meet latency requirements but often blur or suppress fast-moving actors at longer horizons. We present UFMF (Ultra-Fast Multi-modal Forecaster), a real-time forecaster that keeps an attention-free diagonal state-space temporal core while injecting targeted kinematic and geometric structure where lightweight models typically fail. A radar-guided velocity-aware dynamic prior (VADP) decomposes future displacement into a linear extrapolation plus a learned horizon-dependent residual. The resulting per-cell flow fields transport a compact dynamic latent subspace during rollout. Two auxiliary training constraints complement this branch: world-frame dynamic supervision enforces inter-horizon trajectory coherence, and class-conditioned vertical-profile supervision regularizes BEV-to-3D lifting. UFMF operates at 122 FPS on a single RTX 4090 and reaches 0.45 fast-displacement mIoU on nuScenes, outperforming recent occupancy world models at a fraction of their computational cost.