Current 4D occupancy world models use heavy cross-attention or deep recurrence to disentangle ego-motion from scene dynamics. This requires significant computational capacity. However, this effort is largely redundant since the ego-vehicle's intended future motion is provided by the downstream motion planner. We present FreeOcc, a lightweight continuous-time world model that introduces explicit SE(2) geometric decoupling into a conditional flow-matching framework. By mathematically factoring out self-motion, we create a purely dynamic feature space. This space supports attention-free temporal fusion and allows physics-constrained ODEs to operate strictly on residual agent kinematics. To maximize deployment efficiency, FreeOcc introduces an uncertainty-aware blending mechanism. This routes well-observed voxels to a single-pass decoder, reserving the ODE solver exclusively for ambiguous, high-uncertainty regions. Requiring only 28.3M parameters-roughly 40% of comparable models-our optimal ensemble mode (FreeOcc-E) runs at 27 FPS. On nuScenes, FreeOcc achieves state-of-the-art forecasting performance with a 0.47 m Near Field Chamfer Distance (NFCD) and 32.1% mIoU. Furthermore, the model transfers zero-shot to SemanticKITTI. It also proves highly robust, retaining 92.8% of its ideal mIoU under significant sensor packet loss and timestamp jitter.
Figures & tables
Figure 1: Overview of FreeOcc. Previous sweeps are encoded into BEV, fused by a continuous-time temporal aggregator, propagated to future ego frames by SE(2) warping, and decoded by the Discriminative Occupancy Decoder plus the Physics-Aware Continuous Flow Matching. The final prediction blends both paths using decoder uncertainty.
Figure 2: Ego-aware future propagator. At each horizon, the warped BEV—obtained via SE(2) ego-motion alignment—is explicitly added to the residual scene dynamics predicted by a ConvGRU and a residual projector.
Method
1.0s ( τ=2 )
3.0s ( τ=6 )
NFCD
L1
AbsRel
NFCD
L1
AbsRel
4D-Occ † Khurana et al. (2023)
1.41
1.40
10.37
1.40
1.71
13.48
Copilot4D † Zhang et al. (2024)
0.36
1.30
8.58
0.58
1.51
10.38
UnO † Agro et al. (2024)
–
–
–
0.89
1.15
8.50
ViDAR † Yang et al. (2024)
0.85
1.55
11.50
1.18
1.82
14.20
FreeOcc-D (ours)
0.48
1.25
8.15
0.63
1.35
9.20
Table 1: 3D Geometry Forecasting on nuScenes. Evaluated via Near Field Chamfer Distance (NFCD, m) ↓ , L1 Depth Error (m) ↓ , and Absolute Relative Error (AbsRel, %) ↓ .
Method
Forecasting Horizon mIoU (%)
Average Metrics (%)
τ=1
τ=2
τ=3
τ=4
τ=5
τ=6
mIoU
Prec.
Rec.
F1
Static Copy
31.5
22.1
14.8
10.5
8.8
7.5
15.9
34.0
23.5
27.8
4D-Occ ∗ Khurana et al. (2023)
35.8
27.5
22.4
19.0
16.5
14.2
22.6
39.1
31.8
35.1
OccWorld Zheng et al. (2024)
39.7
34.2
27.9
26.4
23.6
21.6
28.9
50.5
40.2
44.8
ViDAR ∗ Yang et al. (2024)
30.5
24.2
20.5
17.8
15.9
14.3
20.5
39.5
29.8
34.1
FreeOcc-D (ours)
40.8
34.3
29.1
25.1
21.8
19.2
28.4
41.3
49.4
44.9
Table 2: 4D Occupancy Forecasting on nuScenes. Evaluated via Class-Agnostic 3D Voxel mIoU (%) ↑ .
Method
#Params
Latency (ms)
FPS
4D-Occ Khurana et al. (2023)
8.59M
73
13.7
OccWorld Zheng et al. (2024)
72.4M
56
17.8
ViDAR Yang et al. (2024)
69.1M
189
5.3
FreeOcc-D (ours)
20.7M
22
45.4
FreeOcc-G (ours)
7.5M
168
6.0
FreeOcc-E (ours)
28.3M
37
27.0
Table 3: Efficiency comparison on nuScenes. We evaluate the parameter count and inference speed of our three variants: D (Decoder-only), G (Generation-only), and E (Ensemble) against representative baselines. All measurements are conducted on a single NVIDIA RTX 4090.
Method
Short-term ( τ≤3 )
Long-term ( τ>3 )
mIoU
Precision
Recall
mIoU
Precision
Recall
Static Copy
14.5
30.2
22.1
7.5
22.5
13.0
4D-Occ Khurana et al. (2023)
15.2
31.8
24.4
8.9
24.1
15.2
OccWorld Zheng et al. (2024)
18.5
38.1
30.1
12.3
31.5
22.1
ViDAR Yang et al. (2024)
6.5
12.4
8.3
2.3
1.5
2.2
FreeOcc-D (ours)
22.0
42.5
39.8
15.5
34.1
33.2
Table 4: Zero-shot transfer from nuScenes to SemanticKITTI (no fine-tuning). We report metrics across short-term and long-term horizons to assess structural preservation under distribution shift.
Figure 3: 4D occupancy forecasting on nuScenes(sample 1720) in BEV and 3D views across six horizons. FreeOcc-E is compared against OccWorld and ViDAR; more qualitative examples are given in Appendix D .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Condition
mIoU ↑
Prec. ↑
Rec. ↑
F1 ↑
4D-Occ Khurana et al. [2023]
Ideal
22.6
39.1
31.8
35.1
Imperfect
14.2
30.5
21.0
24.9
OccWorld Zheng et al. [2024]
Ideal
28.9
50.5
40.2
44.8
Imperfect
19.0
38.5
28.5
32.8
ViDAR Yang et al. [2024]
Ideal
20.5
39.5
29.8
34.1
Imperfect
12.8
28.2
19.5
23.1
Appendix
Table 5: Robustness to sensor temporal imperfections on nuScenes. The imperfect setting adds severe timestamp jitter and 30% history-frame dropout. All metrics are voxel-based.
Variant
mIoU ↑
Precision ↑
F1 ↑
FreeOcc-E (full)
32.1
54.6
49.6
Generation Path Ablations
w/o Flow Matching (FreeOcc-D)
28.4
41.3
44.9
w/o Decoder (FreeOcc-G)
30.3
52.5
47.4
Max blending
31.2
53.4
48.4
Mean blending
30.5
52.6
47.5
Appendix
Table 6: Ablation study on nuScenes validation. Metrics are voxel-based. The results demonstrate that explicit geometric decoupling and physics-aware regularization are the primary drivers of forecasting quality.
Figure 4: BEV visualization of 4D occupancy forecasting (additional scene, sample 0970). Columns span τ=1 to τ=6 , comparing FreeOcc against OccWorld and ViDAR on a dense urban scene with multiple interacting vehicles. FreeOcc-E preserves scene layout and object extent more faithfully over the full 3.0 s horizon, whereas OccWorld and ViDAR drift more noticeably after τ=3 .
Figure 5: Temporal evolution of GT vs. FreeOcc-E (nuScenes sample 1730). Top row: ground-truth occupancy at τ=2 (1.0 s), τ=4 (2.0 s), and τ=6 (3.0 s), each shown as BEV (left) and 3D voxel rendering (right). Bottom row: corresponding FreeOcc-E predictions. Agreement is strongest at short horizons. By τ=6 , the main discrepancies appear in the far field and around dynamic actors, while large static structures remain well preserved.
Figure 6: Long-horizon comparison at τ=6 (3.0 s, nuScenes sample 3380). Ground Truth, OccWorld, FreeOcc-D, and FreeOcc-E are shown in paired BEV and 3D views. At 3.0 s, OccWorld exhibits boundary blur and noise around moving objects. FreeOcc-D slightly over-predicts occupancy, whereas FreeOcc-E preserves sharper boundaries and cleaner free space.
Figure 7: 3D voxel-level uncertainty across prediction horizons (scene-0274, τ=1 to τ=6 ). Each panel renders occupied voxels coloured by the decoder’s predicted aleatoric uncertainty s^v ; cool tones indicate low uncertainty and warm tones indicate high. At short horizons, uncertainty is concentrated near dynamic actors and occupancy boundaries. As the horizon increases, uncertainty expands into the far field and intensifies around moving objects. By τ=6 (3.0 s), most remaining structures are assigned substantially higher uncertainty. This behaviour is consistent with the ensemble design in Section 3.7 , which assigns more weight to the ODE path where decoder confidence weakens.
Figure 8: Noise suppression analysis (nuScenes sample 1290). Paired BEV and 3D views highlighting hallucinated occupancy at long prediction horizons between FreeOcc-D and FreeOcc-E. FreeOcc-E clears vacated space more effectively while preserving plausible future object positions.
Figure 9: BEV uncertainty evolution (nuScenes scene-0220, τ=1 to τ=6 ). Each panel shows the BEV occupancy coloured by the predicted aleatoric uncertainty s^v at horizon τ . Cool colours (cyan/green) indicate low uncertainty; warm colours (orange/red) indicate high uncertainty. Uncertainty increases with horizon and concentrates on dynamic objects and far-field voxels. Similar behaviour appears in Figure 7 .
Figure 10: Zero-shot SemanticKITTI predictions (sample 2070). BEV and 3D occupancy predictions from a nuScenes-trained model evaluated on SemanticKITTI, comparing FreeOcc-E against OccWorld and ViDAR. Despite domain differences (64-beam LiDAR, different driving context, no fine-tuning), FreeOcc-E preserves road-level structure and major obstacles. The denser 64-beam input provides richer geometric detail than nuScenes’s 32-beam data, and FreeOcc adapts without retraining, an advantage of operating on raw 3D geometry rather than appearance-dependent features.
Figure 11: Additional zero-shot SemanticKITTI scene (sample 1080) and noise suppression analysis. Top: 3D occupancy predictions at the maximum forecasting horizon ( τ=6 , 3.0s), comparing our model variants (FreeOcc-D, -G, -E) against Ground Truth. Bottom: BEV temporal evolution at τ=3 and τ=6 . Road geometry, vegetation boundaries, and building walls are consistently captured, reinforcing the cross-domain robustness reported in Table 4 .
Figure 12: Robustness on nuScenes ( τ=4 , 2.0s, sample 2500). Occupancy predictions under ideal and imperfect sensor conditions. FreeOcc-E retains its predictions under severe temporal perturbations, while OccWorld and ViDAR break into fragments and hallucinated clusters when temporal input is degraded.
Figure 13: Cross-domain robustness on SemanticKITTI ( τ=4 , 2.0s, sample 2090). Zero-shot predictions under ideal (top two rows) and imperfect (bottom two rows) sensor conditions. The cross-domain robustness gap compounds both the sensor-domain shift (nuScenes → SemanticKITTI) and the temporal perturbation, yet FreeOcc-E retains recognisable road and object geometry in both cases, while OccWorld and ViDAR suffer severe structural collapse.
Figure 14: Robustness on nuScenes at τ=6 (3.0 s, sample 1600). Long-horizon predictions under ideal (top) and imperfect (bottom) sensor conditions. At the maximum forecasting horizon, OccWorld and ViDAR dissolve into discontiguous fragments, whereas FreeOcc-E preserves the road layout and principal object silhouettes. The gap between ideal and imperfect rows is noticeably smaller for FreeOcc-E than for OccWorld or ViDAR, consistent with the quantitative retention rate in Table 5 .
Figure 15: Cross-domain robustness on SemanticKITTI at τ=6 (3.0 s, sample 2090). The most challenging evaluation setting: simultaneous cross-domain transfer, long-horizon prediction, and degraded sensor input. FreeOcc-E still recovers the overall road layout and major object boundaries, whereas OccWorld and ViDAR suffer near-complete structural collapse.
Figure 16: Rain scene overview (nuScenes scene-0441). Left: front camera image showing wet road, reflections, and rain droplets. Right: LiDAR BEV coloured by height; the intersection geometry and parked vehicles are clearly visible despite rain-induced noise in the point cloud.
Method
Mean (ms)
Worst (ms)
FPS (mean)
OccWorld (ideal, ref.)
56.0
—
17.8
FreeOcc-D (rain)
22.1
23.8
45.2
FreeOcc-E (rain)
37.3
39.1
26.8
Appendix
Table 7: Inference latency on rain sequence (nuScenes scene-0441, 40 keyframes). Both FreeOcc variants maintain stable latency indistinguishable from dry-weather conditions. FreeOcc-E’s worst-case is still 1.4 × faster than our OccWorld run under ideal conditions.
Traj. error @3 s (L2)
0.0 m
0.3 m
0.5 m
1.0 m
2.0 m
4.0 m
FreeOcc-E mIoU
32.1
31.8
31.5
31.1
30.2
28.6
Appendix
Table 8: FreeOcc-E mIoU under injected future-trajectory error on nuScenes.
Trajectory
mIoU
NFCD@3s
Recorded route (paper setting)
32.1
0.47
Self-predicted from history
31.2
0.51
OccWorld (no external traj.)
28.9
—
Appendix
Table 9: FreeOcc-E with a self-predicted trajectory (no future information).
Method
mIoU
Prec.
Rec.
F1
NFCD@3s
Static Copy
15.9
34.0
23.5
27.8
1.11
Warp+Fuse
20.4
35.3
33.1
34.2
1.04
Warp+Fuse+FOV
20.8
36.5
33.4
34.9
0.98
FreeOcc-D
28.4
41.3
49.4
44.9
0.63
FreeOcc-E
32.1
54.6
45.5
49.6
0.47
Appendix
Table 10: Information-matched warp baselines versus FreeOcc on nuScenes.
Method
Static mIoU
Dynamic mIoU
Overall
Dyn. margin vs OccWorld
Warp+Fuse+FOV
26.8
4.2
20.8
−10.3
OccWorld
37.2
14.5
28.9
—
FreeOcc-E
37.5
19.6
32.1
+5.1
Appendix
Table 11: Static and dynamic mIoU on nuScenes.
Method
Modality
mIoU
NFCD@3s
FPS
SparseWorld-TC
Camera
23.7
0.82
9.8
FreeOcc-Cam (ours)
Camera
25.6
0.69
19.4
FreeOcc-E (ours)
LiDAR
30.2
0.47
27.0
Appendix
Table 12: Comparison with SparseWorld-TC on our metrics ( {1,2,3} s horizons).
Configuration
Modality
mIoU
FreeOcc (full)
LiDAR
32.1
FreeOcc-Cam
Camera
26.5
Camera baseline (no warp/CFM/ensemble)
Camera
21.9
Appendix
Table 13: Isolating the modality from the forecasting modules (6-horizon average on nuScenes).
Roadside 4D occupancy forecasting predicts future semantic occupancy in a fixed infrastructure coordinate system. Roadside scenes are globally structured yet locally evolving, requiring comprehensive scene-level context while precisely modeling sparse localized motion. However, existing forecasting paradigms struggle to tackle both requirements. To address this limitation, we propose InfraWorld, a world model formulating roadside occupancy forecasting as selective state evolution. InfraWorld maintains a dense BEV state for the complete scene and persistent motion queries for moving regions, assigning complementary states to global context and local dynamics. Specifically, we propose Dynamic Query Initiation (DQI) to initialize queries in regions supported by motion in the observed history. During rollout, the dense state is updated through Long--Short Temporal Fusion (LST), which combines short-term adjacent-frame continuity with long-term motion trends to preserve stable context while capturing scene evolution. Motion-Aware Query Injection (MAQI) then refines the queries and injects confidence-weighted query information into corresponding BEV tokens, enabling targeted dynamic updates without perturbing global context. Together, InfraWorld concentrates predictive capacity on evolving traffic regions. Extensive experiments on the InfraOcc and Occ3D--nus datasets demonstrate that InfraWorld achieves state-of-the-art performance for roadside 4D occupancy forecasting while retaining its effectiveness under ego-centric settings. Code will be released.
Songkai Wang, Xiaokai Bai, Lei Yang +1
College of Information Science and Electronic Engineering, Zhejiang University. · School of Mechanical and Aerospace Engineering, Nanyang Technological University.
Real-time 4D occupancy forecasting requires reliably extrapolating both ego motion and surrounding actor motion under strict latency budgets. Recent occupancy world models improve long-horizon fidelity through generative decoders and 3D volumetric modeling, yet their computational cost makes real-time deployment challenging. Light-weight BEV forecasters meet latency requirements but often blur or suppress fast-moving actors at longer horizons. We present UFMF (Ultra-Fast Multi-modal Forecaster), a real-time forecaster that keeps an attention-free diagonal state-space temporal core while injecting targeted kinematic and geometric structure where lightweight models typically fail. A radar-guided velocity-aware dynamic prior (VADP) decomposes future displacement into a linear extrapolation plus a learned horizon-dependent residual. The resulting per-cell flow fields transport a compact dynamic latent subspace during rollout. Two auxiliary training constraints complement this branch: world-frame dynamic supervision enforces inter-horizon trajectory coherence, and class-conditioned vertical-profile supervision regularizes BEV-to-3D lifting. UFMF operates at 122 FPS on a single RTX 4090 and reaches 0.45 fast-displacement mIoU on nuScenes, outperforming recent occupancy world models at a fraction of their computational cost.
4D occupancy forecasting models the spatio-temporal evolution of 3D scenes and is crucial for autonomous driving, especially for corner-case simulation. Existing methods often rely on discrete tokenization followed by autoregressive prediction, yet struggle with geometric distortion in static structures and inconsistent temporal coherence over the forecasting horizon. In this work, we propose a Geometry-Aware Spatio-Temporal context modeling method (GAST) for 4D occupancy forecasting, built upon progressive explicit-implicit generation and dual-path spatio-temporal modeling. Specifically, the generation module produces per-frame occupancy with high geometric fidelity and semantic plausibility through pose-driven warping, motion-aware feature modulation, and attention-based feature refinement. Subsequently, the spatio-temporal module enhances spatial consistency through global context aggregation while capturing scene evolution through temporal dynamics extraction. This unified design enables joint optimization of historical reconstruction and future forecasting in an end-to-end manner. Extensive experiments on Occ3D-nuScenes demonstrate the superiority of our method, outperforming the state-of-the-art by 7.67% in mIoU and 6.44% in IoU with a 2.84x speedup, while maintaining strong performance in long-term forecasting.
Sitao Chen, Zhuangwei Zhuang, Hui Luo +2
South China University of Technology, Guangzhou, China · Institute of Optics and Electronics, CAS, Chengdu, China