Real-time 4D occupancy forecasting requires reliably extrapolating both ego motion and surrounding actor motion under strict latency budgets. Recent occupancy world models improve long-horizon fidelity through generative decoders and 3D volumetric modeling, yet their computational cost makes real-time deployment challenging. Light-weight BEV forecasters meet latency requirements but often blur or suppress fast-moving actors at longer horizons. We present UFMF (Ultra-Fast Multi-modal Forecaster), a real-time forecaster that keeps an attention-free diagonal state-space temporal core while injecting targeted kinematic and geometric structure where lightweight models typically fail. A radar-guided velocity-aware dynamic prior (VADP) decomposes future displacement into a linear extrapolation plus a learned horizon-dependent residual. The resulting per-cell flow fields transport a compact dynamic latent subspace during rollout. Two auxiliary training constraints complement this branch: world-frame dynamic supervision enforces inter-horizon trajectory coherence, and class-conditioned vertical-profile supervision regularizes BEV-to-3D lifting. UFMF operates at 122 FPS on a single RTX 4090 and reaches 0.45 fast-displacement mIoU on nuScenes, outperforming recent occupancy world models at a fraction of their computational cost.
Figures & tables
Figure 1: Real-data teaser at the last forecast horizon on a nuScenes validation window. (a) Our lightweight BEV baseline (i.e., the first row in Table 2 : shared backbone without VADP or auxiliary constraints) predicts only a partial future footprint for the boxed fast-moving car. (b) UFMF preserves a fuller, better-aligned footprint.
Figure 2: Overview of UFMF. Semantic history and rasterized radar cues are encoded by a lightweight BEV encoder and diagonal SSM history encoder, then propagated by a future rollout. VADP uses semantic deltas and radar cues to predict Δk=kv^+rk and dynamic support Qk(c,p) , which guide the rollout state before decoding. The prediction output panel shows the lifted voxel forecast, its BEV projection, and the training-only world-frame and height-profile supervision heads.
Method
mIoU(all) ↑
Static ↑
Slow ↑
Fast ↑
FPS ↑ / ms ↓
OccWorld [ 19 ]
0.478
0.571
0.472
0.336
23 / 44
Drive-OccWorld [ 17 ]
0.509
0.601
0.506
0.381
∼ 2 / 647
COME [ 14 ]
0.527
0.618
0.526
0.402
∼ 1 / 1157
UFMF (ours)
0.542
0.632
0.541
0.450
122 / 8
Table 1: Main comparison on a nuScenes [ 2 ] validation set. mIoU(all) averages over all semantic classes, including background. Static/Slow/Fast denote target-actor displacement buckets of <2 m, 2 – 10 m, and ≥10 m, respectively.
Figure 4: 3D qualitative comparison on one representative nuScenes [ 2 ] validation scene over time. Columns show Ground Truth(GT), OccWorld, Drive-OccWorld, COME, and UFMF, while rows correspond to t=1 s, t=2 s, and t=3 s from the same oblique camera viewpoint. UFMF preserves cleaner actor extent and more stable scene structure as the horizon increases, especially around elongated vehicles and sparse dynamic clusters.
Variant
mIoU(all) ↑
Static ↑
Slow ↑
Fast ↑
FPS ↑ / ms ↓
Lightweight BEV baseline
0.433
0.521
0.426
0.298
138 / 7
+ VADP
0.529
0.618
0.524
0.403
122 / 8
+ VADP + Lwf
0.536
0.626
0.531
0.418
122 / 8
+ VADP + Lwf + Ltraj
0.539
0.631
0.538
0.435
122 / 8
+ Lheight (Full UFMF)
0.542
0.632
0.541
0.450
122 / 8
Table 2: Ablation study on the proposed components. VADP mainly improves the fast-displacement bucket, world-frame constraints stabilize slow- and fast-displacement trajectories, and the height regularizer preserves the final overall mIoU without sacrificing runtime.
Figure 5: BEV comparison on one validation scene focused on a single tracked actor from the fast-displacement bucket over time. Columns show GT, OccWorld, Drive-OccWorld, COME, and UFMF, while rows correspond to t=1 s, t=2 s, and t=3 s. Each subfigure is centered on the same tracked object, and the black box marks the region of interest. UFMF better preserves compact support and temporal alignment, whereas the weaker rows show stronger fragmentation, shrinkage, or stray occupancy.
Radar condition
mIoU(all) ↑
Fast ↑
FPS ↑ / ms ↓
Full VADP (UFMF)
0.542
0.450
122/8
Noisy radar ( σ=5 m/s)
0.534
0.431
122/8
Radar-only VADP (no sem. delta)
0.535
0.420
122/8
Sem.-delta-only VADP (No radar)
0.529
0.407
128/8
Lightweight BEV baseline
0.433
0.298
138/7
Table 3: Radar modality isolation. The semantic-delta-only row removes radar from VADP; the radar-only row removes semantic deltas. Both retain the structural VADP mechanism, confirming that the architecture contributes independently of either input modality.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Veh. IoU ↑
Horizon
FPS ↑
Dim.
FIERY [ 5 ]
36.7
2 s
28
2D
PowerBEV [ 9 ]
39.3
2 s
52
2D
UFMF (veh. class)
57.8
3 s
122
3D → 2D
Appendix
Table A1 : 2D vehicle IoU against efficient BEV forecasters on nuScenes. UFMF is evaluated at the harder 3 s horizon.
History source
mIoU(all) ↑
Fast ↑
FPS ↑
GT history (standard)
0.542
0.450
122
Predicted (BEVDet-Occ)
0.521
0.427
122
Appendix
Table B1 : Robustness to predicted (non-ground-truth) history inputs.
Current 4D occupancy world models use heavy cross-attention or deep recurrence to disentangle ego-motion from scene dynamics. This requires significant computational capacity. However, this effort is largely redundant since the ego-vehicle's intended future motion is provided by the downstream motion planner. We present FreeOcc, a lightweight continuous-time world model that introduces explicit SE(2) geometric decoupling into a conditional flow-matching framework. By mathematically factoring out self-motion, we create a purely dynamic feature space. This space supports attention-free temporal fusion and allows physics-constrained ODEs to operate strictly on residual agent kinematics. To maximize deployment efficiency, FreeOcc introduces an uncertainty-aware blending mechanism. This routes well-observed voxels to a single-pass decoder, reserving the ODE solver exclusively for ambiguous, high-uncertainty regions. Requiring only 28.3M parameters-roughly 40% of comparable models-our optimal ensemble mode (FreeOcc-E) runs at 27 FPS. On nuScenes, FreeOcc achieves state-of-the-art forecasting performance with a 0.47 m Near Field Chamfer Distance (NFCD) and 32.1% mIoU. Furthermore, the model transfers zero-shot to SemanticKITTI. It also proves highly robust, retaining 92.8% of its ideal mIoU under significant sensor packet loss and timestamp jitter.
Zeping Zhang, Zhuoya Zhao, Samy Metari +1
University of Ottawa · National Research Council Canada
Roadside 4D occupancy forecasting predicts future semantic occupancy in a fixed infrastructure coordinate system. Roadside scenes are globally structured yet locally evolving, requiring comprehensive scene-level context while precisely modeling sparse localized motion. However, existing forecasting paradigms struggle to tackle both requirements. To address this limitation, we propose InfraWorld, a world model formulating roadside occupancy forecasting as selective state evolution. InfraWorld maintains a dense BEV state for the complete scene and persistent motion queries for moving regions, assigning complementary states to global context and local dynamics. Specifically, we propose Dynamic Query Initiation (DQI) to initialize queries in regions supported by motion in the observed history. During rollout, the dense state is updated through Long--Short Temporal Fusion (LST), which combines short-term adjacent-frame continuity with long-term motion trends to preserve stable context while capturing scene evolution. Motion-Aware Query Injection (MAQI) then refines the queries and injects confidence-weighted query information into corresponding BEV tokens, enabling targeted dynamic updates without perturbing global context. Together, InfraWorld concentrates predictive capacity on evolving traffic regions. Extensive experiments on the InfraOcc and Occ3D--nus datasets demonstrate that InfraWorld achieves state-of-the-art performance for roadside 4D occupancy forecasting while retaining its effectiveness under ego-centric settings. Code will be released.
Songkai Wang, Xiaokai Bai, Lei Yang +1
College of Information Science and Electronic Engineering, Zhejiang University. · School of Mechanical and Aerospace Engineering, Nanyang Technological University.
Camera-only 4D occupancy forecasting enables autonomous vehicles to predict future 3D semantic scenes solely from historical multi-view images, which is critical for driving safety. Even though current methods have achieved good performance, the strong spatial-temporal modeling between the input multi-view frames is still underexplored, which limits the performance of those methods in future 4D forecasting. To address this gap, we introduce a novel framework, InterOCF, for 4D occupancy forecasting that jointly models temporal dynamics in both 3D voxel-based representations and multi-view segmentation sequences, while explicitly incorporating feature interaction between the 2D and 3D branches. Our framework incorporates three core components: 1) A 3D Spatio-Temporal (3DST) module that learns volumetric dynamics from historical voxel states to predict future voxel states; 2) A 2D Spatio-Temporal (2DST) module employing an auxiliary multi-view temporal segmentation forecasting task to enhance temporal semantic dynamics; 3) A Spatio-Temporal Interaction Modeling (STIM) module that enables feature interaction between 2D and 3D representations. Experiments on the nuScenes, Lyft-Level5, and nuScenes-Occupancy datasets show that InterOCF consistently outperforms existing baseline approaches.
Qi Zhang, Xinquan Yu, Kaiyi Zhang +1
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.