Geometry-Aware Spatio-Temporal Context Modeling for 4D Occupancy Forecasting

Sitao Chen; Zhuangwei Zhuang; Hui Luo; Qingyao Wu; Mingkui Tan

cs.CVAug 15, 2026

Geometry-Aware Spatio-Temporal Context Modeling for 4D Occupancy Forecasting

Authors: Sitao Chen, Zhuangwei Zhuang, Hui Luo, Qingyao Wu, Mingkui Tan

Organizations: South China University of Technology, Guangzhou, China · Institute of Optics and Electronics, CAS, Chengdu, China

Abstract

4D occupancy forecasting models the spatio-temporal evolution of 3D scenes and is crucial for autonomous driving, especially for corner-case simulation. Existing methods often rely on discrete tokenization followed by autoregressive prediction, yet struggle with geometric distortion in static structures and inconsistent temporal coherence over the forecasting horizon. In this work, we propose a Geometry-Aware Spatio-Temporal context modeling method (GAST) for 4D occupancy forecasting, built upon progressive explicit-implicit generation and dual-path spatio-temporal modeling. Specifically, the generation module produces per-frame occupancy with high geometric fidelity and semantic plausibility through pose-driven warping, motion-aware feature modulation, and attention-based feature refinement. Subsequently, the spatio-temporal module enhances spatial consistency through global context aggregation while capturing scene evolution through temporal dynamics extraction. This unified design enables joint optimization of historical reconstruction and future forecasting in an end-to-end manner. Extensive experiments on Occ3D-nuScenes demonstrate the superiority of our method, outperforming the state-of-the-art by 7.67% in mIoU and 6.44% in IoU with a 2.84x speedup, while maintaining strong performance in long-term forecasting.

Figures & tables

Figure 1: Comparison with the existing paradigm [ 45 , 16 ] . It adopts a two-stage pipeline that encodes occupancy into discrete tokens and autoregressively generates future tokens. In contrast, our end-to-end framework directly models spatio-temporal representations in continuous BEV space via per-frame generation and cross-frame modeling.

Figure 2: General architecture of GAST. We first encode historical occupancy into BEV features and predict future ego-poses. The generation module outputs per-frame future features via geometric warping, feature modulation, and deformable cross-attention. The spatio-temporal module enhances global consistency through spatial context aggregation while modeling scene evolution via temporal extraction. Finally, an occupancy decoder jointly reconstructs past observations and forecasts future occupancy.

Method	Input	Ego traj.	mIoU (%) $\uparrow$				IoU (%) $\uparrow$
Method	Input	Ego traj.	1s	2s	3s	Avg.	1s	2s	3s	Avg.
OccWorld-D [ 45 ]	Camera	Pred.	11.55	8.10	6.22	8.62	18.90	16.26	14.43	16.53
OccWorld-T [ 45 ]	Camera	Pred.	4.68	3.36	2.63	3.56	9.32	8.23	7.47	8.34
OccWorld-S [ 45 ]	Camera	Pred.	0.28	0.26	0.24	0.26	5.05	5.01	4.95	5.00
OccWorld-F [ 45 ]	Camera	Pred.	8.03	6.91	3.54	6.16	23.62	18.13	15.22	18.99
RenderWorld [ 38 ]	Camera	Pred.	2.83	2.55	2.37	2.58	14.61	13.61	12.98	13.73

Table 1: 4D occupancy forecasting performance on Occ3D-nuScenes validation set. The settings vary by input modality and whether the ego trajectory is predicted (Pred.) or provided as ground truth (GT). “Avg." denotes the average performance across 1s, 2s, and 3s horizons. The bold numbers indicate the best results.

Method	mIoU (%) $\uparrow$
Method	1s	2s	3s	4s	5s	6s	7s	8s	Avg.
DOME [ 7 ]	30.10	21.35	17.36	14.86	12.61	11.03	10.00	9.34	15.83
COME [ 23 ]	33.78	24.57	21.35	18.25	15.84	13.85	12.99	11.96	19.07
GAST (Ours)	36.23	30.61	27.13	24.36	21.98	19.81	17.83	15.96	24.12
Method	IoU (%) $\uparrow$
Method	1s	2s	3s	4s	5s	6s	7s	8s	Avg.

Table 2: Long-term 4D occupancy forecasting performance on Occ3D-nuScenes validation set. Ground-truth 3D occupancy and trajectories are used as inputs. “Avg.” denotes the average performance over the 1-8 second prediction horizon. The bold numbers indicate the best results.

Table 5

Figure 3: Qualitative results of GAST on Occ3D-nuScenes.

Table 7

Table 8

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting
Jul 27, 2026Qi Zhang, Xinquan Yu, Kaiyi Zhang +13D Occupancy Prediction Spatiotemporal Forecasting
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.
InfraWorld: Motion-Aware World Modeling for Roadside 4D Occupancy Forecasting
Oct 10, 2026Songkai Wang, Xiaokai Bai, Lei Yang +1Spatiotemporal Forecasting 3D Occupancy Prediction
College of Information Science and Electronic Engineering, Zhejiang University. · School of Mechanical and Aerospace Engineering, Nanyang Technological University.
GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning
May 17, 2026Cheng Chen, Hao Huang, Saurabh Bagchi3D Gaussian Splatting World Model Learning
Purdue University · New York University Abu Dhabi

Jul 27, 2026J/K move · Enter open · S save

InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting

Qi Zhang, Xinquan Yu, Kaiyi Zhang, Hui Huang

College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.

3D Occupancy Prediction Spatiotemporal Forecasting

Open paper AskPDF arXiv

Camera-only 4D occupancy forecasting enables autonomous vehicles to predict future 3D semantic scenes solely from historical multi-view images, which is critical for driving safety. Even though current methods have achieved good performance, the strong spatial-temporal modeling between the input multi-view frames is still underexplored, which limits the performance of those methods in future 4D forecasting. To address this gap, we introduce a novel framework, InterOCF, for 4D occupancy forecasting that jointly models temporal dynamics in both 3D voxel-based representations and multi-view segmentation sequences, while explicitly incorporating feature interaction between the 2D and 3D branches. Our framework incorporates three core components: 1) A 3D Spatio-Temporal (3DST) module that learns volumetric dynamics from historical voxel states to predict future voxel states; 2) A 2D Spatio-Temporal (2DST) module employing an auxiliary multi-view temporal segmentation forecasting task to enhance temporal semantic dynamics; 3) A Spatio-Temporal Interaction Modeling (STIM) module that enables feature interaction between 2D and 3D representations. Experiments on the nuScenes, Lyft-Level5, and nuScenes-Occupancy datasets show that InterOCF consistently outperforms existing baseline approaches.

Abstract

3D Occupancy Prediction

Explore similar work

InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting

InfraWorld: Motion-Aware World Modeling for Roadside 4D Occupancy Forecasting

GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning