Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking

Maximilian Luz; Rohit Mohan; Thomas Nürnberg; Yakov Miron; Daniele Cattaneo; Abhinav Valada

cs.CVFeb 26, 2026

Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking

Authors: Maximilian Luz, Rohit Mohan, Thomas Nürnberg, Yakov Miron, Daniele Cattaneo, Abhinav Valada

Organizations: Department of Computer Science, University of Freiburg, Germany. · Bosch Research, Robert Bosch GmbH, Germany. · Leon H. Charney School of Marine Sciences, University of Haifa, Israel.

Abstract

Capturing 4D spatiotemporal scene structure is crucial for the safe and reliable operation of robots in dynamic environments. However, existing approaches typically address only part of the problem: they either provide coarse geometric tracking via bounding boxes or detailed 3D occupancy estimates that lack explicit temporal association and instance-level reasoning. In this work, we present Latent Gaussian Splatting (LaGS) for 4D Panoptic Occupancy Tracking (4D-POT). We revisit the underlying representation and model 3D features as a sparse set of feature-bearing Gaussians. These act as dynamic, volume-oriented keypoints that enable spatially continuous, distance-weighted aggregation of multi-view features before being splatted into a voxel grid for decoding. This point-centric formulation enables flexible, data-dependent receptive fields and long-range spatial interactions that are difficult to capture with local and dense voxel-based operators. A hierarchical Gaussian representation further enables multi-scale reasoning by combining global context from coarse super-points with fine-grained detail from higher-resolution streams. Extensive experiments on Occ3D nuScenes and Waymo demonstrate state-of-the-art performance for 4D-POT. We provide code and models at https://lags.cs.uni-freiburg.de/.

Figures & tables

Fig. 1: Illustration of our latent Gaussian representation. Bottom left: panoptic voxel predictions. Center: latent Gaussian features colorized via principal component analysis, splatted to 2D, and overlaid over voxel predictions. Top right: latent Gaussians.

Fig. 2: Architecture overview (left to right). An image encoder ( ) produces image features $\bm{F}$ and depth $\bm{D}$ . Features are lifted via depth to a 3D volume and further encoded into a 3D voxel feature pyramid ( $\bm{V}_{0}$ , $\bm{V}_{2}$ , ). Our latent Gaussian encoder ( ) samples points from the pyramid volumes and processes them in a coarse (left) and fine (right) stream, using self-attention (SA), windowed self-attention (WSA), cross-attention (CA), spatial cross-attention (SCA), and feed-forward networks (FFN). Our novel Serialized Multi-Stream Attention (SMSA) facilitates information exchange between streams. Refined points are decoded as Gaussians ( $\bm{G}_{0}$ , $\bm{G}_{2}$ ) and splatted back to a 3D feature volume, which is then refined to the final voxel volume $\bm{V}$ . Our transformer decoder ( ) then decodes this volume into semantic and instance masks using volume cross-attention (VCA) for efficient query-to-3D-volume attention. Tracking is facilitated by the tracking-by-attention paradigm. We refine track queries by spatio-temporal reasoning before passing them onto the next frame ( ).

					mIoU
Approach	STQ	AQ	STQ ${}_{\text{1}}$	AQ ${}_{\text{1}}$	all	things	stuff	IoU
Per-Frame	$9.0$	$2.5$	$21.8$	$14.7$	$32.5$	$26.4$	$41.2$	$63.2$
MinVIS † [ 10 ]	$11.8$	$4.3$	$21.8$	$14.7$	$32.5$	$26.4$	$41.2$	$63.2$
CTVIS † [ 36 ]	$11.4$	$3.9$	$22.5$	$15.4$	$33.0$	$27.0$	$41.5$	$63.8$
4D-LCA † [ 1 ]	$12.5$	$4.8$	$21.8$	$14.7$	$32.5$	$26.4$	$41.2$	$63.2$
AB3DMOT † [ 34 ]	$13.1$	$5.3$	$21.8$	$14.7$	$32.5$	$26.4$	$41.2$	$63.2$

TABLE I: 4D-POT performance on Occ3D nuScenes.

					mIoU
Approach	STQ	AQ	STQ ${}_{\text{1}}$	AQ ${}_{\text{1}}$	all	things	stuff	IoU
Per-Frame	11.9	4.7	—	—	$30.0$	$32.7$	29.3	—
MinVIS † [ 10 ]	15.0	7.5	—	—	$30.0$	$32.7$	29.3	—
CTVIS † [ 36 ]	16.4	9.3	—	—	28.9	31.9	28.1	—
4D-LCA † [ 1 ]	16.2	8.7	—	—	$30.0$	$32.7$	29.3	—
AB3DMOT † [ 34 ]	18.0	10.8	—	—	$30.0$	$32.7$	29.3	—

TABLE II: 4D-POT performance on Occ3D Waymo.

Approach	Image Backbone	mIoU	IoU
TPVFormer [ 13 ]	ResNet-50	$34.2$	$66.8$
SurroundOcc [ 33 ]	ResNet-101	$34.6$	$65.5$
OccFormer [ 39 ]	ResNet-50	$37.4$	$70.1$
BEVDet4D [ 11 ]	ResNet-50	$39.3$	$73.8$
BEVDet4D + COTR [ 23 ]	ResNet-50	$44.5$	$75.0$
BEVDet4D + COTR [ 23 ]	Swin-B	$46.2$	$74.9$

TABLE III: 3D occupancy prediction performance on Occ3D nuScenes.

Panel of Table 6 — TABLE IV: Ablation study on the latent Gaussian encoder.

Panel of Table 7 — TABLE V: Ablation study on the decoder transformer.

Panel of Figure 8 — Fig. 3: Qualitative results on the Occ3D nuScenes validation split. Our approach shows clear improvements in (1) instance separation, (2) instance association, (3) missing detections, and (4) underconfident detections.

Panel of Figure 9 — Fig. 4: Qualitative results on the Occ3D Waymo validation split. Our approach shows clear improvements in (1) instance separation, (2) instance association, and (3) ID switches.

Panel of Table 10 — TABLE VI: Inference runtime and scaling.

4D Scene Understanding

4D scene understanding reconstructs and reasons about how objects, geometry, and actions change across space and time.

See all 57 papers

Streaming Gaussian Encoding for 4D Panoptic Occupancy Tracking
Jun 29, 2026Maximilian Luz, Thomas Nürnberg, Yakov Miron +1Autonomous Driving 3D Occupancy Prediction
Department of Computer Science, University of Freiburg, Germany. · Bosch Research, Robert Bosch GmbH, Germany. · University of Haifa, Israel
InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting
Jul 27, 2026Qi Zhang, Xinquan Yu, Kaiyi Zhang +13D Occupancy Prediction Spatiotemporal Forecasting
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.
SplatLabel: Pseudo-Labelling through 4D Gaussian Splatting
Sep 24, 2026Nitya Nanvani, Andras Palffy, Holger Caesar3D Gaussian Splatting 3D Semantic Segmentation
Perciv AI, The Netherlands · Delft University of Technology, The Netherlands

Jun 29, 2026J/K move · Enter open · S save

Streaming Gaussian Encoding for 4D Panoptic Occupancy Tracking

Maximilian Luz, Thomas Nürnberg, Yakov Miron, Abhinav Valada

Department of Computer Science, University of Freiburg, Germany. · Bosch Research, Robert Bosch GmbH, Germany. · University of Haifa, Israel

Autonomous Driving 3D Occupancy Prediction 3D Representation Learning

Open paper AskPDF arXiv

Camera-based 4D panoptic occupancy tracking (4D-POT) is a promising paradigm for holistic scene understanding from multi-view imagery, enabling joint reasoning about geometry, semantics, and object identities across time. Recent mask-based pipelines achieve strong performance by propagating instance queries across frames. However, their underlying volumetric representations are typically recomputed at each timestep, limiting geometric temporal consistency, particularly under occlusion and for static scene elements. To address this limitation, we propose a streaming Gaussian encoder that maintains a persistent volumetric scene representation for 4D-POT. Our method models the scene as a fixed-size set of latent Gaussian queries that are propagated via ego-motion compensation and refreshed under a confidence-guided budget constraint. Crucially, we shape Gaussian opacities through depth-based supervision to serve as proxy for visibility, enabling confidence to accumulate as a temporally aggregated measure of persistent scene support. Together with a warmup-based multi-frame training strategy, this yields representation-level temporal coherence beyond decoder-only tracking. Extensive experiments on Occ3D-extended nuScenes and Waymo establish a new state-of-the-art for camera-based 4D-POT, improving tracking consistency with negligible computational overhead while remaining fully compatible with existing mask-based pipelines. We provide code and models at https://sge.cs.uni-freiburg.de.

Abstract

4D Scene Understanding

Explore similar work

Streaming Gaussian Encoding for 4D Panoptic Occupancy Tracking

InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting

SplatLabel: Pseudo-Labelling through 4D Gaussian Splatting