GNFormer: Gaussian Neural Field Transformer for Efficient Camera-based Occupancy Prediction

Guangcheng Chen; Li He; Hong Zhang

cs.ROOct 11, 2026

GNFormer: Gaussian Neural Field Transformer for Efficient Camera-based Occupancy Prediction

Authors: Guangcheng Chen, Li He, Hong Zhang

Organizations: Department of Electronic and Electrical Engineering, Southern University of Science and Technology, China · Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology, Shenzhen, China

Abstract

Camera-based occupancy prediction provides geometric and semantic scene understanding for robot perception and has been adopted in autonomous vehicle and UAV perception systems. Gaussian primitives offer a promising alternative by potentially representing only occupied regions, thereby concentrating computation and representational capacity where needed and reducing wasted computation and memory in empty space. However, existing Gaussian-based methods suffer from poor primitive utilization, as ineffective training objectives leave some Gaussians in empty space, while the limited expressiveness of Gaussian functions restricts their ability to model complex shapes. To address these limitations, we propose a novel 3D representation, Gaussian Neural Field (GNF), which can accurately map voxel positions to semantic occupancies. In GNF, Gaussians and a neural network jointly parameterize the field, where the Gaussians define its spatial support and provide features for neural decoding, while the neural network enables more expressive occupancy modeling than Gaussian functions alone. We further develop the Gaussian Neural Field Transformer (GNFormer), a streamlined and flexible method that predicts GNFs for camera-based occupancy prediction. Across Occ-ScanNet, SSCBench-KITTI-360, and nuScenes-SurroundOcc benchmarks, GNFormer achieves the highest IoU and mIoU and the fastest inference among the compared single-frame methods. In the experiments, GNFormer can represent indoor and outdoor scenes with only 256 and 1024 Gaussians, respectively, substantially fewer primitives than prior Gaussian-based methods. The code will be released.

Figures & tables

Fig. 1: GNFormer predicts semantic occupancy from single-frame images and camera parameters using GNF. The same architecture can be trained and deployed for different scenarios and camera configurations separately.

Fig. 2: Decoding semantics from BEV [ 17 ] and TPV [ 2 ] requires traversing the entire space. Gaussian-based methods [ 8 , 10 ] compute semantics only for regions covered by the Gaussians, using posterior-weighted Gaussian semantics. GNF further employs a neural decoder for semantic prediction while retaining the efficiency of Gaussian-based representations.

Fig. 3: Fitting 2D “R” and “O” occupancy grids (100 $\times$ 100) with 32 Gaussians using different methods. Gaussian centers are shown over the grids with their initial positions shown in the first column. Only Gaussian centers are shown for clarity, and brighter pixels indicate occupied regions. GNF not only drives all Gaussians towards occupied regions but also fits sharp boundaries with an extra MLP.

Fig. 4: (a) Given input images and camera parameters, GNFormer employs Transformer blocks [ 13 ] to transform a set of Gaussians and queries into a GNF. During training, output semantics are supervised using cross-entropy loss and Gaussians are regularized following FLM-Occ [ 10 ] . Note that the camera parameters and positional encoding are omitted for brevity. (b) Each decoder block adds an extra cross-attention layer to enable information exchange among all tokens of $\mathbf{Q}$ and $\mathbf{Z}$ .

Panel of Table 5 — TABLE I: Summary of Datasets and Hyperparameters

Method	Venue	Representation	Number	IoU	mIoU	$\blacksquare$ Ceiling	$\blacksquare$ Floor	$\blacksquare$ Wall	$\blacksquare$ Window	$\blacksquare$ Chair	$\blacksquare$ Bed	$\blacksquare$ Sofa	$\blacksquare$ Table	$\blacksquare$ TVs	$\blacksquare$ Furniture	$\blacksquare$ Objects	FPS
MonoScene [ 1 ]	CVPR22	Dense Voxels	129.6K	41.6	24.6	15.2	44.7	22.4	12.6	26.1	27.0	35.9	28.3	6.6	32.2	19.8	—
ISO [ 14 ]	ECCV24	Dense Voxels	129.6K	42.2	28.7	19.9	41.9	22.4	17.0	29.0	42.4	42.0	29.6	10.6	36.4	24.6	—
DiScene [ 27 ]	RAL25	Gaussians	15.4K	52.0	47.2	45.3	50.6	40.4	36.7	42.3	59.7	62.0	45.6	41.2	52.4	42.7	11.0
EmbodiedOcc [ 3 ]	ICCV25	Gaussians	16.2K	53.6	45.2	39.6	50.4	41.4	31.7	40.9	55.0	61.4	44.0	36.1	53.9	42.2	7.8
SplatSSC [ 4 ]	AAAI26	Gaussians	1200	62.8	51.8	49.1	59.0	48.3	38.8	47.4	62.4	67.0	49.5	42.6	60.7	45.4	8.7
FLM-Occ [ 10 ]	ECCV26	Superquadrics	1024	71.7	64.0	62.5	70.6	58.0	53.1	58.8	74.0	77.9	63.5	55.2	70.8	60.3	25.4

TABLE II: Results on Occ-ScanNet [ 14 ] Validation Set

Method	Venue	Represent.	Number	IoU	mIoU	$\blacksquare$ car	$\blacksquare$ bicycle	$\blacksquare$ motorcycle	$\blacksquare$ truck	$\blacksquare$ other-veh.	$\blacksquare$ person	$\blacksquare$ road	$\blacksquare$ parking	$\blacksquare$ sidewalk	$\blacksquare$ other-grnd	$\blacksquare$ building	$\blacksquare$ fence	$\blacksquare$ vegetation	$\blacksquare$ terrain	$\blacksquare$ pole	$\blacksquare$ traf.-sign	$\blacksquare$ other-struct.	$\blacksquare$ other-object
MonoScene [ 1 ]	CVPR22	Dense Vox	2.1M	37.9	12.3	19.3	0.4	0.6	8.0	2.0	0.9	48.4	11.4	28.1	3.2	32.9	3.5	26.2	16.8	6.9	5.7	4.2	3.1
TPVFormer [ 2 ]	CVPR23	Planes	3	40.2	13.6	21.6	1.1	1.4	8.1	2.6	2.4	53.0	12.0	31.1	3.8	34.8	4.8	30.1	17.5	7.5	5.9	5.5	2.7
OccFormer [ 28 ]	ICCV23	Dense Vox	2.1M	40.3	13.8	22.6	0.7	0.3	9.9	3.8	2.8	54.3	13.4	31.5	3.6	36.4	4.8	31.0	19.5	7.8	8.5	7.0	4.6
GaussianFormer [ 7 ]	ECCV24	Gaussians	38.4K	35.4	12.9	18.9	1.0	4.6	18.1	7.6	3.4	45.5	10.9	25.0	5.3	28.4	5.7	29.5	8.6	3.0	2.3	9.5	5.1
GaussianFormer-2 [ 8 ]	CVPR25	Gaussians	38.4K	38.4	13.9	21.1	2.6	4.2	12.4	5.7	1.6	54.1	11.0	32.3	3.3	32.0	5.0	28.9	17.3	3.6	5.5	5.9	3.5
S2GO-Base* [ 12 ]	ICLR26	Gaussians	3600	40.8	15.1	22.7	1.3	1.7	15.9	5.1	2.1	53.8	13.3	33.4	3.8	35.3	7.2	31.2	21.1	6.4	6.5	6.0	4.2

TABLE III: Results on SSCBench-KITTI-360 [ 15 ] Test Set

Method	Venue	Represent.	Num.	IoU	mIoU	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ const. veh.	$\blacksquare$ motorcycle	$\blacksquare$ pedestrian	$\blacksquare$ traffic cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drive. surf.	$\blacksquare$ other flat	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation	FPS
TPVFormer [ 2 ]	CVPR23	Planes	3	30.9	17.1	16.0	5.3	23.9	27.3	9.8	8.7	7.1	5.2	11.0	19.2	38.9	21.3	24.3	23.2	11.7	20.8	2.9
SurroundOcc [ 16 ]	ICCV23	Dense Vox	640K	31.5	20.3	20.6	11.7	28.1	30.9	10.7	15.1	14.1	12.1	14.4	22.3	37.3	23.7	24.5	22.8	14.9	21.9	3.3
Co-Occ [ 29 ]	RAL24	Sparse Vox	—	30.0	20.3	22.5	11.2	28.6	29.5	9.9	15.8	13.5	8.7	13.6	22.2	24.9	23.1	24.2	24.1	18.0	24.8	—
HybridOcc [ 20 ]	RAL24	Sparse Vox	—	33.1	21.4	22.3	12.1	29.8	32.3	10.9	16.3	14.1	12.7	14.6	24.0	40.4	23.7	26.2	24.5	15.2	22.6	1.9
GaussianFormer-2 [ 8 ]	CVPR25	Gaussians	12.8K	30.6	20.0	20.2	13.0	27.6	30.2	11.2	15.3	12.6	9.6	13.3	22.3	39.7	23.5	25.6	23.2	12.3	20.7	2.8
QuadricFormer [ 11 ]	NeurIPS25	SQuadrics	1600	31.2	20.1	19.6	13.1	27.3	29.6	11.3	16.3	12.7	9.2	12.5	21.2	40.2	24.3	25.7	24.2	13.0	21.9	6.2

TABLE IV: Results on nuScenes-SurroundOcc [ 16 ] Validation Set

Index	Representation			Architecture					Metrics $\uparrow$			Latency (ms) $\downarrow$
Index	$\operatorname{MLP_{s}}$	Gau. Reg.	#Gau	#Tokens/Img	$N_{\text{blk}}$	$B$	BiCA	ViT	IoU	mIoU	FPS	Encoder	Decoder	GNF-to-Voxel	Full
B	✓	✓	1024	1590	3	4	✓	B	37.4	23.0	8.3	44	55	21	120
1		✓	1024	1590	3	4	✓	B	-6.3	-3.5	-1.1	43	54	42	138
2	✓		1024	1590	3	4	✓	B	-3.5	-1.9	-2.5	42	54	74	171
3	✓	✓	512	1590	3	4	✓	B	-0.8	-0.8	+1.2	43	44	18	106
4	✓	✓	2048	1590	3	4	✓	B	+0.3	+0.1	-2.5	43	81	24	148

TABLE V: Ablation Study and Latency Analysis of GNFormer.

ViT Initialization	IoU	mIoU
DepthAnythingV2 [ 33 ]	38.5	24.0
$\pi^{3}$ -X [ 36 ]	38.6	24.0
DINOv2 [ 24 ]	38.7	24.3
VGGT [ 34 ]	38.9	24.3

TABLE VI: Ablation Study of Encoder Initialization

	Density Agg. / Occupied		Feat. Agg. / Full Grid
Method	Mean (ms)	Peak (MiB)	Mean (ms)	Peak (MiB)
Naive PyTorch	60.13	37.25	998.60	346.26
GaussianFormer-2 [ 8 ]	1.14	91.86	2.55	576.16
SuperOcc [ 9 ]	0.85	55.60	2.12	526.28
Ours ( section III-C )	0.43	37.25	6.47	353.27

TABLE VII: Aggregation Efficiency

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

Jul 20, 2026

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction

Vision-only occupancy prediction requires recovering a semantic 3D occupancy field from calibrated surround-view images, where each view provides observations with ambiguous depth along camera rays. Existing methods have progressed from dense structured representations to sparse Gaussian primitives, improving the efficiency of 3D scene representation. However, Gaussian learning still relies primarily on image domain features, which provide limited explicit geometric information for volumetric reasoning. Our key observation is that effective Gaussian occupancy modeling requires not only sparse primitives, but also richer geometric and semantic learning cues. In this paper, we propose VGOcc, which learns visual and geometric cues from foundation models for Gaussian modeling. VGOcc incorporates these cues into primitive initialization and refinement, yielding a representation termed Visual-Geometric Gaussians tailored to semantic occupancy prediction. Specifically, we propose Visual-Geometric Gaussian Birth to form spatially balanced Gaussian centers from ray depth hypotheses, while visual semantic features initialize primitive attributes. Next, we design Pose-Aware Feature Learning to combine foundation tokens with camera embeddings and calibrated ray information. Features from neighboring views are then aggregated at projected 3D locations for each Gaussian refinement stage. Finally, Gaussian decoder refines birth Gaussians with pose-aware features and renders them into semantic occupancy. Experiments on nuScenes demonstrate that VGOcc achieves state-of-the-art performance in vision-only 3D occupancy prediction. Codes will be available at https://github.com/JHLin42in/VGOcc.

Junhong Lin, Xianda Guo, Kangli Wang +4

1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor

Autonomous Driving 3D Occupancy Prediction 3D Scene Representation

Ask PDF

Jul 24, 2026

Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy

Semantic 3D Gaussians provide a compact representation for 3D semantic occupancy prediction by rendering semantic primitives into a voxel volume under voxel-wise supervision. Recent methods have improved the modeling ability and efficiency of this representation through more flexible primitive shapes, geometry-guided initialization, and progressive densification. However, these advances mainly determine how primitives are represented, initialized, or added, and do not explicitly address how to select the most useful Gaussians when their total number must be limited to control memory and computation. This imbalance creates an allocation bottleneck: redundant Gaussians remain in simple regions, while difficult regions receive insufficient semantic support. We propose the Semantic Gaussian Allocation Transformer (SAGFormer), which uses Gaussian attributes and local geometric-semantic features to score candidates and select a fixed final Gaussian set. Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 show that SAGFormer improves occupancy prediction under the evaluated protocols and yields more semantically consistent and better-utilized Gaussian representations. Under similar final counts and raw coverage, it reduces semantic mixing, strengthens class-consistent voxel support, and produces fewer unused Gaussians. The results indicate that explicit capacity allocation is a useful complement to Gaussian refinement for semantic occupancy prediction.

Kanglin Ning, Yiran Zhao, Wenrui Li +4

Harbin Institute of Technology · The Suzhou Research Institute of HIT · The PengChengLab

3D Gaussian Splatting Semantic Occupancy Mapping 3D Occupancy Prediction

Ask PDF

Jul 15, 2026

GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors

Accurate 3D scene understanding is fundamental to embodied intelligence and autonomous driving, where 3D occupancy provides a unified representation of objects, structures, and free space. However, recovering such a complete volumetric representation from visual observations remains challenging, particularly in occluded and unobserved regions. Visual geometry priors offer strong and generalizable geometric cues for addressing this challenge, but their outputs are inherently surface-centric, whereas occupancy prediction requires reasoning about volumetric interiors and free space. To bridge this gap, we introduce GPOcc, which transforms visual geometry priors into occupancy-aware sparse Gaussian representations for efficient and expressive volumetric scene modeling. Building on GPOcc, GPOcc++ models multi-view observations and temporal sequences within a unified framework, allowing spatial and temporal evidence to be handled through the same representation. We further extend GPOcc++ from indoor scenes to outdoor occupancy prediction. Extensive experiments on both indoor and outdoor benchmarks demonstrate consistently strong performance across both multi-view and temporal settings, together with favorable efficiency and generalization. Code will be released at https://github.com/JuIvyy/GPOcc.

Changqing Zhou, Yueru Luo, Yulan Guo +3

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · The School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China +3

3D Occupancy Prediction 3D Scene Representation

Ask PDF

Abstract

3D Occupancy Prediction

Explore similar work

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction

Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy

GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors