GNFormer: Gaussian Neural Field Transformer for Efficient Camera-based Occupancy Prediction

Guangcheng Chen; Li He; Hong Zhang

cs.ROOct 11, 2026

GNFormer: Gaussian Neural Field Transformer for Efficient Camera-based Occupancy Prediction

Authors: Guangcheng Chen, Li He, Hong Zhang

Organizations: Department of Electronic and Electrical Engineering, Southern University of Science and Technology, China · Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology, Shenzhen, China

Abstract

Camera-based occupancy prediction provides geometric and semantic scene understanding for robot perception and has been adopted in autonomous vehicle and UAV perception systems. Gaussian primitives offer a promising alternative by potentially representing only occupied regions, thereby concentrating computation and representational capacity where needed and reducing wasted computation and memory in empty space. However, existing Gaussian-based methods suffer from poor primitive utilization, as ineffective training objectives leave some Gaussians in empty space, while the limited expressiveness of Gaussian functions restricts their ability to model complex shapes. To address these limitations, we propose a novel 3D representation, Gaussian Neural Field (GNF), which can accurately map voxel positions to semantic occupancies. In GNF, Gaussians and a neural network jointly parameterize the field, where the Gaussians define its spatial support and provide features for neural decoding, while the neural network enables more expressive occupancy modeling than Gaussian functions alone. We further develop the Gaussian Neural Field Transformer (GNFormer), a streamlined and flexible method that predicts GNFs for camera-based occupancy prediction. Across Occ-ScanNet, SSCBench-KITTI-360, and nuScenes-SurroundOcc benchmarks, GNFormer achieves the highest IoU and mIoU and the fastest inference among the compared single-frame methods. In the experiments, GNFormer can represent indoor and outdoor scenes with only 256 and 1024 Gaussians, respectively, substantially fewer primitives than prior Gaussian-based methods. The code will be released.

Figures & tables

Fig. 1: GNFormer predicts semantic occupancy from single-frame images and camera parameters using GNF. The same architecture can be trained and deployed for different scenarios and camera configurations separately.

Fig. 2: Decoding semantics from BEV [ 17 ] and TPV [ 2 ] requires traversing the entire space. Gaussian-based methods [ 8 , 10 ] compute semantics only for regions covered by the Gaussians, using posterior-weighted Gaussian semantics. GNF further employs a neural decoder for semantic prediction while retaining the efficiency of Gaussian-based representations.

Fig. 3: Fitting 2D “R” and “O” occupancy grids (100 $\times$ 100) with 32 Gaussians using different methods. Gaussian centers are shown over the grids with their initial positions shown in the first column. Only Gaussian centers are shown for clarity, and brighter pixels indicate occupied regions. GNF not only drives all Gaussians towards occupied regions but also fits sharp boundaries with an extra MLP.

Fig. 4: (a) Given input images and camera parameters, GNFormer employs Transformer blocks [ 13 ] to transform a set of Gaussians and queries into a GNF. During training, output semantics are supervised using cross-entropy loss and Gaussians are regularized following FLM-Occ [ 10 ] . Note that the camera parameters and positional encoding are omitted for brevity. (b) Each decoder block adds an extra cross-attention layer to enable information exchange among all tokens of $\mathbf{Q}$ and $\mathbf{Z}$ .

Panel of Table 5 — TABLE I: Summary of Datasets and Hyperparameters

Method	Venue	Representation	Number	IoU	mIoU	$\blacksquare$ Ceiling	$\blacksquare$ Floor	$\blacksquare$ Wall	$\blacksquare$ Window	$\blacksquare$ Chair	$\blacksquare$ Bed	$\blacksquare$ Sofa	$\blacksquare$ Table	$\blacksquare$ TVs	$\blacksquare$ Furniture	$\blacksquare$ Objects	FPS
MonoScene [ 1 ]	CVPR22	Dense Voxels	129.6K	41.6	24.6	15.2	44.7	22.4	12.6	26.1	27.0	35.9	28.3	6.6	32.2	19.8	—
ISO [ 14 ]	ECCV24	Dense Voxels	129.6K	42.2	28.7	19.9	41.9	22.4	17.0	29.0	42.4	42.0	29.6	10.6	36.4	24.6	—
DiScene [ 27 ]	RAL25	Gaussians	15.4K	52.0	47.2	45.3	50.6	40.4	36.7	42.3	59.7	62.0	45.6	41.2	52.4	42.7	11.0
EmbodiedOcc [ 3 ]	ICCV25	Gaussians	16.2K	53.6	45.2	39.6	50.4	41.4	31.7	40.9	55.0	61.4	44.0	36.1	53.9	42.2	7.8
SplatSSC [ 4 ]	AAAI26	Gaussians	1200	62.8	51.8	49.1	59.0	48.3	38.8	47.4	62.4	67.0	49.5	42.6	60.7	45.4	8.7
FLM-Occ [ 10 ]	ECCV26	Superquadrics	1024	71.7	64.0	62.5	70.6	58.0	53.1	58.8	74.0	77.9	63.5	55.2	70.8	60.3	25.4

TABLE II: Results on Occ-ScanNet [ 14 ] Validation Set

Method	Venue	Represent.	Number	IoU	mIoU	$\blacksquare$ car	$\blacksquare$ bicycle	$\blacksquare$ motorcycle	$\blacksquare$ truck	$\blacksquare$ other-veh.	$\blacksquare$ person	$\blacksquare$ road	$\blacksquare$ parking	$\blacksquare$ sidewalk	$\blacksquare$ other-grnd	$\blacksquare$ building	$\blacksquare$ fence	$\blacksquare$ vegetation	$\blacksquare$ terrain	$\blacksquare$ pole	$\blacksquare$ traf.-sign	$\blacksquare$ other-struct.	$\blacksquare$ other-object
MonoScene [ 1 ]	CVPR22	Dense Vox	2.1M	37.9	12.3	19.3	0.4	0.6	8.0	2.0	0.9	48.4	11.4	28.1	3.2	32.9	3.5	26.2	16.8	6.9	5.7	4.2	3.1
TPVFormer [ 2 ]	CVPR23	Planes	3	40.2	13.6	21.6	1.1	1.4	8.1	2.6	2.4	53.0	12.0	31.1	3.8	34.8	4.8	30.1	17.5	7.5	5.9	5.5	2.7
OccFormer [ 28 ]	ICCV23	Dense Vox	2.1M	40.3	13.8	22.6	0.7	0.3	9.9	3.8	2.8	54.3	13.4	31.5	3.6	36.4	4.8	31.0	19.5	7.8	8.5	7.0	4.6
GaussianFormer [ 7 ]	ECCV24	Gaussians	38.4K	35.4	12.9	18.9	1.0	4.6	18.1	7.6	3.4	45.5	10.9	25.0	5.3	28.4	5.7	29.5	8.6	3.0	2.3	9.5	5.1
GaussianFormer-2 [ 8 ]	CVPR25	Gaussians	38.4K	38.4	13.9	21.1	2.6	4.2	12.4	5.7	1.6	54.1	11.0	32.3	3.3	32.0	5.0	28.9	17.3	3.6	5.5	5.9	3.5
S2GO-Base* [ 12 ]	ICLR26	Gaussians	3600	40.8	15.1	22.7	1.3	1.7	15.9	5.1	2.1	53.8	13.3	33.4	3.8	35.3	7.2	31.2	21.1	6.4	6.5	6.0	4.2

TABLE III: Results on SSCBench-KITTI-360 [ 15 ] Test Set

Method	Venue	Represent.	Num.	IoU	mIoU	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ const. veh.	$\blacksquare$ motorcycle	$\blacksquare$ pedestrian	$\blacksquare$ traffic cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drive. surf.	$\blacksquare$ other flat	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation	FPS
TPVFormer [ 2 ]	CVPR23	Planes	3	30.9	17.1	16.0	5.3	23.9	27.3	9.8	8.7	7.1	5.2	11.0	19.2	38.9	21.3	24.3	23.2	11.7	20.8	2.9
SurroundOcc [ 16 ]	ICCV23	Dense Vox	640K	31.5	20.3	20.6	11.7	28.1	30.9	10.7	15.1	14.1	12.1	14.4	22.3	37.3	23.7	24.5	22.8	14.9	21.9	3.3
Co-Occ [ 29 ]	RAL24	Sparse Vox	—	30.0	20.3	22.5	11.2	28.6	29.5	9.9	15.8	13.5	8.7	13.6	22.2	24.9	23.1	24.2	24.1	18.0	24.8	—
HybridOcc [ 20 ]	RAL24	Sparse Vox	—	33.1	21.4	22.3	12.1	29.8	32.3	10.9	16.3	14.1	12.7	14.6	24.0	40.4	23.7	26.2	24.5	15.2	22.6	1.9
GaussianFormer-2 [ 8 ]	CVPR25	Gaussians	12.8K	30.6	20.0	20.2	13.0	27.6	30.2	11.2	15.3	12.6	9.6	13.3	22.3	39.7	23.5	25.6	23.2	12.3	20.7	2.8
QuadricFormer [ 11 ]	NeurIPS25	SQuadrics	1600	31.2	20.1	19.6	13.1	27.3	29.6	11.3	16.3	12.7	9.2	12.5	21.2	40.2	24.3	25.7	24.2	13.0	21.9	6.2

TABLE IV: Results on nuScenes-SurroundOcc [ 16 ] Validation Set

Index	Representation			Architecture					Metrics $\uparrow$			Latency (ms) $\downarrow$
Index	$\operatorname{MLP_{s}}$	Gau. Reg.	#Gau	#Tokens/Img	$N_{\text{blk}}$	$B$	BiCA	ViT	IoU	mIoU	FPS	Encoder	Decoder	GNF-to-Voxel	Full
B	✓	✓	1024	1590	3	4	✓	B	37.4	23.0	8.3	44	55	21	120
1		✓	1024	1590	3	4	✓	B	-6.3	-3.5	-1.1	43	54	42	138
2	✓		1024	1590	3	4	✓	B	-3.5	-1.9	-2.5	42	54	74	171
3	✓	✓	512	1590	3	4	✓	B	-0.8	-0.8	+1.2	43	44	18	106
4	✓	✓	2048	1590	3	4	✓	B	+0.3	+0.1	-2.5	43	81	24	148

TABLE V: Ablation Study and Latency Analysis of GNFormer.

ViT Initialization	IoU	mIoU
DepthAnythingV2 [ 33 ]	38.5	24.0
$\pi^{3}$ -X [ 36 ]	38.6	24.0
DINOv2 [ 24 ]	38.7	24.3
VGGT [ 34 ]	38.9	24.3

TABLE VI: Ablation Study of Encoder Initialization

	Density Agg. / Occupied		Feat. Agg. / Full Grid
Method	Mean (ms)	Peak (MiB)	Mean (ms)	Peak (MiB)
Naive PyTorch	60.13	37.25	998.60	346.26
GaussianFormer-2 [ 8 ]	1.14	91.86	2.55	576.16
SuperOcc [ 9 ]	0.85	55.60	2.12	526.28
Ours ( section III-C )	0.43	37.25	6.47	353.27

TABLE VII: Aggregation Efficiency

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction
Jul 20, 2026Junhong Lin, Xianda Guo, Kangli Wang +4Autonomous Driving 3D Occupancy Prediction
1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor
Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy
Jul 24, 2026Kanglin Ning, Yiran Zhao, Wenrui Li +43D Gaussian Splatting Semantic Occupancy Mapping
Harbin Institute of Technology · The Suzhou Research Institute of HIT · The PengChengLab
GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors
Jul 15, 2026Changqing Zhou, Yueru Luo, Yulan Guo +33D Occupancy Prediction 3D Scene Representation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · The School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China +3

Jul 20, 2026J/K move · Enter open · S save

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction

Junhong Lin, Xianda Guo, Kangli Wang, Yuqi Ye+3

1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor

Autonomous Driving 3D Occupancy Prediction 3D Scene Representation

Open paper AskPDF arXiv

Vision-only occupancy prediction requires recovering a semantic 3D occupancy field from calibrated surround-view images, where each view provides observations with ambiguous depth along camera rays. Existing methods have progressed from dense structured representations to sparse Gaussian primitives, improving the efficiency of 3D scene representation. However, Gaussian learning still relies primarily on image domain features, which provide limited explicit geometric information for volumetric reasoning. Our key observation is that effective Gaussian occupancy modeling requires not only sparse primitives, but also richer geometric and semantic learning cues. In this paper, we propose VGOcc, which learns visual and geometric cues from foundation models for Gaussian modeling. VGOcc incorporates these cues into primitive initialization and refinement, yielding a representation termed Visual-Geometric Gaussians tailored to semantic occupancy prediction. Specifically, we propose Visual-Geometric Gaussian Birth to form spatially balanced Gaussian centers from ray depth hypotheses, while visual semantic features initialize primitive attributes. Next, we design Pose-Aware Feature Learning to combine foundation tokens with camera embeddings and calibrated ray information. Features from neighboring views are then aggregated at projected 3D locations for each Gaussian refinement stage. Finally, Gaussian decoder refines birth Gaussians with pose-aware features and renders them into semantic occupancy. Experiments on nuScenes demonstrate that VGOcc achieves state-of-the-art performance in vision-only 3D occupancy prediction. Codes will be available at https://github.com/JHLin42in/VGOcc.

Abstract

3D Occupancy Prediction

Explore similar work

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction

Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy

GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors