We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by 13.4% relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.
Figures & tables
Figure 2: Foundation4D architecture. Foundation4D unifies driving perception and reconstruction in a feed-forward model with a shared representation of multi-camera video. A common image encoder and factorized spatiotemporal attention efficiently combine observations across views and time. Task-specific decoders recover camera-to-rig calibration, ego motion, and metric depth; predict 3D objects, velocities, and high-resolution semantic occupancy; and reconstruct scene appearance with 3D Gaussians.
Figure 3: Foundation4D semantic occupancy BEV features. PCA of the upsampled features reveals a separation between occupied and empty regions. All panels show the same sample; PCA colors encode feature variation, while prediction colors denote semantic classes.
Dataset
Clips
Urban
Residential
Highway
Rural
Other
NRE
67,970
16.5%
9.1%
64.5%
7.6%
2.3%
PlanNet
75,921
42.5%
17.3%
20.8%
16.5%
2.9%
MLMCF
59,999
36.5%
10.9%
42.9%
6.9%
2.8%
Table 1: Scene composition of the internal training splits. NRE is dominated by highway driving, while PlanNet emphasizes urban and residential scenes. MLMCF provides a more balanced mix of urban and highway driving, complementing the other two collections. Counts and percentages refer to unique clips within each dataset.
Figure 4: PAI depth and occupancy annotation pipeline. The pipeline combines temporally aggregated LiDAR, cuboid tracks, and image masks to generate camera-view depth maps and vehicle-frame occupancy grids with per-voxel semantic and velocity labels.
Figure 5: Generated depth and semantic occupancy annotations on PAI. Each example shows multi-camera RGB images overlaid with depth (top), followed by unprojected depth colored by RGB, a side view of semantic occupancy, and a forward view of semantic occupancy (bottom, left to right). The shared legend indicates depth in meters and semantic classes.
Model
Waymo
nuScenes
AbsRel ↓
δ1.25↑ (%)
AbsRel ↓
δ1.25↑ (%)
DVGT-1 [ 40 ]
0.20
72.87
0.18
78.97
DVGT-2 [ 23 ]
0.26
70.93
0.22
75.32
LiAuto-GeoX [ 44 ]
0.19
75.72
0.27
62.75
GeoUP [ 3 ]
0.13
87.30
0.19
80.05
Foundation4D
0.09
91.97
0.15
86.44
Table 2: Depth estimation on the Waymo and nuScenes validation splits. Depth is evaluated independently in non-overlapping windows of 48 consecutive frames per camera, sampled at the native frame rates (Waymo: 10 Hz; nuScenes: ≈ 12 Hz).
Dataset
Inference Mode
Model
Trajectory
Rig
Depth
ATE ↓
RTE ↓
RRE ↓
RTE ↓
RRE ↓
mAA@15 ↑
AbsRel ↓
δ1.25↑
(cm)
(cm)
( ∘ )
(cm)
( ∘ )
(%)
(%)
Waymo
Stitching
DVGT-1 [ 40 ]
62.33
3.54
0.06
–
–
–
0.20
73.04
Foundation4D
32.70
3.20
0.06
0.96
1.02
83.37
0.09
92.33
PGO+LM
DVGT-1 [ 40 ]
54.55
2.38
0.05
–
–
–
0.20
73.26
Foundation4D
23.62
1.56
0.06
0.90
1.03
83.54
0.09
92.94
Table 3: Video-level geometry comparison on the Waymo and nuScenes validation data described in Sec. 7.1.1 , at native camera rates (10 Hz and ≈ 12 Hz). Each clip uses one evaluation Sim(3), shared by poses and depth. Both models use our video inference system with 48-frame windows and a 5-frame overlap for stitching and PGO+LM. Rig metrics are omitted (–) because DVGT-1 does not estimate the rig.
Model
Inference Mode
Trajectory
Rig
Depth
ATE ↓
RTE ↓
RRE ↓
RTE ↓
RRE ↓
mAA@15 ↑
AbsRel ↓
δ1.25↑
(cm)
(cm)
( ∘ )
(cm)
( ∘ )
(%)
(%)
VGGT †
Stitching
111.12
8.31
0.122
29.26
0.520
42.50
0.084
94.88
Foundation4D
Stitching
78.06
4.69
0.070
3.90
0.404
88.72
0.081
94.98
PGO+LM
52.75
2.28
0.066
3.92
0.404
88.71
0.079
95.03
Table 4: Geometry results on all 1,253 PAI test clips with six cameras. † denotes internal fine-tuning. Metrics and evaluation alignment follow Sec. 7.1 .
Figure 6: Depth error by ground-truth depth range on PAI, averaged equally over 1,253 clips in each bin. AbsRel is dimensionless; the y-axis starts at 0.05 . Foundation4D stitching uses sky masks; VGGT stitching and PGO+LM do not. Each configuration uses its shared full-clip Sim(3) evaluation scale, with no per-bin alignment. † denotes internal fine-tuning.
Figure 7: Foundation4D depth predictions with PGO+LM from the front-wide and front-tele cameras of a PAI clip. Each full view is followed by a 3.2× enlargement of the region boxed in the RGB image. Absolute depth error and GT-masked predictions use valid GT support; the final column excludes predicted sky and ego-vehicle pixels from the dense prediction. Gray denotes masked pixels. Depth and error colors saturate at 300 m and 30 m, respectively. Predictions retain the shared full-clip pose-derived scale without local realignment.
Figure 8: Video-level point-cloud reconstructions on PAI using Foundation4D with PGO+LM. Each scene accumulates six-camera predictions over an approximately 20 s clip. Before confidence-filtered depth fusion, we remove moving objects using external annotation-assisted masks and suppress sky and ego-vehicle pixels using predicted semantics. Fig. 9 shows a detailed reconstruction of the overpass scene with its RGB input.
Figure 9: Detailed reconstruction of the overpass scene in Fig. 8 (b). The front-wide RGB input at τ=8.67 s is paired with RGB and semantic point-cloud renderings using the corresponding predicted camera pose and calibrated projection. Both renderings show the same geometry, accumulated from all six cameras across the full clip using depth fusion with dynamic-point restoration. White regions contain no rendered points.
Variant
Trajectory
Rig
Depth
ATE ↓
RTE ↓
RRE ↓
RTE ↓
RRE ↓
mAA@15 ↑
AbsRel ↓
δ1.25↑
(cm)
(cm)
( ∘ )
(cm)
( ∘ )
(%)
(%)
(a) Baseline
128.48
5.04
0.086
7.75
0.701
71.78
0.133
83.16
(b) + factorized global attention
149.88
4.87
0.086
7.16
0.808
74.24
0.140
82.25
(c) + rig-ego factorization
141.72
4.31
0.053
5.00
0.796
77.91
0.136
82.51
(d) + metric-scale training
70.67
3.43
0.053
5.88
0.834
76.98
0.129
83.58
Table 5: Cumulative geometry ablation on 50 internal clips with stitching and no semantic masks. Row (c) introduces rig-ego factorization with separate rig and ego losses. ATE uses one multiview Sim(3) alignment per clip, whose scale is also applied to depth.
Variant
Trajectory
Rig
Depth
ATE ↓
RTE ↓
RRE ↓
RTE ↓
RRE ↓
AbsRel ↓
δ1.25↑
(cm)
(cm)
( ∘ )
(cm)
( ∘ )
(%)
Stitching, no mask
69.53
2.94
0.042
5.77
0.802
0.127
83.62
Stitching, sky mask
65.55
2.89
0.042
5.75
0.802
0.127
83.68
PGO
33.11
2.18
0.041
5.64
0.801
0.125
84.02
PGO + LM
31.26
1.21
0.041
5.60
0.801
0.125
84.01
Table 6: Video assembly with stitching and pose graph optimization. All variants use checkpoint (f) and the protocol in Tab. 5 , including one multiview Sim(3) evaluation alignment per clip with a shared pose/depth scale. Sky masks filter stitching correspondences; PGO uses capture-time corrections without pixel-alignment masks. LM denotes the linear-motion prior.
Figure 10: Camera-position errors under different video assembly methods. In this example sequence, each method uses one full-clip multiview Sim(3) alignment shared across four cameras. Signed front-wide residuals (cm) are projected onto the ground-truth forward, left, and world-up axes and plotted against cumulative ground distance (m). Dotted lines mark window starts; all frame samples are shown without smoothing.
Model
Backbone
Detection
Architecture
Backbone Init.
mAP ↑
NDS ↑
PETRv2
ResNet101
nuImages
42.1
52.4
SparseBEV
ResNet101
nuImages
50.1
59.2
Stream3DPPE
VoV-99
FCOS3D
50.0
58.5
RoPETR
VoV-99
FCOS3D
52.9
61.4
StreamPETR
EVA02-L
Cascade R-CNN
52.1
60.8
Table 7: Camera-only 3D object detection performance on the nuScenes validation set (150 scenes). All models are trained using supervision on the 700-scene nuScenes training set.
Figure 11: Foundation4D detection and tracking on the PAI test split. (a) Detection mAP by planar distance from the ego vehicle, with full-evaluation scores shown separately. Each distance interval independently filters predicted and ground-truth box centers. (b) Tracking HOTA over the full evaluation, using center-distance matching. Colors distinguish averages over the five main classes and all twelve classes.
Figure 12: Foundation4D detection and tracking on the PAI dataset. Foundation4D offline tracks are projected into the front-wide camera. Top: the overview at 8s ; the dashed rectangle marks the region shown in the first temporal crop. Bottom: cropped views at 8 , 10 , and 12s . Colors and numeric labels denote track identities. We display the five main classes with confidence at least 0.4 .
Configuration
mAP ↑
HOTA ↑
VoVNet ( 320×800 )
21.4
21.5
DINOv2 ( 532×952 )
30.1
28.8
VGGT ( 518×518 )
27.5
26.6
+ resolution 1036×1036
31.7
30.2
+ deformable attention
32.1
29.3
+ QueryDN / auxiliary YOLOX
34.7
32.5
Table 8: Ablation study for 3D object detection and tracking. All models are trained on four cameras, and all except the last configuration use a reduced 80k steps training schedule. We indicate in parenthesis next to each backbone its image resolution.
Timestamps
Span (s)
mAP ↑
HOTA ↑
2
0.07
37.0
36.7
4
0.20
39.5
39.1
8
0.47
42.1
40.5
16
1.00
44.9
42.0
32
2.07
47.0
43.6
48
3.13
47.6
43.3
Table 9: Temporal-context ablation with a fixed checkpoint. Metrics are five-class averages in percentages; HOTA uses center-distance matching with four-camera visibility-filtered ground truth. Temporal span is (T−1)2/30 seconds for T timestamps at stride two in 30 Hz video.
Table 21
Method
IoU
mIoU
Per-class IoU
barrier
bicycle
bus
car
const. v.
motor.
ped.
t. cone
trailer
truck
drive. surf.
other flat
sidewalk
terrain
manmade
veg.
MonoScene [ 71 ]
18.4
6.9
7.1
3.9
9.3
7.2
5.6
3
5.9
4.4
4.9
4.2
14.9
6.3
7.9
7.4
10
7.6
TPVFormer [ 68 ]
15.3
7.8
9.3
4.1
11.3
10.1
5.2
4.3
5.9
5.3
6.8
6.5
13.6
9
8.3
8
9.2
8.2
CONet [ 11 ]
20.1
12.8
13.2
8.1
15.4
17.2
6.3
11.2
10
8.3
4.7
12.1
31.4
18.8
18.7
16.3
4.8
8.2
OccMamba [ 74 ]
21.7
13.2
13.2
7.4
15.3
17.6
6.1
10.3
10.3
6.5
6.2
13.3
32.9
20.5
19.8
17.4
4.8
8.7
SparseOcc [ 78 ]
21.8
14.1
16.1
9.3
15.1
18.6
7.3
9.4
11.2
9.4
7.2
13
31.8
21.7
20.7
18.8
6.1
10.6
Table 12: Semantic occupancy results on OpenOccupancy [ 11 ] . Scores are percentages.
Figure 13: Semantic occupancy predictions on OpenOccupancy. Each example pairs six camera views with the predicted occupancy and ground truth.
ROI x (m)
IoU
mIoU
Per-class IoU
road
sidewalk
build.
wall
fence
pole
traf.l.
traf.s.
veg.
terrain
person
rider
car
truck
bus
[−20,0]
38.4
18.4
60.6
34.2
8.2
13.7
8.0
10.0
7.9
5.6
13.5
30.1
9.6
14.0
26.9
18.4
15.2
[0,50]
35.0
16.0
56.1
30.0
6.9
12.6
6.2
8.3
7.1
5.0
11.5
25.2
8.3
11.9
21.8
14.9
14.3
[50,100]
22.9
6.9
35.8
16.1
2.4
8.3
2.7
1.1
1.0
0.4
4.2
14.4
0.8
1.1
7.0
4.1
4.3
[−20,100]
31.5
14.1
49.2
25.8
5.9
11.3
5.5
6.8
5.9
3.9
9.7
22.1
7.9
11.3
19.8
13.7
12.8
Table 13: Foundation4D semantic occupancy results on the PAI test set. Scores are percentages. All rows use y∈[−30,30] m and z∈[−3,5] m. In the class headers, build. denotes building, traf.l. denotes traffic light, traf.s. denotes traffic sign, and veg. denotes vegetation.
Figure 14: Semantic occupancy predictions on PAI. Each example pairs six camera views with the predicted occupancy and ground truth.
Version
0–100 m
0–50 m
50–100 m
IoU
mIoU
IoU
mIoU
IoU
mIoU
DINOv2 + LSS + occ. head
24.03
10.37
27.32
12.51
19.68
7.19
DINOv2 + BEV attn. + occ. head
27.08
11.44
29.77
13.57
23.44
8.45
VGGT + BEV attn. + occ. head
27.64
11.94
30.33
14.01
24.01
8.96
+ BEV segmentation
27.46
12.28
30.16
14.54
23.80
8.94
+ Dice loss
27.46
12.51
30.34
14.62
23.61
9.36
Table 14: Architecture and training ablations for semantic occupancy. All configurations use 4 input cameras at 512×512 resolution and an ROI of [0,100]×[−25,25]×[−3,5]m .
Figure 15: Six-camera Gaussian reconstruction with Foundation4D . Each row shows approximately synchronized native-camera renderings of one example scene from the benchmark dataset. Gaussians are predicted feed-forward without per-scene optimization.
Figure 16: Gaussian renders from four example scenes. Each row compares Instant NuRec and Foundation4D at the same target exposure timestamp in the front-wide and cross-left cameras. Labels at the top right give per-image PSNR in dB.
Target set
Region
Instant NuRec
Foundation4D with GT ego pose
Foundation4D with predicted ego pose
PSNR (dB) ↑
SSIM ↑
PSNR (dB) ↑
SSIM ↑
PSNR (dB) ↑
SSIM ↑
Overall
All
22.66
0.78
24.02
0.80
23.94
0.80
Static
22.83
0.78
24.34
0.81
24.27
0.81
Dynamic
20.01
0.72
20.03
0.73
19.98
0.73
Interpolation
All
22.86
0.78
23.98
0.80
23.91
0.80
Static
23.04
0.78
24.30
0.81
24.22
0.80
Table 15: Gaussian reconstruction quality. Both Foundation4D variants use predicted actor tracks, constructed using their respective ego poses. The GT-ego variant uses GT source and rendering poses; the predicted-ego variant uses predicted source and rendering poses. Instant NuRec takes GT ego poses as input. Scores average five common cameras equally.
Figure 17: Scene-level geometry, occupancy, and tracks. Pointmaps (left) and occupancy (right) span a 20-second clip, with car geometry and occupancy accumulated in tracked object frames. Solid cuboids mark the final scene at τ=19.97 s; dashed cuboids and translucent car and pedestrian occupancy show τ=1.50 s. The ego route uses optimized camera poses and fixed camera-to-rig calibration.
Training tasks
Detection
Occupancy
Depth
Trajectory
Rig pose
GEO
DET
OCC
Top-mAP ↑
mIoU ↑
AbsRel ↓
ATE ↓
mAA@3 ↑
RTE ↓
RRE ↓
(%)
(%)
(cm)
(%)
(cm)
( ∘ )
✓
—
—
—
—
0.123
5.00
37.09
7.53
0.898
—
✓
—
40.94
—
—
—
—
—
—
—
—
✓
—
11.00
—
—
—
—
—
✓
✓
—
41.30
—
0.127
4.38
43.17
6.52
0.966
Table 16: Perception performance across training-task combinations. All configurations use four cameras and are trained for 80,000 updates.
Component
T=24
T=36
T=48
Image encoder
2.218
3.322
4.455
Spatiotemporal aggregator
1.248
2.175
3.305
Geometry heads
1.786
2.673
3.559
Shared DET/OCC neck
0.014
0.021
0.028
Detection head
0.090
0.128
0.168
Occupancy head †
1.060
1.576
2.104
Table 17: Forward runtime and peak memory of the unified model. Median times (s) are measured per six-camera window on one H100 80GB; T counts timestamps per camera. Component medians need not sum to the independently measured full-forward median.
Figure 18: Foundation4D in lidarless NuRec. Foundation4D and video-level processing supply the scene priors; NuRec then optimizes a Gaussian scene against the recorded RGB frames. Bottom: example depth, normal, and fused-geometry priors.
Figure 19: NuRec reconstruction with and without Foundation4D predictions as priors. Left: detection and tracking priors used for reconstruction; right: novel views of the optimized NuRec scene. The bottom row shows more consistent tracks and a more complete reconstruction of the crossing white vehicle. This comparison does not isolate individual model changes.
Scope
Without Foundation4D (hours/clip)
With Foundation4D (hours/clip)
Speedup
End-to-end lidarless NuRec
16.0
3.7
4.3×
Camera pose estimation
3.5
0.4
8.8×
Depth estimation and fusion
8.0
1.0
8.0×
Table 18: Workflow turnaround across releases. Times are hours per clip, including startup and teardown. Camera configurations, software, and hardware differ across releases. Rows describe separate processing scopes and are not an additive breakdown.
Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera streams. However, existing image-based frameworks often rely on backbones pretrained for semantic recognition, and introduce 3D geometry through downstream task-specific modules. As a result, their shared representations may fail to preserve explicit metric geometry and consistent 3D scene structure. In this paper, we present a Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes. GeoUP factorizes cross-image interaction into self, temporal, and view attention to capture structurally distinct temporal and cross-view correspondences. It further injects calibration-aware raymap encodings to provide metric scale and camera geometry. The resulting geometry-grounded latent is decoded for metric depth estimation, 3D object detection, and semantic occupancy prediction, corresponding to surface-, instance-, and volume-level readouts of the same 3D scene. Through joint multi-task and multi-dataset training, GeoUP effectively leverages heterogeneous annotations and generalizes across diverse sensor configurations and perception ranges. Extensive experiments on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD demonstrate that GeoUP achieves SOTA performance across detection, occupancy, and depth estimation. These results validate the effectiveness of geometry-grounded representations for unified 3D driving perception.
Longfei Xu, Xiaohui Wang, Zehao Huang +4
School of Computer Science and Engineering Beihang University Beijing, China · School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · School of Artificial Intelligence Beihang University Beijing, China
Feedforward Gaussian Splatting has recently emerged as an efficient paradigm for 4D reconstruction in autonomous driving. However, in unstructured off-road scenes, its performance degrades due to high-frequency geometry, ego-motion jitter, and increased non-rigid dynamics. These factors introduce conflicting Gaussian observations across timestamps, leading to either over-smoothed renderings or structural artifacts. To address this issue, we propose Ground4D, a spatially-grounded 4D feedforward framework for pose-free off-road reconstruction. The key idea is to resolve temporal conflicts through spatially localized conditioning. Specifically, we introduce voxel-grounded temporal Gaussian aggregation, which partitions the canonical Gaussian space into spatial voxels and performs query-conditioned temporal attention within each voxel. Intra-voxel softmax normalization ensures that temporal selectivity and spatial occupancy become mutually reinforcing rather than conflicting. We furthermore introduce surface normal cues as auxiliary geometric guidance to regularize the geometry of Gaussian primitives. Extensive experiments on ORAD-3D and RELLIS-3D demonstrate that Ground4D consistently outperforms existing feedforward methods in reconstruction quality and generalizes zero-shot to unseen off-road domains. Project page and code:https://github.com/wsnbws/Ground4D.
Shuo Wang, Jilin Mei, Fuyang Liu +6
Institute of Computing Technology, Chinese Academy of Sciences China · Xi’an Jiaotong University China · Beijing Institute of Technology China
High-fidelity visual reconstruction and novel-view synthesis are essential for realistic closed-loop evaluation in autonomous driving. While 4D Gaussian Splatting (4DGS) offers a promising balance of accuracy and efficiency, existing per-scene optimization methods require costly iterative refinement, rendering them unscalable for extensive urban environments. Conversely, current feed-forward approaches often suffer from degraded photometric quality. To address these limitations, we propose ReconDrive, a feed-forward framework that leverages and extends the 3D foundation model VGGT for rapid, high-fidelity 4DGS generation. Our architecture introduces two core adaptations to tailor the foundation model to dynamic driving scenes: (1) Hybrid Gaussian Prediction Heads, which decouple the regression of spatial coordinates and appearance attributes to overcome the photometric deficiencies inherent in generalized foundation features; and (2) a Static-Dynamic 4D Composition strategy that explicitly captures temporal motion via velocity modeling to represent complex dynamic environments. Benchmarked on nuScenes, ReconDrive significantly outperforms existing feed-forward baselines in reconstruction, novel-view synthesis, and 3D perception. It achieves performance competitive with per-scene optimization while being orders of magnitude faster, providing a scalable and practical solution for realistic driving simulation.
Haibao Yu, Kuntao Xiao, Jiahang Wang +7
The University of Hong Kong · Tuojing Intelligence · King’s College London +2