Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

NVIDIA; :; Sérgio Agostinho; Maximilian Apodaca; Emmanuel Attia; Guillem Braso; George Dong; Ilya Fedorov

Figure 2: Foundation4D architecture. Foundation4D unifies driving perception and reconstruction in a feed-forward model with a shared representation of multi-camera video. A common image encoder and factorized spatiotemporal attention efficiently combine observations across views and time. Task-specific decoders recover camera-to-rig calibration, ego motion, and metric depth; predict 3D objects, velocities, and high-resolution semantic occupancy; and reconstruct scene appearance with 3D Gaussians.

Figure 3: Foundation4D semantic occupancy BEV features. PCA of the upsampled features reveals a separation between occupied and empty regions. All panels show the same sample; PCA colors encode feature variation, while prediction colors denote semantic classes.

Dataset	Clips	Urban	Residential	Highway	Rural	Other
NRE	67,970	16.5%	9.1%	64.5%	7.6%	2.3%
PlanNet	75,921	42.5%	17.3%	20.8%	16.5%	2.9%
MLMCF	59,999	36.5%	10.9%	42.9%	6.9%	2.8%

Table 1: Scene composition of the internal training splits. NRE is dominated by highway driving, while PlanNet emphasizes urban and residential scenes. MLMCF provides a more balanced mix of urban and highway driving, complementing the other two collections. Counts and percentages refer to unique clips within each dataset.

Figure 4: PAI depth and occupancy annotation pipeline. The pipeline combines temporally aggregated LiDAR, cuboid tracks, and image masks to generate camera-view depth maps and vehicle-frame occupancy grids with per-voxel semantic and velocity labels.

Figure 5: Generated depth and semantic occupancy annotations on PAI. Each example shows multi-camera RGB images overlaid with depth (top), followed by unprojected depth colored by RGB, a side view of semantic occupancy, and a forward view of semantic occupancy (bottom, left to right). The shared legend indicates depth in meters and semantic classes.

Model	Waymo		nuScenes
	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$ (%)	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$ (%)
DVGT-1 [ 40 ]	0.20	72.87	0.18	78.97
DVGT-2 [ 23 ]	0.26	70.93	0.22	75.32
LiAuto-GeoX [ 44 ]	0.19	75.72	0.27	62.75
GeoUP [ 3 ]	0.13	87.30	0.19	80.05
Foundation4D	0.09	91.97	0.15	86.44

Table 2: Depth estimation on the Waymo and nuScenes validation splits. Depth is evaluated independently in non-overlapping windows of 48 consecutive frames per camera, sampled at the native frame rates (Waymo: 10 Hz; nuScenes:

\approx

12 Hz).

Dataset	Inference Mode	Model	Trajectory			Rig			Depth
			ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	mAA@15 $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
			(cm)	(cm)	( ∘ )	(cm)	( ∘ )	(%)		(%)
Waymo	Stitching	DVGT-1 [ 40 ]	62.33	3.54	0.06	–	–	–	0.20	73.04
		Foundation4D	32.70	3.20	0.06	0.96	1.02	83.37	0.09	92.33
	PGO+LM	DVGT-1 [ 40 ]	54.55	2.38	0.05	–	–	–	0.20	73.26
		Foundation4D	23.62	1.56	0.06	0.90	1.03	83.54	0.09	92.94

Table 3: Video-level geometry comparison on the Waymo and nuScenes validation data described in Sec. 7.1.1 , at native camera rates (10 Hz and

\approx

12 Hz). Each clip uses one evaluation Sim(3), shared by poses and depth. Both models use our video inference system with 48-frame windows and a 5-frame overlap for stitching and PGO+LM. Rig metrics are omitted (–) because DVGT-1 does not estimate the rig.

Model	Inference Mode	Trajectory			Rig			Depth
		ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	mAA@15 $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
		(cm)	(cm)	( ∘ )	(cm)	( ∘ )	(%)		(%)
VGGT †	Stitching	111.12	8.31	0.122	29.26	0.520	42.50	0.084	94.88
Foundation4D	Stitching	78.06	4.69	0.070	3.90	0.404	88.72	0.081	94.98
	PGO+LM	52.75	2.28	0.066	3.92	0.404	88.71	0.079	95.03

Table 4: Geometry results on all 1,253 PAI test clips with six cameras. † denotes internal fine-tuning. Metrics and evaluation alignment follow Sec. 7.1 .

Figure 6: Depth error by ground-truth depth range on PAI, averaged equally over 1,253 clips in each bin. AbsRel is dimensionless; the y-axis starts at

0.05

. Foundation4D stitching uses sky masks; VGGT stitching and PGO+LM do not. Each configuration uses its shared full-clip Sim(3) evaluation scale, with no per-bin alignment. † denotes internal fine-tuning.

Figure 7: Foundation4D depth predictions with PGO+LM from the front-wide and front-tele cameras of a PAI clip. Each full view is followed by a

3.2\!\times

enlargement of the region boxed in the RGB image. Absolute depth error and GT-masked predictions use valid GT support; the final column excludes predicted sky and ego-vehicle pixels from the dense prediction. Gray denotes masked pixels. Depth and error colors saturate at

300

m and

30

m, respectively. Predictions retain the shared full-clip pose-derived scale without local realignment.

Figure 8: Video-level point-cloud reconstructions on PAI using Foundation4D with PGO+LM. Each scene accumulates six-camera predictions over an approximately

20

s clip. Before confidence-filtered depth fusion, we remove moving objects using external annotation-assisted masks and suppress sky and ego-vehicle pixels using predicted semantics. Fig. 9 shows a detailed reconstruction of the overpass scene with its RGB input.

Figure 9: Detailed reconstruction of the overpass scene in Fig. 8 (b). The front-wide RGB input at

\tau=8.67

s is paired with RGB and semantic point-cloud renderings using the corresponding predicted camera pose and calibrated projection. Both renderings show the same geometry, accumulated from all six cameras across the full clip using depth fusion with dynamic-point restoration. White regions contain no rendered points.

Variant	Trajectory			Rig			Depth
	ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	mAA@15 $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
	(cm)	(cm)	( ∘ )	(cm)	( ∘ )	(%)		(%)
(a) Baseline	128.48	5.04	0.086	7.75	0.701	71.78	0.133	83.16
(b) + factorized global attention	149.88	4.87	0.086	7.16	0.808	74.24	0.140	82.25
(c) + rig-ego factorization	141.72	4.31	0.053	5.00	0.796	77.91	0.136	82.51
(d) + metric-scale training	70.67	3.43	0.053	5.88	0.834	76.98	0.129	83.58

Table 5: Cumulative geometry ablation on 50 internal clips with stitching and no semantic masks. Row (c) introduces rig-ego factorization with separate rig and ego losses. ATE uses one multiview Sim(3) alignment per clip, whose scale is also applied to depth.

Variant	Trajectory			Rig		Depth
	ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
	(cm)	(cm)	( ∘ )	(cm)	( ∘ )		(%)
Stitching, no mask	69.53	2.94	0.042	5.77	0.802	0.127	83.62
Stitching, sky mask	65.55	2.89	0.042	5.75	0.802	0.127	83.68
PGO	33.11	2.18	0.041	5.64	0.801	0.125	84.02
PGO + LM	31.26	1.21	0.041	5.60	0.801	0.125	84.01

Table 6: Video assembly with stitching and pose graph optimization. All variants use checkpoint (f) and the protocol in Tab. 5 , including one multiview Sim(3) evaluation alignment per clip with a shared pose/depth scale. Sky masks filter stitching correspondences; PGO uses capture-time corrections without pixel-alignment masks. LM denotes the linear-motion prior.

Figure 10: Camera-position errors under different video assembly methods. In this example sequence, each method uses one full-clip multiview Sim(3) alignment shared across four cameras. Signed front-wide residuals (cm) are projected onto the ground-truth forward, left, and world-up axes and plotted against cumulative ground distance (m). Dotted lines mark window starts; all frame samples are shown without smoothing.

Model	Backbone		Detection
Model	Architecture	Backbone Init.	mAP $\uparrow$	NDS $\uparrow$
PETRv2	ResNet101	nuImages	42.1	52.4
SparseBEV	ResNet101	nuImages	50.1	59.2
Stream3DPPE	VoV-99	FCOS3D	50.0	58.5
RoPETR	VoV-99	FCOS3D	52.9	61.4
StreamPETR	EVA02-L	Cascade R-CNN	52.1	60.8

Table 7: Camera-only 3D object detection performance on the nuScenes validation set (150 scenes). All models are trained using supervision on the 700-scene nuScenes training set.

Figure 11: Foundation4D detection and tracking on the PAI test split. (a) Detection mAP by planar distance from the ego vehicle, with full-evaluation scores shown separately. Each distance interval independently filters predicted and ground-truth box centers. (b) Tracking HOTA over the full evaluation, using center-distance matching. Colors distinguish averages over the five main classes and all twelve classes.

Figure 12: Foundation4D detection and tracking on the PAI dataset. Foundation4D offline tracks are projected into the front-wide camera. Top: the overview at

8\,\mathrm{s}

; the dashed rectangle marks the region shown in the first temporal crop. Bottom: cropped views at

8

,

10

, and

12\,\mathrm{s}

. Colors and numeric labels denote track identities. We display the five main classes with confidence at least

0.4

.

Configuration	mAP $\uparrow$	HOTA $\uparrow$
VoVNet ( $320\!\times\!800$ )	21.4	21.5
DINOv2 ( $532\!\times\!952$ )	30.1	28.8
VGGT ( $518\!\times\!518$ )	27.5	26.6
+ resolution $1036\!\times\!1036$	31.7	30.2
+ deformable attention	32.1	29.3
+ QueryDN / auxiliary YOLOX	34.7	32.5

Table 8: Ablation study for 3D object detection and tracking. All models are trained on four cameras, and all except the last configuration use a reduced

80k

steps training schedule. We indicate in parenthesis next to each backbone its image resolution.

Timestamps	Span (s)	mAP $\uparrow$	HOTA $\uparrow$
2	0.07	37.0	36.7
4	0.20	39.5	39.1
8	0.47	42.1	40.5
16	1.00	44.9	42.0
32	2.07	47.0	43.6
48	3.13	47.6	43.3

Table 9: Temporal-context ablation with a fixed checkpoint. Metrics are five-class averages in percentages; HOTA uses center-distance matching with four-camera visibility-filtered ground truth. Temporal span is

(T-1)\,2/30

seconds for

T

timestamps at stride two in 30 Hz video.

Table 21

Method	IoU	mIoU	Per-class IoU
Method	IoU	mIoU	barrier	bicycle	bus	car	const. v.	motor.	ped.	t. cone	trailer	truck	drive. surf.	other flat	sidewalk	terrain	manmade	veg.
MonoScene [ 71 ]	18.4	6.9	7.1	3.9	9.3	7.2	5.6	3	5.9	4.4	4.9	4.2	14.9	6.3	7.9	7.4	10	7.6
TPVFormer [ 68 ]	15.3	7.8	9.3	4.1	11.3	10.1	5.2	4.3	5.9	5.3	6.8	6.5	13.6	9	8.3	8	9.2	8.2
CONet [ 11 ]	20.1	12.8	13.2	8.1	15.4	17.2	6.3	11.2	10	8.3	4.7	12.1	31.4	18.8	18.7	16.3	4.8	8.2
OccMamba [ 74 ]	21.7	13.2	13.2	7.4	15.3	17.6	6.1	10.3	10.3	6.5	6.2	13.3	32.9	20.5	19.8	17.4	4.8	8.7
SparseOcc [ 78 ]	21.8	14.1	16.1	9.3	15.1	18.6	7.3	9.4	11.2	9.4	7.2	13	31.8	21.7	20.7	18.8	6.1	10.6

Table 12: Semantic occupancy results on OpenOccupancy [ 11 ] . Scores are percentages.

Figure 13: Semantic occupancy predictions on OpenOccupancy. Each example pairs six camera views with the predicted occupancy and ground truth.

ROI $x$ (m)	IoU	mIoU	Per-class IoU
ROI $x$ (m)	IoU	mIoU	road	sidewalk	build.	wall	fence	pole	traf.l.	traf.s.	veg.	terrain	person	rider	car	truck	bus
$[-20,0]$	38.4	18.4	60.6	34.2	8.2	13.7	8.0	10.0	7.9	5.6	13.5	30.1	9.6	14.0	26.9	18.4	15.2
$[0,50]$	35.0	16.0	56.1	30.0	6.9	12.6	6.2	8.3	7.1	5.0	11.5	25.2	8.3	11.9	21.8	14.9	14.3
$[50,100]$	22.9	6.9	35.8	16.1	2.4	8.3	2.7	1.1	1.0	0.4	4.2	14.4	0.8	1.1	7.0	4.1	4.3
$[-20,100]$	31.5	14.1	49.2	25.8	5.9	11.3	5.5	6.8	5.9	3.9	9.7	22.1	7.9	11.3	19.8	13.7	12.8

Table 13: Foundation4D semantic occupancy results on the PAI test set. Scores are percentages. All rows use

y\in[-30,30]

m and

z\in[-3,5]

m. In the class headers, build. denotes building, traf.l. denotes traffic light, traf.s. denotes traffic sign, and veg. denotes vegetation.

Figure 14: Semantic occupancy predictions on PAI. Each example pairs six camera views with the predicted occupancy and ground truth.

Version	0–100 m		0–50 m		50–100 m
Version	IoU	mIoU	IoU	mIoU	IoU	mIoU
DINOv2 + LSS + occ. head	24.03	10.37	27.32	12.51	19.68	7.19
DINOv2 + BEV attn. + occ. head	27.08	11.44	29.77	13.57	23.44	8.45
VGGT + BEV attn. + occ. head	27.64	11.94	30.33	14.01	24.01	8.96
+ BEV segmentation	27.46	12.28	30.16	14.54	23.80	8.94
+ Dice loss	27.46	12.51	30.34	14.62	23.61	9.36

Table 14: Architecture and training ablations for semantic occupancy. All configurations use

4

input cameras at

512\times 512

resolution and an ROI of

[0,100]\times[-25,25]\times[-3,5]\,\mathrm{m}

.

Figure 15: Six-camera Gaussian reconstruction with Foundation4D . Each row shows approximately synchronized native-camera renderings of one example scene from the benchmark dataset. Gaussians are predicted feed-forward without per-scene optimization.

Figure 16: Gaussian renders from four example scenes. Each row compares Instant NuRec and Foundation4D at the same target exposure timestamp in the front-wide and cross-left cameras. Labels at the top right give per-image PSNR in dB.

Target set	Region	Instant NuRec		Foundation4D with GT ego pose		Foundation4D with predicted ego pose
		PSNR (dB) $\uparrow$	SSIM $\uparrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$
Overall	All	22.66	0.78	24.02	0.80	23.94	0.80
	Static	22.83	0.78	24.34	0.81	24.27	0.81
	Dynamic	20.01	0.72	20.03	0.73	19.98	0.73
Interpolation	All	22.86	0.78	23.98	0.80	23.91	0.80
	Static	23.04	0.78	24.30	0.81	24.22	0.80

Table 15: Gaussian reconstruction quality. Both Foundation4D variants use predicted actor tracks, constructed using their respective ego poses. The GT-ego variant uses GT source and rendering poses; the predicted-ego variant uses predicted source and rendering poses. Instant NuRec takes GT ego poses as input. Scores average five common cameras equally.

Figure 17: Scene-level geometry, occupancy, and tracks. Pointmaps (left) and occupancy (right) span a 20-second clip, with car geometry and occupancy accumulated in tracked object frames. Solid cuboids mark the final scene at

\tau=19.97

s; dashed cuboids and translucent car and pedestrian occupancy show

\tau=1.50

s. The ego route uses optimized camera poses and fixed camera-to-rig calibration.

Training tasks			Detection	Occupancy	Depth	Trajectory	Rig pose
GEO	DET	OCC	Top-mAP $\uparrow$	mIoU $\uparrow$	AbsRel $\downarrow$	ATE $\downarrow$	mAA@3 $\uparrow$	RTE $\downarrow$	RRE $\downarrow$
			(%)	(%)		(cm)	(%)	(cm)	( ∘ )
$\checkmark$	—	—	—	—	0.123	$5.00$	$37.09$	$7.53$	0.898
—	$\checkmark$	—	$40.94$	—	—	—	—	—	—
—	—	$\checkmark$	—	$11.00$	—	—	—	—	—
$\checkmark$	$\checkmark$	—	$41.30$	—	$0.127$	4.38	43.17	6.52	$0.966$

Table 16: Perception performance across training-task combinations. All configurations use four cameras and are trained for 80,000 updates.

Component	$T=24$	$T=36$	$T=48$
Image encoder	2.218	3.322	4.455
Spatiotemporal aggregator	1.248	2.175	3.305
Geometry heads	1.786	2.673	3.559
Shared DET/OCC neck	0.014	0.021	0.028
Detection head	0.090	0.128	0.168
Occupancy head †	1.060	1.576	2.104

Table 17: Forward runtime and peak memory of the unified model. Median times (s) are measured per six-camera window on one H100 80GB;

T

counts timestamps per camera. Component medians need not sum to the independently measured full-forward median.

Figure 18: Foundation4D in lidarless NuRec. Foundation4D and video-level processing supply the scene priors; NuRec then optimizes a Gaussian scene against the recorded RGB frames. Bottom: example depth, normal, and fused-geometry priors.

Figure 19: NuRec reconstruction with and without Foundation4D predictions as priors. Left: detection and tracking priors used for reconstruction; right: novel views of the optimized NuRec scene. The bottom row shows more consistent tracks and a more complete reconstruction of the crossing white vehicle. This comparison does not isolate individual model changes.

Scope	Without Foundation4D (hours/clip)	With Foundation4D (hours/clip)	Speedup
End-to-end lidarless NuRec	16.0	3.7	$4.3\times$
Camera pose estimation	3.5	0.4	$8.8\times$
Depth estimation and fusion	8.0	1.0	$8.0\times$

Table 18: Workflow turnaround across releases. Times are hours per clip, including startup and teardown. Camera configurations, software, and hardware differ across releases. Rows describe separate processing scopes and are not an additive breakdown.

Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

Abstract

Figures & tables

Autonomous Driving Perception

Geometry-Grounded Unified 3D Perception for Autonomous Driving

Ground4D: Spatially-Grounded Feedforward 4D Reconstruction for Unstructured Off-Road Scenes

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction

Abstract

Autonomous Driving Perception

Explore similar work

Geometry-Grounded Unified 3D Perception for Autonomous Driving

Ground4D: Spatially-Grounded Feedforward 4D Reconstruction for Unstructured Off-Road Scenes

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction