USR-Drive: Unified Driving Scene Representation via Joint Denoising of 3D Gaussians and Boxes

Li-Heng Chen; Haokai Pang; Chengye Su; Jiarun Liu; Qifeng Chen; Ziqian Ni; Jianxin Huang; Shi-Sheng Huang

cs.CVAug 19, 2026

USR-Drive: Unified Driving Scene Representation via Joint Denoising of 3D Gaussians and Boxes

Authors: Li-Heng Chen, Haokai Pang, Chengye Su, Jiarun Liu, Qifeng Chen, Ziqian Ni, Jianxin Huang, Shi-Sheng Huang, +2 more

Organizations: NIO · HKUST · Beijing Normal University

Abstract

Spatial representation learning for autonomous driving aims to map raw visual signals into structured 3D scene representations, where object-centric bounding boxes and rendering-oriented 3D primitives (\eg, 3D Gaussians) serve as two distinct yet highly complementary levels for scene understanding. Existing methods typically treat dynamic reconstruction and instance-level perception as separate tasks, despite their shared goal of estimating the underlying 3D world state. As a result, dynamic reconstruction is under-constrained while 3D detection lacks geometric grounding. To address this gap, we propose USR-Drive, a unified conditional generative framework that, given only posed multi-view driving videos, jointly recovers dense dynamic geometry and instance-level object layouts within a shared scene representation. Specifically, USR-Drive represents dense Gaussian primitives and sparse 3D bounding boxes as two aligned latent token streams and jointly denoises them with a unified multi-modal diffusion Transformer. Unlike prior paradigms that use boxes as external conditions or predict them with detached modules, USR-Drive treats them as mutually constrained state variables with a Unified Positional Encoding (UPE) that aligns heterogeneous tokens within a shared metric spatiotemporal coordinate. Via such unified representation and generative framework, the two modalities reinforce each other: geometry supplies dense metric evidence for box prediction, while boxes provide instance-level structural priors that help preserve spatial consistency and reduce ambiguity in sequential 3D geometric representation. Our approach successfully delivers state-of-the-art results for both dynamic reconstruction and 3D detection on the nuScenes and VKitti datasets.

Figures & tables

Figure 2: Overview of USR-Drive. Stage I (Top): A dual-branch autoencoder compresses dense geometry and slot-aligned 3D boxes into latents $z_{\textrm{geo}}$ and $z_{\textrm{box}}$ , respectively. Stage II (Bottom): A unified MMDiT jointly denoises both latent streams, where the Unified Positional Encoding (UPE) anchors heterogeneous tokens in a common 3D metric space, enabling bidirectional regularization between geometry and layout through shared self-attention. Inference (Right): Starting from pure noise, the MMDiT iteratively denoises the joint latent streams conditioned on reference video latents, and the denoised latents are decoded by the DPT geometry head and the box decoder into explicit dense scene geometry and 3D object layouts.

Figure 3: Unified Positional Encoding. Dense GS tokens $\mathbf{p}^{\mathrm{GS}}_{t,u}$ and sparse box tokens $\mathbf{p}^{\mathrm{Box}}_{t,n}$ are assigned metric 3D anchors and a normalized frame index, which are encoded by shared spatial and temporal Fourier features followed by an MLP. The resulting UPE embeddings are combined with projected geometry and box latents together with learnable modality-aware tokens, aligning heterogeneous modalities into a unified latent space for joint DiT modeling.

Figure 4: Visualization of reconstruction and detection results. Compared to baseline methods, our approach reveals better quality on both reconstruction task (top 2 rows) and detection task (bottom 2 rows).

Scene-level Reconstruction
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	D-RMSE $\downarrow$
VGGT	–	–	–	5.07
DA3	21.75	0.617	0.183	12.16
Pi-3	–	–	–	6.93
AnySplat	25.53	0.803	0.157	17.65
STORM	24.54	0.784	0.267	6.48

Table 1: Scene-level and object-level reconstruction results on the nuScenes dataset. The object-level evaluation is conducted on foreground dynamic objects. We highlight the first , second , and third best results within each reconstruction block. VGGT and Pi-3 only produce 3D point clouds, so rendered-image quality metrics are omitted.

Method	NDS $\uparrow$	mAP $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
BEVDepth	0.475	0.351	0.629	0.267	0.479	0.428	0.198
BEVFormer	0.517	0.416	0.673	0.274	0.372	0.394	0.198
PETRv2	0.456	0.349	0.700	0.275	0.580	0.437	0.187
HoP	0.558	0.454	0.565	0.265	0.327	0.337	0.194
StreamPETR	0.550	0.450	0.613	0.267	0.413	0.265	0.198
RayDN	0.563	0.469	0.579	0.264	0.433	0.256	0.187

Table 2: Results of 3D detection tasks on the nuScenes val set. We inherit the reported results of baseline methods from their official publications ( Zong et al. 2023 ; Liu et al. 2024 ; Ji et al. 2025 ) .

Detection-Only Baselines
Method	Reconstruction		Detection
Method	PSNR $\uparrow$	SSIM $\uparrow$	mAP $\uparrow$	mATE $\downarrow$
RayDN	-	-	0.015	1.024
HoP	-	-	0.022	1.258
StreamPETR	-	-	0.008	1.340
Reconstruction-Only Baselines

Table 3: Zero-shot reconstruction and 3D detection results on VKitti. Our unified model performs both tasks, while baselines specialize in only one.

Joint Denoising and Pipeline Comparison
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mAP $\uparrow$	mATE $\downarrow$
Stage-I Geo-AE	23.43	0.709	0.177	–	–
Decoupled two-stage	21.47	0.655	0.179	0.473	0.701
DA3 + StreamPETR	–	–	–	0.491	0.648
Component and UPE Ablation
Geom. Only	26.37	0.833	0.127	–	–

Table 4: Ablation results on nuScenes val set. Top: joint denoising vs. decoupled pipelines. Bottom: component and UPE ablations. Our full model performs best on both tasks.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Auxiliary variable placement	mAP $\uparrow$	mATE $\downarrow$	PSNR $\uparrow$
No auxiliary	0.223	0.801	27.49
Inside BBox Autoencoder	0.445	0.679	27.31
Separate Learnable Tokens (Ours)	0.552	0.525	27.55

Appendix

Table 5: Ablation on auxiliary layout tokens.

Configuration	PSNR $\uparrow$	SSIM $\uparrow$	mAP $\uparrow$	mATE $\downarrow$
$\lambda_{\textrm{box}}=1.0$	27.42	0.849	0.449	0.624
$\lambda_{\textrm{box}}=3.0$	27.50	0.851	0.491	0.575
$\lambda_{\textrm{box}}=7.0$	27.20	0.841	0.521	0.540
$\lambda_{\textrm{box}}=10.0$	26.38	0.812	0.516	0.548
$\lambda_{\textrm{box}}=5.0$ , w/o Curr.	26.95	0.831	0.473	0.585
$\lambda_{\textrm{box}}=5.0$ , w/ Curr.	27.55	0.853	0.552	0.525

Appendix

Table 6: Ablation on training hyperparameters. We evaluate the bounding-box loss weight

\lambda_{\textrm{box}}

and the high-noise curriculum. The default setting achieves the best balance between geometric fidelity and layout accuracy.

Feature Dim.	Params (B)	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
768	0.356	25.84	0.792	0.138
1024	0.631	26.68	0.825	0.104
1536 (Default)	1.418	27.55	0.853	0.076
2048	2.517	27.48	0.849	0.080

Appendix

Table 7: Ablation on MMDiT hidden dimension. The default 1536-dimensional setting achieves the best trade-off between reconstruction quality and model size.

Figure 5: Visualization of reconstruction results. Compared to baseline methods, our approach reveals better appearance quality on both static scenes (top 2 rows) and dynamic objects (bottom 2 rows).

Figure 6: Visualization of 3D detection results. While previous methods suffer from missed or inaccurate detection, our method achieves better results considering completeness and accuracy.

Figure 7: Visualization of Ablation Studies. The complete model provides the best reconstruction effect and detection quality.

Figure 8: Visualization of 3D detection results on VKitti dataset. While previous methods suffer from missed or inaccurate detection, our method achieves better results considering completeness and accuracy.

Figure 9: Visualization of reconstruction results on VKitti dataset. Compared to baseline methods, our approach reveals better appearance quality.

Figure 10: Visualization of Depth results. Our method achieves best depth estimation on both qualitative and quantitative results.

Explore similar work

Geometry-Grounded Unified 3D Perception for Autonomous Driving
Aug 13, 2026Longfei Xu, Xiaohui Wang, Zehao Huang +4Depth Estimation Autonomous Driving Perception
School of Computer Science and Engineering Beihang University Beijing, China · School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · School of Artificial Intelligence Beihang University Beijing, China
PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations
May 12, 2026Cheng Chi, Xianqi Wang, Hongcheng Luo +93D Reconstruction Multi-View 3D Reconstruction
Xiaomi EV · Huazhong University of Science and Technology · Zhejiang University
LiAuto-GeoX: Efficient Grounded Driving Transformer
Jun 4, 2026Jiawei Lian, Haoyi Sun, Yang Wu +83D ViTs 3D Reconstruction
Nanjing University of Science and Technology · Li Auto Inc. · Northwestern Polytechnical University +1

Aug 13, 2026J/K move · Enter open · S save

Geometry-Grounded Unified 3D Perception for Autonomous Driving

Longfei Xu, Xiaohui Wang, Zehao Huang, Han Li+3

School of Computer Science and Engineering Beihang University Beijing, China · School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · School of Artificial Intelligence Beihang University Beijing, China

Depth Estimation Autonomous Driving Perception 3D Occupancy Prediction 3D Object Detection 3D Representation Learning 3D Scene Understanding

Open paper AskPDF arXiv

Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera streams. However, existing image-based frameworks often rely on backbones pretrained for semantic recognition, and introduce 3D geometry through downstream task-specific modules. As a result, their shared representations may fail to preserve explicit metric geometry and consistent 3D scene structure. In this paper, we present a Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes. GeoUP factorizes cross-image interaction into self, temporal, and view attention to capture structurally distinct temporal and cross-view correspondences. It further injects calibration-aware raymap encodings to provide metric scale and camera geometry. The resulting geometry-grounded latent is decoded for metric depth estimation, 3D object detection, and semantic occupancy prediction, corresponding to surface-, instance-, and volume-level readouts of the same 3D scene. Through joint multi-task and multi-dataset training, GeoUP effectively leverages heterogeneous annotations and generalizes across diverse sensor configurations and perception ranges. Extensive experiments on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD demonstrate that GeoUP achieves SOTA performance across detection, occupancy, and depth estimation. These results validate the effectiveness of geometry-grounded representations for unified 3D driving perception.

Abstract

Explore similar work

Geometry-Grounded Unified 3D Perception for Autonomous Driving

PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations

LiAuto-GeoX: Efficient Grounded Driving Transformer