DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding

Dong Zhuo; Wenzhao Zheng; Sicheng Zuo; Siming Yan; Lu Hou; Jie Zhou; Jiwen Lu

cs.CVMar 19, 2026

DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding

Authors: Dong Zhuo, Wenzhao Zheng, Sicheng Zuo, Siming Yan, Lu Hou, Jie Zhou, Jiwen Lu

Organizations: Tsinghua University · Yinwang Intelligent Technology Co. Ltd.

Abstract

With the growing adoption of vision-language-action models and world models in autonomous driving systems, scalable image tokenization becomes crucial as the interface for the visual modality. However, most existing tokenizers are designed for monocular and 2D scenes, leading to inefficiency and inter-view inconsistency when applied to high-resolution multi-view driving scenes. To address this, we propose DriveTok, an efficient 3D driving scene tokenizer for unified multi-view reconstruction and understanding. DriveTok first obtains semantically rich visual features from vision foundation models and then transforms them into the scene tokens with 3D deformable cross-attention. For decoding, we employ a multi-view transformer to reconstruct multi-view features from the scene tokens and use multiple heads to obtain RGB, depth, and semantic reconstructions. We also add a 3D head directly on the scene tokens for 3D semantic occupancy prediction for better spatial awareness. With the multiple training objectives, DriveTok learns unified scene tokens that integrate semantic, geometric, and textural information for efficient multi-view tokenization. Extensive experiments on the widely used nuScenes dataset demonstrate that the scene tokens from DriveTok perform well on image reconstruction, semantic segmentation, depth prediction, and 3D occupancy prediction tasks.

Figures & tables

Figure 2: Illustration of DriveTok. DriveTok processes multi-view images and scene queries through an encoder-decoder architecture to produce unified scene tokens and generate diverse autonomous driving scene reconstruction and understanding outputs.

Figure 3: Overview of DriveTok. Surround-view images are encoded by a 3D scene encoder. view tokens (with learnable and Plücker-ray embeddings) and scene tokens interact through a spatial-aware multi-view transformer with visibility-guided scene-view attention. Joint pretraining uses image reconstruction, depth prediction, semantic prediction, and occupancy prediction.

Method	Input	PSNR $\uparrow$	SSIM $\uparrow$	Latency (Tok./Full) $\downarrow$	Peak Memory (Tok./Full) $\downarrow$
$\text{VQGAN}^{\ddagger}$ [ 9 ]	$256\times 256$	24.41	0.674	63.31 ms / 77.06 ms	1440.7 MB / 1608.9 MB
ViT-VQGAN [ 56 ]	$256\times 256$	26.96	0.763	59.23 ms / 117.36 ms	1354.4 MB / 1445.1 MB
$\text{VQGAN}^{\dagger}$ [ 9 ]	$256\times 256$	26.77	0.756	63.34 ms / 115.17 ms	1440.7 MB / 1605.8 MB
FlowMo-Lo [ 39 ]	$256\times 256$	23.11	0.696	248.04 ms / 5555.78 ms	8434.8 MB / 8464.5 MB
FlowMo-Hi [ 39 ]	$256\times 256$	27.91	0.793	259.70 ms / 5562.63 ms	8436.6 MB / 8466.6 MB
BEV-VAE [ 5 ]	$6\times 256\times 256$	26.32	0.746	– / –	– / –

Table 1: Comparison of image tokenizers on nuScenes dataset.

\ddagger

denotes VQGAN was trained on Imagenet [ 7 ] ,

\dagger

indicates trained on OpenImages [ 25 ] . Tok. denotes the tokenization stage only (w/o decoder).

Method	Input	absrel $\downarrow$	$\delta<1.25$ $\uparrow$	latency $\downarrow$	Peak Memory $\downarrow$
UniDepthV2-B [ 35 ]	$256\times 704$	0.52	0.30	62.45 ms	2194.65 MB
UniDepthV2-L [ 35 ]	$256\times 704$	0.40	0.31	100.45 ms	3902.32 MB
DepthPro [ 1 ]	$256\times 704$	0.26	0.47	5179.93 ms	30559.55 MB
Metric3D-V2 [ 16 ]	$256\times 704$	0.28	0.69	403.49 ms	2982.53 MB
DriveTok	$6\times 256\times 704$	0.08	0.93	383.53 ms	8562.64 MB

Table 2: Comparison of different monocular depth estimator on nuScenes dataset. We utilize the projected lidar points as ground-truth.

Method	absrel $\downarrow$	$\delta<1.25$ $\uparrow$	Method	absrel $\downarrow$	$\delta<1.25$ $\uparrow$
SurroundDepth [ 51 ]	0.28	0.66	Dist4D [ 13 ]	0.39	0.58
R3D3 [ 40 ]	0.25	0.73	OmniNWM [ 26 ]	0.23	0.81
SelfOcc [ 19 ]	0.23	0.75	DriveTok	0.08	0.93

Table 3: Comparison of different multi-view depth predictors on the nuScenes dataset. We utilize the projected lidar points as ground-truth.

Method	IoU	mIoU	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ const. veh.	$\blacksquare$ motorcycle	$\blacksquare$ pedestrian	$\blacksquare$ traffic cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drive. suf.	$\blacksquare$ other flat	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation
MonoScene [ 3 ]	23.96	7.31	4.03	0.35	8.00	8.04	2.90	0.28	1.16	0.67	4.01	4.35	27.72	5.20	15.13	11.29	9.03	14.86
Atlas [ 33 ]	28.66	15.00	10.64	5.68	19.66	24.94	8.90	8.84	6.47	3.28	10.42	16.21	34.86	15.46	21.89	20.95	11.21	20.54
BEVFormer [ 29 ]	30.50	16.75	14.22	6.58	23.46	28.28	8.66	10.77	6.64	4.05	11.20	17.78	37.28	18.00	22.88	22.17	13.80	22.21
TPVFormer [ 20 ]	30.86	17.10	15.96	5.31	23.86	27.32	9.79	8.74	7.09	5.20	10.97	19.22	38.87	21.25	24.26	23.15	11.73	20.81
OccFormer [ 58 ]	31.39	19.03	18.65	10.41	23.92	30.29	10.31	14.19	13.59	10.13	12.49	20.77	38.78	19.79	24.19	22.21	13.48	21.35
SurroundOcc [ 52 ]	31.49	20.30	20.59	11.68	28.06	30.86	10.70	15.14	14.09	12.06	14.38	22.26	37.29	23.70	24.49	22.77	14.89	21.86

Table 4: 3D semantic occupancy prediction results on nuScenes.

Figure 4: Visualizations of BEV scene tokens and images. We visualize the BEV feature maps with PCA, the ground truth labels for semantic regularization and the decoded images v.s. ground truth images. The PCA result clearly shows that our BEV scene tokens not only learns the complex textures but also models the semantic structure of the driving scenes, avoiding radial patterns in conventional methods.

Setting	Texture		Geometry & Semantics
Setting	PSNR $\uparrow$	SSIM $\uparrow$	xx IoU $\uparrow$	xx mIoU $\uparrow$
w/o Geo-Attn	29.98	0.798	xx 5.32	xx 0.59
w/ Geo-Attn	28.84	0.770	xx 12.81	xx 3.84

Table 5: Ablation on visibility-guided attention. Removing visibility guidance degrades spatial reasoning, causing scene tokens to overfit image textures.

Setting	Tasks				Texture		Geometry & Semantics
Setting	Recon.	Depth	Sem.	Occ.	PSNR $\uparrow$	SSIM $\uparrow$	xx IoU $\uparrow$	xx mIoU $\uparrow$
Image Tokenizer	✓				28.84	0.770	xx 12.81	xx 3.84
Geometry-Aware Model	✓	✓			28.52	0.764	xx 20.72	xx 9.36
Scene-Understanding Model	✓	✓	✓		28.06	0.756	xx 24.05	xx 11.68
Unified Model	✓	✓	✓	✓	27.89	0.747	xx 33.32	xx 20.06

Table 6: Ablation study on the joint task training strategy. Different combinations of reconstruction, depth, semantic, and occupancy objectives reveal a trade-off between texture quality and geometry-semantic understanding, where stronger multi-task supervision improves occupancy prediction performance.

Figure 5: Visualizations of DriveTok in different tasks. We provide a holistic visualization of our DriveTok in diverse autonomous driving scene reconstruction and understanding tasks. They show that DriveTok effectively represents the overall 3D environment by constructing scene tokens, thereby maintaining strong multi-view consistency. Through joint task training, the learned scene tokens not only capture texture details in the image space but also achieve a deeper perception and understanding.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Visualization of visibility-guided mask. Each color corresponds to the region in the BEV plane that is visible from a particular camera, forming six wedge-shaped sectors that meet at the ego-vehicle center. The black region at the center corresponds to areas not visible to any camera. In addition, any colors outside these six primary sectors represent boundary zones where multiple camera frustums overlap.

Figure 7: Visualizations of DriveTok in different tasks. Each column corresponds to a different autonomous driving scenario, including challenging night-time scenes. These results illustrate that DriveTok can generate coherent 2D and 3D predictions across diverse tasks from our unified scene tokens.

Module	Avg. time (ms)	Peak GPU memory (MB)
Image Backbone	13.56	3747.24
FPN	2.01	3771.46
BEV Encoder	72.39	4423.73
Spatial-Aware Decoder	225.85	6476.74
RGB Head	68.70	8456.09
Depth Head	69.72	8562.64

Appendix

Table 7: Average runtime and GPU memory usage of each DriveTok module during forward inference.

Method	Backbone	Input	IOU $\uparrow$	mIOU $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	absrel $\downarrow$	$\delta<1.25$ $\uparrow$
BEVFormer [ 29 ]	ResNet-101	896 $\times$ 1600	30.50	16.75	-	-	-	-
GaussianFormer [ 21 ]	ResNet-101	896 $\times$ 1600	29.83	19.10	-	-	-	-
GaussianFormer-2 [ 18 ]	ResNet-101	896 $\times$ 1600	30.56	20.02	-	-	-	-
QuadricFormer [ 65 ]	ResNet-101	896 $\times$ 1600	31.22	20.12	-	-	-	-
DriveTok	ResNet-101	256 $\times$ 704	32.23 ( $\downarrow~$ 3.2% )	18.83( $\downarrow~$ 6.1% )	25.94( $\downarrow~$ 6.9% )	0.698( $\downarrow~$ 6.6% )	0.10( $\downarrow~$ 25.0% )	0.88( $\downarrow~$ 5.4% )
DriveTok	DINOv3	256 $\times$ 704	33.32	20.06	27.89	0.747	0.08	0.93

Appendix

Table 8: Ablation study on backbone configuration. We compare our DriveTok (equipped with ResNet-101) against state-of-the-art methods using the same backbone. Note that DriveTok uses a significantly lower input resolution (

256\times 704

) compared to the baselines (

896\times 1600

) yet achieves competitive results. The red percentages denote the performance drop when switching from DINOv3 to ResNet-101.

Method	Backbone	IOU $\downarrow$	mIOU $\downarrow$	Latency $\downarrow$	Peak Memory $\downarrow$
GaussianFormer [ 21 ]	ResNet-101	29.83	19.10	372 ms	6229 MB
GaussianFormer-2 [ 18 ]	ResNet-101	30.56	20.02	451 ms	4535 MB
DriveTok	ResNet-101	32.23	18.83	238 ms	7803 MB

Appendix

Table 9: Latency and memory comparison with Occ models.

3D Scene Representation

3D scene representation studies how to encode geometry, appearance, semantics, and motion for reconstruction, rendering, generation, and interaction.

See all 242 papers

SceneTok: A Compressed, Diffusable Token Space for 3D Scenes
Feb 21, 2026Mohammad Asim, Christopher Wewer, Jan Eric Lenssen3D Scene Representation 3D Scene Generation
Max Planck Institute for Informatics, Saarland Informatics Campus
Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning
Jun 1, 2026Ziyang Yao, Zeyu Zhu, YunCheng Jiang +2Image Tokenization World Models
Peking University · Xiaomi EV
NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving
May 29, 2026Jiahui Li, Jiawei Sun, Zixiang Ren +7End-to-End Autonomous Driving Self-Supervised Visual Representation Learning
National University of Singapore · Black Sesame Technologies

Feb 21, 2026J/K move · Enter open · S save

SceneTok: A Compressed, Diffusable Token Space for 3D Scenes

Mohammad Asim, Christopher Wewer, Jan Eric Lenssen

Max Planck Institute for Informatics, Saarland Informatics Campus

3D Scene Representation 3D Scene Generation 3D Diffusion Models Multi-View 3D Reconstruction

Open paper AskPDF arXiv

We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.

Abstract

3D Scene Representation

Explore similar work

SceneTok: A Compressed, Diffusable Token Space for 3D Scenes

Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning

NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving