ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving

Shengzhen Li; Yongkang Li; Hao Gao; Chenxi Wu; Yajing Sun; Lu Hou; Wenyu Liu; Xinggang Wang

cs.CVOct 12, 2026

ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving

Authors: Shengzhen Li, Yongkang Li, Hao Gao, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang

Organizations: Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.

Abstract

End-to-end driving requires rich spatiotemporal representations to support scene understanding and trajectory planning under strict latency constraints. However, most systems initialize their visual encoders from generic image pretraining and densely process all patch tokens for each camera view and input frame, limiting both representation quality and inference efficiency. To address these challenges, we present ConDenseDrive, a knowledge-dense and token-sparse visual encoder for end-to-end autonomous driving. Our three-stage framework first makes the encoder knowledge-dense by distilling complementary visual, semantic, and geometric knowledge from three vision foundation models. It then adapts the distilled encoder to multi-frame end-to-end driving and integrates spatiotemporal context into register tokens for planning. Finally, it makes the encoder token-sparse by applying register-guided token compression, reducing redundant patch computation and yielding a 2.0 $\times$ speedup while improving planning performance. ConDenseDrive achieves 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2, and reaches 37.4 HD-Score in zero-shot evaluation on HUGSIM. Extensive experiments and qualitative analyses demonstrate that visual encoder design provides an effective route to reliable and efficient end-to-end driving.

Figures & tables

Figure 1: Design focuses in end-to-end driving. The bottom row outlines an end-to-end driving pipeline. Representative methods focus on (b) scene representation ( Hu et al., 2023 ; Jiang et al., 2023 ) , (c) trajectory generation ( Chen et al., 2024 ; Liao et al., 2025 ) , and (d) trajectory selection ( Yao et al., 2026 ; Li et al., 2025b ) . ConDenseDrive instead targets (a) visual encoding through multi-teacher distillation and token compression.

Figure 2: Three-stage framework of ConDenseDrive. Multi-teacher visual distillation enriches the student encoder. Multi-frame register adaptation incorporates spatiotemporal context into register tokens for planning. Register-guided token compression then uses the resulting register-to-patch attention to retain informative patch tokens and fuse the rest.

Method	Image Encoder	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Human driver ( Dauner et al., 2024 )	–	100	100	100	99.9	87.5	94.8
UniAD ( Hu et al., 2023 )	ResNet-50	97.8	91.9	92.9	100	78.8	83.4
TransFuser ( Chitta et al., 2022 )	ResNet-34	97.7	92.8	92.8	100	79.2	84.0
DiffusionDrive ( Liao et al., 2025 )	ResNet-34	98.2	96.2	94.7	100	82.2	88.1
Hydra-MDP ( Li et al., 2024 )	V2-99	98.4	97.8	93.9	100	86.5	90.3
GoalFlow ( Xing et al., 2025 )	V2-99	98.4	98.3	94.6	100	85.0	90.3

Table 1: Comparison on NAVSIM v1 navtest split. ConDenseDrive uses

T=2

and

\rho=0.4

. Bold values denote the best result among the listed learned methods.

Panel of Figure 3 — Figure 3: Accuracy–efficiency trade-off and module-wise latency. (a) PDMS versus throughput for $T=1$ and $T=2$ . Markers follow $\rho=1.0,0.9,\ldots,0.3$ from left to right, and the outlined marker denotes the selected $T=2$ , $\rho=0.4$ model. (b) End-to-end latency decomposed into perception, planning, and other costs.

Table 4: Contribution of the three-stage framework. Distill., Temporal, and Compress. denote visual distillation, temporal attention, and token compression, respectively. ID 1 uses the original pretrained DeiT-S.

Rate	Encoder GFLOPs $\downarrow$	FPS $\uparrow$	PDMS $\uparrow$
$\rho=1.0$	2363.62	13.6	94.3
$\rho=0.9$	1951.77	16.2	94.5
$\rho=0.8$	1637.24	18.8	94.4
$\rho=0.7$	1398.12	20.9	94.4
$\rho=0.6$	1215.52	22.7	94.4
$\rho=0.5$	1075.51	24.5	94.5

Table 7: Keep rate (

T=2

Panel of Figure 4 — Figure 4: PCA and depth visualizations. From left to right, columns show the input image, PCA projections of the DeiT-S student and DINOv3 ViT-L/16 teacher, and depth maps from the student and Depth Anything 3-Large teacher.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Parameter	Value
1	Number of epochs	50
	Batch size	128
	Learning rate	$5\times 10^{-4}$
	Weight decay	$10^{-2}$
	Warmup ratio	0.10
	Learning rate schedule	Cosine

Appendix

Table 8: Hyperparameters for ConDenseDrive.

Method	Image Encoder	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
DiffusionDrive ( Liao et al., 2025 )	ResNet-34	97.9	94.6	93.6	100	80.7	86.0
DiffusionDrive + distilled encoder	Distilled DeiT-S	97.9	96.4	94.0	100	82.4	87.9

Appendix

Table 9: Encoder transfer to camera-only DiffusionDrive.

Frames	FPS $\uparrow$	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
$T=1$	25.4	99.1	98.8	96.6	100	91.2	94.1
$T=2$	13.6	99.2	98.9	97.0	100	91.2	94.3
$T=4$	7.3	99.1	98.5	96.8	100	91.1	93.9

Appendix

Table 10: Effect of the number of input frames (

\rho=1.0

Attention	Frames	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
None	$T=1$	99.1	98.8	96.6	100	91.2	94.1
View	$T=1$	98.9	98.5	96.9	100	90.0	93.5
None	$T=2$	99.0	98.7	96.8	100	90.9	94.0
Temporal	$T=2$	99.2	98.9	97.0	100	91.2	94.3
View & temporal	$T=2$	98.9	98.7	96.2	100	90.1	93.7

Appendix

Table 11: View and temporal attention variants (

\rho=1.0

Table 12

Rate	Encoder GFLOPs $\downarrow$	FPS $\uparrow$	PDMS $\uparrow$
$\rho=1.0$	1181.77	25.4	94.1
$\rho=0.9$	975.85	29.5	94.2
$\rho=0.8$	818.59	32.6	94.2
$\rho=0.7$	699.02	35.1	94.3
$\rho=0.6$	607.73	37.4	94.1
$\rho=0.5$	537.72	40.2	94.2

Appendix

Table 14: Keep rate (

T=1

Figure 7: Progressive register-guided token compression, Scene 1. Results use $\rho=0.7$ .

Figure 8: Progressive register-guided token compression, Scene 2. Results use $\rho=0.7$ .

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 222 papers

NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving
May 29, 2026Jiahui Li, Jiawei Sun, Zixiang Ren +7End-to-End Autonomous Driving Self-Supervised Visual Representation Learning
National University of Singapore · Black Sesame Technologies
Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning
Jun 1, 2026Ziyang Yao, Zeyu Zhu, YunCheng Jiang +2Image Tokenization World Models
Peking University · Xiaomi EV
ReDrive: Shaping Representations with World Modeling for End-to-End Driving
Sep 27, 2026Yueting Zhu, Shaoyu Chen, Yuehao Song +4World Model Learning End-to-End Autonomous Driving
Huazhong University of Science & Technology · Horizon Robotics

May 29, 2026J/K move · Enter open · S save

NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving

Jiahui Li, Jiawei Sun, Zixiang Ren, Ming Liu+6

National University of Singapore · Black Sesame Technologies

End-to-End Autonomous Driving Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch tokens into compact scene tokens for downstream trajectory generation and scoring. While these scene tokens form a compact visual bottleneck for the planner, they receive supervision solely from the planning objective, providing limited constraints on the encoded visual information. To address this limitation, we introduce Neural Token Reconstruction (NTR), a representation learning framework to directly constrain the compact scene-token bottleneck in perception-free driving. NTR introduces a self-distillation masked latent reconstruction objective that reconstructs masked patch-level latent features using only compact scene tokens as reconstruction memory. This forces reconstruction gradients to pass exclusively through the scene-token bottleneck, encouraging scene tokens to preserve richer and less redundant visual representations for planning. We further introduce semantic priors derived from foundation-model annotations as a weak semantic interface biasing reconstruction targets toward driving-related structures without introducing explicit perception heads. All auxiliary reconstruction components are removed at inference time, leaving the deployed planner unchanged. NTR achieves state-of-the-art performance on three public autonomous driving benchmarks, including 8.0461 RFS on Waymo E2E and 94.1 PDMS / 90.9 EPDMS on NavSim1&2. The learned scene tokens exhibit lower pairwise redundancy and higher effective rank, indicating that effective bottleneck supervision improves both compact visual representation learning and planning performance.

Abstract

End-to-End Autonomous Driving

Explore similar work

NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving

Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning

ReDrive: Shaping Representations with World Modeling for End-to-End Driving