End-to-end driving requires rich spatiotemporal representations to support scene understanding and trajectory planning under strict latency constraints. However, most systems initialize their visual encoders from generic image pretraining and densely process all patch tokens for each camera view and input frame, limiting both representation quality and inference efficiency. To address these challenges, we present ConDenseDrive, a knowledge-dense and token-sparse visual encoder for end-to-end autonomous driving. Our three-stage framework first makes the encoder knowledge-dense by distilling complementary visual, semantic, and geometric knowledge from three vision foundation models. It then adapts the distilled encoder to multi-frame end-to-end driving and integrates spatiotemporal context into register tokens for planning. Finally, it makes the encoder token-sparse by applying register-guided token compression, reducing redundant patch computation and yielding a 2.0× speedup while improving planning performance. ConDenseDrive achieves 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2, and reaches 37.4 HD-Score in zero-shot evaluation on HUGSIM. Extensive experiments and qualitative analyses demonstrate that visual encoder design provides an effective route to reliable and efficient end-to-end driving.
Figures & tables
Figure 1: Design focuses in end-to-end driving. The bottom row outlines an end-to-end driving pipeline. Representative methods focus on (b) scene representation ( Hu et al., 2023 ; Jiang et al., 2023 ) , (c) trajectory generation ( Chen et al., 2024 ; Liao et al., 2025 ) , and (d) trajectory selection ( Yao et al., 2026 ; Li et al., 2025b ) . ConDenseDrive instead targets (a) visual encoding through multi-teacher distillation and token compression.
Figure 2: Three-stage framework of ConDenseDrive. Multi-teacher visual distillation enriches the student encoder. Multi-frame register adaptation incorporates spatiotemporal context into register tokens for planning. Register-guided token compression then uses the resulting register-to-patch attention to retain informative patch tokens and fuse the rest.
Method
Image Encoder
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
Human driver ( Dauner et al., 2024 )
–
100
100
100
99.9
87.5
94.8
UniAD ( Hu et al., 2023 )
ResNet-50
97.8
91.9
92.9
100
78.8
83.4
TransFuser ( Chitta et al., 2022 )
ResNet-34
97.7
92.8
92.8
100
79.2
84.0
DiffusionDrive ( Liao et al., 2025 )
ResNet-34
98.2
96.2
94.7
100
82.2
88.1
Hydra-MDP ( Li et al., 2024 )
V2-99
98.4
97.8
93.9
100
86.5
90.3
GoalFlow ( Xing et al., 2025 )
V2-99
98.4
98.3
94.6
100
85.0
90.3
Table 1: Comparison on NAVSIM v1 navtest split. ConDenseDrive uses T=2 and ρ=0.4 . Bold values denote the best result among the listed learned methods.
Figure 3: Accuracy–efficiency trade-off and module-wise latency. (a) PDMS versus throughput for T=1 and T=2 . Markers follow ρ=1.0,0.9,…,0.3 from left to right, and the outlined marker denotes the selected T=2 , ρ=0.4 model. (b) End-to-end latency decomposed into perception, planning, and other costs.
Table 4: Contribution of the three-stage framework. Distill., Temporal, and Compress. denote visual distillation, temporal attention, and token compression, respectively. ID 1 uses the original pretrained DeiT-S.
Rate
Encoder GFLOPs ↓
FPS ↑
PDMS ↑
ρ=1.0
2363.62
13.6
94.3
ρ=0.9
1951.77
16.2
94.5
ρ=0.8
1637.24
18.8
94.4
ρ=0.7
1398.12
20.9
94.4
ρ=0.6
1215.52
22.7
94.4
ρ=0.5
1075.51
24.5
94.5
Table 7: Keep rate ( T=2 ).
Figure 4: PCA and depth visualizations. From left to right, columns show the input image, PCA projections of the DeiT-S student and DINOv3 ViT-L/16 teacher, and depth maps from the student and Depth Anything 3-Large teacher.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Parameter
Value
1
Number of epochs
50
Batch size
128
Learning rate
5×10−4
Weight decay
10−2
Warmup ratio
0.10
Learning rate schedule
Cosine
Appendix
Table 8: Hyperparameters for ConDenseDrive.
Method
Image Encoder
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
DiffusionDrive ( Liao et al., 2025 )
ResNet-34
97.9
94.6
93.6
100
80.7
86.0
DiffusionDrive + distilled encoder
Distilled DeiT-S
97.9
96.4
94.0
100
82.4
87.9
Appendix
Table 9: Encoder transfer to camera-only DiffusionDrive.
Frames
FPS ↑
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
T=1
25.4
99.1
98.8
96.6
100
91.2
94.1
T=2
13.6
99.2
98.9
97.0
100
91.2
94.3
T=4
7.3
99.1
98.5
96.8
100
91.1
93.9
Appendix
Table 10: Effect of the number of input frames ( ρ=1.0 ).
Attention
Frames
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
None
T=1
99.1
98.8
96.6
100
91.2
94.1
View
T=1
98.9
98.5
96.9
100
90.0
93.5
None
T=2
99.0
98.7
96.8
100
90.9
94.0
Temporal
T=2
99.2
98.9
97.0
100
91.2
94.3
View & temporal
T=2
98.9
98.7
96.2
100
90.1
93.7
Appendix
Table 11: View and temporal attention variants ( ρ=1.0 ).
Table 12
Rate
Encoder GFLOPs ↓
FPS ↑
PDMS ↑
ρ=1.0
1181.77
25.4
94.1
ρ=0.9
975.85
29.5
94.2
ρ=0.8
818.59
32.6
94.2
ρ=0.7
699.02
35.1
94.3
ρ=0.6
607.73
37.4
94.1
ρ=0.5
537.72
40.2
94.2
Appendix
Table 14: Keep rate ( T=1 ).
Figure 6: PCA and depth visualizations.
Figure 7: Progressive register-guided token compression, Scene 1. Results use ρ=0.7 .
Figure 8: Progressive register-guided token compression, Scene 2. Results use ρ=0.7 .
Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch tokens into compact scene tokens for downstream trajectory generation and scoring. While these scene tokens form a compact visual bottleneck for the planner, they receive supervision solely from the planning objective, providing limited constraints on the encoded visual information. To address this limitation, we introduce Neural Token Reconstruction (NTR), a representation learning framework to directly constrain the compact scene-token bottleneck in perception-free driving. NTR introduces a self-distillation masked latent reconstruction objective that reconstructs masked patch-level latent features using only compact scene tokens as reconstruction memory. This forces reconstruction gradients to pass exclusively through the scene-token bottleneck, encouraging scene tokens to preserve richer and less redundant visual representations for planning. We further introduce semantic priors derived from foundation-model annotations as a weak semantic interface biasing reconstruction targets toward driving-related structures without introducing explicit perception heads. All auxiliary reconstruction components are removed at inference time, leaving the deployed planner unchanged. NTR achieves state-of-the-art performance on three public autonomous driving benchmarks, including 8.0461 RFS on Waymo E2E and 94.1 PDMS / 90.9 EPDMS on NavSim1&2. The learned scene tokens exhibit lower pairwise redundancy and higher effective rank, indicating that effective bottleneck supervision improves both compact visual representation learning and planning performance.
Jiahui Li, Jiawei Sun, Zixiang Ren +7
National University of Singapore · Black Sesame Technologies
Discrete visual tokens should provide a compact representation for both token-based world modeling and planning in autonomous driving. However, most tokenizers are inherited from image generation and are optimized mainly for pixel reconstruction, which may leave a gap between what is easy to generate and what is useful to decode for driving decisions. We present a representation-guided and geometry-enhanced tokenizer that learns discrete tokens under joint supervision. The tokenizer aligns its discrete bottleneck with a frozen DINO feature space through feature decoding, while preserving appearance via RGB reconstruction with perceptual and adversarial losses. To inject geometric state-related cues, we add adjacent-frame depth and relative-pose supervision during training and stabilize joint objectives with multi-codebook quantization. We evaluate the same learned tokens with a lightweight planning readout and a GPT-style next-token world model. Experiments on NAVSIM show improved reconstruction fidelity and representation consistency, competitive planning performance under a fixed decoder, and better generative quality under matched settings.
Driving policies require capabilities of scene understanding and future evolution prediction. To achieve this goal, current end-to-end models typically construct complex perception-planning pipelines or introduce world models that explicitly predict future states, resulting in a complex system architecture. Inspired by the transferability of general-purpose visual representations, we argue that combining sufficiently strong visual representations with representation world modeling can support effective planning without relying on complex inference-time auxiliary modules. Based on this insight, we present ReDrive, an end-to-end driving framework that strengthens planning-oriented visual features via future representation prediction. To achieve this, ReDrive adopts a three-stage training pipeline consisting of driving video pretraining, joint world-modeling and planning training, and planner adaptation. This yields a strong planning-oriented representation and a high-performance planner, while requiring neither auxiliary perception modules nor future prediction at inference time. Experiments on NAVSIM demonstrate strong performance, achieving 91.0 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2. These results show that shaping representations with world modeling is sufficient to enable high-performance end-to-end planning while retaining a simple encoder-planner inference pipeline.
Yueting Zhu, Shaoyu Chen, Yuehao Song +4
Huazhong University of Science & Technology · Horizon Robotics