Organizations: College of Intelligent Robotics and Advanced Manufacturing, Fudan University, China · Institute for AI Industry Research (AIR), Tsinghua University · Institute of Automation, Chinese Academy of Sciences · Didi Chuxing
Global navigation information and local scene understanding are two crucial components of autonomous driving systems. However, our experimental results indicate that many end-to-end autonomous driving systems tend to over-rely on local scene understanding while failing to utilize global navigation information. These systems exhibit weak correlation between their planning capabilities and navigation input, and struggle to perform navigation-following in complex scenarios. To overcome this limitation, we propose the Sequential Navigation Guidance (SNG) framework, an efficient representation of global navigation information based on real-world navigation patterns. The SNG encompasses both navigation paths for constraining long-term trajectories and turn-by-turn (TBT) information for real-time decision-making logic. We constructed the SNG-QA dataset, a visual question answering (VQA) dataset based on SNG that aligns global and local planning. Additionally, we introduce an efficient model SNG-VLA that fuses local planning with global planning. The SNG-VLA achieves state-of-the-art performance through precise navigation information modeling without requiring auxiliary loss functions from perception tasks. Project page: SNG-VLA
Figures & tables
Fig. 1 : We demonstrate the impact of introducing perturbations to the driving command and the effectiveness of the SNG. The command ablation experiments comprise five distinct experimental settings, with Transfuser-command-ablation representing the statistical results across all five experiments. Detailed data are presented in Table IV . The utilization of random or erroneous navigation information demonstrates minimal impact on model performance. SNG exhibits significant performance improvement over the baseline approach.
Fig. 2 : We demonstrate erroneous examples in the annotation process of driving commands. (a) Incorrect annotation occurs in roundabouts due to longitudinal displacement. (b) The annotation information fails to comprehend lane-changing behavior, resulting in causal confusion.
Fig. 3 : Overview of our pipeline. Sequential navigation guidance is consists of navigation path and TBT information. SNG-QA comprises three subtasks: global planning, local planning, and trajectory planning. The architecture of our model is divided into two parts: the multimodal feature fusion encoder and the unified transformer backbone.
Method
Input
Navigation
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
UniAD [ 21 ]
C & L
CM
97.8
91.9
92.9
100
78.8
83.4
PARA-Drive [ 10 ]
C & L
CM
97.9
92.4
93.0
99.8
79.3
84.0
LTF [ 8 ]
C & L
CM
97.4
92.8
92.4
100
79.0
83.8
Transfuser [ 8 ]
C & L
CM
97.7
92.8
92.8
100
79.2
84.0
Transfuser †
C & L
SNG
97.8
94.5
93.5
100
80.0
85.5
DRAMA [ 34 ]
C & L
CM
98.0
93.1
94.8
100
80.1
85.5
TABLE I: Comparison on NAVSIM navtest split with closed-loop metrics . † represents the results with SNG input. CM represents Driving Command. C-single represents front view image. PDM score (PDMS) [ 9 ] is weighted aggregation of several sub-scores: no at-fault collisions (NC), drivable area compliance (DAC), time-to-collision (TTC), comfort (Comf.), and ego progress (EP).
Method
Open-loop Metric
Closed-loop Metric
Latency
Avg. L2 ↓
Driving Score ↑
Success Rate (%) ↑
Efficiency ↑
Comfortness ↑
AD-MLP [ 14 ]
3.64
18.05
0.00
48.45
22.63
3ms
UniAD-Tiny [ 21 ]
0.80
40.73
13.18
123.92
47.04
420.4ms
UniAD-Base [ 21 ]
0.73
45.81
16.36
129.21
43.58
663.4ms
VAD [ 6 ]
0.91
42.35
15.00
157.94
46.01
278.3ms
DriveTransformer-Large [ 37 ]
0.62
63.46
35.01
100.64
20.78
211.7ms
TABLE II : Open-loop and Closed-loop Results in Bench2Drive . All results are trained on the base training set. Avg. L2 is averaged over the predictions in 2 seconds under 2Hz. * denotes expert feature distillation. Latency is measured on the A6000.
Method
Ability ↑
M
O
EB
GW
TS
Mean
AD-MLP [ 14 ]
0.0
0.0
0.0
0.0
4.4
0.9
UniAD-Tiny [ 21 ]
8.9
9.3
20.0
20.0
15.4
14.7
UniAD-Base [ 21 ]
14.1
17.8
21.7
10.0
14.2
15.6
VAD [ 6 ]
8.1
24.4
18.6
20.0
19.2
18.1
DriveTransformer [ 37 ]
17.6
35.0
48.4
40.0
52.1
38.6
TABLE III: Multi-Ability Results in Bench2Drive . All results are trained on the base training set. * denotes expert feature distillation.M: Merging, O: Overtaking, EB: Emergency Brake, GW: Give Way, TS: Traffic Sign, M: Mean.
Fig. 4 : Qualitative analysis of real-world scenarios. Navigation paths are augmented with substantial noise before being fed into the model to mitigate the influence of privileged information.
Fig. 5 : We demonstrate the impact of introducing noise to the driving command on the predicted trajectory during the inference process of the Transfuser [ 8 ] . The experimental results are based on the official checkpoint [ 9 ] .
Command
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
Original
97.7
92.8
92.8
100
79.2
84.0
None
97.7
93.5
92.9
100
78.7
84.4
Random
97.8
93.4
93.3
100
79.2
84.7
Left
97.4
93.4
92.5
100
79.2
84.3
Right
97.7
93.5
93.2
100
78.9
84.4
Forward
97.7
93.2
93.1
100
78.9
84.2
TABLE IV : Command ablation results in NAVSIM . We conducted experiments on NAVSIM [ 9 ] using Transfuser [ 8 ] by modifying the input driving commands. Original: ground truth driving command; None: no driving command added; Random: random driving command; Left, Right, Forward: fixed driving commands. All results are obtained under identical training parameters. The introduction of noise into the driving command has minimal impact on the final planning performance.
ID
Navigation Path
TBT Information
Driving Command
NC ↑
DAC ↑
PDMS ↑
0
−
−
−
97.2
95.1
85.9
1
−
−
✓
97.5
95.3
86.1
2
−
✓
−
97.6
95.2
86.4
3
2×20
−
−
97.8
95.1
86.4
4
2×20
✓
−
97.5
96.1
87.6
5
4×10
−
−
97.5
96.6
87.7
TABLE V : Ablation of navigation information representation. We conduct ablation studies on the sampling interval of the navigation path, TBT information and driving commands.
Fig. 6 : Qualitative analysis of real-world scenarios.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
College of Automotive Engineering, Jilin University · The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · ReeFocus AI Technology