Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.
Figures & tables
Figure 1: LinkVLA achieves both higher performance and lower latency in closed-loop evaluation and equipped with superior instruction-following capability.
Figure 2: An overview of the LinkVLA architecture. The model comprises a pretrained InternViT [ 4 ] visual backbone and a Qwen2-0.5B [ 43 ] LLM. At its core, LinkVLA unifies language tokens and action tokens (for navigation points and trajectories) into a single, shared codebook. Training is driven by a unified objective for both language-action understanding and generation, ensuring deep semantic alignment. Inference employs an efficient coarse-to-fine process: first, (a) the model predicts the trajectory endpoint, which is then (b) interpolated into coarse waypoints before being (c) refined into the final, smooth trajectory.
Figure 3: Illustration of the action understanding ( Left ) and the action generation ( Right ).
Method
Closed-loop metrics ↑
Multi-Ability (%) ↑
DS
SR (%)
Efficiency
Comfort.
Merging
Overtake
Brake
Give-Way
Traffic-Sign
Mean
TCP* [ 53 ]
40.70
15.00
54.26
47.80
16.18
20.00
20.00
10.00
6.99
14.63
TCP-ctrl*
30.47
7.27
55.97
51.51
10.29
4.44
10.00
10.00
6.45
8.23
TCP-traj*
59.90
30.00
76.54
18.08
8.89
24.29
51.67
40.00
46.28
34.22
ThinkTwice* [ 18 ]
62.44
31.23
69.33
16.22
27.38
18.42
35.82
50.00
54.23
37.17
DriveAdapter* [ 17 ]
64.22
33.08
70.22
16.01
28.82
26.38
48.76
50.00
56.43
42.08
Table 1: Main results and multi-ability in Bench2Drive . * denote expert feature distillation.
ID
Method
Type
Latency ↓
Driving Score ↑
1
Orion [ 8 ]
VAE
65ms
77.74
2
Simlingo [ 37 ]
MLP
34ms
85.07
3
Ours
AR
361ms
90.66
4
Ours
C2F
48ms
91.01
Table 2: Performance and Latency Comparison. All metrics are evaluated on the CARLA benchmark. Latency is the average inference time per step, measured on H20 GPU.
Figure 4: Visualization in challenging environment with various language instructions. The generated trajectory accurately adheres to the language instruction while remaining safe and feasible within the complex environment.
ID
Token
C2F
Align.
Success Rate (%) ↑
Faster
Slower
Target Speed
Lane Change
Object
Stop
Mean
1
✗
✗
✗
81.42
61.83
66.27
75.53
74.69
60.93
70.11
2
✓
✗
✗
88.44
65.24
63.37
88.49
84.34
99.88
81.63
3
✓
✓
✗
93.16
55.86
69.24
95.45
85.38
92.14
81.87
4
✓
✓
✓
96.48
65.57
74.73
97.42
91.41
97.34
87.16
Table 3: Instruction-following evaluation on Action Dreaming dataset [ 37 ] . Align. refers to alignment with unified training.
Table 8
Method
Driving Score ↑
Success Rate(%) ↑
Ours w/o. soft label
90.85
72.73
Ours w/. soft label
91.01
74.55
Table 6: Effect of soft label in close-loop evaluation.
Method
Driving Score ↑
Success Rate(%) ↑
GPS target point
91.01
74.55
Navigation command
91.25
73.18
Table 7: Effect of different navigation forms.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Comparison of uniform and log token grids, with the corresponding waypoint distributions under each grid.
Method
Driving Score ↑
Success Rate (%) ↑
k = 5.0 (5,656 tokens)
91.01
74.55
k = 10.0 (7,245 tokens)
89.85
70.45
Appendix
Table S1: Effect of the number of action tokens (controlled by k ) in closed-loop evaluation [ 7 ] .
Method
Driving Score ↑
Success Rate (%) ↑
σ=1.2
91.01
74.55
σ=3.0
89.73
69.55
Appendix
Table S2: Effect of the spread scale parameter σ in the spatial soft-labeling in closed-loop evaluation [ 7 ] .
Figure S2 : Qualitative results of our proposed model during closed-loop evaluation in the CARLA simulator. The figure showcases representative driving scenarios, such as navigating intersections and avoiding obstacles.