AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding

Yuqi Ye; Caigao Jiang; Zhuonan Wang; Congyuan Pei; Chao Qu; Changhao Peng; Junhong Lin; Qize Yu

cs.CVOct 12, 2026

AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding

Authors: Yuqi Ye, Caigao Jiang, Zhuonan Wang, Congyuan Pei, Chao Qu, Changhao Peng, Junhong Lin, Qize Yu, +7 more

Organizations: Peking University · XPeng Motors · Fudan University

Abstract

Vision-language-action (VLA) models show strong potential for end-to-end autonomous driving by leveraging world knowledge and understanding to excel in long-tail scenarios. However, current VLAs have two limitations: 1) Degeneration into vision-action (VA) models. For low latency, regression/flow-based methods directly generate actions but underuse VLM understanding, causing unsafe, unexplainable actions. 2) Autoregressive high latency. Autoregressive methods sequentially generate chain-of-thought (CoT) and action tokens, fully leveraging VLM understanding, but high latency hinders deployment. To enable low-latency deployment while using CoT to assist action generation, we propose AUGUstus, an Autonomous-driving VLA model with Unified Generation and Understanding. We unify diffusion-based action generation and autoregressive CoT understanding in one model, where understanding only serves generation. Joint CoT supervision preserves VLM spatial understanding in shared representation for safer, more accurate trajectories. AUGUstus-mode orders action generation before CoT, so CoT supervises reliable actions during training while inference needs only action output, enabling low latency. AUGUstus-GRPO, a unified GRPO algorithm, jointly optimizes action and CoT, improving action performance. Experiments show AUGUstus achieves 94.6 PDMS on NAVSIMv1 and 90.7 EPDMS on NAVSIMv2, SOTA planning without scorer module. On nuScenes, it achieves average L2 error 0.27 m and average collision rate 0.05%, best among compared methods. To validate AUGUstus practicality, we train on in-house data with 512 NVIDIA H800 GPUs for ~2 days, evaluate on an internal 1,000 km dataset, and preliminarily deploy on a real vehicle, achieving 12 Hz operation. Project page: https://augustus-ad.github.io

Figures & tables

Figure 1: The difference between AUGUstus and other autonomous-driving VLA paradigms. Our method unifies generation and understanding under AUGUstus-mode, so that joint CoT supervision preserves the VLM’s understanding in the shared representation, yielding safer, more accurate actions with an RL-friendly action distribution and low-latency deployment.

Figure 2: Overview of AUGUstus. AUGUstus takes video and ego status as input and uses one shared transformer to first generate clean action and then understand a CoT containing key-object bounding boxes and future position. Action tokens use bidirectional attention and cannot attend to the CoT tokens. At inference, AUGUstus directly outputs the action to ensure low latency.

Figure 3

Figure 5

End-to-end small models
Method	Ref	RL	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	PDMS $\uparrow$
Human	-	✗	100.0	100.0	87.5	100.0	100.0	94.8
TransFuser ( Chitta et al., 2022 )	TPAMI’23	✗	97.8	92.6	78.9	92.9	99.9	83.9
PARA-Drive ( Weng et al., 2024 )	CVPR’24	✗	97.9	92.4	79.3	93.0	99.8	84.0
Hydra-MDP ( Li et al., 2024 )	CVPR’24	✗	98.3	96.0	78.7	94.6	100.0	86.5
WoTE ( Guan et al., 2025 )	ICCV’25	✗	98.5	96.8	81.9	94.9	100.0	88.3

Table 1: Performance comparison on NAVSIMv1 benchmark.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$ False / True
Human	100.0	100.0	99.8	100.0	87.4	100.0	100.0	98.1	90.1	90.3 / 94.5
Ego Status MLP	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0 / –
Transfuser ( Chitta et al., 2022 )	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7 / 84.0
Hydra-MDP++ ( Li et al., 2024 )	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4 / –
DiffusionDrive ( Liao et al., 2025 )	98.2	96.2	99.5	99.8	87.4	97.3	96.9	98.4	87.7	84.7 / 88.2
WoTE ( Guan et al., 2025 )	98.5	96.8	98.8	99.8	86.1	97.9	95.5	98.3	82.9	84.2 / 87.7

Table 2: Performance comparison on NAVSIMv2 benchmark.

Method	Ref	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Method	Ref	1s	2s	3s	Avg.	1s	2s	3s	Avg.
ST-P3 ( Hu et al., 2022 )	ECCV’22	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
VAD ( Jiang et al., 2023 )	ICCV’23	0.17	0.34	0.60	0.37	0.07	0.10	0.24	0.14
UniAD ( Hu et al., 2023 )	CVPR’23	0.44	0.67	0.96	0.69	0.04	0.08	0.23	0.12
GPT-Driver ( Mao et al., 2023 )	arXiv’23	0.20	0.40	0.70	0.44	0.04	0.12	0.36	0.17
DriveVLM ( Tian et al., 2024 )	CoRL’25	0.18	0.34	0.68	0.40	0.10	0.22	0.45	0.27

Table 3: Performance comparison on nuScenes benchmark.

Table 9

Setting	PDMS $\uparrow$	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	ADE (m) $\downarrow$	Time (ms) $\downarrow$
No CoT, Only Action	89.2	98.8	96.8	98.4	83.5	95.8	100.0	0.530	73.60
CoT first, Action second	90.6	99.1	97.8	98.6	84.2	96.6	100.0	0.521	4587
AUGUstus-mode (Ours)	90.6	99.0	97.9	98.5	84.2	96.5	100.0	0.520	73.60

Table 6: Ablation Study of CoT and AUGUstus-mode.

Setting	PDMS $\uparrow$	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$
SFT with AUGUstus-mode	90.6	99.0	97.9	84.2	96.4	100.0
+ AUGUstus-GRPO Stage I	94.0	98.6	98.7	91.4	96.6	100.0
+ AUGUstus-GRPO Stage II	94.6	98.9	98.9	91.9	97.1	100.0
+ JaT-GRPO, without CoT-GRPO	92.6	98.0	97.6	91.8	94.1	100.0

Table 7: Ablation Study of AUGUstus-GRPO.

Figure 6: CoT attention visualization. CoT supervision anchors the action-relevant attention on the nearby risk object (46% in-box), whereas the no-CoT model misses it (0%).

Panel of Figure 7 — Figure 7: Visualization. Left: Trajectory visualization with and without CoT. Right: Trajectory visualization compared with other methods.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Detailed AUGUstus module. Left: Action Projector. A normalized noisy trajectory

x_{t}\in\mathbb{R}^{8\times 3}

and a sinusoidal time embedding are projected, concatenated, and passed through Linear–SiLU layers to form

e_{\mathrm{act}}\in\mathbb{R}^{8\times D}

. Center: shared decoder. Visual tokens

V

, input text

L

, noisy action tokens

A

, and CoT text share the transformer; the action block is bidirectional, and the segment-specific time modulation uses

t{=}0

for context/CoT and the sampled

t

for action tokens. Each layer contains attention, MLP, residual additions, and AdaRMSNorm. The inset expands the adaptive scale/shift

s,b

and the pretrained RMSNorm weight

w

. Right: Action Decoder. LayerNorm–Linear–SiLU–LayerNorm–Linear decoding maps

h_{\mathrm{act}}\in\mathbb{R}^{8\times D}

\hat{x}_{0}\in\mathbb{R}^{8\times 3}

	Flow-GRPO ( Liu et al., 2026a )	DanceGRPO ( Xue et al., 2025 )	JaT-GRPO (ours)
$g(t)^{2}$	$a^{2}\tfrac{t}{1-t}$	$\eta^{2}$	$\lambda t$
$c_{x}$	$1-\frac{\Delta t}{t}\big(1+\frac{a^{2}}{2(1-t)}\big)$	$1-\frac{\Delta t}{t}\big(1+\frac{\eta^{2}}{2t}\big)$	$1-\frac{\Delta t}{t}\big(1+\frac{\lambda}{2}\big)$
$c_{0}$	$\frac{\Delta t}{t}\big(1+\frac{a^{2}}{2}\big)$	$\frac{\Delta t}{t}\big(1+\frac{\eta^{2}(1-t)}{2t}\big)$	$\frac{\Delta t}{t}\big(1+\frac{\lambda(1-t)}{2}\big)$
$\sigma_{s}$	$a\sqrt{\frac{t\,\Delta t}{1-t}}$	$\eta\sqrt{\Delta t}$	$\sqrt{\lambda t\,\Delta t}$
$t{=}1,\ \Delta t{=}1$	$c_{x}\!\to\!-\infty$ , $\sigma_{s}\!\to\!\infty$ †	$c_{x}{=}-\frac{\eta^{2}}{2},\ c_{0}{=}1,\ \sigma_{s}{=}\eta$	$c_{x}{=}-\frac{\lambda}{2},\ c_{0}{=}1,\ \sigma_{s}{=}\sqrt{\lambda}$
single-step policy	undefined	$\mathcal{N}(\hat{x}_{0}-\frac{\eta^{2}}{2}x_{1},\eta^{2}I)$	$\mathcal{N}(\hat{x}_{0}-\frac{\lambda}{2}x_{1},\lambda I)$

Appendix

Table 8: Kernel coefficients per schedule (Eq. ( 8 ); one reverse step

t\to t-\Delta t

). The first row names each method and the second row gives its diffusion schedule

g(t)^{2}

; the remaining rows give the induced kernel coefficients and endpoint behavior.

\dagger

Flow-GRPO implementations substitute a clamp

\sigma_{\max}

for

\sigma_{s}

near

t{=}1

CoT type	Template	PDMS $\uparrow$	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	ADE $\downarrow$	Time (ms)
No CoT	action-only training	89.2	98.8	96.8	98.4	83.5	95.8	100.0	0.530
Planning	slow, keep lane, pass after crossing	90.1	98.9	97.4	98.5	83.8	96.1	100.0	0.522
Reasoning	perception + future bins + maneuver	89.6	98.7	96.9	98.4	83.4	96.1	100.0	0.534
Perception	car [x1,y1,x2,y2]; ped. [x1,y1,x2,y2]	90.3	98.9	97.5	98.6	84.3	96.1	100.0	0.517	73.60
Perception + Prediction	boxes + future bins, no maneuver sentence	90.3	98.9	97.6	98.6	84.1	96.3	100.0	0.513
Short-prediction	atomic vocabulary encoding	89.1	98.7	96.7	98.3	83.1	95.4	100.0	0.559

Appendix

Table 9: CoT content ablation (complete metrics and supervision formats). All variants share the Section 4.1 protocol and are evaluated with the deterministic one-step evaluator on the full navtest split; only the CoT content changes. PDMS is reported together with its full sub-metric breakdown: NC, DAC, DDC, EP, TTC, Comfort, and ADE. NC: no-collision, DAC: drivable-area compliance, DDC: driving-direction compliance, EP: ego progress, TTC: time-to-collision, Comfort: comfort, ADE: average L2 displacement. The default prediction CoT (risk-object boxes + future positions) is bolded ; the last row is the CoT-first ordering control (identical recipe and prediction content, reverse token order). Inference time is reported once for the shared 73.60 ms action-only path and separately for the CoT-first path.

Figure 9: The three CoT contents compared on one NAVSIM right-turn scene (ground-truth annotations). The three rows are three alternative supervision targets, not a serial pipeline: each is trained alone on the same scene and action, and the model never consumes one row’s output to produce another’s. One pastel outer frame encloses the title, annotated scene, and JSON answer of each CoT variant. (a) Perception CoT : a 2-D box and a class label for every salient agent (vehicle: blue, bicycle: orange, pedestrian: violet) on the $2048\times 512$ deployment canvas. (b) Prediction CoT : each box is annotated with its future relative position and a motion arrow; this is the default target. (c) Planning CoT : the ground-truth $4$ s trajectory (green) projected onto the camera strip, with a BEV inset (ego-left on the left) showing the top-down right-turn arc.

Panel of Figure 13 — Figure 13: Three-stage RL visualization. Each case shows the same scene under SFT, RL Stage I, and RL Stage II. Stage I moves the path toward the desired progress profile; Stage II retains that progress while increasing the safety margin. The displayed case-level PDMS values are illustrative qualitative traces, whereas the aggregate values are in Table 7 .

Path or module	Mean (ms)	Fraction
Vision encoder and M-RoPE	40.20	54.6%
Context prefix, eager	46.04	34.8%
One-step denoising, eager	45.70	34.6%
LLM stage, CUDA-graph replay	33.19	45.1%
End-to-end, eager	131.88	–
End-to-end, CUDA graph	73.67	13.6 Hz

Appendix

Table 10: Measured batch-one latency on one H800.

Panel of Figure 14 — Figure 14: Public-set qualitative comparison (part 1/2). Each panel contrasts ReCogDrive, AutoDrive-P 3 , and AUGUstus on the same public scene. The single AUGUstus trajectory is both collision-free and route compliant in these examples.

Panel of Figure 15 — Figure 15: Public-set qualitative comparison (part 2/2). Three additional public scenes comparing ReCogDrive, AutoDrive-P 3 , and AUGUstus under the same visualization protocol.

Panel of Figure 16 — Figure 16: Internal 1,000 km qualitative examples. Representative long-tail interactions and camera conditions from the private evaluation set.

Panel of Figure 17 — Figure 17: Internal 1,000 km qualitative examples. Representative long-tail interactions and camera conditions from the private evaluation set. (part 2/2).

Panel of Figure 18 — Figure 18: Internal qualitative comparison of VLM+regression, VLM+flow matching, and AUGUstus (part 1/2).

Panel of Figure 19 — Figure 19: Internal qualitative comparison of VLM+regression, VLM+flow matching, and AUGUstus (part 2/2).

Explore similar work

Unifying Language-Action Understanding and Generation for Autonomous Driving
Mar 2, 2026Xinyang Wang, Qian Liu, Wenjie Ding +7Vision-Language-Action Models End-to-End Autonomous Driving
State Key Lab of CAD&CG, Zhejiang University · Li Auto
Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous Driving VLM Distillation
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
WCog-VLA: A Dual-Level World-Cognitive Vision-Language-Action Model for End-to-End Autonomous Driving
Jul 9, 2026Xuerun Yan, Zhexi Lian, Nuoheng Zhang +5End-to-End Autonomous Driving World Models
Tongji University, China · Nanyang Technological University, Singapore

Mar 2, 2026J/K move · Enter open · S save

Unifying Language-Action Understanding and Generation for Autonomous Driving

Xinyang Wang, Qian Liu, Wenjie Ding, Zhao Yang+6

State Key Lab of CAD&CG, Zhejiang University · Li Auto

Vision-Language-Action Models End-to-End Autonomous Driving Efficient VLA Model Inference

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.

Abstract

Explore similar work

Unifying Language-Action Understanding and Generation for Autonomous Driving

Teaching Vision-Language-Action Models What to See and Where to Look

WCog-VLA: A Dual-Level World-Cognitive Vision-Language-Action Model for End-to-End Autonomous Driving