Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks

Satyajeet Das; Aaron Buxbaum; Niels Joubert; Gaurav S. Sukhatme

cs.ROSep 29, 2026

Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks

Authors: Satyajeet Das, Aaron Buxbaum, Niels Joubert, Gaurav S. Sukhatme

Organizations: Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA · Stack AV, Pittsburgh, PA, USA

Abstract

Class 8 trucks differ from passenger cars in geometry, dynamics, and maneuvering requirements. As a result, vision-language-action (VLA) models trained for passenger vehicles do not readily transfer to Class 8 trucks, particularly in unstructured scenarios such as accident scenes and construction zones. Rather than training a truck-driving VLA from scratch, we propose an adapt-then-steer strategy that adapts an off-the-shelf VLA to generate trajectories for Class-8 trucks in these challenging scenarios. In the adapt stage, we use NVIDIA's Alpamayo 1.5 as the base model, fine-tuning only its action-generation stack on a few hundred real-world construction and accident-related highway scenarios. In the steer stage, we introduce Flow Velocity Steering (FVS) to further refine the model's predictions while holding the adapted VLA fixed. FVS is a compact, flow-time-conditioned residual module that adds learned corrections to the action-space flow velocity used to update the action sequence at each generation step. In open-loop evaluation on a scenario-disjoint held-out set, targeted fine-tuning more than halves single-candidate average displacement error (ADE) and final displacement error (FDE) over the entire 6.4 s horizon compared to the base model. Using the same targeted demonstrations, FVS further reduces the fine-tuned model's full-horizon ADE and FDE by 13.9% and 16.5%, respectively. At matched data budgets, targeted supervision yields 19-26% lower full-horizon ADE than general truck-driving supervision, while the targeted model remains competitive with a model fine-tuned on approximately 65 times as many general truck-driving scenarios. These results support adapt-then-steer for data-efficient vehicle-domain transfer to Class 8 trucks. Our project website is available at https://truckvla.github.io.

Figures & tables

Fig. 1: Motivation and qualitative example of truck-domain adaptation. (a)-(b) Under the same reference path, a passenger car clears the work-zone closure, whereas trailer off-tracking causes the Class 8 truck’s swept envelope to enter the closure. (c)-(d) In a held-out accident-related event, off-the-shelf Alpamayo 1.5 deviates from the logged expert truck trajectory, whereas the adapted model follows it more closely. Adapted denotes Targeted Truck-SFT followed by FVS. All displayed trajectories in (c)-(d) span $6.4\,\mathrm{s}$ . The schematics in (a)-(b) are illustrative and not to scale.

Fig. 2: Overview of the proposed adapt-then-steer pipeline. Stage I fine-tunes the action-generation stack with the vision-language backbone frozen; Stage II freezes the adapted policy and trains FVS to correct its action-space flow velocity throughout the Euler rollout before fixed trajectory decoding.

	$K_{\mathrm{cand}}=1$ : single candidate				$K_{\mathrm{cand}}=16$ : oracle best-of-16
Method	ADE@1s	ADE@3s	ADE@6.4s	FDE@6.4s	minADE@1s	minADE@3s	minADE@6.4s	minFDE@6.4s
CTRV	$0.1643$	$0.7003$	$2.0430$	$5.8412$	—	—	—	—
Base	$0.1530\pm 0.0018$	$0.6836\pm 0.0440$	$2.6358\pm 0.1493$	$7.1830\pm 0.2868$	$0.0621\pm 0.0001$	$0.2556\pm 0.0082$	$0.9402\pm 0.0038$	$2.3849\pm 0.0118$
Targeted Truck-SFT	$0.0600\pm 0.0008$	$0.3120\pm 0.0081$	$1.1650\pm 0.0184$	$3.1590\pm 0.0213$	$0.0413\pm 0.0004$	$0.1855\pm 0.0029$	$0.6968\pm 0.0115$	$1.8118\pm 0.0398$
General Truck-SFT	$0.0584\pm 0.0017$	$0.3146\pm 0.0194$	$1.2868\pm 0.0049$	$3.7404\pm 0.0824$	$0.0361\pm 0.0001$	$0.1659\pm 0.0051$	$0.6634\pm 0.0044$	$1.7652\pm 0.0240$
General $\rightarrow$ Targeted SFT	$0.0580\pm 0.0017$	$0.3121\pm 0.0168$	$1.2539\pm 0.0422$	$3.6012\pm 0.0703$	$0.0371\pm 0.0008$	$0.1716\pm 0.0045$	$0.6697\pm 0.0009$	$1.7587\pm 0.0313$

TABLE I: Trajectory accuracy on the 26-scenario held-out truck long-tail test set.

K_{\mathrm{cand}}=1

evaluates one candidate, whereas

K_{\mathrm{cand}}=16

reports oracle best-of-16

\operatorname{minADE}/\operatorname{minFDE}

. Learned-model values are mean

\pm

sample standard deviation across inference-noise seeds. CTRV is deterministic and reported only for

K_{\mathrm{cand}}=1

. Errors are in metres; lower is better. Bold indicates the lowest mean in each column. Final-action residual and FVS models use the same frozen Targeted Truck-SFT checkpoint.

Method	Lateral ADE $\downarrow$	Longitudinal ADE $\downarrow$	Heading Error $\downarrow$
CTRV	$0.7958$	$1.6934$	$1.6162$
Base	$1.0204\pm 0.0638$	$2.1706\pm 0.2306$	$3.2926\pm 0.8549$
Targeted Truck-SFT	$0.4749\pm 0.0130$	$0.9323\pm 0.0197$	$0.8851\pm 0.0232$
General Truck-SFT	$0.4832\pm 0.0152$	$1.0929\pm 0.0402$	$1.1377\pm 0.0459$
General $\rightarrow$ Targeted SFT	$0.4932\pm 0.0140$	$1.0385\pm 0.0298$	$1.1192\pm 0.0553$
Final-Action Residual (MLP)	$0.4634\pm 0.0070$	$0.9115\pm 0.0115$	$0.8590\pm 0.0172$

TABLE II: Supporting

K_{\mathrm{cand}}=1

trajectory-geometry metrics at

6.4\,\mathrm{s}

on the held-out test set. Learned-model values are mean

\pm

sample standard deviation across inference-noise seeds; CTRV is deterministic. Lateral and longitudinal ADE are in metres; final heading error is in degrees. Lower is better; bold indicates the lowest mean in each column.

Fig. 3: Data efficiency of truck adaptation. ADE@ $6.4\,\mathrm{s}$ versus training-set size for targeted long-tail and general truck-driving supervision. Curves show mean performance across three independently trained subset runs at each evaluated size; shaded bands denote $\pm 1$ standard deviation. The blue star and orange diamond denote the full-data Targeted Truck-SFT (229 scenarios) and General Truck-SFT (14,891 scenarios) checkpoints, respectively, shown as separate reference points. The dashed gray line denotes the zero-shot Base model.

Fig. 4: Illustrative adapt-then-steer results on held-out truck scenarios. Columns I–IV progress chronologically, with Base Chain-of-Causation (CoC) annotations. Rows compare Base, Targeted Truck-SFT, and Transformer FVS (top to bottom) against the logged truck trajectory (yellow dashed), using $K_{\mathrm{cand}}=1$ and a shared inference-noise seed.

Explore similar work

Jul 2, 2026cs.CV

Teaching Vision-Language-Action Models What to See and Where to Look

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Yuguang Yang, Canyu Chen, Zhewen Tan +10

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5

Diffusion-Based Vision-Language-Actions Autonomous Driving Drives Computer Vision Dataset Distillation

Ask PDF

May 22, 2026cs.CL

Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking

N

stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to

0.32

m (a

22\%

improvement). When integrated with SGLang, our framework delivers

12\times

throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.

Kewei Zhang, Jin Wang, Sensen Gao +9

1Peking University · 3The University of Hong Kong · 2NVIDIA +1

Autonomous Driving Drives Diffusion-Based Vision-Language-Actions Robust Trajectory

Ask PDF

Sep 23, 2026cs.RO

Less Language, More Latents: Annotation-Efficient VLAs for Driving

Vision-language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-language instructions: while camera streams and expert trajectories are logged at scale, language annotations (e.g., turn left at the intersection) remain scarce and expensive to acquire. To address this challenge, we introduce Latent Action Driving Annotations (LADA), a three-stage pipeline that transforms abundant unlabelled observation-trajectory pairs into a substrate for language-conditioned control. First, we train a latent action model with a vector-quantised bottleneck, producing a compact codebook of high-level vehicle intents. Second, a small language-annotated subset is used to train a vision-language translator to map observations and language instructions into this codebook. Third, we train a driving VLA on observation-latent-action pairs over the full unlabelled corpus. Using fewer than 5% of language annotations and without leveraging any auxiliary chain-of-thought reasoning or visual question answering streams, LADA achieves a Driving Score of 87.98 and a Success Rate of 70.46% on the closed-loop Bench2Drive benchmark, matching or surpassing fully supervised baselines.

Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania

Robert Bosch GmbH, Germany · Five AI Ltd., United Kingdom

Diffusion-Based Vision-Language-Actions Latent Action Models High-Level Source Code

Ask PDF