Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks

Satyajeet Das; Aaron Buxbaum; Niels Joubert; Gaurav S. Sukhatme

cs.ROSep 29, 2026

Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks

Authors: Satyajeet Das, Aaron Buxbaum, Niels Joubert, Gaurav S. Sukhatme

Organizations: Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA · Stack AV, Pittsburgh, PA, USA

Abstract

Class 8 trucks differ from passenger cars in geometry, dynamics, and maneuvering requirements. As a result, vision-language-action (VLA) models trained for passenger vehicles do not readily transfer to Class 8 trucks, particularly in unstructured scenarios such as accident scenes and construction zones. Rather than training a truck-driving VLA from scratch, we propose an adapt-then-steer strategy that adapts an off-the-shelf VLA to generate trajectories for Class-8 trucks in these challenging scenarios. In the adapt stage, we use NVIDIA's Alpamayo 1.5 as the base model, fine-tuning only its action-generation stack on a few hundred real-world construction and accident-related highway scenarios. In the steer stage, we introduce Flow Velocity Steering (FVS) to further refine the model's predictions while holding the adapted VLA fixed. FVS is a compact, flow-time-conditioned residual module that adds learned corrections to the action-space flow velocity used to update the action sequence at each generation step. In open-loop evaluation on a scenario-disjoint held-out set, targeted fine-tuning more than halves single-candidate average displacement error (ADE) and final displacement error (FDE) over the entire 6.4 s horizon compared to the base model. Using the same targeted demonstrations, FVS further reduces the fine-tuned model's full-horizon ADE and FDE by 13.9% and 16.5%, respectively. At matched data budgets, targeted supervision yields 19-26% lower full-horizon ADE than general truck-driving supervision, while the targeted model remains competitive with a model fine-tuned on approximately 65 times as many general truck-driving scenarios. These results support adapt-then-steer for data-efficient vehicle-domain transfer to Class 8 trucks. Our project website is available at https://truckvla.github.io.

Figures & tables

Fig. 1: Motivation and qualitative example of truck-domain adaptation. (a)-(b) Under the same reference path, a passenger car clears the work-zone closure, whereas trailer off-tracking causes the Class 8 truck’s swept envelope to enter the closure. (c)-(d) In a held-out accident-related event, off-the-shelf Alpamayo 1.5 deviates from the logged expert truck trajectory, whereas the adapted model follows it more closely. Adapted denotes Targeted Truck-SFT followed by FVS. All displayed trajectories in (c)-(d) span $6.4\,\mathrm{s}$ . The schematics in (a)-(b) are illustrative and not to scale.

Fig. 2: Overview of the proposed adapt-then-steer pipeline. Stage I fine-tunes the action-generation stack with the vision-language backbone frozen; Stage II freezes the adapted policy and trains FVS to correct its action-space flow velocity throughout the Euler rollout before fixed trajectory decoding.

	$K_{\mathrm{cand}}=1$ : single candidate				$K_{\mathrm{cand}}=16$ : oracle best-of-16
Method	ADE@1s	ADE@3s	ADE@6.4s	FDE@6.4s	minADE@1s	minADE@3s	minADE@6.4s	minFDE@6.4s
CTRV	$0.1643$	$0.7003$	$2.0430$	$5.8412$	—	—	—	—
Base	$0.1530\pm 0.0018$	$0.6836\pm 0.0440$	$2.6358\pm 0.1493$	$7.1830\pm 0.2868$	$0.0621\pm 0.0001$	$0.2556\pm 0.0082$	$0.9402\pm 0.0038$	$2.3849\pm 0.0118$
Targeted Truck-SFT	$0.0600\pm 0.0008$	$0.3120\pm 0.0081$	$1.1650\pm 0.0184$	$3.1590\pm 0.0213$	$0.0413\pm 0.0004$	$0.1855\pm 0.0029$	$0.6968\pm 0.0115$	$1.8118\pm 0.0398$
General Truck-SFT	$0.0584\pm 0.0017$	$0.3146\pm 0.0194$	$1.2868\pm 0.0049$	$3.7404\pm 0.0824$	$0.0361\pm 0.0001$	$0.1659\pm 0.0051$	$0.6634\pm 0.0044$	$1.7652\pm 0.0240$
General $\rightarrow$ Targeted SFT	$0.0580\pm 0.0017$	$0.3121\pm 0.0168$	$1.2539\pm 0.0422$	$3.6012\pm 0.0703$	$0.0371\pm 0.0008$	$0.1716\pm 0.0045$	$0.6697\pm 0.0009$	$1.7587\pm 0.0313$

TABLE I: Trajectory accuracy on the 26-scenario held-out truck long-tail test set.

K_{\mathrm{cand}}=1

evaluates one candidate, whereas

K_{\mathrm{cand}}=16

reports oracle best-of-16

\operatorname{minADE}/\operatorname{minFDE}

. Learned-model values are mean

\pm

sample standard deviation across inference-noise seeds. CTRV is deterministic and reported only for

K_{\mathrm{cand}}=1

. Errors are in metres; lower is better. Bold indicates the lowest mean in each column. Final-action residual and FVS models use the same frozen Targeted Truck-SFT checkpoint.

Method	Lateral ADE $\downarrow$	Longitudinal ADE $\downarrow$	Heading Error $\downarrow$
CTRV	$0.7958$	$1.6934$	$1.6162$
Base	$1.0204\pm 0.0638$	$2.1706\pm 0.2306$	$3.2926\pm 0.8549$
Targeted Truck-SFT	$0.4749\pm 0.0130$	$0.9323\pm 0.0197$	$0.8851\pm 0.0232$
General Truck-SFT	$0.4832\pm 0.0152$	$1.0929\pm 0.0402$	$1.1377\pm 0.0459$
General $\rightarrow$ Targeted SFT	$0.4932\pm 0.0140$	$1.0385\pm 0.0298$	$1.1192\pm 0.0553$
Final-Action Residual (MLP)	$0.4634\pm 0.0070$	$0.9115\pm 0.0115$	$0.8590\pm 0.0172$

TABLE II: Supporting

K_{\mathrm{cand}}=1

trajectory-geometry metrics at

6.4\,\mathrm{s}

on the held-out test set. Learned-model values are mean

\pm

sample standard deviation across inference-noise seeds; CTRV is deterministic. Lateral and longitudinal ADE are in metres; final heading error is in degrees. Lower is better; bold indicates the lowest mean in each column.

Fig. 3: Data efficiency of truck adaptation. ADE@ $6.4\,\mathrm{s}$ versus training-set size for targeted long-tail and general truck-driving supervision. Curves show mean performance across three independently trained subset runs at each evaluated size; shaded bands denote $\pm 1$ standard deviation. The blue star and orange diamond denote the full-data Targeted Truck-SFT (229 scenarios) and General Truck-SFT (14,891 scenarios) checkpoints, respectively, shown as separate reference points. The dashed gray line denotes the zero-shot Base model.

Fig. 4: Illustrative adapt-then-steer results on held-out truck scenarios. Columns I–IV progress chronologically, with Base Chain-of-Causation (CoC) annotations. Rows compare Base, Targeted Truck-SFT, and Transformer FVS (top to bottom) against the logged truck trajectory (yellow dashed), using $K_{\mathrm{cand}}=1$ and a shared inference-noise seed.

Explore similar work

Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10Diffusion-Based Vision-Language-Actions Autonomous Driving
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
May 22, 2026Kewei Zhang, Jin Wang, Sensen Gao +9Autonomous Driving Drives
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
Less Language, More Latents: Annotation-Efficient VLAs for Driving
Sep 23, 2026Alexey Zakharov, Kemal Oksuz, Puneet K. DokaniaDiffusion-Based Vision-Language-Actions Latent Action Models
Robert Bosch GmbH, Germany · Five AI Ltd., United Kingdom

Jul 2, 2026, cs.CVJ/K move · Enter open · S save

Teaching Vision-Language-Action Models What to See and Where to Look

Yuguang Yang, Canyu Chen, Zhewen Tan, Yizhi Wang+9

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University+5

Diffusion-Based Vision-Language-Actions Autonomous Driving Drives Computer Vision Dataset Distillation

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Abstract

Explore similar work

Teaching Vision-Language-Action Models What to See and Where to Look

Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

Less Language, More Latents: Annotation-Efficient VLAs for Driving