FastRoverVLA: Fast Vision-Language-Action Execution for Off-Road Autonomous Driving
Authors: Changtao Zhao, Yingxue Guan, Li Wang, Shijie Zhao, Yuhang Zhang, Shengkai Zhang, Qing Wang, Ziying Song
Organizations: School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China · Chongqing Innovation Center, Beijing Institute of Technology, Chongqing 401120, China · School of Mechanical Engineering and Intelligent Equipment, Chang’an University · School of Electronic and Information Engineering, Beijing Jiaotong University · School of Artificial Intelligence (School of Software), Yanshan University
Off-road vision-language-action (VLA) models generate scene-conditioned plans, but frequent trajectory updates can repeatedly invoke expensive multimodal inference. Reusing a prediction reduces this demand while leaving its commands exposed to changes in vehicle motion. FastRoverVLA addresses this mismatch by decoupling visual plan generation from motion-conditioned execution. Its training-free interface maintains decoded plans between visual predictions. A speed-curvature action buffer is aligned with measured motion at each update, integrated into a local trajectory, and advanced by consuming the current action. The corrected remainder is written back to the buffer, preserving motion corrections across successive updates. A bounded refresh rule normalizes speed and curvature variations within a newly decoded plan and assigns a short or long reuse budget according to the resulting plan-roughness score. The interface requires neither backbone features nor an additional learned driving policy. Over a 3 s horizon, independent real-request evaluation achieves average displacement error (ADE) and final displacement error (FDE) of 0.328/0.596 m on ORAD-3D and 0.266/0.526 m on RELLIS-3D, reducing FDE by 18.0% and 36.7% at request rates of 6.0% and 4.8%, respectively. These runs retain lower errors across four backbones and both datasets and reduce Qwen3-VL request-processing cost by factors of 16.6 and 20.8 under serial replay, while mechanism and motion-only controls identify persistent state correction as the main source of the observed accuracy benefit. The source code and evaluation materials are publicly available at https://github.com/ADEPT-ORG/FastRoverVLA.
Figures & tables
Figure 1: Motivation of FastRoverVLA. (a) Frame-synchronous prediction. (b) State mismatch during direct reuse. (c) Recurrent state alignment and write-back between plan refreshes. The separated paths depict distinct planning and trajectory-update rates; the arrows indicate scheduling dependencies.
Figure 2: FastRoverVLA framework. A frozen VLM initializes a speed–curvature buffer from images, ego motion, and an instruction. Consume–align–write-back updates produce integrated waypoints between budgeted refreshes (Section 3.3 ). The control panel denotes a potential downstream interface; evaluation uses waypoints and logged states, without a vehicle controller.
Figure 3: Plan-variation-based bounded refresh. Speed and curvature variations are normalized and aggregated into the plan-roughness score zt . The threshold τ selects a short reuse budget Ks for higher variation and a long reuse budget Kl otherwise.
Method
L 2 error ↓
ADE ↓
RMSE ↓
P95FDE ↓
1 s
2 s
3 s
OFF-EMMA [ 41 ]
0.168
0.428
0.805
0.395
0.336
2.068
ReCogDrive [ 11 ]
0.292
0.693
1.365
0.692
0.586
3.306
AutoVLA [ 8 ]
0.293
1.217
3.089
1.235
1.159
5.293
SimLingo [ 22 ]
2.411
8.201
14.067
6.955
5.951
20.018
DriveMoE [ 10 ]
1.347
3.763
6.219
3.206
2.684
10.480
Table 1: RELLIS-3D system comparison under the input and horizon adaptations in Section 4.1 . FastRoverVLA has the lowest displayed errors. Frozen weights; 3 s horizon; errors: m ( ↓ ); bold: column minima.
Figure 4: Frame-level 3 s ADE on RELLIS-3D under the adaptations in Table 1 . Curves: peak-normalized densities in log-error space; thick bars: interquartile ranges; thin lines: 5th–95th percentiles; ticks: medians; diamonds: arithmetic means.
Backbone
Execution
ORAD-3D
RELLIS-3D
ADE ↓
FDE ↓
P95FDE ↓
U/R ↑
ADE ↓
FDE ↓
P95FDE ↓
U/R ↑
Qwen3-VL-2B
Every update
0.384
0.726
1.736
1.0
0.387
0.820
1.933
1.0
Fixed interval
0.330
0.599
1.478
12.2
0.289
0.584
1.507
8.8
FastRoverVLA
0.327
0.593
1.445
17.2
0.266
0.526
1.375
21.0
MiniCPM-V-2.6
Every update
0.390
0.740
1.732
1.0
0.457
0.905
1.886
1.0
Fixed interval
0.327
0.595
1.478
12.2
0.292
0.574
1.454
8.8
Table 2: Recurrent execution improves all eight backbone–dataset pairs under shared frozen predictions. Errors: m ( ↓ ); U/R: updates per admitted plan ( ↑ ). Bold: within-backbone displayed optima, including ties.
Control
ORAD-3D
RELLIS-3D
ADE ↓
FDE ↓
RMSE ↓
P95FDE ↓
ADE ↓
FDE ↓
RMSE ↓
P95FDE ↓
Shared VLA-derived refresh times
FastRoverVLA
0.327
0.593
0.267
1.445
0.266
0.526
0.222
1.375
Direct stale reuse
0.500
0.906
0.404
2.260
0.471
0.901
0.387
2.646
No correction during reuse
0.507
0.924
0.411
2.330
0.455
0.876
0.375
2.511
No per-update alignment ( β=1 )
0.454
0.852
0.374
2.064
0.419
0.844
0.353
2.360
Table 3: State alignment and write-back improve reuse. Upper block: shared refresh times; the no-reuse-correction control also shares refresh-time processing. Lower block: no requests or scheduled refresh. Errors: m ( ↓ ); bold: displayed column minima, including ties.
Figure 5: Matched Qwen3-VL replay. (a,b) ADE densities use a shared within-dataset bandwidth and reflection at zero. (c,d) Mean FDE by reuse age; both policies have identical refresh-time trajectories, but the control omits subsequent correction. Shading marks ages beyond the six-action horizon.
Image input
All frames
Turning subset
ADE
FDE
ADE
FDE
ORAD-3D
True
0.385
0.727
0.468
1.014
Gray
0.384
0.726
0.464
1.005
Donor
0.385
0.728
0.465
1.008
RELLIS-3D
Table 4: Image-content controls at fresh-plan initialization. Turning subsets: Section 4 . Errors: m ( ↓ ); bold: dataset/group minima.
Setting
ORAD-3D
RELLIS-3D
ADE ↓
FDE ↓
P95FDE ↓
Rate (%) ↓
ADE ↓
FDE ↓
P95FDE ↓
Rate (%) ↓
Timing and cross-dataset rule transfer
Fixed interval
0.330
0.599
1.478
8.2
0.289
0.584
1.507
11.4
Uniform, matched budget
0.329
0.597
1.519
5.8
0.268
0.530
1.377
4.8
Local plan-conditioned rule
0.327
0.593
1.445
5.8
0.266
0.526
1.375
4.8
Transferred frozen rule
0.331
0.603
1.531
5.3
0.272
0.541
1.391
5.7
Table 5: Refresh timing, signals, and reuse caps in frozen-Qwen3 replay. R/C/Q: initial-action residual, output incompleteness, and plan roughness. Errors: m ( ↓ ); rate: admissions/update (%); bold: displayed block minima.
Figure 6: ORAD-3D trajectories from matched Qwen3-VL predictions. Camera and BEV views show six waypoints; curves interpolate them for display. Boxes: schematic ego poses; gray points: synchronized LiDAR. Camera projections use per-frame calibration and a 0.65 m display lift. Labels give every-update / FastRoverVLA FDE.
Dataset
Metric
Reduction
Interval (95%)
Improved
ORAD-3D
ADE
0.055
[0.030, 0.078]
10/11
FDE
0.125
[0.073, 0.175]
10/11
RELLIS-3D
ADE
0.111
[0.092, 0.138]
3/3
FDE
0.271
[0.239, 0.328]
3/3
Table 6: Sequence-balanced error reductions (every-update minus FastRoverVLA, m ↑ ). Intervals: 10,000 whole-sequence bootstrap resamples; improved: sequences with lower error.
Controlled execution
Archived heading
Past-only heading
ADE ↓
FDE ↓
ADE ↓
FDE ↓
Every update
0.387
0.820
0.438
0.903
Fixed interval
0.289
0.584
0.339
0.666
FastRoverVLA
0.266
0.526
0.316
0.606
No per-update alignment
0.419
0.844
0.449
0.883
No correction write-back
0.371
0.738
0.414
0.805
Table 7: Past-only heading preserves the execution effect on RELLIS-3D. Only the scoring frame changes. Errors: m ( ↓ ); bold: upper-panel column minima and lower-panel within-backbone minima.
Backbone
Execution
ORAD-3D
RELLIS-3D
ADE ↓
FDE ↓
Req. (%) ↓
C ↓
ADE ↓
FDE ↓
Req. (%) ↓
C ↓
Qwen3-VL-2B
Every update
0.385
0.727
100.0
355.4
0.391
0.831
100.0
443.9
Fixed interval
0.330
0.599
8.2
28.0
0.287
0.581
11.4
50.1
FastRoverVLA
0.328
0.596
6.0
21.4
0.266
0.526
4.8
21.3
MiniCPM-V-2.6
Every update
0.397
0.754
100.0
1336.1
0.364
0.753
100.0
1392.8
Fixed interval
0.332
0.604
8.2
108.1
0.280
0.558
11.4
157.3
Table 8: Independent real-request execution improves accuracy and reduces cost relative to every-update prediction. Errors: m ( ↓ ); Req.: requests/update (%); C: request-processing ms/update. Bold: within-backbone minima, including ties.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking N stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to 0.32m (a 22% improvement). When integrated with SGLang, our framework delivers 12× throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.
Kewei Zhang, Jin Wang, Sensen Gao +9
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
State-of-the-art vision-language-action models (VLA) for autonomous driving face critical limitations: excessive parameter counts, inefficient high-resolution image processing, and lack of temporal memory. We introduce Fast and EffectIVE VLA (FIVE-VLA) to address these through two key contributions. First, we employ an efficient vision encoder that processes high-resolution (448×896) images while generating only 98 tokens, over 5× fewer than existing approaches, and bypass text generation entirely for single-pass trajectory prediction. Second, we propose Recurrent Action Memory (RAM), a lightweight module that conditions action prediction on previous action tokens, providing temporal context critical for manoeuvres such as overtaking and emergency braking. With only 641M parameters, FIVE-VLA completes ∼10% more routes without traffic rule infractions than the previous state-of-the-art VLA on the challenging Bench2Drive closed-loop driving benchmark. Non-reactive open-loop simulation on the large-scale real-world NVIDIA Physical AI AV dataset shows 10.2% and 7.7% lower collision-violation rates than SimLingo in single- and four-view settings, respectively. Additionally, FIVE-VLA runs at ∼30 fps on an A100 and ∼4 fps on a T4 GPU (proxy to an edge device), representing an 8-30× speedup over previous methods.
Kemal Oksuz, Alexandru Buburuzan, Yuhan Yao +1
Robert Bosch GmbH, Germany · Five AI Ltd., United Kingdom
Driving Vision-Language-Action Models (Driving VLAs) aim to use language to improve end-to-end planning, but the language-action gap limits this promise. We propose DriveMA, a Driving VLA framework built on verifiable meta-actions, which summarize future ego motion into compact language-domain intentions and can be constructed from expert trajectories with a trajectory-grounded annotation pipeline and can be verified against generated trajectories through rule-based projection. DriveMA exploits this verifiability with action-centric supervised training and a data-efficient turn-level credit assignment reinforcement learning framework, explicitly aligning high-level decisions with low-level trajectory planning through dense rewards and precise credit assignment. DriveMA sets a new state of the art on the Waymo Open Dataset Vision-based E2E Driving, achieving a Rater Feedback Score of 8.060 with a 2B model and further improving it to 8.079 with a 4B model; it also obtains competitive closed-loop planning performance on NAVSIM. These results show that even a simple meta-action interface can achieve state-of-the-art planning when made verifiable and optimized for language-action alignment. Code, data, and models are available at https://tsinghua-mars-lab.github.io/DriveMA.
Weicheng Zheng, Yixin Huang, Qiao Sun +2
1Shanghai Qi Zhi Institute · 3Tongji University · 2IIIS Tsinghua University