Fleets of driving vision-language-action (VLA) models are developed one domain at a time: each operator finetunes a shared base model on private data, producing specialized experts that are costly to consolidate into a single model. Weight merging loses accuracy as heterogeneous domains accumulate, and dense mixture-of-experts routing is prohibitively expensive to combine them. We present DeltaForge, to our knowledge the first post-hoc mixture-of-experts VLA for driving, which consolidates N independently finetuned per-domain models into a single multi-domain model from their weights and a small proxy set alone. When different operators finetune the same base model, their weight updates end up sharing a lot of structure. We exploit this by decomposing all the updates together using singular value decomposition (SVD), splitting each one into a shared part (the same across all domains) and a tiny domain-specific part. The router then only needs to work with those tiny domain-specific pieces, not the full expert weights, which is what keeps the approach efficient. Then two cooperating clip-level blind routers, one over the VLM reasoning experts and one over the flow-matching dynamics experts, are trained from soft labels on the proxy dataset. Across six diverse domains spanning urban, off-road, and highway driving on cars, trucks, and all-terrain vehicles (ATVs), DeltaForge approaches the accuracy of the per-domain experts and surpasses every merging, routing, centralized, and federated baseline at close to single-model parameter and compute cost. Adding a new domain requires only projecting its weight update onto the shared basis with no end-to-end retraining needed. The resulting model also generalizes to domain combinations it was never explicitly trained on.
Figures & tables
Fig. 1: DeltaForge consolidates independently finetuned per-domain driving VLAs into one model from their weights and a small proxy set. A joint SVD factorizes the per-domain deltas into shared skill bases and low-rank cores that clip-level blind routers compose at inference.
Fig. 2: DeltaForge system workflow. Independently finetuned per-domain VLAs are jointly factorized by PEF into a frozen shared skill basis and per-domain low-rank cores. Two cooperating clip-level blind routers then select and mix the cores at inference, one at the input for VLM reasoning experts and one intermediate for flow-matching dynamics experts. Composition is trained on a small proxy set, and a new domain opts in by projecting its delta onto the frozen basis.
Fig. 3: DeltaForge architecture and composition training. (a) Two-level clip-level routing of low-rank cores at inference. (b) Soft-oracle router training with a KL objective. (c) Core-expert finetuning against the task loss.
Domain
Dataset
Scenario
Train
Eval
Urban China
ONCE [ 21 ]
urban, right-hand
24,885
2,769
Urban Japan
CoVLA [ 22 ]
urban, left-hand
26,987
2,893
Off-road ATV
TartanDrive [ 23 ]
forest trails, ATV
17,470
1,484
Off-road car
ORAD-3D [ 24 ]
China, off-road, car
34,801
4,897
Highway truck EU
TruckScenes [ 25 ]
EU highway, truck
33,305
3,172
Base
PhysicalAI-AV [ 26 ]
US/EU, four cameras
19,545
3,000
TABLE I: Domains and datasets. Train/eval are windowed samples after scene-level splitting. The lower block is used only for onboarding and zero-shot evaluation.
Method
Venue
minADE@5 s (m) ↓
Data
Params (B) ↓
PAI base
ONCE China
CoVLA Japan
Tartan ATV
ORAD off-road
TruckScenes highway truck
Mean
Active
Total
Zero-shot base (lower bound)
–
0.9568
2.1763
4.4319
2.5019
1.0800
6.6409
2.9646
none
11.08
11.08
Per-domain experts (upper bound)
–
0.9568
1.0354
1.6828
1.2294
0.5141
0.4601
0.9798
own domain
11.08
66.47
Centralized SFT (pooled data)
–
0.8098
1.2461
1.7701
1.3092
0.9206
0.6217
1.1129
all
11.08
11.08
Federated SFT (FedAdam [ 30 ] )
ICLR’21
0.8076
1.2237
1.7172
1.3457
0.6991
0.8585
1.1086
all (local)
11.08
11.08
Model soup [ 8 ]
ICML’22
1.4376
1.4184
2.0058
1.3878
0.9029
2.6261
1.6298
none
11.08
11.08
TABLE II: End-to-end minADE@5 s (m, best-of-6, lower is better) on the six domains and their mean. PAI is the base domain. Data is the training data a method needs beyond the finetuned weights. Bold and underline mark the best and second-best weight-based method per column.
Component
Params
Trained
Base VLM + action expert
11,079 M
frozen
Shared bases U,V (VLM + FM)
1,257 M
closed form
Embeddings, heads, norms
+0
merged
Routers ( ×2 )
2.1 M
✓
Cores + projections / domain
42.0 M
✓
Total ( N=5 )
12,547 M
TABLE III: Parameter breakdown of the deployed DeltaForge system ( N=5 ). The shared bases come in closed form from the joint SVD, so only the cores and the two routers are trained, about 1% of the total.
Params (B) ↓
GFLOPs / token ↓
Method
formula
N=5
N=10
formula
top-1
top-2
top-5
Per-domain experts
11.08(N+1)
66.5
121.9
20.9
20.9
–
–
BTX
11.08+7.26N
47.4
83.7
6.35+14.52k
20.9
35.4
79.0
DeltaForge (ours)
12.34+0.042N
12.55
12.76
23.39+0.08k
23.5
23.6
23.8
TABLE IV: Deployment cost versus the number of domains N and routed experts k (dense base 20.9 GFLOPs/token). DeltaForge stays near single-model cost as both grow, unlike the linear scaling of separate experts and BTX.
Unseen: Torc TruckDrive in pool: MAN TruckScenes
Model
minADE
Base (zero-shot)
8.9862
Own-domain finetune
0.8225
DeltaForge top-1 routing
1.0627
DeltaForge top-2 routing
0.9969
Unseen: BeVIS onboarded: ParkRecon3D
TABLE V: Generalization to unseen domains (minADE@5 s, meters). Top: a truck skill already in the pool (MAN TruckScenes) transfers zero-shot to the unseen Torc TruckDrive. Bottom: onboarding the related ParkRecon3D parking domain serves the never-seen BeVIS below its own finetuned expert, without training on it.
Fig. 4: Router behavior. Each column is one router, with rows showing its input-feature PCA (top), soft routing (middle), and hard routing (bottom), the confusion matrices row-normalized by true domain. The input router (left) recognizes the domain and picks the corresponding route 97% of the time, while the intermediate router (right) routes to the correct domain only 40% of the time, indicating that the dynamics experts are interchangeable.
Fig. 5: Task-vector spectrum per module. (a,b) Per-layer top-1 energy of attention and FFN in the VLM and the action expert, red marks the low, near-orthogonal band. (c) Mean pairwise cosine similarity of the domain deltas per module: attention and FFN near zero (route), embeddings and norms high (merge).
Vision-Language-Action (VLA) models promise to bring end-to-end reasoning to autonomous driving, but their computational cost remains far too high for real-time control. The core challenge is structural: VLA inference is not a single bottleneck but a cascade of four. Visual encoding wastes compute on overlapping video frames; language-model prefill recomputes context that could be carried over from the previous timestep; reasoning tokens are generated serially despite low entropy; and flow-matching denoising applies uniform compute to a non-uniform velocity field. Addressing any one stage in isolation leaves the others untouched. We propose FlashDrive, an algorithm-system co-design framework that targets all four stages simultaneously. Our key insight is that each bottleneck admits a distinct, lightweight algorithmic shortcut: temporal overlap enables streaming KV-cache reuse across frames; the low per-token entropy and strong intra-block correlations of driving-domain reasoning make a non-autoregressive diffusion drafter highly effective for speculative decoding; and the velocity field's structure---sharp at the endpoints, flat in the middle---permits adaptive step caching that concentrates compute where it matters. Layered on system-level CUDA Graph compilation and kernel fusion, these techniques compound. Applied to Alpamayo 1.5-10B with W4A8 quantization, FlashDrive reduces end-to-end latency from 717ms to 151ms (4.7x) while leaving accuracy essentially unchanged: [email protected] shifts by only 0.08m, minADE1 improves, and closed-loop collision and off-road rates improve in simulation. By raising a 10B-parameter reasoning VLA from 1.4Hz to 6.6Hz on a single GPU, FlashDrive moves end-to-end autonomous driving substantially closer to real-time deployment.
Zekai Li, Yihao Liang, Hongfei Zhang +3
1UC San Diego · 2Princeton · 3Independent Researcher
End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking N stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to 0.32m (a 22% improvement). When integrated with SGLang, our framework delivers 12× throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.
Kewei Zhang, Jin Wang, Sensen Gao +9
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
Vision-Language-Action (VLA) models for autonomous driving increasingly adopt generative planners trained with imitation learning followed by reinforcement learning. Diffusion-based planners suffer from modality alignment difficulties, low training efficiency, and limited generalization. Token-based planners are plagued by cumulative causal errors and irreversible decoding. In summary, the two dominant paradigms exhibit complementary strengths and weaknesses. In this paper, we propose DriveFine, a masked diffusion VLA model that combines flexible decoding with self-correction capabilities. In particular, we design a novel plug-and-play block-MoE, which seamlessly injects a refinement expert on top of the generation expert. By enabling explicit expert selection during inference and gradient blocking during training, the two experts are fully decoupled, preserving the foundational capabilities and generic patterns of the pretrained weights, which highlights the flexibility and extensibility of the block-MoE design. Furthermore, we design a hybrid reinforcement learning strategy that encourages effective exploration of refinement expert while maintaining training stability. Extensive experiments on NAVSIM v1, v2, and Navhard benchmarks demonstrate that DriveFine exhibits strong efficacy and robustness. The code will be released at https://github.com/MSunDYY/DriveFine.
Chenxu Dang, Sining Ang, Yongkang Li +7
Huazhong University of Science and Technology · Xiaomi EV · Institute for AI Industry Research (AIR), Tsinghua University