Fleets of driving vision-language-action (VLA) models are developed one domain at a time: each operator finetunes a shared base model on private data, producing specialized experts that are costly to consolidate into a single model. Weight merging loses accuracy as heterogeneous domains accumulate, and dense mixture-of-experts routing is prohibitively expensive to combine them. We present DeltaForge, to our knowledge the first post-hoc mixture-of-experts VLA for driving, which consolidates N independently finetuned per-domain models into a single multi-domain model from their weights and a small proxy set alone. When different operators finetune the same base model, their weight updates end up sharing a lot of structure. We exploit this by decomposing all the updates together using singular value decomposition (SVD), splitting each one into a shared part (the same across all domains) and a tiny domain-specific part. The router then only needs to work with those tiny domain-specific pieces, not the full expert weights, which is what keeps the approach efficient. Then two cooperating clip-level blind routers, one over the VLM reasoning experts and one over the flow-matching dynamics experts, are trained from soft labels on the proxy dataset. Across six diverse domains spanning urban, off-road, and highway driving on cars, trucks, and all-terrain vehicles (ATVs), DeltaForge approaches the accuracy of the per-domain experts and surpasses every merging, routing, centralized, and federated baseline at close to single-model parameter and compute cost. Adding a new domain requires only projecting its weight update onto the shared basis with no end-to-end retraining needed. The resulting model also generalizes to domain combinations it was never explicitly trained on.
Figures & tables
Fig. 1: DeltaForge consolidates independently finetuned per-domain driving VLAs into one model from their weights and a small proxy set. A joint SVD factorizes the per-domain deltas into shared skill bases and low-rank cores that clip-level blind routers compose at inference.
Fig. 2: DeltaForge system workflow. Independently finetuned per-domain VLAs are jointly factorized by PEF into a frozen shared skill basis and per-domain low-rank cores. Two cooperating clip-level blind routers then select and mix the cores at inference, one at the input for VLM reasoning experts and one intermediate for flow-matching dynamics experts. Composition is trained on a small proxy set, and a new domain opts in by projecting its delta onto the frozen basis.
Fig. 3: DeltaForge architecture and composition training. (a) Two-level clip-level routing of low-rank cores at inference. (b) Soft-oracle router training with a KL objective. (c) Core-expert finetuning against the task loss.
Domain
Dataset
Scenario
Train
Eval
Urban China
ONCE [ 21 ]
urban, right-hand
24,885
2,769
Urban Japan
CoVLA [ 22 ]
urban, left-hand
26,987
2,893
Off-road ATV
TartanDrive [ 23 ]
forest trails, ATV
17,470
1,484
Off-road car
ORAD-3D [ 24 ]
China, off-road, car
34,801
4,897
Highway truck EU
TruckScenes [ 25 ]
EU highway, truck
33,305
3,172
Base
PhysicalAI-AV [ 26 ]
US/EU, four cameras
19,545
3,000
TABLE I: Domains and datasets. Train/eval are windowed samples after scene-level splitting. The lower block is used only for onboarding and zero-shot evaluation.
Method
Venue
minADE@5 s (m) ↓
Data
Params (B) ↓
PAI base
ONCE China
CoVLA Japan
Tartan ATV
ORAD off-road
TruckScenes highway truck
Mean
Active
Total
Zero-shot base (lower bound)
–
0.9568
2.1763
4.4319
2.5019
1.0800
6.6409
2.9646
none
11.08
11.08
Per-domain experts (upper bound)
–
0.9568
1.0354
1.6828
1.2294
0.5141
0.4601
0.9798
own domain
11.08
66.47
Centralized SFT (pooled data)
–
0.8098
1.2461
1.7701
1.3092
0.9206
0.6217
1.1129
all
11.08
11.08
Federated SFT (FedAdam [ 30 ] )
ICLR’21
0.8076
1.2237
1.7172
1.3457
0.6991
0.8585
1.1086
all (local)
11.08
11.08
Model soup [ 8 ]
ICML’22
1.4376
1.4184
2.0058
1.3878
0.9029
2.6261
1.6298
none
11.08
11.08
TABLE II: End-to-end minADE@5 s (m, best-of-6, lower is better) on the six domains and their mean. PAI is the base domain. Data is the training data a method needs beyond the finetuned weights. Bold and underline mark the best and second-best weight-based method per column.
Component
Params
Trained
Base VLM + action expert
11,079 M
frozen
Shared bases U,V (VLM + FM)
1,257 M
closed form
Embeddings, heads, norms
+0
merged
Routers ( ×2 )
2.1 M
✓
Cores + projections / domain
42.0 M
✓
Total ( N=5 )
12,547 M
TABLE III: Parameter breakdown of the deployed DeltaForge system ( N=5 ). The shared bases come in closed form from the joint SVD, so only the cores and the two routers are trained, about 1% of the total.
Params (B) ↓
GFLOPs / token ↓
Method
formula
N=5
N=10
formula
top-1
top-2
top-5
Per-domain experts
11.08(N+1)
66.5
121.9
20.9
20.9
–
–
BTX
11.08+7.26N
47.4
83.7
6.35+14.52k
20.9
35.4
79.0
DeltaForge (ours)
12.34+0.042N
12.55
12.76
23.39+0.08k
23.5
23.6
23.8
TABLE IV: Deployment cost versus the number of domains N and routed experts k (dense base 20.9 GFLOPs/token). DeltaForge stays near single-model cost as both grow, unlike the linear scaling of separate experts and BTX.
Unseen: Torc TruckDrive in pool: MAN TruckScenes
Model
minADE
Base (zero-shot)
8.9862
Own-domain finetune
0.8225
DeltaForge top-1 routing
1.0627
DeltaForge top-2 routing
0.9969
Unseen: BeVIS onboarded: ParkRecon3D
TABLE V: Generalization to unseen domains (minADE@5 s, meters). Top: a truck skill already in the pool (MAN TruckScenes) transfers zero-shot to the unseen Torc TruckDrive. Bottom: onboarding the related ParkRecon3D parking domain serves the never-seen BeVIS below its own finetuned expert, without training on it.
Fig. 4: Router behavior. Each column is one router, with rows showing its input-feature PCA (top), soft routing (middle), and hard routing (bottom), the confusion matrices row-normalized by true domain. The input router (left) recognizes the domain and picks the corresponding route 97% of the time, while the intermediate router (right) routes to the correct domain only 40% of the time, indicating that the dynamics experts are interchangeable.
Fig. 5: Task-vector spectrum per module. (a,b) Per-layer top-1 energy of attention and FFN in the VLM and the action expert, red marks the low, near-orthogonal band. (c) Mean pairwise cosine similarity of the domain deltas per module: attention and FFN near zero (route), embeddings and norms high (merge).