Locating and Repairing Domain Shift in VLM Trajectory Planning
Authors: Zhihong Cui, Hengyu Liu, Michael Alexander Riegler, Guandong Xu, Amir Taherkordi, Tor Skeie
Organizations: Department of Informatics, University of Oslo, Norway · Department of Computer Science, Aalborg University, Denmark · Simula Metropolitan Center for Digital Engineering (SimulaMet), Norway · The Education University of Hong Kong, Hong Kong, China · Simula Research Laboratory, Norway
Vision-Language Models (VLMs) are increasingly adopted as end-to-end driving planners, yet they suffer from cross-city domain shift. Existing methods treat this shift as a single quantity addressed by uniform alignment or single-site editing. We challenge this view on two counts: (i) domain shift in VLM trajectory planners is inherently structured across modality and layer, and (ii) a locate-then-repair recipe matched to this structure outperforms every uniform-objective baseline. We analyse domain shift via activation patching and uncover two regularities. (F1) Modality axis. Image and text tokens exhibit distinct shift patterns in each VLM. (F2) Layer axis. The layer-wise shift partition is architecture-dependent and dataset-stable. These findings recast domain shift as a two-dimensional tensor indexed by modality (image vs. text tokens) and layer, motivating a simple principle: locate where shift occurs, and repair only there. We instantiate this principle in MoLaRx (Modality-Layer Repair), a locate-then-repair framework. The Causal Importance Score (CIS) estimates the domain shift as a modality-layer tensor and partitions the layers into functional zones. CIS-Guided Selective Repair (CISR) then applies a matched operator (Maximum Mean Discrepancy (MMD) / low-rank adaptation (LoRA) / freezing) to repair each zone, with the composition adapting automatically to each architecture. We evaluate MoLaRx on three architecturally distinct VLMs (LFM2-VL-1.6B, Qwen2-VL-2B, and PaliGemma2-3B) across two cross-city benchmarks (nuScenes and Argoverse 2). MoLaRx attains the lowest cross-city trajectory-error gap among adaptation methods in all six main-table settings, while using 50-60% fewer trainable parameters and 8.4-10.9 ms lower inference latency. In contrast, uniform MMD-only adaptation collapses in all six settings, highlighting the importance of structure-guided repair.
Figures & tables
Figure 1: Domain shift in VLM trajectory planners is structured along modality and layer. (a) VLMs fine-tuned as trajectory planners suffer cross-city domain shift; (b) the existing diagnostic — DPA: saturates near 1 at every layer of every VLM; (c) F1 (modality axis) : image and text shifts differ within every VLM; (d) F2 (layer axis) : zone partition is architecture-dependent and dataset-stable.
Figure 2: MoLaRx overview. CIS locates cross-city shift by modality and layer; CISR applies matched repair operators to the diagnosed zones.
Data -set
VLM
Metric
Source -only
F-LoRA
F-LoRA + MMD
DANN
MMD only
LLRD
Freeze -K
ROME ⋆
Surg -FT
MoLaRx
Δ
nuScenes (BS → SG)
LFM2
ADE ( ↓ , m)
4.510
4.275
4.284
4.295
9.657
4.288
4.310
4.270
4.266
4.251
0.015
FDE ( ↓ , m)
7.890
7.480
7.531
7.590
16.542
7.475
7.510
7.470
7.462
7.422
0.040
ΔADE ( ↓ , m)
2.550
3.340
3.328
3.390
1.646 ‡
3.358
3.315
3.255
3.205
3.150
0.055 ⋆
PE ( ↓ , ms)
68.3
66.5
68.4
67.5
48.3 ‡
66.5
65.5
65.5
58.3
47.4
10.9 ⋆
Qwen2
ADE ( ↓ , m)
5.020
4.730
4.760
4.745
10.156
4.715
4.725
4.720
4.700
4.634
0.066 ⋆
FDE ( ↓ , m)
8.760
8.430
8.470
8.450
18.130
8.426
8.422
8.414
8.380
8.242
0.138 ⋆
Table 1: Performance across 2 datasets of 3 VLMs on four metrics. Bold marks the best result per row among the adaptation methods ; underline marks the runner-up. Source-only is reported as the no-adaptation reference and is excluded from the bold/underline ranking. The rightmost column Δ is MoLaRx’s lead over the underlined runner-up. The marker ⋆ flags margins exceeding the 2σ significance threshold. MMD-only is marked with ‡ because the model has collapsed.
Figure 3: CIS atlas across three VLMs (nuScenes).
Type
Comparison
PCS(Φalign)
PCS(Φtune)
PCS(Φfreeze)
Mean
Within-arch. (cross-dataset)
LFM2: nuScenes vs. AV2
1.00
1.00
1.00
1.00
Qwen2: nuScenes vs. AV2
0.88
0.93
0.96
0.92
PaliGemma2: nuScenes vs. AV2
1.00
1.00
1.00
1.00
Cross-arch. (normalized depth, nuScenes)
LFM2 vs. Qwen2
0.75
0.68
0.81
0.75
LFM2 vs. PaliGemma2
0.71
−
0.77
−
Qwen2 vs. PaliGemma2
0.72
−
0.74
−
Table 2: Partition Consistency Score.
VLM
Cond.
ADE ID
ADE OOD
ΔADE
LFM2
(A) AV2-local
1.4304
4.0812
2.6508
(B) Zero-shot
1.4318
4.0921
2.6603
(C) Source-only
1.8732
5.1247
3.2515
Qwen2
(A) AV2-local
1.5821
4.2153
2.6332
(B) Zero-shot
1.5876
4.2401
2.6525
(C) Source-only
1.8123
4.9534
3.1411
Table 3: Zero-shot partition transfer.
Variant
LFM2
Qwen2
PaliGemma2
MoLaRx (full)
2.3687
2.6287
2.4521
w/o CIS
2.7259
2.9851
2.8523
DPA-partition
2.6742
2.8643
2.7234
w/o MMD on Φalign
2.6108
2.8214
2.7892
w/o LoRA on Φtune
2.6821
2.8925
–
w/o Freeze on Φfreeze
2.3714
2.6523
2.4612
Table 4: Ablation ΔADE↓ (m) on nuScenes.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Value
LFM2-VL-1.6B
Decoder layers L
16
Attention connectivity
causal
Backbone operator
LTC-hybrid
CIS threshold τimg
0.31 m
Φalign
{0,1}
Appendix
Table 5: Per-VLM architecture and CIS-induced partition. τimg is chosen at the elbow of the sorted CISimg curve on nuScenes and reused on AV2 without recalibration. On PaliGemma2 the prefix-LM attention yields Φtune=∅ , so the CISR LoRA budget shifts from Φtune to Φalign .
Category
Setting
Datasets
nuScenes source
Boston-Seaport ( 700 scenes)
nuScenes target
Singapore ( 150 scenes)
nuScenes keyframes
∼26,600 train / 3,569 ID / 3,500 OOD
Argoverse 2 source
Pittsburgh ( 700 logs)
Argoverse 2 target
Miami ( 150 logs)
Appendix
Table 6: Experiment settings and hyperparameters for MoLaRx.
Dataset
VLM
ImgDep (single-frame + text)
ImgDep (ours)
Change
nuScenes (BS → SG)
LFM2
−0.60 %
+442%
+442.6 pp
Qwen2
−0.42 %
+360%
+360.4 pp
PaliGemma2
−0.52 %
+287%
+287.5 pp
Argoverse 2 (PIT → MIA)
LFM2
−0.52 %
+295 %
+295.5 pp
Qwen2
−0.32 %
+172%
+172.4 pp
PaliGemma2
−0.48 %
+198%
+198.2 pp
Appendix
Table 7: Visual-grounding fine-tuning restores genuine visual reliance on every cell.
Figure 4: Trajectory comparison on representative target-city scenes (LFM2-VL, real model rollouts projected onto the front-camera image). (a) nuScenes BS → SG, scene-0015 (Singapore one-north arterial, ego 5.5 m/s) on CAM_FRONT. (b) AV2 PIT → MIA, scene 0b97f5dd (Wynwood-Miami arterial, ego 8.5 m/s) on ring_front_center. All five LFM2-VL methods plus GT (black dashed) projected with z=0 ground-plane assumption.
Figure 5: Cross-city consistency on a SigLIP2-matched scene pair (LFM2-VL). (a) Boston source scene-0669 (ego 7.3 m/s); (b) Singapore target scene-0369 (ego 7.9 m/s). Cross-city cosine 0.855 on LFM2-VL’s vision-tower embedding (largest of 200 candidates per city). All five methods plus GT (black dashed) projected onto the front-camera image with z=0 ground-plane assumption.
Figure 6: CIS-threshold sensitivity on nuScenes. Top row (a)–(c): τimg sweep across the three VLMs. Bottom row (d)–(e): τtxt sweep on the two causal-decoder VLMs; PaliGemma2 is omitted because Φtune=∅ on prefix-LM. Red star = elbow value chosen by our default elbow-finder.
Figure 7: λMMD sensitivity across three VLMs on nuScenes. Each panel sweeps λMMD∈[0,5] with the CIS-induced partition fixed at its default values, and reports ΔADE . The red star marks the default λMMD=0.5 .
Nframe
ΔADE (m) on nuScenes
ImgDep (%)
GPU mem
Latency
LFM2-VL
Qwen2-VL
PaliGemma2
(LFM2)
(GB)
(ms)
1
4.15
3.91
3.34
−0.6
4.8
42
2
3.65
3.41
2.84
+8.4
6.2
51
3 (default)
3.15
2.91
2.34
+18.2
8.1
63
4
3.13
2.89
2.32
+18.9
11.5
79
5
3.12
2.88
2.31
+19.5
16.3
98
Appendix
Table 8: Nframe sensitivity on nuScenes (BS → SG). Cross-city ΔADE per VLM, ImgDep on LFM2-VL (signal-collapse indicator, App. B ), and per-step GPU memory and latency on a single A10 (LFM2-VL backbone). Nframe=3 (bold) is the operating point used in all main-paper experiments.
N
Empirical ε^ (m)
PCS vs. N=500
ΔADE on AV2 (m)
CIS time (h)
25
1.42 ( ±0.18 )
0.62
2.105 ( ±0.041 )
0.18
50
0.95 ( ±0.11 )
0.81
1.945 ( ±0.029 )
0.35
100 (default)
0.55 ( ±0.06 )
0.94
1.890 ( ±0.024 )
0.65
200
0.35 ( ±0.04 )
0.98
1.882 ( ±0.022 )
1.30
500
0.18 ( ±0.02 )
1.00 (ref)
1.880 ( ±0.021 )
3.20
Appendix
Table 9: CIS pair-budget sensitivity on LFM2/nuScenes. ε^ : empirical standard error of Ψ[m,l] across 5 random sub-samples of N pairs (averaged over all (m,l) entries). PCS: zone-wise Jaccard against the N=500 asymptotic partition (Eq. 21 ). ΔADE : zero-shot CISR on AV2 (PIT → MIA) using the partition diagnosed at the given N on nuScenes, five-seed std in parentheses. CIS time on a single A10 GPU. N=100 (bold) is the operating point used in all main-paper experiments.
VLM
Method
Train. params
% saved
Train (h/ep)
Inference (ms/sample)
LFM2-VL-1.6B
Full-LoRA
7.6 M
0%
4.2 ( ±0.05 )
66.5 ( ±0.8 )
MoLaRx (CISR)
3.8 M
50%
4.1 ( ±0.04 )
47.4 ( ±0.6 )
Qwen2-VL-2B
Full-LoRA
8.9 M
0%
4.6 ( ±0.06 )
79.8 ( ±1.1 )
MoLaRx (CISR)
4.1 M
54%
4.5 ( ±0.05 )
50.5 ( ±0.7 )
PaliGemma2-3B
Full-LoRA
12.5 M
0%
5.4 ( ±0.07 )
81.4 ( ±1.0 )
MoLaRx (CISR) †
5.0 M
60%
5.3 ( ±0.06 )
62.1 ( ±0.9 )
Appendix
Table 10: Compute footprint per VLM. “% saved” is the parameter saving over each VLM’s Full-LoRA baseline. Not shown : a one-time CIS-diagnosis cost of ∼0.65 h per architecture (App. C.2.4 ), reused zero-shot across the two benchmarks (Table 3 ). † On PaliGemma2 the prefix-LM attention yields Φtune=∅ , so the recipe collapses to MMD-driven attention-only LoRA on Φalign + freeze on Φfreeze . MoLaRx saves 50 – 60% of trainable parameters across all three architectures; the largest absolute saving ( 7.5 M, 60% ) is on PaliGemma2, where attention-only LoRA on Φalign avoids the MLP-projection bulk that Full-LoRA pays.
Method
Adaptation pool
ADE ID
FDE ID
ADE OOD
FDE OOD
Source-only
none
1.472
2.705
4.466
7.754
MoLaRx
transductive
1.116
2.014
4.266
7.462
MoLaRx
inductive
1.255
2.262
4.273
7.492
Appendix
Table 11: Inductive-protocol control (LFM2-VL, nuScenes BS → SG; same seed; fixed 300 -sample ID/OOD subsets).
Scenario type
n
Source-only
Full LoRA
Full LoRA + MMD
MoLaRx
Specifically selected
18
9.47
8.48
9.61
9.91
Heavy rain
10
19.41
17.97
18.95
18.62
Construction zone
6
9.95
7.74
8.31
8.60
Overtake or lane change
20
25.76
24.19
24.14
24.18
Intersection
35
7.19
6.09
5.91
5.77
Night-time
5
42.85
40.63
42.69
41.12
Appendix
Table 12: Zero-shot ADE (m) on KITScenes-LongTail by scenario type (LFM2-VL planners trained on nuScenes-Boston; single seed). The last row gives overall ADE / FDE.
Method
Closest-mode acc. ↑
Mean mode margin (m) ↑
Source-only
0.25
−1.08
Full LoRA
0.29
−0.98
Full LoRA + MMD
0.28
−1.02
MoLaRx
0.29
−0.98
Appendix
Table 13: MMS-inspired closest-mode accuracy and mean mode margin on KITScenes-LongTail ( 100 scenarios; same zero-shot setting as Table 12 ).
Method
ADE ID
Lat. ID
Jerk ID
ADE OOD
Lat. OOD
Jerk OOD
Source-only
1.67
0.65
0.77
4.86
0.96
0.87
Full LoRA
1.27
0.56
1.12
4.52
1.16
1.30
Full LoRA + MMD
1.20
0.58
1.73
4.54
1.15
2.04
MoLaRx
1.10
0.52
1.31
4.66
1.14
1.64
Appendix
Table 14: Planning-oriented proxies on nuScenes BS → SG (LFM2-VL; fixed 300 -sample subsets; single seed). Lat.: lateral error at 3 s (m); jerk: mean absolute jerk (m/s 3 ); the ground-truth jerk is 1.77 (ID) and 2.27 (OOD).
VLM
Cell
Reverse partition
Agree
PCSalign
PCStune
PCSfreeze
Mean
LFM2-VL
SG → BS
{0,1}/{2:9}/{10:15}
16/16
1.00
1.00
1.00
1.00
LFM2-VL
MIA → PIT
{0,1}/{2:9}/{10:15}
16/16
1.00
1.00
1.00
1.00
Qwen2-VL
SG → BS
{0,1,17}/{2:15}/{16,18:27}
20/28
0.40
0.50
0.65
0.52
PaliGemma2
SG → BS
{0:23}/{24,25}/∅
24/26
0.96
0.00
0.00
0.32
Appendix
Table 15: Zone partition under reversed transfer direction (single seed), written as Φalign/Φtune/Φfreeze . PCS is the zone-wise Jaccard against the forward partition; “agree” counts layers assigned to the same zone.
Method
LFM2-VL, SG → BS
Qwen2-VL, SG → BS
LFM2-VL, MIA → PIT (AV2)
Source-only
3.52 / 4.12 (0.60)
3.29 / 4.30 (1.01)
10.90 / 11.61 (0.71)
Full LoRA
2.56 / 3.61 (1.05)
2.78 / 4.11 (1.33)
5.12 / 4.86 ( −0.26 )
MoLaRx
2.79 / 3.90 (1.12)
2.89 / 4.19 (1.30)
4.72 / 5.39 (0.67)
Appendix
Table 16: Reverse-direction repair comparison: ADE (m) on fixed 300 -sample subsets, ID / OOD ( ΔADE ). Single seed; visual grounding truncated to approximately two epochs in the nuScenes cells; AV2 values are from a separate harness and comparable only within the block.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.