Locating and Repairing Domain Shift in VLM Trajectory Planning
Authors: Zhihong Cui, Hengyu Liu, Michael Alexander Riegler, Guandong Xu, Amir Taherkordi, Tor Skeie
Organizations: Department of Informatics, University of Oslo, Norway · Department of Computer Science, Aalborg University, Denmark · Simula Metropolitan Center for Digital Engineering (SimulaMet), Norway · The Education University of Hong Kong, Hong Kong, China · Simula Research Laboratory, Norway
Vision-Language Models (VLMs) are increasingly adopted as end-to-end driving planners, yet they suffer from cross-city domain shift. Existing methods treat this shift as a single quantity addressed by uniform alignment or single-site editing. We challenge this view on two counts: (i) domain shift in VLM trajectory planners is inherently structured across modality and layer, and (ii) a locate-then-repair recipe matched to this structure outperforms every uniform-objective baseline. We analyse domain shift via activation patching and uncover two regularities. (F1) Modality axis. Image and text tokens exhibit distinct shift patterns in each VLM. (F2) Layer axis. The layer-wise shift partition is architecture-dependent and dataset-stable. These findings recast domain shift as a two-dimensional tensor indexed by modality (image vs. text tokens) and layer, motivating a simple principle: locate where shift occurs, and repair only there. We instantiate this principle in MoLaRx (Modality-Layer Repair), a locate-then-repair framework. The Causal Importance Score (CIS) estimates the domain shift as a modality-layer tensor and partitions the layers into functional zones. CIS-Guided Selective Repair (CISR) then applies a matched operator (Maximum Mean Discrepancy (MMD) / low-rank adaptation (LoRA) / freezing) to repair each zone, with the composition adapting automatically to each architecture. We evaluate MoLaRx on three architecturally distinct VLMs (LFM2-VL-1.6B, Qwen2-VL-2B, and PaliGemma2-3B) across two cross-city benchmarks (nuScenes and Argoverse 2). MoLaRx attains the lowest cross-city trajectory-error gap among adaptation methods in all six main-table settings, while using 50-60% fewer trainable parameters and 8.4-10.9 ms lower inference latency. In contrast, uniform MMD-only adaptation collapses in all six settings, highlighting the importance of structure-guided repair.
Figures & tables
Figure 1: Domain shift in VLM trajectory planners is structured along modality and layer. (a) VLMs fine-tuned as trajectory planners suffer cross-city domain shift; (b) the existing diagnostic — DPA: saturates near 1 at every layer of every VLM; (c) F1 (modality axis) : image and text shifts differ within every VLM; (d) F2 (layer axis) : zone partition is architecture-dependent and dataset-stable.
Figure 2: MoLaRx overview. CIS locates cross-city shift by modality and layer; CISR applies matched repair operators to the diagnosed zones.
Data -set
VLM
Metric
Source -only
F-LoRA
F-LoRA + MMD
DANN
MMD only
LLRD
Freeze -K
ROME ⋆
Surg -FT
MoLaRx
Δ
nuScenes (BS → SG)
LFM2
ADE ( ↓ , m)
4.510
4.275
4.284
4.295
9.657
4.288
4.310
4.270
4.266
4.251
0.015
FDE ( ↓ , m)
7.890
7.480
7.531
7.590
16.542
7.475
7.510
7.470
7.462
7.422
0.040
ΔADE ( ↓ , m)
2.550
3.340
3.328
3.390
1.646 ‡
3.358
3.315
3.255
3.205
3.150
0.055 ⋆
PE ( ↓ , ms)
68.3
66.5
68.4
67.5
48.3 ‡
66.5
65.5
65.5
58.3
47.4
10.9 ⋆
Qwen2
ADE ( ↓ , m)
5.020
4.730
4.760
4.745
10.156
4.715
4.725
4.720
4.700
4.634
0.066 ⋆
FDE ( ↓ , m)
8.760
8.430
8.470
8.450
18.130
8.426
8.422
8.414
8.380
8.242
0.138 ⋆
Table 1: Performance across 2 datasets of 3 VLMs on four metrics. Bold marks the best result per row among the adaptation methods ; underline marks the runner-up. Source-only is reported as the no-adaptation reference and is excluded from the bold/underline ranking. The rightmost column Δ is MoLaRx’s lead over the underlined runner-up. The marker ⋆ flags margins exceeding the 2σ significance threshold. MMD-only is marked with ‡ because the model has collapsed.
Figure 3: CIS atlas across three VLMs (nuScenes).
Type
Comparison
PCS(Φalign)
PCS(Φtune)
PCS(Φfreeze)
Mean
Within-arch. (cross-dataset)
LFM2: nuScenes vs. AV2
1.00
1.00
1.00
1.00
Qwen2: nuScenes vs. AV2
0.88
0.93
0.96
0.92
PaliGemma2: nuScenes vs. AV2
1.00
1.00
1.00
1.00
Cross-arch. (normalized depth, nuScenes)
LFM2 vs. Qwen2
0.75
0.68
0.81
0.75
LFM2 vs. PaliGemma2
0.71
−
0.77
−
Qwen2 vs. PaliGemma2
0.72
−
0.74
−
Table 2: Partition Consistency Score.
VLM
Cond.
ADE ID
ADE OOD
ΔADE
LFM2
(A) AV2-local
1.4304
4.0812
2.6508
(B) Zero-shot
1.4318
4.0921
2.6603
(C) Source-only
1.8732
5.1247
3.2515
Qwen2
(A) AV2-local
1.5821
4.2153
2.6332
(B) Zero-shot
1.5876
4.2401
2.6525
(C) Source-only
1.8123
4.9534
3.1411
Table 3: Zero-shot partition transfer.
Variant
LFM2
Qwen2
PaliGemma2
MoLaRx (full)
2.3687
2.6287
2.4521
w/o CIS
2.7259
2.9851
2.8523
DPA-partition
2.6742
2.8643
2.7234
w/o MMD on Φalign
2.6108
2.8214
2.7892
w/o LoRA on Φtune
2.6821
2.8925
–
w/o Freeze on Φfreeze
2.3714
2.6523
2.4612
Table 4: Ablation ΔADE↓ (m) on nuScenes.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Value
LFM2-VL-1.6B
Decoder layers L
16
Attention connectivity
causal
Backbone operator
LTC-hybrid
CIS threshold τimg
0.31 m
Φalign
{0,1}
Appendix
Table 5: Per-VLM architecture and CIS-induced partition. τimg is chosen at the elbow of the sorted CISimg curve on nuScenes and reused on AV2 without recalibration. On PaliGemma2 the prefix-LM attention yields Φtune=∅ , so the CISR LoRA budget shifts from Φtune to Φalign .
Category
Setting
Datasets
nuScenes source
Boston-Seaport ( 700 scenes)
nuScenes target
Singapore ( 150 scenes)
nuScenes keyframes
∼26,600 train / 3,569 ID / 3,500 OOD
Argoverse 2 source
Pittsburgh ( 700 logs)
Argoverse 2 target
Miami ( 150 logs)
Appendix
Table 6: Experiment settings and hyperparameters for MoLaRx.
Dataset
VLM
ImgDep (single-frame + text)
ImgDep (ours)
Change
nuScenes (BS → SG)
LFM2
−0.60 %
+442%
+442.6 pp
Qwen2
−0.42 %
+360%
+360.4 pp
PaliGemma2
−0.52 %
+287%
+287.5 pp
Argoverse 2 (PIT → MIA)
LFM2
−0.52 %
+295 %
+295.5 pp
Qwen2
−0.32 %
+172%
+172.4 pp
PaliGemma2
−0.48 %
+198%
+198.2 pp
Appendix
Table 7: Visual-grounding fine-tuning restores genuine visual reliance on every cell.
Figure 4: Trajectory comparison on representative target-city scenes (LFM2-VL, real model rollouts projected onto the front-camera image). (a) nuScenes BS → SG, scene-0015 (Singapore one-north arterial, ego 5.5 m/s) on CAM_FRONT. (b) AV2 PIT → MIA, scene 0b97f5dd (Wynwood-Miami arterial, ego 8.5 m/s) on ring_front_center. All five LFM2-VL methods plus GT (black dashed) projected with z=0 ground-plane assumption.
Figure 5: Cross-city consistency on a SigLIP2-matched scene pair (LFM2-VL). (a) Boston source scene-0669 (ego 7.3 m/s); (b) Singapore target scene-0369 (ego 7.9 m/s). Cross-city cosine 0.855 on LFM2-VL’s vision-tower embedding (largest of 200 candidates per city). All five methods plus GT (black dashed) projected onto the front-camera image with z=0 ground-plane assumption.
Figure 6: CIS-threshold sensitivity on nuScenes. Top row (a)–(c): τimg sweep across the three VLMs. Bottom row (d)–(e): τtxt sweep on the two causal-decoder VLMs; PaliGemma2 is omitted because Φtune=∅ on prefix-LM. Red star = elbow value chosen by our default elbow-finder.
Figure 7: λMMD sensitivity across three VLMs on nuScenes. Each panel sweeps λMMD∈[0,5] with the CIS-induced partition fixed at its default values, and reports ΔADE . The red star marks the default λMMD=0.5 .
Nframe
ΔADE (m) on nuScenes
ImgDep (%)
GPU mem
Latency
LFM2-VL
Qwen2-VL
PaliGemma2
(LFM2)
(GB)
(ms)
1
4.15
3.91
3.34
−0.6
4.8
42
2
3.65
3.41
2.84
+8.4
6.2
51
3 (default)
3.15
2.91
2.34
+18.2
8.1
63
4
3.13
2.89
2.32
+18.9
11.5
79
5
3.12
2.88
2.31
+19.5
16.3
98
Appendix
Table 8: Nframe sensitivity on nuScenes (BS → SG). Cross-city ΔADE per VLM, ImgDep on LFM2-VL (signal-collapse indicator, App. B ), and per-step GPU memory and latency on a single A10 (LFM2-VL backbone). Nframe=3 (bold) is the operating point used in all main-paper experiments.
N
Empirical ε^ (m)
PCS vs. N=500
ΔADE on AV2 (m)
CIS time (h)
25
1.42 ( ±0.18 )
0.62
2.105 ( ±0.041 )
0.18
50
0.95 ( ±0.11 )
0.81
1.945 ( ±0.029 )
0.35
100 (default)
0.55 ( ±0.06 )
0.94
1.890 ( ±0.024 )
0.65
200
0.35 ( ±0.04 )
0.98
1.882 ( ±0.022 )
1.30
500
0.18 ( ±0.02 )
1.00 (ref)
1.880 ( ±0.021 )
3.20
Appendix
Table 9: CIS pair-budget sensitivity on LFM2/nuScenes. ε^ : empirical standard error of Ψ[m,l] across 5 random sub-samples of N pairs (averaged over all (m,l) entries). PCS: zone-wise Jaccard against the N=500 asymptotic partition (Eq. 21 ). ΔADE : zero-shot CISR on AV2 (PIT → MIA) using the partition diagnosed at the given N on nuScenes, five-seed std in parentheses. CIS time on a single A10 GPU. N=100 (bold) is the operating point used in all main-paper experiments.
VLM
Method
Train. params
% saved
Train (h/ep)
Inference (ms/sample)
LFM2-VL-1.6B
Full-LoRA
7.6 M
0%
4.2 ( ±0.05 )
66.5 ( ±0.8 )
MoLaRx (CISR)
3.8 M
50%
4.1 ( ±0.04 )
47.4 ( ±0.6 )
Qwen2-VL-2B
Full-LoRA
8.9 M
0%
4.6 ( ±0.06 )
79.8 ( ±1.1 )
MoLaRx (CISR)
4.1 M
54%
4.5 ( ±0.05 )
50.5 ( ±0.7 )
PaliGemma2-3B
Full-LoRA
12.5 M
0%
5.4 ( ±0.07 )
81.4 ( ±1.0 )
MoLaRx (CISR) †
5.0 M
60%
5.3 ( ±0.06 )
62.1 ( ±0.9 )
Appendix
Table 10: Compute footprint per VLM. “% saved” is the parameter saving over each VLM’s Full-LoRA baseline. Not shown : a one-time CIS-diagnosis cost of ∼0.65 h per architecture (App. C.2.4 ), reused zero-shot across the two benchmarks (Table 3 ). † On PaliGemma2 the prefix-LM attention yields Φtune=∅ , so the recipe collapses to MMD-driven attention-only LoRA on Φalign + freeze on Φfreeze . MoLaRx saves 50 – 60% of trainable parameters across all three architectures; the largest absolute saving ( 7.5 M, 60% ) is on PaliGemma2, where attention-only LoRA on Φalign avoids the MLP-projection bulk that Full-LoRA pays.
Method
Adaptation pool
ADE ID
FDE ID
ADE OOD
FDE OOD
Source-only
none
1.472
2.705
4.466
7.754
MoLaRx
transductive
1.116
2.014
4.266
7.462
MoLaRx
inductive
1.255
2.262
4.273
7.492
Appendix
Table 11: Inductive-protocol control (LFM2-VL, nuScenes BS → SG; same seed; fixed 300 -sample ID/OOD subsets).
Scenario type
n
Source-only
Full LoRA
Full LoRA + MMD
MoLaRx
Specifically selected
18
9.47
8.48
9.61
9.91
Heavy rain
10
19.41
17.97
18.95
18.62
Construction zone
6
9.95
7.74
8.31
8.60
Overtake or lane change
20
25.76
24.19
24.14
24.18
Intersection
35
7.19
6.09
5.91
5.77
Night-time
5
42.85
40.63
42.69
41.12
Appendix
Table 12: Zero-shot ADE (m) on KITScenes-LongTail by scenario type (LFM2-VL planners trained on nuScenes-Boston; single seed). The last row gives overall ADE / FDE.
Method
Closest-mode acc. ↑
Mean mode margin (m) ↑
Source-only
0.25
−1.08
Full LoRA
0.29
−0.98
Full LoRA + MMD
0.28
−1.02
MoLaRx
0.29
−0.98
Appendix
Table 13: MMS-inspired closest-mode accuracy and mean mode margin on KITScenes-LongTail ( 100 scenarios; same zero-shot setting as Table 12 ).
Method
ADE ID
Lat. ID
Jerk ID
ADE OOD
Lat. OOD
Jerk OOD
Source-only
1.67
0.65
0.77
4.86
0.96
0.87
Full LoRA
1.27
0.56
1.12
4.52
1.16
1.30
Full LoRA + MMD
1.20
0.58
1.73
4.54
1.15
2.04
MoLaRx
1.10
0.52
1.31
4.66
1.14
1.64
Appendix
Table 14: Planning-oriented proxies on nuScenes BS → SG (LFM2-VL; fixed 300 -sample subsets; single seed). Lat.: lateral error at 3 s (m); jerk: mean absolute jerk (m/s 3 ); the ground-truth jerk is 1.77 (ID) and 2.27 (OOD).
VLM
Cell
Reverse partition
Agree
PCSalign
PCStune
PCSfreeze
Mean
LFM2-VL
SG → BS
{0,1}/{2:9}/{10:15}
16/16
1.00
1.00
1.00
1.00
LFM2-VL
MIA → PIT
{0,1}/{2:9}/{10:15}
16/16
1.00
1.00
1.00
1.00
Qwen2-VL
SG → BS
{0,1,17}/{2:15}/{16,18:27}
20/28
0.40
0.50
0.65
0.52
PaliGemma2
SG → BS
{0:23}/{24,25}/∅
24/26
0.96
0.00
0.00
0.32
Appendix
Table 15: Zone partition under reversed transfer direction (single seed), written as Φalign/Φtune/Φfreeze . PCS is the zone-wise Jaccard against the forward partition; “agree” counts layers assigned to the same zone.
Method
LFM2-VL, SG → BS
Qwen2-VL, SG → BS
LFM2-VL, MIA → PIT (AV2)
Source-only
3.52 / 4.12 (0.60)
3.29 / 4.30 (1.01)
10.90 / 11.61 (0.71)
Full LoRA
2.56 / 3.61 (1.05)
2.78 / 4.11 (1.33)
5.12 / 4.86 ( −0.26 )
MoLaRx
2.79 / 3.90 (1.12)
2.89 / 4.19 (1.30)
4.72 / 5.39 (0.67)
Appendix
Table 16: Reverse-direction repair comparison: ADE (m) on fixed 300 -sample subsets, ID / OOD ( ΔADE ). Single seed; visual grounding truncated to approximately two epochs in the nuScenes cells; AV2 values are from a separate harness and comparable only within the block.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.
Dominick Reilly, Manish Kumar Govind, Le Xue +1
University of North Carolina at Charlotte · Elorian AI
Vision-Language-Action (VLA) models often fail to perform the same learned tasks under environmental shifts, such as changes in camera pose and shifts to a different but similar robot (e.g., from Panda to UR5e). Adapting these models to the shifted environment (i.e., target domain) often requires training on multiple demonstrations for each task, which are costly to collect. To reduce the burden of data curation and training, we propose an analogy-based method that adapts VLA models under environmental shifts through weight vector arithmetic with domain-specific information addition, named Domain ARiThmetic (DART). Unlike prior approaches, DART requires collecting only a single demonstration, enabling efficient adaptation. To accurately isolate domain-specific information for addition, DART performs subspace alignment between singular components in weight vectors to filter out noisy components. In both simulated and real-world experiments, DART outperforms existing VLA adaptation methods in one-shot scenarios across diverse visual and embodiment shifts. Code is available at https://github.com/snumprlab/dart.
Taewook Kang, Taeheon Kim, Donghyun Shin +1
Seoul National University · ECE, IPAI and ASRI in SNU
We present a system that uses a Vision-Language Model (VLM) as a diagnostic agent for adapting a detect-to-track pipeline to a new target domain without access to target-domain labels. Rather than optimizing against annotated metrics, the VLM directly inspects rendered tracking outputs, identifies visual failure modes, and recommends parameter updates through an iterative tuning loop. We first demonstrate that ground-truth-supervised hyperparameter transfer can be brittle. On MOT17->MOT20, applying a source-derived oracle configuration reduces mean HOTA by 0.090, from a target-domain ceiling of 0.357, to 0.267. Without using any target-domain labels, our VLM-based tuner recovers 67.8% of this lost headroom, finishing within 0.029 HOTA of the target ceiling; on the highest-density target sequence, it recovers up to 86.7%. We further show that label-free Bayesian optimization with handcrafted proxy objectives struggles under large domain shifts and can degrade configurations that are already strong. In contrast, the VLM tuner acts selectively: when its visual diagnosis reveals no clear failure mode, it declines to modify the configuration, preserving performance on easy transfers while improving hard ones. Finally, we characterize the conditions under which this approach succeeds, namely, when domain shift manifests through exposed detection-level parameters, versus where it is less effective, such as MOT17->DanceTrack, where the source oracle is already near-optimal.
Daniel Davila, Ravikumar Balakrishnan, Mike Cochran