cs.CVNov 7, 2026

Locating and Repairing Domain Shift in VLM Trajectory Planning

Authors: Zhihong Cui, Hengyu Liu, Michael Alexander Riegler, Guandong Xu, Amir Taherkordi, Tor Skeie

Organizations: Department of Informatics, University of Oslo, Norway · Department of Computer Science, Aalborg University, Denmark · Simula Metropolitan Center for Digital Engineering (SimulaMet), Norway · The Education University of Hong Kong, Hong Kong, China · 

Abstract

Vision-Language Models (VLMs) are increasingly adopted as end-to-end driving planners, yet they suffer from cross-city domain shift. Existing methods treat this shift as a single quantity addressed by uniform alignment or single-site editing. We challenge this view on two counts: (i) domain shift in VLM trajectory planners is inherently structured across modality and layer, and (ii) a locate-then-repair recipe matched to this structure outperforms every uniform-objective baseline. We analyse domain shift via activation patching and uncover two regularities. (F1) Modality axis. Image and text tokens exhibit distinct shift patterns in each VLM. (F2) Layer axis. The layer-wise shift partition is architecture-dependent and dataset-stable. These findings recast domain shift as a two-dimensional tensor indexed by modality (image vs. text tokens) and layer, motivating a simple principle: locate where shift occurs, and repair only there. We instantiate this principle in MoLaRx (Modality-Layer Repair), a locate-then-repair framework. The Causal Importance Score (CIS) estimates the domain shift as a modality-layer tensor and partitions the layers into functional zones. CIS-Guided Selective Repair (CISR) then applies a matched operator (Maximum Mean Discrepancy (MMD) / low-rank adaptation (LoRA) / freezing) to repair each zone, with the composition adapting automatically to each architecture. We evaluate MoLaRx on three architecturally distinct VLMs (LFM2-VL-1.6B, Qwen2-VL-2B, and PaliGemma2-3B) across two cross-city benchmarks (nuScenes and Argoverse 2). MoLaRx attains the lowest cross-city trajectory-error gap among adaptation methods in all six main-table settings, while using 50-60% fewer trainable parameters and 8.4-10.9 ms lower inference latency. In contrast, uniform MMD-only adaptation collapses in all six settings, highlighting the importance of structure-guided repair.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

    Oct 15, 2025Dominick Reilly, Manish Kumar Govind, Le Xue +1Domain AdaptationVLM Adaptation

  2. Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts

    Jul 1, 2026Taewook Kang, Taeheon Kim, Donghyun Shin +1Domain AdaptationVision-Language-Action Models

  3. Cross-Domain Tracker Adaptation Without Target-Domain Labels via Vision-Language Agents

    Sep 4, 2026Daniel Davila, Ravikumar Balakrishnan, Mike CochranMulti-Object TrackingUnsupervised Domain Adaptation