Cross-Domain Tracker Adaptation Without Target-Domain Labels via Vision-Language Agents

Daniel Davila; Ravikumar Balakrishnan; Mike Cochran

cs.CVSep 4, 2026

Cross-Domain Tracker Adaptation Without Target-Domain Labels via Vision-Language Agents

Authors: Daniel Davila, Ravikumar Balakrishnan, Mike Cochran

Organizations: Cisco Systems, Inc. San Jose, CA

Abstract

We present a system that uses a Vision-Language Model (VLM) as a diagnostic agent for adapting a detect-to-track pipeline to a new target domain without access to target-domain labels. Rather than optimizing against annotated metrics, the VLM directly inspects rendered tracking outputs, identifies visual failure modes, and recommends parameter updates through an iterative tuning loop. We first demonstrate that ground-truth-supervised hyperparameter transfer can be brittle. On MOT17->MOT20, applying a source-derived oracle configuration reduces mean HOTA by 0.090, from a target-domain ceiling of 0.357, to 0.267. Without using any target-domain labels, our VLM-based tuner recovers 67.8% of this lost headroom, finishing within 0.029 HOTA of the target ceiling; on the highest-density target sequence, it recovers up to 86.7%. We further show that label-free Bayesian optimization with handcrafted proxy objectives struggles under large domain shifts and can degrade configurations that are already strong. In contrast, the VLM tuner acts selectively: when its visual diagnosis reveals no clear failure mode, it declines to modify the configuration, preserving performance on easy transfers while improving hard ones. Finally, we characterize the conditions under which this approach succeeds, namely, when domain shift manifests through exposed detection-level parameters, versus where it is less effective, such as MOT17->DanceTrack, where the source oracle is already near-optimal.

Figures & tables

Figure 1: Cross-domain tracker adaptation without target-domain labels. (Left) A MOT20-03 frame under MOT17-tuned parameters, and then again after VLM-guided adaptation on the target sequence without target-domain labels. Tuning increases number of persons tracked in difficult scene. (Right) Per-sequence HOTA on MOT20: our VLM-tuned configuration (blue) closes most of the gap between oracle transfer (red) and the target per-sequence oracle ceiling (gray hatched), with the largest gains on the hardest sequences MOT20-03/05.

Figure 2: Overview of VLM Tracker Tuner for cross-domain tracker adaptation without target-domain ground truth.

Figure 3: Detection-phase visual diagnostic suite. Each window of tracker output is rendered into five panel types delivered to the VLM as a single multimodal prompt: a full-scene dot panel and a zoomed dot panel (left) for recall evidence, above- and below-threshold collages (right) for precision and quality-cliff evidence, and an NMS-orphaned strip (bottom right) attributing suppressed detections to nms_thresh .

Figure 5: Tracking-phase visual diagnostic suite. Three panel types per window: (left) the scene/dot panel for recall evidence; (middle) track-break visualizations showing spawns near a dead-track endpoint to test whether a track was broken and re-established; and (right) track-approach visualizations showing likely crossover points, to check for ID switching.

Failure mode	Primary parameter response
id_switch	$\downarrow$ match_thresh
ghost_tracks	$\uparrow$ min_hits , $\uparrow$ spawn_thresh
fragmented_tracks	$\uparrow$ max_lost_age ; then $\downarrow$ velocity_decay
frequent_id_switches	$\uparrow$ match_thresh , $\downarrow$ max_lost_age
track_drift	adjust velocity_decay

Table 1: Tracking-phase failure-mode-to-parameter mapping. Arrows indicate direction; magnitude is sized by lifecycle statistics (death

\rightarrow

birth pairs, conf-at-death sequences). Secondary levers and fallback rules omitted; see released prompts.

Oracle Source	Target	Target Oracle	Oracle xfer	BO-proxy	VLM (ours)
MOT17	MOT20	0.357	0.267	0.305	0.328
MOT20	MOT20	0.357	0.338	0.322	0.349
DT	MOT20	0.357	0.312	0.305	0.326
MOT17	DT	0.501	0.467	0.431	0.452
MOT20	DT	0.501	0.428	0.412	0.441
DT	DT	0.501	0.460	0.402	0.477

Table 2: Cross-domain transfer and recovery. We compute a source domain configuration oracle for each of MOT17, MOT20 and DanceTrack (DT). Each row shows four metrics for each combination of source→target domain transfer: The oracle computed on the target domain (near-optimal ceiling for that domain); the results of transferring the source domain oracle to that new domain; the results of a Bayesian Optimization of the new target domain using the proxy metric in Equation 1 ; and finally the results of allowing the VLM to optimize tracker performance on the new domain, starting from the transferred source oracle. Our VLM-based tracker tuner excels at overcoming large domain shifts, such as between MOT17→MOT20 and MOT17/20→DanceTrack, while avoiding catastrophic regressions for transfers where the starting conditions are already near optimal such as all →MOT17 transfers. For same-domain rows (e.g. MOT17

\rightarrow

MOT17), oracle xfer applies a single configuration pooled across the source sequences, whereas target oracle averages per-sequence fits; the diagonal gap between them therefore reflects per-sequence specialization, not domain shift.

Source	Condition	01	02	03	05	Mean
MOT17	Oracle transfer	0.427	0.356	0.190	0.095	0.267
MOT20	Oracle transfer	0.446	0.388	0.312	0.206	0.338
DanceTrack	Oracle transfer	0.424	0.374	0.289	0.160	0.312
MOT17	VLM from oracle	0.429	0.360	0.327	0.195	0.328
MOT20	VLM from oracle	0.446	0.393	0.312	0.247	0.349
DanceTrack	VLM from oracle	0.407	0.369	0.333	0.195	0.326

Table 3: MOT20 domain transfer per sequence (HOTA). The first three rows show the performance of various source oracles transferred to each of the MOT20 video sequences, which HOTA numbers displayed across the columns. The bottom 3 rows show the results of running our VLM tracker tuner from these oracle transfer starting points, recovering performance significantly on the more complex scenes (MOT20-03 and -05). Bold marks the best non-ceiling result per column.

M17 $\rightarrow$ M20	01	02	03	05	Mean
Oracle xfer	.427	.356	.190	.095	.267
VLM (m $\pm\sigma$ )	.406 $\pm$ .006	.346 $\pm$ .006	.320 $\pm$ .017	.192 $\pm$ .002	.316 $\pm$ .005

Table 4: We re-run the M17

\rightarrow

M20 5 times to determine whether the performance gains at temperature 1 are noise. The largest gains, in MOT20-03 and -05, clear

2\sigma

while the simpler videos -01 and -02 a stable tendency not to drift parameters away from good starting points.

Condition	01	02	03	05	Mean
Transfer from MOT17 Oracle	0.427	0.356	0.190	0.095	0.267
+ Detection agent only	0.398	0.335	0.312	0.190	0.309
+ Tracking agent only	0.402	0.346	0.230	0.131	0.277
+ Both agents	0.429	0.360	0.327	0.195	0.328
Target per-sequence oracle	0.459	0.372	0.348	0.248	0.357

Table 5: Agent Ablation: MOT17

{\to}

MOT20 transfer. All VLM conditions start from the MOT17 oracle config (same as oracle transfer). Detection agent only tunes

\{\mathrm{conf\_thresh},\mathrm{nms\_thresh}\}

; tracking agent only tunes

\{\mathrm{match\_thresh},\mathrm{spawn\_thresh},\mathrm{max\_lost\_age},\mathrm{velocity\_decay},\mathrm{min\_hits},\mathrm{appearance\_thresh}\}

. The detection agent is responsible for the majority of the gain (

+

0.042 of

+

0.061 total), concentrated on the two hardest sequences (MOT20-03/05) where conf mismatch is catastrophic. The tracking agent contributes

+

0.010 in isolation and a marginal

+

0.019 when applied after detection, and the agents compose near-additively (

0.042+0.010\approx 0.061

). Bold marks the best non-ceiling result per column.

Condition	01	02	03	05	Mean
Oracle transfer	0.427	0.356	0.190	0.095	0.267
VLM text-only	0.407	0.348	0.235	0.131	0.281
VLM visual-only	0.449	0.340	0.312	0.219	0.330
VLM text+vision	0.429	0.360	0.327	0.195	0.328
Target oracle (ceiling)	0.459	0.372	0.348	0.248	0.357

Table 6: Signal modality ablation on MOT17

\rightarrow

MOT20. Text-only (scalars without visual panels) barely improves on oracle transfer; visual-only (panels without scalars) matches the full system within noise. The text-only agent improves the hardest sequences modestly via tracking-side intervention but cannot diagnose the detection-stage failures that drive the cross-domain gap. Bold marks the best non-ceiling result per column.

Condition	04	05	07	10	19	34	47	58	Mean
MOT17 Oracle	0.51	0.48	0.51	0.61	0.32	0.38	0.33	0.59	0.47
+ Det agent	0.34	0.56	0.50	0.47	0.27	0.38	0.30	0.55	0.42
+ Tracking agent	0.39	0.56	0.49	0.47	0.28	0.36	0.37	0.44	0.42
+ Both agents	0.35	0.43	0.50	0.61	0.34	0.39	0.40	0.59	0.45

Table 7: Agent Ablation: MOT17

{\to}

DanceTrack transfer (HOTA). All VLM conditions start from the MOT17 oracle config. Unlike the MOT20 case, neither agent alone improves on oracle transfer (

-

0.046 each): the MOT17 oracle (match

=

0.90, age

=

15) is already near-optimal for DanceTrack’s fast motion, so each isolated agent perturbs away from it without sufficient feedback to self-correct. The full two-phase loop limits the regression to

-

0.015 through inter-phase correction. DT-0047 is the revealing exception: the tracking agent alone yields

+

0.033 over oracle transfer (0.367 vs. 0.334) while the detection agent hurts (

-

0.033). This sequence has reliable detections but identity fragmentation. The tracking agent correctly tightens match/spawn, but the detection agent incorrectly intervenes on conf and destroys the gain. Bold marks the best VLM result per column.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models
Oct 15, 2025Dominick Reilly, Manish Kumar Govind, Le Xue +1Domain Adaptation VLM Adaptation
University of North Carolina at Charlotte · Elorian AI
Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking
Jun 28, 2026Xiao Wang, Liye Jin, Dan Xu +5Vision-Language Models Visual Object Tracking
School of Computer Science and Technology, Anhui University, Hefei 230601, China · School of Electronic and Information Engineering, Anhui University, Hefei 230601, China · Harbin Institute of Technology, Shenzhen, China +3
ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection
Mar 12, 2026Mattia Bernardi, Chiara Cappellino, Matteo Mosconi +3Open-Vocabulary Object Detection Vision Foundation Model Adaptation
University of Modena and Reggio Emilia

Oct 15, 2025J/K move · Enter open · S save

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

Dominick Reilly, Manish Kumar Govind, Le Xue, Srijan Das

University of North Carolina at Charlotte · Elorian AI

Domain Adaptation VLM Adaptation VLMs for Robotics Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.

Abstract

Multi-Object Tracking

Explore similar work

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection