DISTRACT: Degradation-Indexed Study of Textual Reference Across Competing Targets

Dan Zlotnikov; Ohad Ben-Shahar

cs.CVOct 20, 2026

DISTRACT: Degradation-Indexed Study of Textual Reference Across Competing Targets

Authors: Dan Zlotnikov, Ohad Ben-Shahar

Organizations: Ben-Gurion University of the Negev, Israel

Abstract

Vision-language trackers (VLTs) accept descriptions intended to assist tracking, but accepting text does not establish that its content changes tracking decisions, particularly as visual evidence deteriorates. We introduce DISTRACT, a matched-intervention benchmark that holds video, target, and distractor fixed while crossing four text priors with four image degradations along a controlled severity axis. We evaluate five VLTs on four datasets using behavioral contrasts, layerwise representation probes, and activation patching. Molmo2 and JointNLT exhibit directed text use. Some effects contract under degradation, whereas others show no significant endpoint change after multiplicity correction. UVLTrack, MMTrack, and DUTrack respond only weakly to changes in text semantics, with no directed behavioral effect surviving FDR correction in the tested settings. Model-internal analyses reveal that at decision-near states, correct-versus-conflicting text is decodable in Molmo2 and JointNLT but not in the other three systems. In the responsive models, decodability peaks at late states where replacement produces little behavioral change, whereas replacing selected earlier states redirects conflict-associated trajectories toward the target. Because matched-correct donors do not significantly outperform unrelated donors, these interventions identify earlier intervention-sensitive states without establishing target-specific semantic rescue. DISTRACT thus exposes when and where text affects tracking under degraded visual evidence.

Figures & tables

Figure 1: A real MeViS target–distractor pair (green = target; red = distractor) at the clean initialization frame and the terminal severity ( $S_{5}$ ) of blur, noise, pixelation, and compression. Descriptions: target, “girl in red sitting on the right side of the table”; distractor, “The can next to the cake”.

Dataset	Characterization	Pairs	Seq.
MeViS	referring seg., masks	120	120
DanceTrack	dance MOT, similar look	60	13
SportsMOT	athlete MOT, fast motion	30	19
PersonPath22	pedestrian MOT, crowded	30	20

Table 1: Per-dataset pair and source-video coverage. Pairs is the number of selected target–distractor pairs. Seq. is the number of distinct source videos selected.

Perturbation	Fixed schedule values, $S0$ – $S5$
Blur, $k_{s}$	$0,\ .004,\ .008,\ .016,\ .032,\ .064$
Noise, $\sigma_{s}$	$0,\ .12,\ .20,\ .30,\ .45,\ 1.5$
Pixelation, $r_{s}$	$1,\ .35,\ .25,\ .18,\ .12,\ .04$
Compression, $q_{s}$	no-op $,\ 15,\ 10,\ 7,\ 3,\ 1$

Table 2: Fixed severity schedule.

	Molmo2			JointNLT			UVLTrack			MMTrack			DUTrack
Dataset	$C_{T}$	$B_{T}$	$N_{T}$	$C_{T}$	$B_{T}$	$N_{T}$	$C_{T}$	$B_{T}$	$N_{T}$	$C_{T}$	$B_{T}$	$N_{T}$	$C_{T}$	$B_{T}$	$N_{T}$
DanceTrack	$\underline{+25\%}$	$\underline{+13\%}$	$\underline{+8\%}$	$\underline{+11\%}$	$\underline{+11\%}$	$+6\%$	$+1\%$	$-1\%$	$+2\%$	$-1\%$	$+0\%$	$+0\%$	$+0\%$	$+1\%$	$+0\%$
SportsMOT	$+6\%$	$+0\%$	$+4\%$	$-2\%$	$\underline{-11\%}$	$-6\%$	$-1\%$	$+1\%$	$-1\%$	$+1\%$	$-2\%$	$-4\%$	$-1\%$	$-2\%$	$+0\%$
PersonPath22	$+14\%$	$+1\%$	$+4\%$	$\underline{+27\%}$	$\underline{+18\%}$	$\underline{+17\%}$	$+5\%$	$\underline{+5\%}$	$+5\%$	$+2\%$	$+1\%$	$\underline{+3\%}$	$+0\%$	$-2\%$	$\underline{+0\%}^{\dagger}$
MeViS	$\underline{+8\%}$	$\underline{+4\%}$	$+2\%$	$\underline{+14\%}$	$\underline{+11\%}$	$\underline{+8\%}$	$+2\%$	$+1\%$	$-1\%$	$+0\%$	$+0\%$	$+1\%$	$+0\%$	$+0\%$	$+0\%$

Table 3: Clean-input (

S_{0}

) text effects for all five systems on all four datasets. Underlined: 95% source-video cluster-bootstrap CI excludes zero. Among UVLTrack/MMTrack/DUTrack: 0/12

C_{T}

, 1/12

B_{T}

, 2/12

N_{T}

CIs exclude zero; the sole

B_{T}

case (UVLTrack/PersonPath22) fails the clustered sign-flip test (

p=.08

, BH

q=.668

), so is not treated as a reproducible effect. † Near-zero-variance bootstrap artifact, not interpreted.

Figure 2: Representative compression trajectories on PersonPath22. Molmo2 (left) separates text conditions in target accuracy and distractor capture; DUTrack (right) remains nearly unchanged.

Figure 3: Mean severity-integrated absolute target-accuracy effect ( $\mathrm{IAD}(c_{1},c_{2})$ , Eq. 4 ) across all four perturbations and four datasets, per model. Top row: JointNLT, Molmo2. Bottom row: UVLTrack, MMTrack, DUTrack, same color scale. Condition labels abbreviate neutral/unrelated/correct/conflicting.

Contraction level (of 4 perturbation ramps)	Cases (of 14)
4/4	1
1–3/4	3
0/4	10

Table 4: FDR-significant contraction toward zero, across the 14 target-accuracy cases with a CI-supported clean-input signal.

Model	Decision-near stage	Text BA	95% CI	$q$
Molmo2	final coord. query (L36)	$0.574$	[ $0.543$ , $0.605$ ]	$0.015$
JointNLT	late-fusion $T{-}D$	$0.695$	[ $0.647$ , $0.744$ ]	$0.002$
UVLTrack	pre-box-head search	$0.530$	[ $0.505$ , $0.554$ ]	$0.174$
MMTrack	coord. decoder input	$0.488$	[ $0.469$ , $0.509$ ]	$0.898$
DUTrack	pre-CENTER-head visual	$0.504$	[ $0.499$ , $0.512$ ]	$0.506$

Table 5: Correct-vs-conflicting Text BA (

0.5

=chance) at each system’s decision-near hooked stage (115-video MeViS manifest). Underlined: FDR-significant (

q<.05

Figure 4: Exhaustive correct-vs-conflicting Text BA sweeps for the three behaviorally small-effect systems. Colors denote functional stage: text encoder/projection (blue), pre-fusion vision (gray), multimodal fusion (green), and localization/pre-box-head (purple). Filled circles mark FDR-significant states; titles report the number of FDR-surviving states and the total evaluated.

Figure 5: Exhaustive correct-vs-conflicting Text BA sweeps for Molmo2 and JointNLT. Solid trajectory segments indicate the functional state at their left endpoint: text encoder/projection (blue), pre-fusion vision (gray), multimodal fusion (green), and localization/pre-box-head (purple).

Figure 6: Text-condition decodability and matched-correct state-replacement effect across depth. Red, left ordinate: held-out Text BA. Blue, right ordinate: mean paired change in target–distractor margin after replacing a conflicting-run state with its same-video correct-run state. Both evaluated on the persistent-capture stratum. Dashed horizontal lines mark chance Text BA and zero replacement effect.

Figure 7: Matched-correct state replacement redirects conflict-associated trajectories in both responsive systems, each shown at its best-recovery site; sites maximize the observed capture-stratum effect on the displayed data. Each line connects the unpatched conflicting run with its correct-text-state patch; blue: distractor-to-target transition, red: none.

Figure 8: JointNLT’s pre-localization state–output association (Pearson $r$ ) across $S_{0}$ – $S_{5}$ ; dashed line: zero.

Perturbation	late vis. sep.	late fused-text BA	output assoc. $r$
blur	$.173$	$.582$	$-.009$ [ $-.111,.089$ ]
noise	$.042$	$.566$	$\underline{+.268}$ [ $.160,.363$ ]
pixelation	$.148$	$.539$	$+.091$ [ $-.051,.225$ ]
compression	$.253$	$.577$	$\underline{+.254}$ [ $.145,.353$ ]

Table 6: Terminal

S_{5}

JointNLT summaries. Underlined: CI excludes zero.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

System	Native operation	Pinned artifact used	Source / download URL	License status
Molmo2	8B VLM; clean point plus native tracking prompt $\rightarrow$ point trajectory	allenai/Molmo2-8B , revision e28fa285	https://huggingface.co/allenai/Molmo2-8B https://github.com/allenai/molmo2	Apache-2.0
JointNLT	BERT text encoder and visual–language fusion; clean box $\rightarrow$ box trajectory	commit 94b338e1 ; JointNLT_ep0300. pth.tar ; swin_b_ep300_track	https://github.com/lizhou-cs/JointNLT https://drive.google.com/drive/folders/ 1ZKBDhgXgV1GLSzFxw5fqpqd ZuX1hjfQe?usp=share_link	MIT
UVLTrack	Language-conditioned box tracker; clean box and raw expression $\rightarrow$ box trajectory	commit 6ca34055 ; UVLTrack_ep0300. pth.tar (baseline-base)	https://github.com/OpenSpaceAI/UVLTrack https://drive.google.com/drive/folders/ 1UZTrGcL3YlxvNpHi0wKsO_s KsTYuYnFo?usp=sharing	MIT
MMTrack	Language-conditioned multimodal tracker; clean box and raw expression $\rightarrow$ box trajectory	commit 4586b013 ; MMTrack_ep0150. pth.tar (baseline)	https://github.com/Azong-HQU/MMTrack https://drive.google.com/file/d/ 1kM8qxwJHJg4HuZNPPTMFZZTG EdtASVAf/view?usp=sharing	MIT
DUTrack	Description-conditioned box tracker; static external expression plus clean box $\rightarrow$ box trajectory	commit 490d9985 ; DUTrack_ep0045. pth.tar ( dutrack_256_full )	https://github.com/GXNU-ZhongLab/ DUTrack https://drive.google.com/drive/folders/ 1edieIddvSzy9219F4WShajv Y4UmkKJV4?usp=sharing	No license file in pinned repo

Appendix

Table 1: Models, pinned artifacts, upstream source URLs, and stated license status.

Dataset	Source media and annotations used	Access URL(s)	License / use terms
MeViS	120 pairs from 120 source videos. Native human-written motion expressions and masks provide both text and geometry. Source media are obtained from the upstream release.	https://github.com/henghuiding/MeViS	Upstream data: CC BY-NC-SA 4.0 https://creativecommons. org/licenses/ by-nc-sa/4.0/ ; not redistributed here
DanceTrack	60 pairs. DanceTrack provides source video and identities; Molmo2-VideoTrackEval provides human-written expressions and point/mask trajectories.	https://dancetrack.github.io/ https://huggingface.co/datasets/allenai/ Molmo2-VideoTrackEval	Source video: non-commercial research; DanceTrack notes that Internet-sourced media are not project property. Molmo2-VideoTrackEval expressions/masks/points: ODC-BY-1.0
SportsMOT	30 pairs. SportsMOT provides source video and identities; Molmo2-VideoTrackEval provides human-written expressions and point/mask trajectories.	https://github.com/MCG-NJU/SportsMOT https://huggingface.co/datasets/allenai/ Molmo2-VideoTrackEval	Source video: CC BY-NC 4.0 https://creativecommons. org/licenses/ by-nc/4.0/ Molmo2-VideoTrackEval expressions/masks/points: ODC-BY-1.0
PersonPath22	30 pairs. PersonPath22 provides source video and identities; Molmo2-VideoTrackEval provides human-written expressions and point/mask trajectories.	https://github.com/amazon-science/ tracking-dataset https://huggingface.co/datasets/allenai/ Molmo2-VideoTrackEval	Source release: CC BY-NC 4.0; some externally sourced videos are not redistributed by the provider. Molmo2-VideoTrackEval expressions/masks/points: ODC-BY-1.0

Appendix

Table 2: Dataset sources, annotation provenance, access URLs, and applicable upstream terms. Molmo2-VideoTrackEval annotation terms are distinct from source-media terms.

Dataset	Effect	Naive (selected)	IPW	Full pool
DanceTrack	$B_{T}$	$+12.8\%$	$+11.3\%$	$+11.4\%$
	$N_{T}$	$+7.6\%$	$+7.3\%$	$+9.4\%$
	$C_{T}$	$+25.3\%$	$+21.5\%$	$+18.0\%$
SportsMOT	$B_{T}$	$+0.2\%$	$+0.2\%$	$+1.2\%$
	$N_{T}$	$+3.8\%$	$+2.2\%$	$+2.6\%$
	$C_{T}$	$+5.5\%$	$+1.6\%$	$+4.9\%$

Appendix

Table 3: Molmo2 clean-input

B_{T}

N_{T}

, and

C_{T}

: unweighted stress-subset estimate, inverse-probability-weighted estimate, and candidate-pool mean.

Condition	Molmo2 input after its fixed scaffold	Specialized trackers
Neutral	no appended identity description	object
Unrelated	“The target is: $e_{U}$ ”	$e_{U}$
Correct	“The target is: $e_{T}$ ”	$e_{T}$
Conflicting	“The target is: $e_{D}$ ”	$e_{D}$

Appendix

Table 4: Text input under each condition for Molmo2 and the specialized trackers.

System	Initialization	Text interface	Output	Reset safeguard
Molmo2	clean target point	native point-tracking scaffold	point trajectory	fresh generation state
JointNLT	clean tight box	raw expression, native track mode	box trajectory	ROI queue, template features, tracker state
UVLTrack	clean tight box	raw expression	box trajectory	state, frame counter, score and output cache
MMTrack	clean tight box	raw expression	box trajectory	adapter/tracker and decode state
DUTrack	clean tight box	static external expression	box trajectory	dynamic description and tracker state

Appendix

Table 5: Native inputs, outputs, and reset state.

Perturbation	raw $n$	$S_{1}$	$S_{2}$	$S_{3}$	$S_{4}$	$S_{5}$
Blur	8	2	3	4	5	6
Noise	9	4	5	6	7	8
Pixelation	8	1	2	3	4	7
Compression	7	2	3	4	5	6

Appendix

Table 7: Raw implementation indices used for each displayed

S_{0}

–

S_{5}

severity.

Figure 1: Displayed $S_{0}$ – $S_{5}$ perturbation schedule on one MeViS frame.

Figure 64: Endpoint change ( $S_{5}-S_{0}$ ) in responsive-system target-accuracy effects; asterisks mark FDR-significant endpoint tests.

Figure 65: Full severity trajectories for the three model–dataset–effect cases with FDR-supported endpoint contraction.

Model	Dataset	Eff.	$S_{0}$	Blur	Noise	Pixel.	Compr.
JointNLT	MeViS	$C_{T}$	+14.2	$-13.8^{*}$	$-13.9^{*}$	$-9.9^{*}$	$-5.2$
JointNLT	PersonPath22	$C_{T}$	+26.9	$-28.0^{*}$	$-23.5^{*}$	$-29.9^{*}$	$-24.2^{*}$
Molmo2	DanceTrack	$B_{T}$	+12.8	$-13.5$	$-11.0$	$-13.9$	-6.4
JointNLT	MeViS	$B_{T}$	+10.6	$-11.1^{*}$	$-8.1^{*}$	$-9.6^{*}$	-4.1
Molmo2	DanceTrack	$C_{T}$	+25.3	-21.4	$-24.3$	$-25.8$	-16.0
JointNLT	MeViS	$N_{T}$	+7.8	$-8.9^{*}$	$-7.8^{*}$	-5.1	-3.4

Appendix

Table 9: Target-accuracy effects with a CI-supported clean-input value. Values after

S_{0}

are signed endpoint changes; asterisks mark FDR-significant endpoint tests.

Model	Dataset	$S_{0}$	Blur	Noise	Pixel.	Compr.
JointNLT	MeViS	+9.2	$-8.2^{*}$	$-7.3^{*}$	$-7.0^{*}$	-1.6
Molmo2	DanceTrack	+14.1	-7.6	$-11.0$	$-12.8$	-6.1
Molmo2	MeViS	+5.6	-4.4	-4.8	$-5.6^{*}$	+2.3
Molmo2	PersonPath22	+11.3	-12.6	-11.8	-11.9	-3.0

Appendix

Table 10: Distractor-capture effects with a CI-supported clean-input value. Values after

S_{0}

are signed endpoint changes; asterisks mark FDR-significant endpoint tests.

	$S_{0}$			$S_{5}$ (pooled, 4 perturbations)
Model	$A_{T}$	$A_{D}$	other/absent	$A_{T}$	$A_{D}$	other/absent
Molmo2	67.9%	2.4%	29.7%	37.4%	3.7%	58.8%
JointNLT	43.6%	9.0%	47.4%	19.3%	7.3%	73.4%
UVLTrack	64.4%	6.1%	29.5%	27.4%	7.6%	65.0%
MMTrack	62.9%	6.4%	30.7%	24.2%	7.0%	68.8%
DUTrack	68.1%	4.8%	27.1%	33.2%	6.6%	60.2%

Appendix

Table 11: Correct-text outcome rates, pooled across datasets at

S_{0}

and across datasets and perturbations at

S_{5}

. “other/absent” combines the two non-target, non-distractor labels.

Figure 66: JointNLT MeViS internal trajectories under blur. Panels show visual separation, Text BA, and grounding-to-output association; error bars are 95% source-video bootstrap CIs.

Figure 67: JointNLT MeViS internal trajectories under noise.

Figure 68: JointNLT MeViS internal trajectories under pixelation.

Figure 69: JointNLT MeViS internal trajectories under compression.

$B_{T}$
Model	Dataset	point	box-IoU	diff
UVLTrack	DanceTrack	$-0.8\%$	$+0.4\%$	$+1.2$
UVLTrack	SportsMOT	$+1.2\%$	$+0.5\%$	$-0.7$
UVLTrack	PersonPath22	$\underline{+5.3\%}$	$+4.1\%$	$-1.2$
UVLTrack	MeViS	$+0.6\%$	$+1.3\%$	$+0.7$
MMTrack	DanceTrack	$-0.3\%$	$+1.3\%$	$+1.7$

Appendix

Table 16: Clean-input

B_{T}

and

S_{D}

under point-in-mask and box-IoU scoring. “diff” is box-IoU minus point, in percentage points; underlining denotes a 95% source-video bootstrap CI excluding zero.

Figure 70: Outcome-independent pool: target-accuracy IAD, averaged equally across the three MOT sources and four perturbations over paper $S_{0}$ – $S_{5}$ . Labels abbreviate neutral/unrelated/correct/conflicting.

Multimodal Visual Object Tracking

See all 21 papers

May 17, 2026

Beyond Detection: A Structure-Aware Framework for Scene Text Tracking

Modern visual object trackers show impressive results on general targets, yet their performance drops substantially when dealing with scene text. Although currently underexplored, tracking text in videos is essential for dynamic text manipulations such as segmentation, removal, and editing. To fill this gap, this paper formalizes this specific task as Scene Text Tracking and presents the first systematic work for it. We identify three primary challenges in this task: 1) severe geometric distortions from perspective shifts, 2) high visual ambiguity across different instances, and 3) high sensitivity to fine-grained structural details. To address these issues, we propose SymTrack, a unified detection-free framework with synergistic dual-branch design. It integrates a Cross-Expert Calibration mechanism to reduce semantic bias, along with a Predictive Token Rectification mechanism to correct structural imbalances, complemented by an Adaptive Inference Engine that stabilizes predictions under motion constraints. Considering the lack of dedicated benchmarks for this task, we utilize three datasets from video text spotting to construct a benchmark with high-quality annotations. Extensive experiments demonstrate that SymTrack sets the new state-of-the-art on all three benchmarks, outperforming previous best trackers by up to 11.97% AUC on

\text{BOVText}_{\text{SOT}}

. Overall, our work promotes efficient and thorough text tracking, paving the way toward more generalized video text manipulation.

Chenmin Yu, Liu Yu, Daiqing Wu +3

1VCIP & TMCC & DISSec, College of Computer Science & College of Cryptology and Cyber Science, Nankai University · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China.

Visual Object Tracking Multi-Object Tracking

Ask PDF

Jun 28, 2026

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness. Existing studies have verified that adaptively optimizing textual descriptions throughout the tracking process can effectively mitigate the semantic-visual mismatch induced by dynamic variations in target appearance, position, and other inherent attributes. Nevertheless, mainstream methods that directly generate textual information via sequence models or large language models inevitably suffer from inherent defects, including erroneous target updating, excessive background distraction, and pervasive hallucination artifacts. To address the aforementioned limitations, this paper proposes a novel language dependency parsing mechanism to precisely distill core tracking principal components, encompassing target objects, semantic concepts, and background contextual information. On this basis, we perform component-aware adaptive textual description updates by exploiting the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL. By integrating the proposed elaborately designed modules into the baseline framework, our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks, including TNL2K, LaSOT, TNLLT, and OTB-LANG. The source code and pre-trained models will be released at https://github.com/Event-AHU/Open_VLTrack.

Xiao Wang, Liye Jin, Dan Xu +5

School of Computer Science and Technology, Anhui University, Hefei 230601, China · School of Electronic and Information Engineering, Anhui University, Hefei 230601, China · Harbin Institute of Technology, Shenzhen, China +3

Vision-Language Models Visual Object Tracking

Ask PDF

May 15, 2026

VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following

Vision-language models (VLMs) perform strongly on multimodal benchmarks, yet their ability to reliably follow continuous visual structure remains unclear. We focus on path following, a basic operation that requires following a selected path through successive visual continuations. Before considering complex factors such as crossings or overlaps, we first test whether current VLMs can robustly follow a simple continuous line under controlled settings that remove such confounds. Across frontier models, we find that tracing is already fragile, with this vulnerability manifesting as unstable path preservation in the presence of other nearby paths. Analysis of internal representations shows that information needed to determine the correct path continuation becomes weaker, with this degradation already emerging in the visual encoding stage. Common remedies do not resolve this weakness: model scaling provides limited improvement, additional reasoning provides uneven gains at substantial cost, and explicit tracing instructions fail to stabilize performance. The same path-preservation failures and sensitivity to nearby paths persist in visually richer and more realistic settings, suggesting that current VLMs lack a robust mechanism for following a selected path.

Hyesoo Hong, Min Soo Kim, Wonje Jeung +3

Yonsei University

VLM Evaluation Visual Navigation VLM Robustness Visual Spatial Reasoning VLM Reasoning

Ask PDF

Abstract

Multimodal Visual Object Tracking

Explore similar work

Beyond Detection: A Structure-Aware Framework for Scene Text Tracking

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following