Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation

Yuqing Huang; Guotian Zeng; Zhenqiao Yuan; Zhenyu He; Xin Li; Yaowei Wang; Ming-Hsuan Yang

cs.CVApr 2, 2026

Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation

Authors: Yuqing Huang, Guotian Zeng, Zhenqiao Yuan, Zhenyu He, Xin Li, Yaowei Wang, Ming-Hsuan Yang

Organizations: Pengcheng Laboratory · Harbin Institute of Technology, Shenzhen · South China University of Technology · Pazhou Lab (Huangpu) ·

Abstract

Existing visual trackers mainly operate in a non-interactive, fire-and-forget manner, making them impractical for real-world scenarios that require human-in-the-loop adaptation. To overcome this limitation, we introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands. To support research in this direction, we make three main contributions. First, we present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions. Second, we propose a comprehensive evaluation protocol and evaluate 25 representative trackers, showing that state-of-the-art methods fail in interactive scenarios; strong performance on conventional benchmarks does not transfer. Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly. Our benchmark, protocol, and baseline establish a foundation for developing more intelligent, adaptive, and collaborative tracking systems, bridging the gap between automated perception and human guidance. The full benchmark, tracking results, and analysis are available at https://github.com/NorahGreen/InteractTrack.git.

Figures & tables

Benchmark	Year	Videos	Mean Frames	Total Frames	Interactive Label	Scenarios Label
OTB-2013 [ 64 ]	2013	50	578	29K	✗	✗
OTB-2015 [ 65 ]	2015	100	590	59K	✗	✗
TC-128 [ 37 ]	2015	128	429	55K	✗	✗
NUS-PRO [ 32 ]	2016	365	371	135K	✗	✗
UAV123 [ 45 ]	2016	123	915	113K	✗	✗
UAV20L [ 45 ]	2016	20	2,934	59K	✗	✗

Table 1: Comparison of visual tracking benchmarks. This table summarizes key statistics of each test set and highlights support for interactive evaluation and multi-scenario coverage.

Figure 2: Overview of the interactive evaluation protocol. Each video is divided into segments by user prompts. Dashed lines denote ground-truth trajectories, and colored curves represent tracker predictions. At each prompt, the tracker must either update its prediction or switch targets based on user input. $\checkmark$ and $\times$ denote correct and incorrect responses, respectively.

Figure 3: Overview of the proposed Interactive Memory-Augmented Tracking (IMAT) framework. The user provides natural-language descriptions of the target to the interactive perception module, which performs reasoning-guided grounding. The cognitive arbitration module then compares the grounded result with the prediction of the tracker to either correct the trajectory and update the memory banks or confirm the current tracking state.

1 \Block 2-1 Type	\Block 2-1 Algorithm	\Block 2-1 Interactiveness	\Block 2-1 Responsiveness	\Block 1-2 Perception Capability		\Block 1-3 Tracking Capability
				Accuracy	Precision	AUC	Precision	Norm Precision
	Ours	45.25	41.20	52.78	49.63	45.86	49.63	60.90
\Block 3-1VLT	JointNLT [ 85 ]	30.66	36.67	44.33	43.08	19.81	16.16	30.44
	DUTrack [ 35 ]	39.52	37.83	48.82	47.09	41.70	44.85	56.55
	SUTrack [ 7 ]	40.90	38.04	49.25	48.38	44.25	47.23	58.26
\Block 5-1VOS	SAM2 [ 51 ]	41.43	35.93	47.43	45.58	42.22	45.12	56.42

Table 2: Combined evaluation on the InteractTrack test set under the same interactive protocol. Methods are grouped by paradigm (VLT/VOS/VOT/RVOS); best results in each column are in bold .

Panel of Figure 4 — Figure 4: Evaluation results on the six scenarios using the success rate in the InteractTrack benchmark. These figures show that the proposed baseline performs strongly across all six scenarios.

Model Variant	Responsiveness (%)	Interactiveness (%)
w/o IPM (no language)	35.93	41.43
w/o Memory	37.72	41.52
w/o CAM	38.03	42.09
w/ Naive IoU Arbitration	39.55	43.02
IMAT (ours)	41.20	45.25

Table 3: Ablation study on the InteractTrack benchmark.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Representative sequences from the daily activities scenario. This scenario includes everyday indoor and outdoor activities captured from a third-person viewpoint. The sequences depict interactions such as playing with pets, handling household objects, and casual human motion under moderate viewpoint shifts. The highlighted sequence illustrates target-switch interaction cues.

Figure 6: Representative sequences from the sports analysis scenario. The sequence illustrates multiple interaction points in a basketball game, including target initialization, ball-focused attention during passes, shot-following, and explicit target switching among players. At each key timestamp, the user provides updated instructions that reflect the evolving visual context—shifting attention from the ball handler to the ball itself, following the shot trajectory, and then redirecting focus to a specific player. The highlighted sequence illustrates target-switch and part-tracking interaction cues as user instructions evolve across multiple stages.

Figure 7: Representative sequences from the UAV tracking scenario. This scenario includes aerial-view videos characterized by large scale variation, long-range target motion, and perspective distortions typical of drone footage. The sequences feature vehicles, pedestrians, and small moving objects captured at varying altitudes and along diverse trajectories. The highlighted sequence illustrates challenges such as drastic scale changes, temporary target disappearance, and viewpoint transitions during UAV maneuvers. Other examples demonstrate additional difficulties common in aerial surveillance, including low-resolution targets, cluttered backgrounds, and complex scene geometries.

Figure 8: Representative sequences from the surveillance scenario. This scenario features crowded street environments captured from a third-person viewpoint. The sequences depict interactions such as monitoring individuals in dense crowds, shifting attention between people and carried objects, and handling frequent occlusions and complex motion patterns. The highlighted sequence illustrates target-switch interaction cues.

Figure 9: Representative sequences from the wildlife monitoring scenario. This scenario includes animal behaviors captured in natural environments. The sequences depict interactions such as observing feeding behaviors, shifting attention between different animals, and focusing on objects being manipulated by the animals.

Figure 10: Representative sequences from the other scenario. This scenario contains diverse scenes that do not fall into the previous categories, including industrial, laboratory, and indoor environments. The sequences illustrate various interaction points where user instructions shift attention among different people, objects, and activities.

Type	Tracker	Architecture	Where
VLT	JointNLT	Trans.	CVPR’23
VLT	DUTrack	Trans.	CVPR’25
	SUTrack	Trans.	AAAI’25
VOS	SAM2	Trans.	ICLR’25
	HIM2SAM	Trans.	PRCV’25
	SAMURAI	Trans.	arxiv’25

Appendix

Table 4: Categorization of the evaluated tracking methods by task type, architectural family, and publication venue.

Figure 11: Visualization of challenging cases in InteractTrack. It demonstrates that our proposed method can effectively handle complex interactive scenarios and accurately track the target, whereas traditional trackers may lose the target.

Panel of Figure 12 — Figure 12: Precision evaluation results across the six scenarios in the InteractTrack benchmark. The plots show that the proposed baseline achieves strong and consistent precision performance in all scenarios.

Figure 13: Failure cases. Even with updated user instructions, all trackers struggle with heavy occlusion and background clutter, ultimately losing the target despite interactive guidance.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 171 papers

CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models
Mar 24, 2026Youzhi Liu, Li Gao, Liu Liu +2Game-Theoretic RL Reinforcement Learning
Amap, Alibaba Group
Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Tracking
Aug 1, 2026Wenrui Cai, Yuzhe Li, Qingjie Liu +1Multimodal Visual Object Tracking Visual Object Tracking
State Key Laboratory of Virtual Reality Technology and System, Beihang University. · Beihang University. · School of Computer Science and Engineering, Beihang University. +1
Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking
Jun 28, 2026Xiao Wang, Liye Jin, Dan Xu +5Vision-Language Models Visual Object Tracking
School of Computer Science and Technology, Anhui University, Hefei 230601, China · School of Electronic and Information Engineering, Anhui University, Hefei 230601, China · Harbin Institute of Technology, Shenzhen, China +3

Mar 24, 2026J/K move · Enter open · S save

CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models

Youzhi Liu, Li Gao, Liu Liu, Mingyang Lv+1

Amap, Alibaba Group

Game-Theoretic RL Reinforcement Learning VLMs for Robotics Robustness of VLA Models Sequential MARL

Open paper AskPDF arXiv

Embodied Visual Tracking (EVT), a core dynamic task in embodied intelligence, requires an agent to precisely follow a language-specified target. Yet most existing methods rely on single-agent imitation learning, suffering from costly expert data and limited generalization due to static training environments. Inspired by competition-driven capability evolution, we propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework that trains agents in a dynamic adversarial setting with competitive subtasks, yielding stronger adaptive planning and interference-resilient strategies. We further introduce CoMaTrack-Bench, the first open-source Habitat-based benchmark protocol and episode set for language-conditioned competitive EVT featuring dynamic dueling, featuring game scenarios between a tracker and adaptive opponents across diverse environments and instructions, enabling standardized robustness evaluation under active adversarial interactions. Experiments show that CoMaTrack achieves state-of-the-art results on both standard benchmarks and CoMaTrack-Bench. Notably, a 3B VLM trained with our framework surpasses previous single-agent imitation learning methods based on 7B models on the challenging EVT-Bench, achieving 92.1% in STT, 74.2% in DT, and 57.5% in AT. The benchmark code will be available at https://github.com/wlqcode/CoMaTrack-Bench.

Abstract

Visual Object Tracking

Explore similar work

CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models

Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Tracking

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking