Overview

Visual object tracking estimates where a target moves over time, from a single object in a video to many objects across cameras, modalities, or 3D scenes. Common approaches match Siamese-network or Transformer features between frames, associate detections with track histories, and use motion models, geometric constraints, or memory to handle appearance changes, occlusion, and reappearance. Recent work also adapts video and segmentation foundation models, language guidance, event cameras, and learned memory control, with evaluations spanning benchmarks such as LaSOT, GOT-10k, MOT17, UAV tracking datasets, and specialized medical or 3D settings. A central challenge is making trackers reliable beyond standard benchmark conditions—especially over long absences, under severe occlusion or domain shifts—while keeping inference efficient and evaluation representative of real deployments.

Latest papers

182 papers9 papers in the last four weeks, down 65% on the four weeks before. 0.1% of all new papers.

Jul 5, 2024

SSP-GNN: Learning to Track via Bilevel Optimization

We propose a graph-based tracking formulation for multi-object tracking (MOT) where target detections contain kinematic information and re-identification features (attributes). Our method applies a successive shortest paths (SSP) algorithm to a tracking graph defined over a batch of frames. The edge costs in this tracking graph are computed via a message-passing network, a graph neural network (GNN) variant. The parameters of the GNN, and hence, the tracker, are learned end-to-end on a training set of example ground-truth tracks and detections. Specifically, learning takes the form of bilevel optimization guided by our novel loss function. We evaluate our algorithm on simulated scenarios to understand its sensitivity to scenario aspects and model hyperparameters. Across varied scenario complexities, our method compares favorably to a strong baseline.
Jun 7, 2024

Referring Multi-Object Tracking with Diverse Dynamic Expressions

Referring understanding is a fundamental task that bridges natural language and visual content by localizing objects described in free-form expressions. However, existing works are constrained by limited language expressiveness, lacking the capacity to model object dynamics in spatial numbers and temporal states. To address these limitations, we introduce a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking, comprehensively accounting for variations in object quantity and temporal semantics. Along with RMOT, we introduce a RMOT benchmark named Refer-KITTI-V2, featuring scalable and diverse language expressions. To efficiently generate high-quality annotations covering object dynamics with minimal manual effort, we propose a semi-automatic labeling pipeline that formulates a total of 9,758 language prompts. In addition, we propose TempRMOT, an elegant end-to-end Transformer-based framework for RMOT. At its core is a query-driven Temporal Enhancement Module that represents each object as a Transformer query, enabling long-term spatial-temporal interactions with other objects and past frames to efficiently refine these queries. TempRMOT achieves state-of-the-art performance on both Refer-KITTI and Refer-KITTI-V2, demonstrating the effectiveness of our approach. The source code and dataset is available at https://github.com/zyn213/TempRMOT.