Refer-CityFlow and M3Track: A Benchmark Dataset and Reference Framework for Language-Driven Multi-Target Multi-Camera Tracking

Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMOT approaches decompose object grounding and tracking into separated modules and exhibit limited performance due to the scarcity of training videos, ambiguous annotations, and restricted domains. In this work, we introduce STORM, an end-to-end MLLM that jointly performs grounding and tracking within a unified framework, eliminating external detectors and enabling coherent reasoning over appearance, motion, and language. To improve data efficiency, we propose a task-composition learning (TCL) strategy that decomposes RMOT into image grounding and object tracking, allowing STORM to leverage data-rich sub-tasks and learn structured spatial--temporal reasoning. We further construct STORM-Bench, a new RMOT dataset with accurate trajectories and diverse, unambiguous referring expressions generated through a bottom-up annotation pipeline. Extensive experiments show that STORM achieves state-of-the-art performance on image grounding, single-object tracking, and RMOT benchmarks, demonstrating strong generalization and robust spatial--temporal grounding in complex real-world scenarios. STORM-Bench is released at https://github.com/amazon-science/storm-referring-multi-object-grounding.

Omnidirectional cameras provide 360° spatial coverage, making them increasingly valuable in applications such as autonomous driving and video surveillance. While Multi-Object Tracking (MOT) and its language-guided extension, Referring Multi-Object Tracking (RMOT), have achieved notable progress, existing methods rely on conventional cameras with limited fields of view, causing critical contextual cues to be lost when targets move outside the frame. This fundamentally limits the model's ability to interpret long-horizon language descriptions involving sequential actions, spatial relations, and group behaviors. In this work, we propose Omnidirectional Referring Multi-Object Tracking (ORMOT), a novel task extending RMOT to omnidirectional imagery, where 360° coverage ensures complete scene context for accurate language-guided tracking. To advance this task, we construct ORSet, a dataset comprising 27 omnidirectional scenes, 848 language descriptions, and 3,401 annotated objects. Furthermore, we propose ORTrack, an LVLM-driven framework that enables zero-shot language-guided detection and robust cross-frame association in complex 360° environments. Experiments on ORSet demonstrate that ORTrack achieves state-of-the-art performance, providing a strong baseline for future research. The dataset and code will be open-sourced at https://github.com/chen-si-jia/ORMOT.

Referring multi-object tracking (RMOT) extends tracking from category-driven perception to language-guided understanding by grounding object trajectories in natural-language expressions. Despite recent progress, existing RMOT studies are largely conducted under in-domain settings, leaving the robustness of language-conditioned tracking under inevitable visual domain shifts unexplored. In this paper, we study Cross-Domain Referring Multi-Object Tracking (CD-RMOT), a new and challenging problem that evaluates whether an RMOT model trained on a labeled source domain can reliably follow natural-language expressions in an unlabeled target domain with different visual conditions. To support systematic study, we construct CD-RMOT-Bench, a unified benchmark that combines real clear-domain referring tracking data, aligned digital-twin variants, and real adverse-domain videos. CD-RMOT-Bench enables both controlled weather/viewpoint shift analysis and realistic synthetic-real transfer evaluation under a shared RMOT protocol. Further, we provide a Query-Centric Adaptation (QCA) framework, designed to stabilize the query space that bridges visual trajectories and referring expressions. Extensive experiments reveal that domain shifts severely degrade RMOT performance, where the failure is not merely caused by object detection errors but more critically by unstable expression-conditioned temporal association and target selection. QCA establishes a strong baseline, while CD-RMOT-Bench opens a new direction for robust language-guided tracking across visual domains.