Refer-CityFlow and M3Track: A Benchmark Dataset and Reference Framework for Language-Driven Multi-Target Multi-Camera Tracking
Authors: Phuong T. Nguyen, Hao Vo, Tam Bang, Hoang K. Tran, Hoang-Minh Dau, Mahshid Malazizi, Tuan-Anh Hoang, Mina Sartipi, Thanh-Nam Doan, Ngan Le, Hoang H. Nguyen
Organizations: University of Tennessee at Chattanooga · University of Arkansas · RMIT International University Vietnam · Independent Researcher
Referring multi-object tracking enables users to specify targets through natural language, but existing Refer-MOT benchmarks primarily focus on single-camera settings. We formalize Referring Multi-Target Multi-Camera Tracking (Refer-MTMCT), where free-form queries specify identity-consistent trajectories to be recovered across a network of sequential, sparse-overlap cameras. We introduce Refer-CityFlow, a benchmark built on CityFlow's global cross-camera identity annotations and augmented with language queries requiring joint language grounding, cross-camera association, and spatio-temporal reasoning. Refer-CityFlow covers 96 videos from 40 roadside cameras with 660+ identity-consistent cross-camera trajectories, 4,362 trajectory-level descriptions, and 7,699 queries, including 1,699 real-user queries collected from 50 participants. Reference experiments on the four-camera Scenario 02 show that single-camera referring trackers reach 14.95-18.28 HOTA on real-user queries, while the evaluated cross-view method achieves about 0.05 HOTA in direct transfer. To establish reference performance, we provide M3Track, a modular multimodal reference framework in which a large language model (LLM)-based agent orchestrates interchangeable detection, tracking, and cross-camera association modules. M3Track reaches 44.10 HOTA and 65.40 IDF1 on the same real-user query set. The project repository is available at https://github.com/phuongnguyen194/Refer-CityFlow-and--M3Track.
Figures & tables
Figure 1: Refer-CityFlow and the Refer-MTMCT setting. Top : one free-form query resolves to an identity-consistent trajectory across sequential cameras (Cam 6 → 7 → 9). Bottom : comparison with prior referring settings: (a) RMOT operates within one camera; (b) CRMOT [ 7 ] uses synchronized overlapping views of one scene; and (c) Refer-MTMCT spans a sequential, sparse-overlap camera network, where the query defines the target set and combines appearance, cross-camera, and spatio-temporal constraints.
Cameras / videos
Total frames
Global trajectories
40 / 96
116,000+
660+
Descriptions
Queries (AI / real-user)
Query-to-trajectory
4,362
6,000 / 1,699
many-to-many
Table 1: Summary statistics of Refer-CityFlow .
Type
Dataset
Scenes
Cameras
Queries
Language
Refer-MOT
Refer-KITTI [ 27 ]
Street-level vehicles only
On-vehicle
Spatial
49 unique words
Refer-KITTI-V2 [ 30 ]
Street-level driving
On-vehicle
Spatial, motion
Wider than Refer-KITTI
Refer-UE-City [ 16 ]
Synthetic urban
Roadside
Spatial, action
Limited style
Refer-Dance [ 8 ]
Indoor dancing
Dancing
Group motion
48 distinct expr.
Multi-cam retrieval
CityFlow-NL [ 10 ]
Urban traffic
Roadside
Appearance, motion (retrieval)
Trajectory captions
Cross-view referring
CRTrack [ 7 ]
Outdoor pedestrian
Multi-view, one scene
Appearance only
221 descriptions
Table 2: Qualitative comparison of referring-tracking datasets. Prior referring-tracking datasets predominantly target single-camera or synchronized-view settings; Refer-CityFlow targets sequential multi-camera tracking with structured spatio-temporal and real-user queries. Refer-KITTI-V2 and the TempRMOT baseline share one source paper [ 30 ] . CityFlow-NL and CRTrack are listed apart from Refer-MTMCT: the former ranks precomputed trajectories rather than driving the tracking, and the latter observes one scene from simultaneous overlapping views rather than a camera network.
Figure 2: Quantitative dataset-scale comparison. Coverage of Refer-CityFlow and prior tracking datasets in frames, videos, queries, and trajectory descriptions.
Format
Capability
Example constraints
QF-1
Appearance
Color, vehicle type
QF-2
Direction
Motion direction
QF-3
Zone
Entry, exit, or transition zone
QF-4
Action
Going straight, turning, stopping
QF-5
Relation
Relation to a nearby object
QF-6
Lane
Lane-level position
Table 3: The six AI-generated query formats. Each format is built around one constraint type; queries may combine several.
Source
Type
Query
AI-generated
Appearance + reappearance
“Track all white trucks that appear in camera 16, and also appear at camera 17.”
Zonal + temporal (compositional)
“Track all the black sedans that are moving from zone 22d to zone 22b in camera 22 from 15.8 seconds to 25.3 seconds, and also appear at cameras 16,17,18.”
Real-user
Raw, underspecified
“Track all green minivans that move from camera 6 to camera 9 now and yesterday from 2 PM to 5 PM.”
Clarification and resolved form
System: “Do you mean the cameras in the center of Chattanooga?” User: “Yes, Chattanooga Center.” Resolved fields: minivan, green; path 6, 7, 8, 9; current and yesterday 2–5 PM.
Table 4: Example AI-generated and real-user queries. The examples illustrate different query constraints, including an underspecified real-user query and its clarified form.
Figure 3: Real-user query collection and refinement. Participants freely describe a target; when the query is underspecified, the agent asks targeted follow-up questions until it can be resolved into an executable form, which is run against the ground-truth database to obtain the target set. Both the raw and resolved queries are retained for evaluation.
Figure 4: A sample query and overview of the M3Track framework. The LLM-Agent Module parses the query and, when it is underspecified, interactively clarifies it with the user ( blue : user query; orange : the analyzer requests missing information; red : agent interaction). The resolved query then drives video and time-window selection, single-camera perception, and cross-camera trajectory association.
Model
Detector
Training
HOTA
DetA
AssA
IDF1
TransRMOT [ 27 ]
Def. DETR
MS COCO
8.55
3.19
24.75
3.54
TempRMOT [ 30 ]
Def. DETR
MS COCO
9.19
3.27
31.94
3.16
M3Track (ours)
YOLOv5+SVM
COCO + att. fil.
24.15
13.39
48.65
26.28
YOLO26-m+SVM
COCO + att. fil.
26.79
17.37
45.82
32.13
Qwen3-VL
Open-vocab.
29.03
19.30
47.84
38.58
Grounding-DINO
Open-vocab.
28.76
17.42
49.38
37.72
Table 5: Performance on AI-generated queries in Refer-CityFlow Scenario 02. “SVM”: classifier for attribute filtering (att. fil.).
Model
Detector
Training
HOTA
DetA
AssA
IDF1
TransRMOT [ 27 ]
Def. DETR
MS COCO
18.28
10.38
33.96
7.62
TempRMOT [ 30 ]
Def. DETR
MS COCO
14.95
6.46
37.49
9.27
CRTrack [ 7 ]
APTM
Frozen
∼ 0.05
∼ 0.00
∼ 1.55
∼ 0.00
M3Track (ours)
YOLOv5+SVM
COCO + att. fil.
42.93
34.15
56.19
56.15
YOLO26-m+SVM
COCO + att. fil.
34.56
27.58
47.67
43.98
Qwen3-VL
Open-vocab.
43.30
36.20
53.95
64.75
Table 6: Performance on real-user queries (open-vocabulary, human-authored) in Refer-CityFlow Scenario 02. Single-camera baselines and the cross-view CRTrack are run directly on Refer-CityFlow . “SVM”: classifier for attribute filtering (att. fil.). See the supplement for a positional-subset ablation.
Model
Accuracy
Est. Cost (USD)
Amazon Nova Micro
99.4%
0.110
DeepSeek V3.2
97.1%
1.828
Meta Llama 3.1 8B
100.0%
0.609
GPT 4o-mini
100.0%
1.690
Table 7: LLM parsing accuracy and estimated cost across four open and proprietary models, measured on the directional (QF-2) query set; a query counts as correct only if every extracted field matches the ground truth.
National Key Laboratory of Science and Technology on Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China