STORM: End-to-End Referring Multi-Object Tracking in Videos

Zijia Lu; Jingru Yi; Jue Wang; Yuxiao Chen; Junwen Chen; Xinyu Li; Davide Modolo

cs.CVApr 12, 2026

STORM: End-to-End Referring Multi-Object Tracking in Videos

Authors: Zijia Lu, Jingru Yi, Jue Wang, Yuxiao Chen, Junwen Chen, Xinyu Li, Davide Modolo

Organizations: Amazon

Abstract

Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMOT approaches decompose object grounding and tracking into separated modules and exhibit limited performance due to the scarcity of training videos, ambiguous annotations, and restricted domains. In this work, we introduce STORM, an end-to-end MLLM that jointly performs grounding and tracking within a unified framework, eliminating external detectors and enabling coherent reasoning over appearance, motion, and language. To improve data efficiency, we propose a task-composition learning (TCL) strategy that decomposes RMOT into image grounding and object tracking, allowing STORM to leverage data-rich sub-tasks and learn structured spatial--temporal reasoning. We further construct STORM-Bench, a new RMOT dataset with accurate trajectories and diverse, unambiguous referring expressions generated through a bottom-up annotation pipeline. Extensive experiments show that STORM achieves state-of-the-art performance on image grounding, single-object tracking, and RMOT benchmarks, demonstrating strong generalization and robust spatial--temporal grounding in complex real-world scenarios. STORM-Bench is released at https://github.com/amazon-science/storm-referring-multi-object-grounding.

Figures & tables

Figure 1: Overview of STORM method. The STORM adopts a LLaVA-style MLLM architecture [ 47 ] , enabling video understanding on top of a large language model (LLM). Due to the scarcity of RMOT data, we propose a Task-Composition Learning (TCL) recipe. As illustrated in the figure on the right, TCL employs a two-stage training process. Stage 1 transfers subtask knowledge (i.e., object grounding and single-object tracking, SOT) to the referring object tracking task. In Stage 2, the model is fine-tuned on our proposed STORM-Bench dataset, further enhancing performance on the referring multi-object tracking (RMOT) task with limited videos.

Figure 2: Visualization of videos from our STORM-Bench dataset. Bounding boxes in different colors represent different tracklets. The dataset includes a wide range of challenging scenarios: top—significant scale changes for the same target; middle—multi-object tracking with clear prompts under weaker conditions; and bottom—multi-target tracking of similar object types but different instances under stronger, more specific conditions.

	# Expressions	# Words	# Length	MO	CE
LaSOT [ 21 ]	1.4K	9.8K	7.0	$\times$	$\times$
Elysium [ 68 ]	1.3M	2.7M	2.1	$\times$	$\times$
Refer-KITTI-V2 [ 82 ]	9.8K	63.9K	6.6	$\checkmark$	$\times$
LaMOT [ 39 ]	7.5K	28K	3.8	$\checkmark$	$\times$
Ours	0.2M	2.4M	12.2	$\checkmark$	$\checkmark$

Table 1: STORM-Bench dataset statistics and comparison with other referring tracking datasets. MO: Multi-Objects; CE: Complex-expression.

	RefCOCO	RefCOCO+	RefCOCOg
	val/testA/testB	val/testA/testB	val/test
Shik-7B [ 15 ]	87.0/90.6/80.2	81.6/87.4/72.1	82.3/82.2
Shik-13B [ 15 ]	87.8/91.1/81.8	82.9 /87.8/74.4	82.6/83.2
M-GPT2 [ 13 ]	88.7/91.7/ 85.3	80.0/85.1/ 74.5	84.4 /84.7
Ferret [ 78 ]	87.5/91.4/82.5	80.8/87.4/73.1	83.9/84.8
G-GPT [ 40 ]	88.0/91.6/82.5	81.6/ 87.2/73.2	81.7/82.0

Table 2: Referring image object grounding performance on RefCOCO, RefCOCO+, and RefCOCOg.

	Elysium (RSOT)			Elysium (SOT)
	AUC	P	P norm	AUC	P	P norm
Elys [ 68 ]	83.3	89.1	90.0	88.7	94.6	93.8
Ours	84.1	89.7	93.2	89.8	96.4	97.8
Elys*	87.5	94.5	93.7	88.7	94.6	93.8
Ours*	87.4	95.3	97.2	89.8	96.4	97.8

Table 3: Single-object tracking (SOT) and referring single-object tracking (RSOT) results on Elysium. * denotes the experiments with our prompts (longer and more comprehensive).

	HOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$	LocA $\uparrow$	IDF1 $\uparrow$	IDP $\uparrow$	IDR $\uparrow$	MOTA $\uparrow$	IDsw $\downarrow$
Grounding DINO [ 37 ] $\dagger$	31.7	17.8	56.5	88.1	27.3	68.5	17.0	15.4	2953
Qwen2.5-VL [ 2 ] $\dagger$	37.9	23.2	62.6	76.6	38.6	77.9	25.5	22.4	2767
VisionLLMv2 [ 28 ] $\dagger$	45.3	35.5	58.9	75.3	44.6	53.9	38.1	18.7	12097
LaMOT [ 39 ]	46.7	39.0	56.1	89.2	48.8	71.5	37.1	37.6	7972
Ours*	34.8	21.4	56.8	63.7	36.4	54.2	27.3	15.9	2832
Ours	66.7	55.3	82.6	81.4	78.3	78.4	78.1	57.1	183

Table 4: Referring multi-object tracking (RMOT) performance on STORM-Bench.

\dagger

indicates that OCSORT is used as the tracker. *denotes the model trained on STORM-Bench only.

	Training Data Scale				Test Task Performance
ROW	IG	SOT	RSOT	RMOT	SOT	RSOT	RMOT
1	1M	0M	0.1M	0K	83.4	77.4	36.9
2	1M	0M	1M	0K	89.8	84.1	43.3
3	1M	0.9M	0.1M	0K	91.3	83.0	41.8
4	1M	0M	1M	15K	91.3	83.0	66.7

Table 5: Ablation on task-composition training using image grounding (IG), single-object tracking (SOT), and referring single-object tracking (RSOT) data.“M” and “K” denote million and thousand, respectively.

Figure 3: Visualization of complex referring expressions involving attributes and relations (e.g., spatial relations between objects). Our model maintains consistent identities and precise localization across challenging scenarios.

RMOT Training Data	5K	10K	12K	15K
RMOT Performance (HOTA)	26.4	53.4	63.3	66.7

Table 6: Effect of the scale of RMOT training data in TCL.

Figure 4: Comparison of STORM, Qwen2.5-VL, and VisionLLMv2 on referring multi-object tracking using examples from STORM-Bench. STORM closely follows the query prompt and accurately localizes and tracks all objects that satisfy the referring expression over time.

#frames	32	64	128	256	64	64	64	64
#targets	1	1	1	1	1	2	4	8
time (s)	9.7	19.7	36.4	76.4	19.7	34.9	67.4	143.3
FPS	3.3	3.2	3.3	3.3	3.2	1.8	1.0	0.4

Table 7: Efficiency analysis of STORM with an 8B MLLM backbone. The first four columns vary video length with one target, while the last four vary the number of targets at 64 frames.

Figure 5: Word cloud of the referring expressions in the STORM-Bench dataset.

Figure 6: Distribution of object classes in STORM-Bench.

Figure 7: Model results with the referring expression the baby and the pink toy above it .

Figure 8: Model results with the referring expression the baby and the blue high chair beneath the baby .

Figure 9: Model results with the referring expression the cup next to the dish with fan-shaped meat arrangement .

Figure 10: Model results with the referring expression the adult wearing a pink and white patterned shirt and the black dog .

Figure 11: Model results with the referring expression the table with white cloth and the wooden table .

Explore similar work

CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking
Jul 28, 2026Xiangqun Zhang, Likai Wang, Zekun Qian +2Synthetic-to-Real Domain Adaptation Cross-Domain Transfer Learning
School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China
DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking
Feb 4, 2026Sijia Chen, Shaofeng Liang, Lijuan Ma +7Multi-Object Tracking Multimodal Large Language Models
Huazhong University of Science and Technology · South-Central Minzu University
COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking
May 14, 2026Shukun Jia, Shiyu Hu, Yipei Wang +3Counterfactual Learning Multi-Object Tracking
School of Automation, Southeast University, Nanjing, China · Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, China · School of Physical & Mathematical Sciences, Nanyang Technological University, Singapore +1

Jul 28, 2026J/K move · Enter open · S save

CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking

Xiangqun Zhang, Likai Wang, Zekun Qian, Ruize Han+1

School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China

Synthetic-to-Real Domain Adaptation Cross-Domain Transfer Learning Multi-Object Tracking

Open paper AskPDF arXiv

Referring multi-object tracking (RMOT) extends tracking from category-driven perception to language-guided understanding by grounding object trajectories in natural-language expressions. Despite recent progress, existing RMOT studies are largely conducted under in-domain settings, leaving the robustness of language-conditioned tracking under inevitable visual domain shifts unexplored. In this paper, we study Cross-Domain Referring Multi-Object Tracking (CD-RMOT), a new and challenging problem that evaluates whether an RMOT model trained on a labeled source domain can reliably follow natural-language expressions in an unlabeled target domain with different visual conditions. To support systematic study, we construct CD-RMOT-Bench, a unified benchmark that combines real clear-domain referring tracking data, aligned digital-twin variants, and real adverse-domain videos. CD-RMOT-Bench enables both controlled weather/viewpoint shift analysis and realistic synthetic-real transfer evaluation under a shared RMOT protocol. Further, we provide a Query-Centric Adaptation (QCA) framework, designed to stabilize the query space that bridges visual trajectories and referring expressions. Extensive experiments reveal that domain shifts severely degrade RMOT performance, where the failure is not merely caused by object detection errors but more critically by unstable expression-conditioned temporal association and target selection. QCA establishes a strong baseline, while CD-RMOT-Bench opens a new direction for robust language-guided tracking across visual domains.

Abstract

Explore similar work

CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking