Referring Multi-Object Tracking in Moving-Camera Videos via Global Motion Compensation

Hsin-Chen Pai; Jyun-Kai Wang; Yi-Cheng Peng; Wei-Ta Chu

cs.CVOct 3, 2026

Referring Multi-Object Tracking in Moving-Camera Videos via Global Motion Compensation

Authors: Hsin-Chen Pai, Jyun-Kai Wang, Yi-Cheng Peng, Wei-Ta Chu

Organizations: National Cheng Kung University, Tainan, Taiwan

Abstract

Referring multi-object tracking (RMOT) takes a video and a language expression as input and tracks all referred objects. Many tracking requirements involve how an object moves rather than how it appears. However, in a video captured by a moving camera, a parked vehicle may appear to move, while a moving vehicle may show little displacement. Existing RMOT methods relate motion with text but do not explicitly remove camera-induced motion. In this paper, we propose extracting residual motion across frames by estimating camera motion in driver-view videos and compare motion characteristics with the query expression. We consider the motion-matching extent and integrate it with the RMOT method's prediction result through late fusion. In the evaluation, we verify the performance gain of taking the motion compensation module as a plug-in across different RMOT hosts.

Figures & tables

Figure 1: The proposed RMOT framework with the global motion compensation module. The host model (iKUN in this case) predicts an object-language matching score. The GMC module provides an extra matching score to adjust matching results.

Host	$\alpha_{\mathrm{mot}}$ (moving class)	$\alpha_{\mathrm{oth}}$ (other class)
iKUN	1.0	0.1
TransRMOT	0.7	0.5
FlexHook V1	7	7
FlexHook V2	5	5

Table 1: Fusion weightings

\alpha_{c}

selected by leave-one-sequence-out cross-validation.

Refer-KITTI
Method	HOTA	DetA	AssA
iKUN [ 3 ]	44.56	32.04	62.48
+ GMC	45.304	32.90	62.94
TransRMOT [ 17 ]	45.757 ∗	36.64	57.40
+ GMC	46.149	36.95	57.89
FlexHook V1 [ 6 ]	53.824	43.35	66.92

Table 2: Pooled HOTA, DetA, and AssA on the Refer-KITTI test sets. Each host row gives the baseline (HOTA as published; DetA and AssA from our reproduction); the + GMC row gives the fused model (mean over three seeds). The better value of each pair is in bold. ∗ denotes our reproduction using the official implementation, under the host’s own frame convention (Sec. 4.1 ).

Host	Baseline	Performance gain
iKUN	27.70	$+9.44\pm 0.92$
TransRMOT	33.88	$+1.43\pm 0.13$
FlexHook V2	38.56	$+0.69\pm 0.04$
FlexHook V1	47.90	$+0.43\pm 0.19$

Table 3: Moving-class HOTA (moving-class expressions:

21/150

for iKUN, TransRMOT, and FlexHook V1,

111/862

for FlexHook V2),

n{=}3

seeds. Rows are ordered by baseline moving-class HOTA.

Variant	Moving HOTA	Others HOTA	Pooled HOTA
Native host (w/o GMC)	27.70	45.87	44.54
Full module (w. GMC)	36.99	46.05	45.28
single $\alpha$ (no routing)	32.42	45.99	44.95
$-$ ego (raw velocity only)	32.37	45.77	44.61
$-$ multiscale (single gap)	35.14	45.96	45.00

Table 4: Per-class HOTA ablation on iKUN (

n{=}5

seeds); single

\alpha

\alpha_{\mathrm{mot}}{=}\alpha_{\mathrm{oth}}{=}0.35

(LOSO). Others

=

all expressions outside the moving class.

Figure 2: “Moving cars” (Refer-KITTI seq 0005). The parked car (red) sweeps across the image as the camera passes, while the moving car (green) barely shifts. Raw image motion misclassifies both; the ego-compensated residual recovers both from the same host score.

Camera Motion Understanding

See all 19 papers

STORM: End-to-End Referring Multi-Object Tracking in Videos
Apr 12, 2026Zijia Lu, Jingru Yi, Jue Wang +4Referring MOT Multi-Object Tracking
Amazon
Referring Multi-Object Tracking with Diverse Dynamic Expressions
Jun 7, 2024Yani Zhang, Dongming Wu, Wencheng Han +3Referring MOT Multi-Object Tracking
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China. · Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong, China. · State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau, Macau, China. +1
CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking
Jul 28, 2026Xiangqun Zhang, Likai Wang, Zekun Qian +2Synthetic-to-Real Domain Adaptation Referring MOT
School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China

Apr 12, 2026J/K move · Enter open · S save

STORM: End-to-End Referring Multi-Object Tracking in Videos

Zijia Lu, Jingru Yi, Jue Wang, Yuxiao Chen+3

Amazon

Referring MOT Multi-Object Tracking Multimodal Visual Object Tracking Multimodal Grounding Unified Multimodal Models

Open paper AskPDF arXiv

Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMOT approaches decompose object grounding and tracking into separated modules and exhibit limited performance due to the scarcity of training videos, ambiguous annotations, and restricted domains. In this work, we introduce STORM, an end-to-end MLLM that jointly performs grounding and tracking within a unified framework, eliminating external detectors and enabling coherent reasoning over appearance, motion, and language. To improve data efficiency, we propose a task-composition learning (TCL) strategy that decomposes RMOT into image grounding and object tracking, allowing STORM to leverage data-rich sub-tasks and learn structured spatial--temporal reasoning. We further construct STORM-Bench, a new RMOT dataset with accurate trajectories and diverse, unambiguous referring expressions generated through a bottom-up annotation pipeline. Extensive experiments show that STORM achieves state-of-the-art performance on image grounding, single-object tracking, and RMOT benchmarks, demonstrating strong generalization and robust spatial--temporal grounding in complex real-world scenarios. STORM-Bench is released at https://github.com/amazon-science/storm-referring-multi-object-grounding.

Abstract

Camera Motion Understanding

Explore similar work

STORM: End-to-End Referring Multi-Object Tracking in Videos

Referring Multi-Object Tracking with Diverse Dynamic Expressions

CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking