cs.CVJun 7, 2024
SaveReferring Multi-Object Tracking with Diverse Dynamic Expressions
Organizations: School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China. · Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong, China. · State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau, Macau, China. · College of Information Technology, United Arab Emirates University, Al Ain, UAE.
Abstract
Referring understanding is a fundamental task that bridges natural language and visual content by localizing objects described in free-form expressions. However, existing works are constrained by limited language expressiveness, lacking the capacity to model object dynamics in spatial numbers and temporal states. To address these limitations, we introduce a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking, comprehensively accounting for variations in object quantity and temporal semantics. Along with RMOT, we introduce a RMOT benchmark named Refer-KITTI-V2, featuring scalable and diverse language expressions. To efficiently generate high-quality annotations covering object dynamics with minimal manual effort, we propose a semi-automatic labeling pipeline that formulates a total of 9,758 language prompts. In addition, we propose TempRMOT, an elegant end-to-end Transformer-based framework for RMOT. At its core is a query-driven Temporal Enhancement Module that represents each object as a Transformer query, enabling long-term spatial-temporal interactions with other objects and past frames to efficiently refine these queries. TempRMOT achieves state-of-the-art performance on both Refer-KITTI and Refer-KITTI-V2, demonstrating the effectiveness of our approach. The source code and dataset is available at https://github.com/zyn213/TempRMOT.
Figures & tables
Fig. 1 : Representative examples from RMOT . The expression query can refer to multiple objects of interest (a), and captures the short-term status with accurate labels (b).
Fig. 3 : Language Prompt Annotation Pipeline consists of three steps: language item collection, prompt generation, and prompt expansion. Firstly, we use an efficient labeling tool to associate instances in each video with language elements at low human cost. Then, we manually create accurate language descriptions. Finally, leveraging the powerful language understanding capabilities of large language models, we expand the new annotations with language descriptions.
Fig. 5 : Statistics of Refer-KITTI-V2 and Refer-Animal. (a) Distribution of the proportion of referenced objects. For example, in Refer-KITTI-V2, 57.17% of the expressions refer to objects that make up 0-20% of the total objects in the video. (b) Distribution of object number. (c) Distinct expressions comparison.
Fig. 6 : The statistics of expressions in Refer-KITTI-V2.
Fig. 7 : Overall architecture of our TempRMOT. The proposed framework comprises two key components: a Transformer-based RMOT module and a temporal enhancement module. The Transformer-based RMOT accepts a video frame, a language expression, and a set of learnable queries as input. Its output comprises a set of embeddings corresponding to the objects. The temporal enhancement module harnesses a memory mechanism to integrate temporal information for query enhancement. Our loss includes detection loss , based on image-level detection and spatio-temporal loss based on temporal modeling.
Fig. 8 : The details of temporal enhancement module. It contains two parts: temporal decoder and object decoder.
| Method | E | HOTA | DetA | AssA | DetRe | DetPr | AssRe | AssPr | LocA |
| FairMOT † [ 72 ] | ✗ | 22.78 | 14.43 | 39.11 | 16.44 | 45.48 | 43.05 | 71.65 | 74.77 |
| DeepSORT [ 73 ] | ✗ | 25.59 | 19.76 | 34.31 | 26.38 | 36.93 | 39.55 | 61.05 | 71.34 |
| ByteTrack † [ 74 ] | ✗ | 24.95 | 15.50 | 43.11 | 18.25 | 43.48 | 48.64 | 70.72 | 73.90 |
| CStrack [ 75 ] | ✗ | 27.91 | 20.65 | 39.10 | 33.76 | 32.61 | 43.12 | 71.82 | 79.51 |
| TransTrack [ 76 ] | ✗ | 32.77 | 23.31 | 45.71 | 32.33 | 42.23 | 49.99 | 78.74 | 79.48 |
| TrackFormer [ 77 ] | ✗ | 33.26 | 25.44 | 45.87 | 35.21 | 42.10 | 50.26 | 78.92 | 79.63 |
TABLE III: Comparison with state-of-the-art methods on Refer-KITTI. “E” means “End to End”. † represents results after frame correction. The best results are in bold.
| Method | E | HOTA | DetA | AssA | DetRe | DetPr | AssRe | AssPr | LocA |
| Refer-KITTI-V2 dataset | |||||||||
| FairMOT [ 72 ] | ✗ | 22.53 | 15.80 | 32.82 | 20.60 | 37.03 | 36.21 | 71.94 | 78.28 |
| ByteTrack [ 74 ] | ✗ | 24.59 | 16.78 | 36.63 | 22.60 | 36.18 | 41.00 | 69.63 | 78.00 |
| iKUN [ 19 ] | ✗ | 10.32 | 2.17 | 49.77 | 2.36 | 19.75 | 58.48 | 68.64 | 74.56 |
| ReferGPT [ 28 ] | ✗ | 30.12 | 15.69 | 59.02 | 21.55 | 34.41 | 74.59 | 68.20 | 79.76 |
| TransRMOT [ 18 ] | ✓ | 31.00 | 19.40 | 49.68 | 36.41 | 28.97 | 54.59 | 82.29 | 89.82 |
TABLE IV: Comparison with state-of-the-art methods on Refer-KITTI-V2 and KITTI. “E” means “End to End”. The best results are in bold.
TABLE V : Ablation studies of our proposed TempRMOT on Refer-KITTI-V2.
Fig. 9 : Representative examples of Refer-KITTI . It provides high-diversity scenes and high-quality annotations referred to by expressions.
Fig. 10 : Representative examples of Refer-KITTI-V2. It possesses the advantages of complex motion states, linguistic flexibility, and multi-perspective descriptions, showing strong semantic diversity.
Fig. 11 : Visualization of predictions from TransRMOT and our TempRMOT. TempRMOT not only accurately understands instructions related to motion without wrong identification, but also successfully detects and tracks objects.
| Method | HOTA | DetA | AssA | DetRe | DetPr |
| Refer-BDD dataset | |||||
| MOTRv2 [ 70 ] | 35.35 | 22.88 | 55.24 | 46.35 | 30.33 |
| CO-MOT [ 83 ] | 36.72 | 24.88 | 54.73 | 30.01 | 56.95 |
| TransRMOT [ 18 ] | 34.79 | 26.22 | 47.56 | 38.46 | 41.54 |
| EchoTrack [ 20 ] | 38.00 | 28.57 | 51.24 | 36.60 | 54.25 |
| TempRMOT(Ours) | 41.23 | 30.10 | 57.91 | 38.09 | 56.55 |
TABLE VI: Comparison with state-of-the-art methods on Refer-BDD and Refer-Dance. The best results are in bold.
| Temp. | Object. | PE | HOTA | DetA | AssA | DetRe |
| ✗ | ✗ | ✗ | 31.00 | 19.40 | 49.68 | 36.41 |
| ✓ | ✗ | ✗ | 33.52 | 22.12 | 52.86 | 32.61 |
| ✗ | ✓ | ✗ | 32.90 | 20.64 | 52.78 | 28.87 |
| ✓ | ✓ | ✗ | 34.43 | 22.28 | 53.32 | 32.72 |
| ✓ | ✓ | ✓ | 35.04 | 22.97 | 53.58 | 34.23 |
TABLE VII: Ablation study on temporal enhancement module. “Temp.”, “Object.” and “PE” denote the temporal decoder, object decoder, and positional embeddings, respectively.
| Method | HOTA | DetA | AssA | DetRe | DetPr |
| iKUN [ 19 ] | 8.90 | 1.61 | 49.25 | 1.69 | 24.90 |
| TransRMOT [ 18 ] | 20.41 | 9.47 | 44.12 | 10.43 | 49.58 |
| TempRMOT(Ours) | 22.47 | 10.13 | 50.60 | 12.12 | 37.32 |
TABLE VIII: Comparison on implicit expressions
| Expression Type | HOTA | DetA | AssA | DetRe | DetPr |
| Human Only | 27.69 | 15.83 | 48.54 | 20.88 | 38.90 |
| Human + LLM | 35.04 | 22.97 | 53.58 | 34.23 | 40.41 |
TABLE IX: Effect of LLM-Generated Expressions.
| Memory Design | HOTA | DetA | AssA | DetRe | DetPr |
| Baseline | 31.00 | 19.40 | 49.68 | 36.41 | 28.97 |
| Memory Pooling | 31.31 | 20.83 | 47.21 | 34.57 | 33.89 |
| Convolution | 31.52 | 22.33 | 44.65 | 32.09 | 41.58 |
| GRU [ 84 ] | 31.88 | 22.57 | 45.18 | 32.37 | 41.95 |
| Ours | 35.04 | 22.97 | 53.58 | 34.23 | 40.41 |
TABLE X: Ablation study on temporal enhancement module
| Train | Val | HOTA | DetA | AssA |
| Refer-Dance [ 19 ] | GroOT [ 45 ] | 2.46 | 0.49 | 13.57 |
| Refer-KITTI [ 18 ] | GroOT [ 45 ] | 6.75 | 1.25 | 38.69 |
| Refer-KITTI-V2 | GroOT [ 45 ] | 12.01 | 2.91 | 49.80 |
TABLE XI: Ablation study on cross data generalization.
| Method | Params | FLOPs | Inference | HOTA |
| TempRMOT | 49.6M | 294.0G | 144.18ms | 35.04 |
| wo Temp. | 42.8M | 288.9G | 136.77ms | 31.00 |
TABLE XII: Computational cost of the temporal module
| Expression Type | HOTA | DetA | AssA | DetRe | DetPr |
| Human Only | 30.09 | 19.09 | 47.59 | 32.29 | 31.41 |
| Human + LLM | 37.90 | 28.34 | 50.82 | 39.36 | 49.35 |
TABLE XIII: Study on Free-Form Human Expressions.
Fig. 12 : Representative examples of Refer-Animal.
Fig. 13 : Failure cases of TempRMOT.
| Yani Zhang received the BS degree in the School of Computer Science from Wuhan University in 2024. She is currently working toward the ME degree with the School of Computer Science, Wuhan University. Her current research interests include vision-language learning, embodied perception and multi-modal LLMs. |
| Dongming Wu is currently a Postdoctoral Fellow with the Multimedia Lab (MMLab) at the Chinese University of Hong Kong. He received the Ph.D. degree in Computer Science from Beijing Institute of Technology in 2025. His current research interests include vision-language learning, and multi-modal LLMs. He has published more than 10 papers in refereed conference proceedings and journals such as IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), ICML, ICLR, CVPR, ICCV, ECCV, etc. |
| Wencheng Han is currently a Full Professor with the School of Control Science and Engineering, Shandong University, Jinan, China. He received the Ph.D. degree in computer science from the University of Macau, Macao, China, in 2025, where he subsequently worked as a Postdoctoral Fellow with the State Key Laboratory of Internet of Things for Smart City (SKL-IOTSC) from 2025 to 2026. His research interests include computer vision, autonomous driving, robot control, and artificial intelligence-generated content (AIGC). He has authored or co-authored more than 30 papers in leading journals and conferences, including IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), CVPR, ICCV, and ECCV. |
| Xingping Dong received the BS degree from Xiamen University, in 2012, and the Ph.D. degree from Beijing Institute of Technology, in 2019. He is currently a professor with the School of Computer Science, Wuhan University. Before that, he was a research scientist with the Inception Institute of Artificial Intelligence (IIAI), UAE. From 2016 to 2018, he was a joint PhD candidate with the College of Engineering and Computer Science, Australian National University. His current research interests include computer vision, autonomous driving, and deep learning. |
| Shengcai Liao is an Associate Professor in the College of Computing and Artificial Intelligence (CCAI) in UAEU. He is an IEEE Fellow and IAPR Fellow. He was a Principal Scientist in IIAI, G42 in UAE during 2018-2024, and an Associate Professor in Institute of Automation, Chinese Academy of Sciences (CASIA) from 2014 to 2018. He received B.S. degree in mathematics from Sun Yat-sen University in 2005 and Ph.D. degree from CASIA in 2010. He was a Postdoc in Michigan State University during 2010-2012. His research interests include object detection, recognition, image generation, especially face and person analysis and synthesis. He was awarded Best Paper/Best Student Paper in ICB 2006/2007/2015, and CCBR 2016, Best/Outstanding Reviewer in IJCB 2014 and CVPR 2019/2021. He served as a Program Chair for IJCB 2022, Area Chair for ICPR 2016, ICB 2016/2018, CVPR 2022/2023, ICCV 2023, and NeurIPS 2023. |
| Bo Du (M’10–SM’15) is currently a professor with the School of Computer Science and Institute of Artificial Intelligence, Wuhan University. He is also the director of National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan, China. He has more than 80 research papers published in the IEEE Transactions on Image Processing (TIP), IEEE Transactions on Cybernetics (TCYB), IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAMI), IEEE Transactions on Geoscience and Remote Sensing (TGRS), etc. Twenty-four of them are ESI hot papers or highly cited papers. His major research interests include machine learning, computer vision, and image processing. He won IEEE Geoscience and Remote Sensing Society 2020 Transactions Prize Paper Award. He won the IJCAI (International Joint Conferences on Artificial Intelligence) Distinguished Paper Prize, IEEE Data Fusion Contest Champion, and IEEE Workshop on Hyperspectral Image and Signal Processing Best Paper Award in 2018. He regularly serves as a senior PC member of IJCAI and AAAI. He served as area chair for ICPR. |