IMM-based Multiple Object Tracking using a State Prediction Neural Network

Chan-Bin Lim; Dong-Hee Paek; Seung-Hyun Kong

cs.ROSep 10, 2026

IMM-based Multiple Object Tracking using a State Prediction Neural Network

Authors: Chan-Bin Lim, Dong-Hee Paek, Seung-Hyun Kong

Organizations: Robotics Program Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea, 34051 · Mechanical Engineering Research Institute Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea, 34141 · Cho Chun Shik Graduate School of Mobility Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea, 34051

Abstract

Object tracking is essential for autonomous vehicles to avoid obstacles and plan routes. Radar maintains detection performance even in adverse weather and can measure relative velocity through the Doppler effect, making it well suited for object tracking. In this paper, we propose a data-driven state PRedictor-based Interacting Multiple Model tracking method (PR-IMM) that improves nonlinear object-motion representation while preserving the stability and interpretability of physics-based motion models. The proposed method employs a transformer-based PRediction model (PR) that incorporates radar Doppler measurements to predict object displacement. The PR model is integrated into the IMM as a mode alongside the CV, CA, and CT motion models, and their prior positions are dynamically combined according to the mode probabilities. Experimental results show that PR-IMM reduces position-estimation error by 57.3% over the IMM and by 16.5% over the PR, while reducing ID switches by 25.3% and improving IDF1 by 9.6% over the IMM.

Figures & tables

Split	Sequences	Frames	Car object frames	Pedestrian object frames
Train	130	32,555	49,349	60,216
Validation	6	1,291	1,591	1,744
Evaluation	22	4,784	11,307	6,559

TABLE I: Dataset split and class composition.

Model	$\mathbf{RMSE_{x}\downarrow}$ [m]	$\mathbf{RMSE_{y}\downarrow}$ [m]	$\mathbf{RMSE_{xy}\downarrow}$ [m]
CV	4.884	3.801	6.189
CA	2.241	2.068	3.050
CT	4.872	3.545	6.025
PR	0.971	0.822	1.272
IMM (CV+CA+CT)	1.827	1.685	2.485
PR-IMM (Ours)	0.799	0.698	1.062

TABLE II: One-step prior prediction RMSE.

Scenario	CV	CA	CT	PR	IMM	Ours
Straight	4.007	1.308	5.119	0.633	1.038	0.584
Accel./Decel.	8.387	2.323	8.694	1.029	2.099	0.984
Left/Right Turn	7.055	2.999	6.781	1.024	2.291	0.904
Nonlinear	7.433	3.896	7.164	1.270	3.143	1.106

TABLE III: Scenario-wise prior prediction RMSE.

Model	$\mathbf{IDSW\downarrow}$	$\mathbf{IDF1\uparrow}$
CV	954	0.7583
CA	700	0.8075
CT	901	0.7571
PR	915	0.8795
IMM (CV+CA+CT)	695	0.8228
PR-IMM (Ours)	519	0.9020

TABLE IV: Overall multiple-object-tracking performance (GT object frames = 17,866).

Fig. 3: Qualitative tracking results on the RadarScenes dataset. (a) Radar point cloud with annotated vehicle points. (b) CV/CA/CT-based IMM result. (c) PR model result. (d) Proposed PR-IMM result. In (b)–(d), light thick lines denote ground-truth trajectories, whereas dark thin lines denote estimated trajectories.

Scenario (GT object frames)	Metric	PR	IMM	PR-IMM (Ours)
Close Proximity (1,284)	IDSW $\downarrow$	54	43	34
Close Proximity (1,284)	IDF1 $\uparrow$	0.8769	0.8100	0.8941
Crossing Motion (687)	IDSW $\downarrow$	20	21	11
Crossing Motion (687)	IDF1 $\uparrow$	0.9127	0.8486	0.9258
10% Miss (17,866)	IDSW $\downarrow$	1,245	836	711
10% Miss (17,866)	IDF1 $\uparrow$	0.7316	0.7185	0.7610

TABLE V: Scenario-wise multiple-object-tracking performance.

Fig. 4: Mean mode probability of PR-IMM.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 194 papers

IMMNet: Hybrid Fusion of Model-based and Data-driven Approaches for Maneuvering Target Tracking
Jul 15, 2026Yixuan Zhao, Chaoqun Yang, Lin Gao +2Bayesian Filtering
School of Automation, Southeast University, Nanjing, 210096, China · School of ICE, University of Electronic Science and Technology of China, Chengdu, 611731, China · Faculty of Artificial Intelligence, Shanghai University of Electric Power, Shanghai, 201300, China +1
Radar-Informed 3D Multi-Object Tracking under Adverse Conditions
Apr 15, 2026Bingxue Xu, Emil Hedemalm, Ajinkya Khoche +13D MOT Multimodal Robustness
KTH Royal Institute of Technology, Stockholm 10044, Sweden.
Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking
Sep 8, 2026Jue Wang, Xuan Wang, Hao Zhou +6Camera-Radar Fusion Video Object Segmentation
Harbin Institute of Technology, Shenzhen, China · Great Bay University, Dongguan, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

Jul 15, 2026J/K move · Enter open · S save

IMMNet: Hybrid Fusion of Model-based and Data-driven Approaches for Maneuvering Target Tracking

Yixuan Zhao, Chaoqun Yang, Lin Gao, Yongxiao Tian+1

School of Automation, Southeast University, Nanjing, 210096, China · School of ICE, University of Electronic Science and Technology of China, Chengdu, 611731, China · Faculty of Artificial Intelligence, Shanghai University of Electric Power, Shanghai, 201300, China+1

Bayesian Filtering

Open paper AskPDF arXiv

Maneuvering target tracking in three-dimensional space remains a challenging problem due to complex motion dynamics and model mismatch. To address this, this paper proposes a hybrid model/data-driven algorithm named IMMNet, which integrates the interpretable structure of the interacting multiple model (IMM) algorithm with learnable neural components. Unlike end-to-end black-box methods, the proposed IMMNet algorithm not only can preserve the Bayesian inference mechanism that is essential for real-time radar applications, but also can adaptively learn motion patterns and noise characteristics from data. Extensive experiments demonstrate that the proposed IMMNet algorithm consistently outperforms the existing algorithms across various scenarios, validating it as a robust, interpretable, and practical solution for maneuvering target tracking.

Abstract

Multi-Object Tracking

Explore similar work

IMMNet: Hybrid Fusion of Model-based and Data-driven Approaches for Maneuvering Target Tracking

Radar-Informed 3D Multi-Object Tracking under Adverse Conditions

Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking