S3KF: Spherical State-Space Kalman Filtering for Panoramic 3D Multi-Object Tracking

Zhongyuan Liu; Shaonan Yu; Jianping Li; Pengfei Wan; Xinhang Xu; Pengfei Wang; Maggie Y. Gao; Lihua Xie

cs.ROMar 29, 2026

S3KF: Spherical State-Space Kalman Filtering for Panoramic 3D Multi-Object Tracking

Authors: Zhongyuan Liu, Shaonan Yu, Jianping Li, Pengfei Wan, Xinhang Xu, Pengfei Wang, Maggie Y. Gao, Lihua Xie

Organizations: CertaintyX · the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · ST Engineering, Singapore · the School of Civil and Environmental Engineering, Nanyang Technological University, Singapore

Abstract

Panoramic multi-object tracking is important for industrial safety monitoring, wide-area robotic perception, and infrastructure-light deployment in large workspaces. In these settings, the sensing system must provide full-surround coverage, metric geometric cues, and stable target association under wide field-of-view distortion and occlusion. Existing image-plane trackers are tightly coupled to the camera projection and become unreliable in panoramic imagery, while conventional Euclidean 3D formulations introduce redundant directional parameters and do not naturally unify angular, scale, and depth estimation. In this paper, we present $\mathbf{S^3KF}$ , a panoramic 3D multi-object tracking framework built on a motorized rotating LiDAR and a quad-fisheye camera rig. The key idea is a geometry-consistent state representation on the unit sphere $\mathbb{S}^2$ , where object bearing is modeled by a two-degree-of-freedom tangent-plane parameterization and jointly estimated with box scale and depth dynamics. Based on this state, we derive an extended spherical Kalman filtering pipeline that fuses panoramic camera detections with LiDAR depth observations for multimodal tracking. We further establish a map-based ground-truth generation pipeline using wearable localization devices registered to a shared global LiDAR map, enabling quantitative evaluation without motion-capture infrastructure. Experiments on self-collected real-world sequences show decimeter-level planar tracking accuracy, improved identity continuity over a 2D panoramic baseline in dynamic scenes, and real-time onboard operation on a Jetson AGX Orin platform. These results indicate that the proposed framework is a practical solution for panoramic perception and industrial-scale multi-object tracking.The project page can be found at https://kafeiyin00.github.io/S3KF/.

Figures & tables

Fig. 2: Hardware of the proposed panoramic sensing system.

Fig. 3: Workflow of the proposed panoramic 3D multi-object tracking. Arrows indicate the data flow: the quad-fisheye camera array provides panoramic 2D detections (target bearing and bounding box), and the rotating LiDAR provides metric depth measurements; both modalities are transformed into the unified spherical state representation and fused in the extended spherical Kalman filter for joint state estimation and data association.

Fig. 4: $\mathbf{g}\in\mathbb{S}^{2}$ denotes the true bearing vector of the object. Because $\mathbb{S}^{2}$ is a nonlinear manifold, we employ a local tangent parameterization for filtering.

Fig. 5: The hardware of the localization system for each agent. Each person is wearing a LiDAR and a wireless module, which will be used to generate a ground-truth trajectory .

Stage	Median (ms)	P95 (ms)	Max (ms)
Panoramic projection (CPU)	35.0	37.0	66.7
Detection (TensorRT FP16)	17.6	20.9	–
Tracking (EKF + association)	2.4	3.3	11.0
Serial total	$\approx$ 55.0	$\approx$ 61	–

TABLE I: Per-stage runtime on the Jetson AGX Orin (559 frames).

Seq.	Motion	Radial	Tangential	Vertical	3D RMSE
Seq1	Static	$-0.060$ /0.142	$-0.014$ /0.043	$+0.039$ /0.064	0.161
Seq2	CW	$-0.122$ /0.150	$+0.016$ /0.075	$+0.063$ /0.075	0.183
Seq3	CCW	$-0.083$ /0.119	$-0.093$ /0.122	$+0.092$ /0.105	0.200
Seq4	Fast + occl.	$-0.108$ /0.158	$+0.066$ /0.127	$+0.043$ /0.077	0.217
Seq5	Fast radial	$-0.033$ /0.139	$-0.073$ /0.149	$+0.101$ /0.128	0.241
All	–	$-0.086$ /0.142	$-0.016$ /0.105	$+0.064$ /0.088	0.197

TABLE II: 3D error decomposition per sequence (m): bias/RMSE, using per-target trajectory pairing under the head-top-aligned protocol of Section V-B. The motion attribute of each sequence is indicated. (The per-frame nearest-neighbor pairing with a 5 m gate used in Table IV yields a merged RMSE of 0.276 m.)

Seq.	Method	MOTA	MOTP	IDF1	HOTA	IDS
Seq1	Ours	0.997	0.114	0.999	0.915	0
Seq1	Baseline	0.952	0.087	0.976	0.922	0
Seq2	Ours	0.989	0.173	0.994	0.854	0
Seq2	Baseline	0.907	0.260	0.540	0.520	5
Seq3	Ours	0.997	0.190	0.998	0.844	0
Seq3	Baseline	0.710	0.311	0.439	0.420	8

TABLE III: MOT metrics per sequence (

d_{\max}=1.0

m).

Fig. 6: 3D tracking error over time for the three targets in (a)–(e) Seq1–Seq5 on a common axis; dotted lines mark the per-target RMSE (m), also listed in the inset table.

Fig. 7: Evaluation on the robot dog in a real-world outdoor environment. (a) Objects standing. (b) Objects moving clockwise. (c) Objects moving counterclockwise. (d) Objects suddenly changing direction.

Fig. 8: Evaluation on the drone in a real-world outdoor environment. (a) Objects standing. (b) Objects moving clockwise. (c) Objects moving counterclockwise. (d) Objects suddenly changing direction.

Variant	3D RMSE (m)	std	P95	P99	f $>$ 0.4	MOTA	IDF1	HOTA	AssA	IDS
Full S 3 KF	0.276	0.204	0.339	0.661	2.8%	0.983	0.992	0.856	1.000	0
w/o depth fusion	0.381	0.226	0.645	0.734	19.4%	0.981	0.976	0.735	0.980	1
Cartesian EKF (height-coupled)	0.279	0.204	0.344	0.667	3.2%	0.983	0.992	0.853	1.000	0
w/o seam-aware IoU	0.302	0.230	0.348	0.749	3.2%	0.977	0.911	0.809	0.911	17
w/o blind-zone gain decay	0.283	0.208	0.389	0.643	4.6%	0.983	0.992	0.851	1.000	0
2D image-plane KF baseline	0.825	0.705	2.652	3.160	20.8%	0.860	0.649	0.623	0.697	20

TABLE IV: Ablation on the five sequences (merged,

n=6894

paired frames). Error statistics use a 5 m gate with per-frame nearest-neighbor pairing (drift counted as error); MOT metrics use

d_{\max}=1.0

Method	3D RMSE (m)	P95	f $>$ 0.4	MOTA	IDF1	AssA	IDS
Full S 3 KF	0.276	0.339	2.8%	0.983	0.992	1.000	0
2D image-plane KF	0.825	2.652	20.8%	0.860	0.649	0.697	20
Cartesian 3D KF	0.612	1.316	23.3%	0.841	0.729	0.777	24

TABLE V: Comparison with stronger baselines (merged over the five sequences; protocol as in Table IV ).

Sequence	ByteTrack (2D)	Proposed tracker
Ground-Outdoor	26	3
Ground-Indoor	18	3
Aerial-Outdoor	27	3
Aerial-Indoor	27	7

TABLE VI: Track-ID proliferation on four additional panoramic sequences: maximum assigned track ID per sequence.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9: Process-noise sweep of the Cartesian baselines over $q\in[0.01,100]$ (five sequences merged and per-sequence): merged 3D RMSE, P99, and the fraction of frames above 0.4 m, with the selected operating point marked. No single $q$ is optimal across all sequences.

Symbol	Description
$\mathbb{S}^{2}$	Unit sphere used to represent object bearing directions.
$\mathbf{g}_{k}\in\mathbb{S}^{2}$	Object bearing vector at time step $k$ .
$\bar{\mathbf{g}}_{k}\in\mathbb{S}^{2}$	Reference bearing used to define the local tangent space.
$T_{\bar{\mathbf{g}}_{k}}\mathbb{S}^{2}$	Tangent space of $\mathbb{S}^{2}$ at $\bar{\mathbf{g}}_{k}$ .
$\mathbf{B}_{k}=[\mathbf{b}_{1,k},\mathbf{b}_{2,k}]$	Orthonormal basis spanning $T_{\bar{\mathbf{g}}_{k}}\mathbb{S}^{2}$ .
$\mathbf{w}_{k}=[w_{1,k},w_{2,k}]^{\top}$	Local tangent-plane coordinates of the bearing state.

Appendix

TABLE VII: Notation used in the spherical tracking formulation.

Parameter	Value
Detector	YOLO11-m, TensorRT FP16, $640\times 640$ , COCO-pretrained (no fine-tuning)
Confidence / NMS threshold	0.50 / 0.65
Panorama	$1280\times 640$ equirectangular, vertical FoV $\pm 90^{\circ}$
Camera model	Kalibr omni (Mei) + radtan, 4 fisheye
Sensor rates	image 15.1 Hz; LiDAR 10 Hz; IMU $\approx$ 200 Hz; encoder 167 Hz
track / high / match thresholds	0.30 / 0.50 / 0.80 (low-score 0.50; unconfirmed 0.70)

Appendix

TABLE VIII: Key parameters of the implementation.

Seq.	Method	MOTA	MOTP (m)	IDF1	HOTA	DetA	AssA	FP	FN	IDS	Frag	MT/PT/ML
Seq1	Ours	0.997	0.114	0.999	0.915	0.869	1.000	4	0	0	0	3/0/0
Seq1	Baseline	0.952	0.087	0.976	0.922	0.881	1.000	2	73	0	0	3/0/0
Seq2	Ours	0.989	0.173	0.994	0.854	0.810	1.000	4	13	0	9	3/0/0
Seq2	Baseline	0.907	0.260	0.540	0.520	0.623	0.468	29	105	5	17	3/0/0
Seq3	Ours	0.997	0.190	0.998	0.844	0.799	1.000	3	2	0	2	3/0/0
Seq3	Baseline	0.710	0.311	0.439	0.420	0.457	0.482	120	338	8	14	2/1/0

Appendix

TABLE IX: Complete MOT metrics per sequence and merged (

d_{\max}=1.0

m). MOTP is in meters; FP/FN: false-positive/false-negative frames; Frag: fragmentations; MT/PT/ML: numbers of mostly-tracked, partially-tracked, and mostly-lost targets. HOTA is reported together with its detection (DetA) and association (AssA) components, following [ 30 ] .

3D MOT

3D multi-object tracking (3D MOT) estimates object locations and identities over time from cameras, LiDAR, radar, and other sensors.

See all 40 papers

CylindTrack: Depth-Aware Cylindrical Motion Modeling for Panoramic Multi-Object Tracking
Jun 29, 2026Buyin Deng, Kai Luo, Lingxin Huang +5Multi-Object Tracking
the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · the School of Advanced Technology, Xi’an Jiaotong-Liverpool University, China · Suzhou VSDeep Intelligent Technology Co., Ltd., China +1
Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking
Sep 1, 2026Orcun Cetintas, Guillem Brasó, Tim Meinhardt +13D MOT Multi-Object Tracking
1NVIDIA · 2Technical University of Munich
PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views
Jun 25, 2026Zhisong Xu, Takeshi OishiMulti-View Consistency 3D Gaussian Splatting
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan

Jun 29, 2026J/K move · Enter open · S save

CylindTrack: Depth-Aware Cylindrical Motion Modeling for Panoramic Multi-Object Tracking

Buyin Deng, Kai Luo, Lingxin Huang, Xinqi Liu+4

the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · the School of Advanced Technology, Xi’an Jiaotong-Liverpool University, China · Suzhou VSDeep Intelligent Technology Co., Ltd., China+1

Multi-Object Tracking

Open paper AskPDF arXiv

Multi-Object Tracking (MOT) is essential for embodied perception, where panoramic cameras provide a 360° field of view to reduce blind spots and keep surrounding targets observable for longer durations. However, panoramic MOT requires motion estimates and association cues consistent with its imaging geometry. In videos represented in equirectangular projection (ERP), horizontal coordinates wrap at the image seam, while frame-wise relative-depth estimates may fluctuate over time. These inconsistencies can disrupt trajectory prediction and identity association. To address these challenges, we propose CylindTrack, a depth-aware cylindrical tracking-by-detection framework that integrates continuous angular motion modeling with temporally filtered relative-depth cues. Specifically, Depth-Temporal Trajectory Modeling (DTM) promotes instance depth from an isolated frame-wise cue into temporally filtered trajectory-level states. Spherical Spatio-Temporal Consistency Learning (SSTC) combines a Temporal Mixer and Spherical Geometry-aware Attention to enhance temporal coherence and panoramic geometric alignment in depth-aware representations. Furthermore, a Topology-Aware Cylindrical Motion Model (TCMM) lifts horizontal motion into a continuous angular state space, enabling seam-consistent motion prediction and association. By jointly modeling trajectory-level depth consistency and panoramic topology, CylindTrack improves identity preservation and trajectory continuity in challenging panoramic scenes. Experiments on QuadTrack and JRDB demonstrate improvements in identity-oriented metrics with practical efficiency, achieving HOTA scores of 33.67 and 31.12, IDF1 scores of 40.45 and 34.33, and tracking speeds of 28.56 and 21.34 FPS, respectively. The source code will be released at https://github.com/warriordby/CylindTrack.

Abstract

3D MOT

Explore similar work

CylindTrack: Depth-Aware Cylindrical Motion Modeling for Panoramic Multi-Object Tracking

Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking

PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views