A social robot has to keep track of the people around it over time to plan its motion, a task handled by 3D multi-object tracking. For the LiDAR modality, trackers usually sit on top of a detector and link its detections across frames with a separate motion model or re-identification network. We instead show how a 3D LiDAR detector itself can be turned into a tracker by adding an instance-level contrastive objective on its intermediate voxel features. Each embedding is read from the voxels inside a detected box, and boxes are matched across frames by cosine similarity and the Hungarian algorithm, without a motion model. The detector backbone is fine-tuned together with this objective, which learns identity-aware features while leaving detection accuracy essen- tially intact. Because the method uses only the voxel features that Li- DAR detectors already produce, it is backbone-agnostic. On the JRDB 3D tracking leaderboard our LiDAR-only tracker ranks first on HOTA (38.58) and IDF1 (37.99), 13.04 and 9.30 points above the next pub- lished method, and second on MOTA and OSPA. Code is available at https://hermak56.github.io/CLIFT/.
Figures & tables
Figure 1: The CLIFT pipeline. A single forward pass through the 3D detector produces both predicted boxes (top, detection branch) and the voxel-level feature map. For each box, the tracking branch (bottom, this work) gathers the voxels inside it, mean-pools their features, projects and L2 -normalizes them into an instance embedding eit , and the online tracker associates embeddings across frames by cosine similarity and the Hungarian algorithm. The backbone is either frozen or jointly fine-tuned; the detection head is never modified.
Metric
Config A
Config E
HOTA ↑
30.70
38.58
DetA ↑
47.10
46.04
AssA ↑
20.07
32.41
AssPr ↑
32.96
70.06
MOTA ↑
40.24
42.55
IDF1 ↑
27.99
37.99
Table 1: JRDB 3D tracking test set. Configuration A is the frozen-backbone baseline; Configuration E uses joint backbone fine-tuning. Best per row in bold. Arrows give the direction of improvement. The transition from A to E adds 7.9 HOTA points and reduces identity switches by about 60%, at a small detection cost. The OSPA (2) row is the one exception, analyzed in Section 4.4 .
Method
HOTA ↑
MOTA ↑
IDF1 ↑
OSPA (2) ↓
DRFDFF †
24.78
43.37
27.47
0.903
ELPTNet [ 14 ]
25.54
32.65
28.69
0.783
S2MAT [ 13 ]
18.93
24.21
21.34
0.975
SS3D-MOT [ 23 ]
15.80
22.96
14.48
0.982
PC-DAN [ 22 ]
15.57
22.57
13.89
0.976
AB3DMOT [ 43 ]
11.81
19.34
10.67
0.971
Table 2: JRDB 3D tracking leaderboard. Best in bold, second best underlined. Baseline values are taken from the public leaderboard. † Entry on the public JRDB 3D tracking leaderboard [ 28 ] without an associated publication.
Config
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
IDSW ↓
A
26.94
37.07
20.82
56.88
27.89
84,959
B
27.03
37.40
20.83
59.35
27.54
63,678
C
32.91
37.26
30.22
64.03
37.07
23,235
D
33.11
37.31
30.41
64.18
37.21
22,890
E
34.43
37.25
32.99
64.27
39.23
21,147
Table 3: Validation-set ablation over training configurations on the DCCLA backbone. Best per column in bold. A: same-frame augmentation, frozen. B: temporal pairs. C: K=4 multi-frame. D: A → B → C curriculum. E: joint fine-tune. The B → C step is the largest jump; E adds a further gain on top of C; DetA is stable throughout.
intra ↑
top-1 ↑
Backbone
t+1
t+10
t+1
t+10
frozen
0.936 / 0.872
0.836 / 0.749
0.687 / 0.851
0.419 / 0.594
joint
0.954 / 0.935
0.875 / 0.842
0.773 / 0.927
0.496 / 0.759
Table 4: Feature probe on the held-out validation sequences. Cells give detector features / embedding. Joint fine-tuning improves both, and the margin grows with the temporal gap.
LiDAR-based 3D Multi-Object Tracking (MOT) typically relies solely on geometric information, which is often insufficient to distinguish between targets during prolonged occlusions or in crowded human-populated environments. While integrating RGB-based Re-Identification (ReID) offers a theoretical solution for preserving identity context, existing approaches often rely on computationally expensive parallel detectors that hinder real-time robot responsiveness. This work presents a systematic study of image-based ReID in online 3D MOT, utilizing a lightweight projection-based framework to decouple geometric and appearance modeling for mobile robots. A comprehensive analysis of feature extraction architectures is conducted, employing lightweight CNNs and Vision Transformers, and evaluating various multi-modal data association strategies to balance computational latency with robust tracking. Experiments on the Pedestrian class of the KITTI dataset reveal that naive linear fusion, of appearance and motion costs, degrades performance due to visual noise. Conversely, a cascaded matching strategy successfully recovers occluded tracks without compromising overall precision, effectively preventing identity switches to maintain human-robot interaction continuity. We show that lightweight architectures can offer an optimal trade-off between the low latency required for safe navigation and the discriminative power needed for social awareness.
Eduardo Borges, Luís Garrote, Urbano J. Nunes
Institute of Systems and Robotics, Department of Electrical and Computer Engineering, University of Coimbra, Portugal
LiDAR-camera 3D multi-object tracking (MOT) combines rich visual semantics with accurate depth cues to improve trajectory consistency and tracking reliability. In practice, however, LiDAR and cameras operate at different sampling rates. To maintain temporal alignment, existing data pipelines usually synchronize heterogeneous sensor streams and annotate them at a reduced shared frequency, forcing most prior methods to perform spatial fusion only at synchronized timestamps through projection-based or learnable cross-sensor association. As a result, abundant asynchronous observations remain underexploited, despite their potential to support more frequent association and more robust trajectory estimation over short temporal intervals. To address this limitation, we propose Fusion-Poly, a spatial-temporal fusion framework for 3D MOT that integrates asynchronous LiDAR and camera data. Fusion-Poly associates trajectories with multi-modal observations at synchronized timestamps and with single-modal observations at asynchronous timestamps, enabling higher-frequency updates of motion and existence states. The framework contains three key components: a frequency-aware cascade matching module that adapts to synchronized and asynchronous frames according to available detection modalities; a frequency-aware trajectory estimation module that maintains trajectories through high-frequency motion prediction, differential updates, and confidence-calibrated lifecycle management; and a full-state observation alignment module that improves cross-modal consistency at synchronized timestamps by optimizing image-projection errors. On the nuScenes test set, Fusion-Poly achieves 76.5% AMOTA, establishing a new state of the art among tracking-by-detection 3D MOT methods. Extensive ablation studies further validate the effectiveness of each component. Code will be released.
Xian Wu, Yitao Wu, Xiaoyu Li +3
State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150006, China.
Safe autonomous agents and mobile robots need fast real time 3D perception, especially for vulnerable road users (VRUs) such as pedestrians. We introduce a new bird's eye view (BEV) encoding, which maps the full 3D LiDAR point cloud into a light-weight 2D BEV tensor with three height bands. We explicitly reformulate 3D detection as a 2D detection problem and then reconstruct 3D boxes from the BEV outputs. A single network detects cars, pedestrians, and cyclists in one pass. The backbone uses area attention at deep stages, a hierarchical bidirectional neck over P1 to P4 fuses context and detail, and the head predicts oriented boxes with distribution focal learning for side offsets and a rotated IoU loss. Training applies a small vertical re bin and a mild reflectance jitter in channel space to resist memorization. We use an interquartile range (IQR) filter to remove noisy and outlier LiDAR points during 3D reconstruction. On KITTI dataset, TriBand-BEV attains 58.7/52.6/47.2 pedestrian BEV AP(%) for easy, moderate, and hard at 49 FPS on a single consumer GPU, surpassing Complex-YOLO, with gains of +12.6%, +7.5%, and +3.1%. Qualitative scenes show stable detection under occlusion. The pipeline is compact and ready for real time robotic deployment. Our source code is publicly available on GitHub.
Mohammad Khoshkdahan, Alexey Vinel
Karlsruhe Institute of Technology · Karlsruhe, Germany