Reliable 3D person detection is important for rescue robotics to give first responders situational awareness. Yet most multimodal 3D detection research targets automotive scenarios in which pedestrians mostly appear in a narrow range of upright, near-road poses. It is therefore unclear how much of the reported performance transfers to rescue settings, as a person may be lying, partially occluded, or in unstructured indoor and outdoor environments. To investigate this gap, we introduce TriSAR-3D, a multimodal person detection dataset recorded with a high-resolution 128-channel LiDAR, an RGB camera, and a thermal camera. It captures varied poses in indoor as well as outdoor locations at different times of day to study their influence on 3D object detector performance. Furthermore, we propose a novel trimodal 3D detection pipeline, PaintFusion, that fuses PointPainting-based evidence using a simple MLP. We show that the derived method reaches 65.4 AP@50 and thereby outperforms strong LiDAR-only (CenterPoint, 53.4), RGB+LiDAR (BEVFusion, 60.5), and adapted trimodal (Tri-BEVFusion, 60.7) baselines on the dataset. At the same time, it is substantially faster than other fusion-based methods such as BEVFusion, with a per-frame latency of 51.6 ms compared to 101 ms. Additionally, we test how the performance of different 3D object detectors generalizes to unseen pose categories. Removing lying instances from training data collapses the lying-pose AP@50 to near zero across all evaluated detectors. The results indicate that dataset diversity exerts a large effect on 3D object detection performance. The dataset and code are available at https://github.com/MarkusKramer1/3D-Trimodal-People-Detection.
Figures & tables
Fig. 1: TriSAR-3D: synchronized RGB, thermal, and LiDAR recordings of people in varied poses, indoors and outdoors, by day and at night.
Figure 2Figure 3
Fig. 6: Representative dataset examples with 3D ground-truth bounding boxes on synchronized RGB images (top row), thermal images (middle row), and LiDAR point clouds (bottom row). From left to right: indoor hallway, outdoor forest at night, outdoor sitting pose, indoor lying person, and outdoor night scene.
Fig. 7: Overview of the proposed PaintFusion detection network. RGB and thermal images are processed by separate 2D instance segmentation streams. The resulting features are projected onto the LiDAR point cloud, scored per point, and fed into a PointPillars encoder with a CenterPoint detection head for 3D box prediction.
Figure 6
Regime
Method
Overall
AP@50 by pose
Standing
Lying
Walking
Sitting
LiDAR-only
PointPillars [ 11 ]
47.1 / 40.6
20.6 / 8.3
11.0 / 0.1
59.8 / 59.9
27.1 / 19.6
PV-RCNN [ 12 ]
38.6 / 39.7
20.2 / 13.8
12.0 / 0.3
44.1 / 47.7
25.3 / 31.6
SECOND [ 10 ]
32.6 / 28.3
18.0 / 5.2
22.1 / 0.0
41.2 / 39.5
14.2 / 19.1
CenterPoint [ 13 ]
53.4 / 49.4
21.9 / 11.1
23.8 / 0.1
62.5 / 62.6
33.9 / 37.0
RGB+LiDAR
BEVFusion [ 14 ]
60.5 / 54.8
32.3 / 25.6
22.3 / 0.2
62.6 / 71.0
45.0 / 32.9
TABLE III: Impact of removing lying instances from training on AP@50. Each entry lists performance for full training / training without lying instances
Fig. 9: Absolute AP@50 gain of our method and the two LiDAR-RGB fusion baselines over the mean of four LiDAR-only methods (bars, left axis) and mean LiDAR points per person instance (right axis).
Multi-modal fusion has emerged as a promising paradigm for accurate 3D object detection. However, performance degrades substantially when deployed in target domains different from training. In this work, focusing on dual-branch proposal-level detectors, we identify two factors that limit robust cross-domain generalization: 1) in challenging domains such as rain or nighttime, one modality may undergo severe degradation; 2) the LiDAR branch often dominates the detection process, leading to systematic underutilization of visual cues and vulnerability when point clouds are compromised. To address these challenges, we propose three components. First, Query-Decoupled Loss provides independent supervision for 2D-only, 3D-only, and fused queries, rebalancing gradient flow across modalities. Second, LiDAR-Guided Depth Prior augments 2D queries with instance-aware geometric priors through probabilistic fusion of image-predicted and LiDAR-derived depth distributions, improving their spatial initialization. Third, Complementary Cross-Modal Masking applies complementary spatial masks to the image and point cloud, encouraging queries from both modalities to compete within the fused decoder and thereby promoting adaptive fusion. Extensive experiments demonstrate substantial gains over state-of-the-art baselines while preserving source-domain performance. Code and models are publicly available at https://github.com/IMPL-Lab/CCF.
Tri-Perspective View (TPV) representations describe 3D scene structure through top, side, and front feature planes, but existing TPV lifting is primarily camera-based, leaving the depth of sampled image evidence ambiguous along projected camera rays. We propose RLG-TPV, a multimodal TPV framework for camera-radar 3D object detection in which radar and training-time LiDAR provide complementary geometric guidance during representation construction. A ray-guided deformable-attention lift weights sampled image features using LiDAR-supervised camera depth probabilities and radar frustum occupancy, while radar additionally refines the depth distribution before lifting. Because conventional radar provides limited elevation information, LiDAR-derived class-occupancy targets supervise the side and front planes during training; the corresponding heads are removed at inference, so deployment requires only cameras and radar. For temporal aggregation, Doppler-guided temporal fusion aligns past features using a motion field anchored by measured radar radial velocity, with gating that limits warping in regions without supported motion. An RCS-aware radar scatter further allows radar evidence to spread over spatial neighborhoods conditioned on radar cross section. On the nuScenes validation set, RLG-TPV achieves 0.4981 mAP and 0.5959 NDS, reducing orientation and velocity error by 31.9% and 30.7% relative to the published CRN baseline. Ablation studies show that ray-level geometric guidance is a major contributor to the final performance.
Ahmet Mete Dokgoz, A. Enes Doruk, Hasan F. Ates
Department of Artificial Intelligence and Data Engineering, Ozyegin University, Istanbul, Turkiye
This paper investigates person detection and tracking in an industrial indoor workspace using a LiDAR mounted on an overhead crane. The overhead viewpoint introduces a strong domain shift from common vehicle-centric LiDAR benchmarks, and limited availability of suitable public training data. Henceforth, we curate a site-specific overhead LiDAR dataset with 3D human bounding-box annotations and adapt selected candidate 3D detectors under a unified training and evaluation protocol. We further integrate lightweight tracking-by-detection using AB3DMOT and SimpleTrack to maintain person identities over time. Detection performance is reported with distance-sliced evaluation to quantify the practical operating envelope of the sensing setup. The best adapted detector configurations achieve average precision (AP) up to 0.84 within a 5.0 m horizontal radius, increasing to 0.97 at 1.0 m, with VoxelNeXt and SECOND emerging as the most reliable backbones across this range. The acquired results contribute in bridging the domain gap between standard driving datasets and overhead sensing for person detection and tracking. We also report latency measurements, highlighting practical real-time feasibility. Finally, we release our dataset and implementations in GitHub to support further research
Nilusha Jayawickrama, Henrik Toikka, Risto Ojala
School of Engineering, Aalto University, Espoo, Finland · School of Electrical Engineering, Aalto University, Espoo, Finland