cs.CVDec 18, 2023
SaveLong-Tailed 3D Detection via Multi-Modal Fusion
Organizations: Department of Computer Science, Zhejiang University. · Computing & Mathematical Sciences, Caltech. · Toyota Research Institute. · Robotics Institute, Carnegie Mellon University. ·
Abstract
Contemporary autonomous vehicle (AV) benchmarks have significantly advanced multimodal (LiDAR+RGB) 3D detection. However, despite the naturally long-tailed distribution of object classes, existing benchmarking protocols primarily focus on frequent categories (e.g., pedestrian and car), largely overlooking rare but safety-critical classes such as stroller and emergency vehicle. In practice, reliable detection of both common and rare classes is essential for safe autonomous driving. We formalize this problem as Long-Tailed 3D Detection (LT3D), where evaluation encompasses all annotated classes, including rare ones. To address LT3D, we introduce hierarchical losses that promote feature sharing across classes, diagnostic metrics that assign partial credit to semantically reasonable mistakes with respect to the semantic hierarchy (e.g., confusing a child with an adult), and a multimodal late-fusion (MMLF) framework to fuse detections. In particular, we show that rare-class accuracy benefits substantially from MMLF of independently trained uni-modal LiDAR and RGB detectors. Because of the modular design, unlike prevailing end-to-end trained multi-modal detectors that require paired LiDAR-RGB data, MMLF enables the use of advanced unimodal detectors that are trained on large-scale uni-modal datasets with sufficient data for rare classes. Lastly, we examine three fundamental design choices in MMLF, including the RGB detector representation (2D vs. 3D), cross-modal association (3D vs. image plane), and fusion strategy. We find that 2D RGB detectors recognize rare classes more reliably than 3D RGB detectors, image-plane association is more robust to depth estimation errors, and probabilistic score-calibrated fusion consistently yields the best performance. Extensive experiments on nuScenes and Argoverse2 demonstrate substantial improvements of MMLF, establishing a new state of the art.
Figures & tables
Figure 1: Left: According to the histogram of per-class object counts, the nuScenes benchmark ( Caesar et al., 2020 ) focuses on the common classes in cyan (e.g., car and barrier ) and ignores rare ones in red (e.g., stroller and debris ). Moreover, the benchmark creates a superclass pedestrian by grouping multiple classes in green , including the common class adult and several rare ones (e.g., child and police-officer ). Such a grouping complicates the analysis of detection performance as pedestrian performance is dominated by adult . Right: Furthermore, the ignored superclass pushable-pullable also contains diverse objects such as shopping-cart , dolly , luggage and trash-can as shown in the top row. Arguably, AVs should detect rare classes as they affect AV behaviors. To gain a comprehensive analysis of performance, we report performance in the main paper primarily for three groups of classes ( Many , Medium , and Few shown in the left panel) and provide per-class performance in the appendix.
Figure 2: The nuScenes dataset defines a hierarchy for the annotated classes (Fig. 1 ). We mark common classes in white and rare classes in gold. The standard nuScenes benchmark makes two choices for dealing with rare classes: (1) ignoring them (e.g., stroller and pushable-pullable ), or (2) grouping them into super classes (e.g., adult , child , construction-worker , police-officer are grouped as pedestrian ). As the pedestrian class is dominated by adult (Fig. 1 ), the standard benchmarking protocol masks the challenge of detecting rare classes like child and police-officer .
Figure 3: We adopt a Multi-Modal Late-Fusion (MMLF) framework for LT3D by fusing RGB and LiDAR uni-modal detectors. We rigorously examine three fundamental design choices within MMLF (Fig. 4 ) and propose a simple method that fuses detections produced by a 2D RGB-detector and a 3D LiDAR-detector. Our method achieves 57.0 mAP on the nuScenes LT3D benchmark, significantly improving over the recent 3D detector CCF ( Wu et al., 2026 ) by 9.9 mAP and the LT3D dedicated approach FOMO-3D ( Yang et al., 2025 ) by 2.4 mAP (Table 1 ).
Figure 4: We examine three key components in multi-modal late-fusion (MMLF) from first principles: A. whether to train a 2D or a 3D RGB detector, B. whether to match uni-modal detections on the 2D image plane or in the 3D bird’s-eye-view (BEV), and C. how to optimally fuse these detections. We show that using 2D RGB detectors, matching on the 2D image plane, and probabilistically fusing score-calibrated detections achieves state-of-the-art LT3D performance (Table 3 and Table 4 ).
Figure 5: We exploit the semantic hierarchy defined in the nuScenes dataset (Fig. 2 ) to train detectors by predicting class labels at multiple levels for an object: the fine-grained label (e.g., child ), the coarse class (e.g., pedestrian ), and the root-level class object . That said, the classes in the hierarchy are no longer mutually exclusive. This allows us to use a sigmoid focal loss to learn a separate heatmap for each class.
Figure 6: The proposed method Multi-Modal Filtering (MMF) effectively removes high-scoring false-positive LiDAR detections. The green boxes are ground-truth strollers , while the blue boxes are stroller detections from the LiDAR-based detector CenterPoint ( Yin et al., 2021a ) ( left ) and RGB-based detector FCOS3D ( Wang et al., 2021 ) ( mid ). MMF removes LiDAR detections not within meters of any RGB detection, shown in the red region of right , and keeps all other LiDAR detections.
Figure 7: We analyze the MMF approach. Rare classes are most often confused by the common class (in blue ) in each superclass: ( left ) Vehicle is dominated by car , ( mid ) Pedestrian is dominated by adult , and ( right ) Movable is dominated by barrier . Note that class confusions are quite reasonable. Car is often mistaken for truck . Similarly, truck , construction-vehicle and emergency-vehicle are most often mistaken for car . Bicycle and motorcycle are sometimes misclassified as car , presumably because they are sometimes spatially close to cars . Adults have similar appearance to police-officer and construction-worker , and they are often co-localized with child and stroller .
Figure 8: Our Multi-Modal Late-Fusion (MMLF) strategy takes 3D LiDAR and 2D RGB detections as input (Fig. 4 A), matches the detections on the image plane (Fig. 4 B), and probabilistically fuses score-calibrated detections towards the final 3D detections.
| Method | Modality | All | Many | Medium | Few |
| FCOS3D ( Wang et al., 2021 ) (ICCVW’21) | C | 20.9 | 39.0 | 23.3 | 2.9 |
| BEVFormer ( Li et al., 2022b ) (ECCV’22) | C | 27.3 | 52.3 | 31.6 | 1.4 |
| PolarFormer ( Jiang et al., 2023 ) (AAAI’23) | C | 28.0 | 54.0 | 31.6 | 2.2 |
| DenseBEV ( Dähling et al., 2026 ) (WACV’26) | C | 29.7 | 54.9 | 33.1 | 4.7 |
| CenterPoint ( Yin et al., 2021a ) (CVPR’21) | L | 39.2 | 76.4 | 43.1 | 3.5 |
| TransFusion-L ( Bai et al., 2022 ) (CVPR’22) | L | 38.5 | 68.5 | 42.8 | 8.4 |
Table 1: Benchmarking results on nuScenes . We denote the RGB and LiDAR modalities by C and L, respectively. Our simple late-fusion method MMF, which uses detections of the 3D RGB-based detector FCOS3D to filter detections of the 3D LiDAR-based detector CenterPoint (improved by using our grouping-free head and training by our hierarchy loss), rivals most RGB-only, LiDAR-only and multimodal 3D detectors. Furthermore, our final method MMLF, which probabilistically fuses 3D LiDAR and 2D RGB detections, performs the best among non-oracle methods.
Figure 9: We demonstrate how our multi-modal late-fusion (MMLF) approach improves LT3D by fusing 2D RGB detections (from Co-DINO) and 3D LiDAR detections (from CenterPoint). In the three examples, MMLF correctly relabels detections which are geometrically similar (w.r.t size and shape) in LiDAR but are visually distinct in RGB, such as bus -vs- truck , adult -vs- stroller , and adult -vs- child . Refer to Appendix N for more visualizations.
| Method | Modality | All | Many | Medium | Few |
| FCOS3D | C | 20.9 | 39.0 | 23.3 | 2.9 |
| PointPillars (PP) | L | 30.0 | 64.2 | 28.4 | 3.4 |
| + Hierarchy Loss | L | 31.2 | 66.4 | 30.4 | 2.9 |
| + MMF(PP, FCOS3D) | C+L | 35.8 | 66.2 | 41.0 | 4.4 |
| CenterPoint (CP) | L | 39.2 | 76.4 | 43.1 | 3.5 |
| + Hierarchy Loss | L | 40.4 | 77.1 | 45.1 | 4.3 |
Table 2: Training with the semantic hierarchy improves both PointPillars ( Lang et al., 2019 ) and CenterPoint ( Yin et al., 2021a ) for LT3D by 1 mAP over All classes. Further, MMF improves on Medium and Few classes by 1 13 mAP, showing the benefit of late fusion for LT3D.
| Method | Fusion in 3D | Fusion in 2D | ||||||
| All | Many | Medium | Few | All | Many | Medium | Few | |
| CenterPoint | 40.4 | 77.1 | 45.1 | 4.3 | 40.4 | 77.1 | 45.1 | 4.3 |
| + FCOS3D | 42.9 | 76.6 | 48.7 | 8.1 | 42.6 | 75.0 | 49.4 | 7.7 |
| + BEVFormer | 43.2 | 76.9 | 50.8 | 6.3 | 42.8 | 75.2 | 51.4 | 5.7 |
| + PolarFormer | 42.8 | 76.8 | 50.0 | 6.1 | 42.6 | 75.1 | 51.1 | 5.6 |
| + YOLOV7 | 40.1 | 76.1 | 43.8 | 5.8 | 48.6 | 76.7 | 57.1 | 15.3 |
Table 3: Fusing uni-modal detections in 3D vs. on the 2D image plane . We evaluate the impact of fusing 3D LiDAR detections (from CenterPoint trained with our hierarchy loss) with 2D RGB vs. and 3D RGB detections, in 3D vs. on the image plane. We match and filter detections in 3D using center distance (MMF), and match and filter detections in the 2D image plane using IoU (MMLF). When fusing in 3D, we follow ( Wilson et al., 2020 ; Khurana et al., 2024 ) that inflate 2D detections to 3D using LiDAR points within their box frustums. When fusing on the 2D image plane, we project 3D detections to the image plane using provided sensor extrinsics. When fusing 3D RGB detections, similar results are achieved regardless of whether matching is performed in 3D or on the 2D image plane. Unsurprisingly, inflating 2D RGB detections for matching in 3D performs worse than matching 3D RGB detections in 3D. Yet, late-fusion between LiDAR-based 3D detections and RGB-based 2D detections in the image plane (bottom right panel) significantly improves performance for classes with medium and few examples by 10 mAP. Refer to Appendix D for more results.
| Method | All | Many | Medium | Few | |
| CenterPoint | 39.2 | 76.4 | 43.1 | 3.5 | |
| + Hierarchy Loss | +1.2 | 40.4 | 77.1 | 45.1 | 4.3 |
| + MMLF w/ Co-DINO | +7.7 | 48.1 | 78.8 | 52.9 | 16.9 |
| + External Data | +4.5 | 52.6 | 79.8 | 58.1 | 23.5 |
| + Score Calibration | +2.3 | 54.9 | 80.2 | 60.7 | 27.1 |
| + Probabilistic Fusion | +2.1 | 57.0 | 80.7 | 64.0 | 29.2 |
Table 4: Adding our developed components progressively improves LT3D performance, confirming that (1) 2D RGB detectors are better suited for our MMLF, (2) matching projected 3D LiDAR detections on the 2D image plane outperforms matching 2D RGB detections inflated to 3D, and (3) score calibration prior and probabilistic fusion improve performance. A majority of performance improvements is attributed to fusing detections of the 2D RGB detector Co-DINO on the 2D image plane. Moreover, training with external data notably improves rare class accuracy by 6.6 mAP.
Figure 10: Training 2D detectors with more data (e.g., nuScenes + nuImages vs. nuScenes only) and using better 2D detectors (e.g., Co-DINO, GroundingDINO, DINO, compared with YOLOV7) improve performance not only on 2D detection but also on the 3D detection achieved by our MMLF.
| Method | Modality | Time | Memory | mAP |
| FCOS3D ( Wang et al., 2021 ) | C | 89 | 4.2 GB | 20.9 |
| BEVFormer ( Li et al., 2022b ) | C | 327 | 6.4 GB | 27.3 |
| CenterPoint ( Yin et al., 2021a ) | L | 323 | 4.0 GB | 40.4 |
| FSHNet ( Liu et al., 2025a ) | L | 296 | 4.8 GB | 41.6 |
| CMT ( Yan et al., 2023 ) | L+C | 520 | 9.0 GB | 44.4 |
| TransFusion ( Bai et al., 2022 ) | L+C | 367 | 5.6 GB | 39.8 |
Table 5: Comparison of inference efficiency with respect to wall-clock time (ms) and GPU consumption. We use NVIDIA A6000 GPUs to measure the computation cost. Compared with other multimodal 3D detectors, MMLF is not only competitive in inference efficiency, but also significantly better in performance (mAP).
| Methods | Truck | Adult | Bicycle | Traffic cone | ||||||||||||
| near | far | sparse | dense | near | far | sparse | dense | near | far | sparse | dense | near | far | sparse | dense | |
| CenterPoint | 65.6 | 35.6 | 30.1 | 71.8 | 87.4 | 57.8 | 74.5 | 89.6 | 54.7 | 3.4 | 16.4 | 67.6 | 75.7 | N/A | 56.3 | 83.7 |
| FSHNet | 68.5 | 38.5 | 34.0 | 74.4 | 88.3 | 59.3 | 76.1 | 90.8 | 62.1 | 7.8 | 24.3 | 72.1 | 78.7 | N/A | 61.1 | 85.1 |
| CMT | 65.8 | 42.5 | 39.1 | 69.9 | 88.6 | 59.7 | 76.1 | 91.3 | 70.0 | 19.6 | 43.2 | 76.7 | 80.1 | N/A | 63.8 | 84.8 |
| CCF | 67.2 | 42.9 | 39.5 | 71.1 | 88.8 | 60.8 | 77.1 | 91.7 | 72.2 | 19.7 | 44.7 | 78.2 | 80.8 | N/A | 64.8 | 85.8 |
| MMLF | 71.2 | 45.2 | 40.1 | 76.1 | 90.2 | 63.9 | 80.1 | 93.3 | 81.3 | 26.5 | 58.2 | 88.4 | 82.3 | N/A | 66.8 | 86.7 |
Table 6: Breakdown results on objects from common (in top panel) and rare (in bottom panel) classes, with respect to distance and LiDAR density. We define far-field and near-field objects when they are 30m and 30m away from the ego vehicle, respectively. To define dense or sparse LiDAR returns, we consider a per-class threshold to evenly divide the test instances of a class. We provide the results for all the classes in Table 19 of the Appendix. The results demonstrate that our method boosts performance across most common and rare classes, both far-field and near-field objects, and both objects with dense and sparse LiDAR returns.
| Method | Car | Adult | Truck | CV | Bicycle | MC | Child | CW | Stroller | PP | |
| CenterPoint ( Yin et al., 2021a ) | LCA=0 | 82.4 | 81.2 | 49.4 | 19.7 | 33.6 | 48.9 | 0.1 | 14.2 | 0.1 | 21.7 |
| (original) | LCA=1 | 83.9 | 82.0 | 58.7 | 20.5 | 35.2 | 50.5 | 0.1 | 18.3 | 0.1 | 22.0 |
| LCA=2 | 84.0 | 82.4 | 58.8 | 20.7 | 36.4 | 51.0 | 0.1 | 19.5 | 0.1 | 22.6 | |
| CenterPoint (w/ our grouping-free head) | LCA=0 | 88.1 | 86.3 | 62.7 | 24.5 | 48.5 | 62.8 | 0.1 | 22.2 | 4.3 | 32.7 |
| LCA=1 | 89.0 | 87.1 | 71.6 | 26.7 | 50.2 | 64.7 | 0.1 | 29.4 | 4.5 | 32.9 | |
| LCA=2 | 89.1 | 87.5 | 71.7 | 26.8 | 51.1 | 65.2 | 0.1 | 30.5 | 4.8 | 33.4 |
Table 7: Diagnosis using the mAP H metric on selected classes . We analyze the LiDAR-based 3D detector CenterPoint ( Yin et al., 2021a ) and multi-modal detector CMT ( Yan et al., 2023 ) , with our proposed grouping-free detector head, hierarchy loss, and MMLF (using the 2D RGB detector Co-DINO ( Zong et al., 2023 ) ). Comparing the rows of LCA=0 for CenterPoint and CMT, we see that MMLF significantly improves on rare classes (in blue ), e.g., construction-vehicle (CV), bicycle, motorcycle (MC), construction-worker (CW), stroller, and pushable-pullable (PP). Moreover, performance increases significantly from LCA=0 to LCA=1 compared against LCA=1 to LCA=2, suggesting that objects from rare classes are often detected but misclassified as sibling classes.
Figure 11: Both our MMLF (columns 1-3) and TransFusion ( Bai et al., 2022 ) (columns 4 -5) fail on these cases. In the first and second row, the 2D RGB detector Co-DINO detects the heavily occluded cars but 3D LiDAR detector fails to detect them. As a result, our MMLF misses these cars because it throws away unmatched RGB detections which do not have accurate 3D information. In the third row, we see that, although both the LiDAR and RGB detectors fire on the object (whose ground-truth label is police-officer ), the LiDAR detector classifies it as adult and the RGB detector classifies it as construction-worker . As a result, the final label is incorrectly predicted. TransFusion also misclassifies this object, confusing it with adult . Refer to Appendix N for more visualizations.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| CenterPoint | Multi-Head | Car | Ped. | Barrier | TC | Truck | Bus | Trailer | CV | MC | Bicyc. |
| Original | ✓ | 87.7 | 87.7 | 70.7 | 74.0 | 63.6 | 72.7 | 45.1 | 26.3 | 64.7 | 47.9 |
| 89.1 | 88.4 | 70.8 | 74.3 | 64.8 | 72.9 | 42.0 | 25.7 | 65.9 | 53.6 | ||
| for LT3D | ✓ | 82.4 | — | 62.0 | 60.1 | 49.4 | 55.7 | 28.9 | 19.7 | 48.9 | 33.6 |
| 88.1 | — | 72.4 | 72.7 | 62.7 | 70.8 | 40.2 | 24.5 | 62.8 | 48.5 | ||
| Adult | PP | CW | Debris | Child | Stroller | PO | EV | PM | All | ||
| Original | ✓ | — | — | — | — | — | — | — | — | — | 64.0 |
Table 8: Our proposed grouping-free detector head architecture consistently outperforms grouping-based approaches on both the standard and LT3D benchmarks. We note that sub-optimal grouping strategies (such as those adopted for LT3D) may yield significantly diminished performance, whereas optimized grouping strategies (such as those adopted for the standard setup) have comparable performance to the grouping-free approach. Note, TC is traffic-cone , CV is construction vehicle , MC is motorcycle , PP is pushable-pullable , CW is construction-worker , and PO is police-officer . We highlight Medium and Few classes in blue .
| Method | Hier. | Many | Medium | Few | All |
| CenterPoint (w/o Hier.) | n/a | 76.4 | 43.1 | 3.5 | 39.2 |
| CenterPoint w/ Hier. | (a) | 77.1 | 45.1 | 4.3 | 40.4 |
| (b) | 76.4 | 45.0 | 5.3 | 40.5 | |
| (c) | 76.5 | 45.2 | 5.2 | 40.6 | |
| (d) | 74.5 | 43.5 | 5.6 | 39.5 |
Table 9: Different variants achieve similar performance. We note that other methods do improve accuracy in the tail by sacrificing performance in the head, suggesting that hybrid approaches that apply different techniques for head-vs-tail classes may further improve accuracy. Unlike ( Li et al., 2020 ; Wu et al., 2019 ) which requires a strict label hierarchy, our approach is not limited to a hierarchy.
| Method | All | Many | Medium | Few |
| FCOS3D ( 3D RGB Detector) | 18.3 | 36.0 | 21.1 | 0.2 |
| BEVFormer ( 3D RGB Detector) | 23.0 | 40.8 | 28.2 | 2.1 |
| PolarFormer ( 3D RGB Detector) | 20.7 | 37.5 | 25.2 | 1.6 |
| DenseBEV ( 3D RGB Detector) | 24.4 | 41.2 | 30.1 | 3.7 |
| YOLOV7 ( 2D RGB Detector) | 42.3 | 61.1 | 46.3 | 22.0 |
| DINO ( 2D RGB Detector) | 48.9 | 67.6 | 52.9 | 28.7 |
Table 10: Evaluating 2D detection performance . Simply projecting 3D detections from RGB-only 3D detectors to 2D image plane yields considerably lower 2D detection mAP across all classes. However, 2D RGB detectors achieve higher performance. Intuitively, 2D RGB detectors achieve better classification performance and predict tighter fitting bounding boxes than those derived by projecting 3D detections to the image plane. This explains why fusing detections from 2D RGB detectors yields better LT3D performance than detections from 3D RGB detectors. All detectors in this table are only trained on the nuScenes train-set. Refer to Table 11 for per-class results of ensembling the strong 2D RGB detectors with the 3D LiDAR detector CenterPoint.
| Class name | w/ YOLOV7 | w/ DINO | w/ GD | w/ Co-DINO |
| Personal Mobility | 18.9 | 35.8 | 37.1 | 42.3 |
| Emergency Vehicle | 12.5 | 21.1 | 24.7 | 28.0 |
| Police Officer | 10.9 | 14.4 | 15.9 | 16.1 |
| Stroller | 27.5 | 36.9 | 40.8 | 45.4 |
| Child | 16.7 | 24.9 | 28.6 | 32.1 |
| Debris | 5.3 | 6.1 | 8.9 | 11.4 |
Table 11: Using a stronger 2D detector improves 3D detection by MMLF. We compare YOLOV7, DINO, GroundingDINO, and Co-DINO as the 2D detector, applied along with the 3D LiDAR detector CenterPoint ( Yin et al., 2021a ) in our MMLF. Results demonstrate that using stronger 2D detectors, especially Co-DINO, yields significantly better 3D detection performance on all classes, particularly on medium and few classes. Refer to Fig. 1 for the distribution of these classes.
| Method | All | Many | Medium | Few |
| CenterPoint w/ Hier. | 40.4 | 77.1 | 45.1 | 4.3 |
| MMLF(CenterPoint, Co-DINO) | 57.0 | 80.7 | 64.0 | 29.2 |
| BEVFusion-L | 42.5 | 72.5 | 48.0 | 10.6 |
| MMLF(BEVFusion-L, Co-DINO) | 56.2 | 80.1 | 63.2 | 28.1 |
| CMT-L | 34.7 | 73.4 | 35.9 | 1.1 |
| MMLF(CMT-L, Co-DINO) | 57.1 | 81.2 | 63.8 | 29.2 |
Table 12: Transformer-based 3D LiDAR detector . We evaluate the late-fusion performance of BEVFusion-L and CMT-L (LiDAR-only) with Co-DINO, and find that our late fusion strategy improves over BEVFusion-L by 13.7 mAP overall and CMT-L by 22.4 mAP. This suggests that our late-fusion approach generalizes across different 3D detector architectures.
| Class name | BevFormer | Deepinteraction | CMT | MMLF (CenterPoint, Co-DINO) |
| ( Li et al., 2022b ) | ( Yang et al., 2022 ) | ( Yan et al., 2023 ) | (Ours) | |
| Personal Mobility | 0.7 | 2.5 | 4.1 | 42.3 |
| Emergency Vehicle | 0.2 | 1.5 | 6.9 | 28.0 |
| Police Officer | 0.9 | 0.0 | 7.3 | 16.1 |
| Stroller | 1.1 | 30.9 | 0.9 | 45.4 |
| Child | 3.9 | 6.0 | 4.7 | 32.1 |
Table 13: Per-class performance on nuScenes. Our approach MMLF fuses detections from the LiDAR-based 3D detector CenterPoint and the RGB-based 2D detector Co-DINO, achieving the highest per-class AP on 16 out of 18 classes. It is worth noting that, compared to CMT, our approach improves construction worker by 29.5 AP, stroller by 44.5 AP, and pushable-pullable by 23.1 AP.
| FCOS3D | BEVFormer | CP | TransFusion | DeepInteraction | CMT | MMLF(CP, Co-DINO) | |
| ( Wang et al., 2021 ) | ( Li et al., 2022b ) | ( Yin et al., 2021a ) | ( Bai et al., 2022 ) | ( Yang et al., 2022 ) | ( Yan et al., 2023 ) | (Ours) | |
| mAP | 20.9 | 27.3 | 39.2 | 39.8 | 43.7 | 44.4 | 57.0 |
| NDS | 30.4 | 38.8 | 54.9 | 53.9 | 54.4 | 55.9 | 62.8 |
Table 14: AP vs. NDS results . We compare methods reported in the main paper w.r.t nuScenes Detection Score (NDS) and mAP. We find that the methods show broadly consistent rankings on both metrics, with MMLF ranking first under both mAP and NDS.
| Method | Modality | 0-40% | 40-60% | 60-80% | 80-100% |
| CenterPoint (CP) | L | 84.1 | 90.9 | 92.8 | 96.2 |
| CMT | C + L | 84.3 | 91.6 | 93.1 | 96.1 |
| MMLF(CP, Co-DINO) | C + L | 83.2 | 90.4 | 92.5 | 95.9 |
Table 15: Average recall for occluded targets at different visibility levels . We evaluate average recall at four visibility levels (defined by nuScenes). Perhaps unsurprisingly, detection accuracy drops as objects become more occluded. Note that visibility attributes are not predicted by our model and are manually annotated for the ground truth, making it difficult to measure precision.
| Method | Modality | 0-10m | 10-20m | 20-30m | Full Range |
| CenterPoint (CP) | L | 45.5 | 40.1 | 26.7 | 40.4 |
| CMT | C + L | 42.0 | 43.2 | 31.4 | 44.4 |
| MMLF(CP, Co-DINO) | C + L | 57.7 | 61.5 | 39.3 | 57.0 |
Table 16: Detection accuracy at different distance thresholds . We evaluate detection accuracy at three distance ranges. Detecting objects closer to the ego-vehicle is easier (due to higher LiDAR point density) than detecting objects further away.
| Method | Modality | Vehicle | Pedestrian | Movable | Object |
| CenterPoint | L | 85.5 | 95.5 | 90.7 | 89.3 |
| CMT | C + L | 86.4 | 91.2 | 87.2 | 87.9 |
Table 17: Average recall between detectors . CenterPoint (LiDAR-only) achieves higher recall than CMT for the Pedestrian (LCA=1), Movable (LCA=1), and Object (LCA=2) superclasses.
| Translation Noise | 0.01m | 0.02m | 0.03m | 0.04m |
| CMT | 44.4 | 44.3 | 44.3 | 44.2 |
| MMLF(CP, Co-DINO) | 57.0 | 56.9 | 56.8 | 56.8 |
| Rotation Noise | 1 ∘ | 2 ∘ | 3 ∘ | 4 ∘ |
| CMT | 44.3 | 43.1 | 42.7 | 41.1 |
| MMLF(CP, Co-DINO) | 54.3 | 53.4 | 51.7 | 51.3 |
| Translation Noise | 0.01m | 0.02m | 0.03m | 0.04m |
Table 18: Impact of calibration noise . We apply a small amount of additive Gaussian noise to the translation and rotation calibration matrices to measure the impact of imprecise calibration on late fusion. While translation noise has little effect on detection accuracy, rotation noise significantly impacts performance. As expected, combining both translation and rotation noise degrades accuracy further.
| Methods | near | far | sparse | dense |
| CenterPoint | 93.7 | 65.9 | 70.5 | 96.5 |
| FSHNet | 94.1 | 67.8 | 72.5 | 96.9 |
| CMT | 93.6 | 69.8 | 73.6 | 96.2 |
| CCF | 93.8 | 70.2 | 73.8 | 96.4 |
| MMLF | 93.5 | 71.4 | 73.7 | 96.1 |
Table 19: Detailed evaluation results of various methods across 18 obstacle categories.
| Method | Modality | All | Many | Medium | Few |
| FCOS3D | C | 14.6 | 27.4 | 17.0 | 7.8 |
| DenseBEV | C | 26.1 | 35.8 | 28.9 | 19.6 |
| CenterPoint (CP) | L | 44.0 | 77.4 | 46.9 | 30.2 |
| FSHNet | L | 45.1 | 78.2 | 47.0 | 32.7 |
| CCF | C + L | 49.9 | 81.8 | 52.0 | 37.6 |
| MMF(CP, FCOS3D) | C + L | 48.4 | 79.0 | 51.4 | 35.3 |
Table 20: Results on the Argoverse 2 dataset . We present results on AV2 evaluated at 50m. The RGB-only 3D detector FCOS3D ( Wang et al., 2021 ) achieves poor performance, likely due to inaccurate depth estimates. In contrast, the LiDAR-only detector CenterPoint (CP) ( Yin et al., 2021a ) achieves strong performance on all classes. Our multi-modal fusion approach significantly improves over CenterPoint, achieving a 14.6 mAP improvement averaged over all classes. Importantly, late-fusion with 2D detector Co-DINO ( Zong et al., 2023 ) performs best by a notable margin. These results on AV2 are consistent with those on nuScenes, demonstrating the general applicability of our approach.
Figure 12: Left: According to the histogram of per-class object counts (on the left ), classes in Argoverse 2.0 (AV2) follow a long tailed distribution. Following ( Liu et al., 2019a ) and nuScenes, we report performance for three groups of classes based on their cardinality (split by dotted lines): Many , Medium , and Few . As AV2 does not provide a class hierarchy, we construct one by referring to the nuScenes hierarchy (cf. Fig. 2 on the right).
| Method | Car | Truck | Trailer | Bus | CV | Bicy. | MC | Ped. | TC | Barrier | mAP | NDS |
| DeepInteraction ( Yang et al., 2022 ) (NeurIPS’22) | 88.5 | 64.4 | 44.6 | 79.2 | 30.1 | 67.8 | 79.0 | 88.9 | 80.0 | 76.4 | 69.9 | 72.6 |
| FSHNet ( Liu et al., 2025a ) (CVPR’25) | 88.7 | 61.4 | 47.8 | 79.3 | 26.3 | 60.5 | 76.7 | 89.3 | 78.6 | 72.3 | 68.1 | 71.7 |
| DenseBEV ( Dähling et al., 2026 ) (WACV’26) | 63.1 | 38.6 | 17.1 | 46.2 | 12.9 | 48.2 | 45.6 | 53.2 | 63.5 | 60.6 | 44.9 | 54.9 |
| CenterPoint ( Yin et al., 2021a ) (CVPR’21) | 87.7 | 61.6 | 43.7 | 73.4 | 28.4 | 49.5 | 65.9 | 87.1 | 75.7 | 71.3 | 66.6 | 64.4 |
| MMLF(CenterPoint, Co-DINO) | 88.5 | 68.1 | 49.4 | 79.2 | 40.0 | 77.3 | 80.7 | 89.9 | 82.3 | 75.6 | 73.1 | 74.6 |
| CMT ( Yan et al., 2023 ) (ICCV’23) | 88.6 | 65.1 | 46.5 | 76.4 | 36.9 | 66.7 | 76.2 | 88.1 | 80.1 | 77.9 | 70.3 | 72.9 |
Table 21: 3D detection performance on the standard nuScenes benchmark. We evaluate the impact of late fusion on the standard 10-class validation benchmark. Although our work focuses on improving detection accuracy in the tail, our proposed approach also improves performance on the standard benchmark. Notably, fusing CenterPoint ( Yin et al., 2021a ) with Co-DINO ( Zong et al., 2023 ) improves mAP and NDS by 6.5 and 10.2 percentage points, respectively. Fusing CMT ( Yan et al., 2023 ) , a multimodal 3D detector, with Co-DINO yields improvements of 2.7 mAP and 1.9 NDS, improving all standard classes, with the largest gains on bicycle , motorcycle , truck , trailer , and bus . We posit that CMT already effectively uses multimodal training data for the common classes of the standard nuScenes benchmark, yielding diminishing returns with late fusion. CV denotes construction vehicle , MC denotes motorcycle , Ped. denotes pedestrian , and TC denotes traffic cone .
Figure 13: We visualize the output of our late-fusion method, which combines 3D LiDAR detections from CenterPoint and 2D RGB detections from Co-DINO. In all cases, we find that the 2D RGB detector is able to correct classification errors from the 3D LiDAR detector, improving overall performance. Importantly, score calibration and probabilistic ensembling increase the confidence of the final prediction.
Figure 14: Three examples demonstrate how our multi-modal late-fusion (MMLF) approach improves LT3D by ensembling 2D RGB detections (from Co-DINO) and 3D LiDAR detections (from CenterPoint ( Yin et al., 2021a ) ). In all examples, MMLF correctly relabels detections which are geometrically similar (w.r.t size and shape) in LiDAR but are visually distinct in RGB, such as bus -vs- truck , adult -vs- stroller , and adult -vs- child . This figure complements Fig. 9 in the main paper.
Figure 15: Both our MMLF method (column 1) and TransFusion ( Bai et al., 2022 ) (column 2) share the same failure cases. In the first and second row, the 2D RGB detector Co-DINO detects the heavily occluded cars but 3D LiDAR detector fails to detect them. As a result, our MMLF misses these cars because it throws away unmatched RGB detections which do not have accurate 3D information. In the third row, we see that, although both the LiDAR and RGB detectors fire on the object (whose ground-truth label is police-officer ), the LiDAR detector classifies it as adult and RGB detector classifies it as construction-worker . As a result, the final detection is incorrect w.r.t the predicted label. TransFusion also misclassifies this object as an adult . This figure complements Fig. 11 in the main paper.
More in this topic
3D Object Detection
3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.
See all 181 papers