Authors: Yechi Ma, Neehar Peri, Achal Dave, Wei Hua, Deva Ramanan, Shu Kong
Organizations: Department of Computer Science, Zhejiang University. · Computing & Mathematical Sciences, Caltech. · Toyota Research Institute. · Robotics Institute, Carnegie Mellon University. · Faculty of Information Science and Computing, University of Macau.
Contemporary autonomous vehicle (AV) benchmarks have significantly advanced multimodal (LiDAR+RGB) 3D detection. However, despite the naturally long-tailed distribution of object classes, existing benchmarking protocols primarily focus on frequent categories (e.g., pedestrian and car), largely overlooking rare but safety-critical classes such as stroller and emergency vehicle. In practice, reliable detection of both common and rare classes is essential for safe autonomous driving. We formalize this problem as Long-Tailed 3D Detection (LT3D), where evaluation encompasses all annotated classes, including rare ones. To address LT3D, we introduce hierarchical losses that promote feature sharing across classes, diagnostic metrics that assign partial credit to semantically reasonable mistakes with respect to the semantic hierarchy (e.g., confusing a child with an adult), and a multimodal late-fusion (MMLF) framework to fuse detections. In particular, we show that rare-class accuracy benefits substantially from MMLF of independently trained uni-modal LiDAR and RGB detectors. Because of the modular design, unlike prevailing end-to-end trained multi-modal detectors that require paired LiDAR-RGB data, MMLF enables the use of advanced unimodal detectors that are trained on large-scale uni-modal datasets with sufficient data for rare classes. Lastly, we examine three fundamental design choices in MMLF, including the RGB detector representation (2D vs. 3D), cross-modal association (3D vs. image plane), and fusion strategy. We find that 2D RGB detectors recognize rare classes more reliably than 3D RGB detectors, image-plane association is more robust to depth estimation errors, and probabilistic score-calibrated fusion consistently yields the best performance. Extensive experiments on nuScenes and Argoverse2 demonstrate substantial improvements of MMLF, establishing a new state of the art.
Figures & tables
Figure 1: Left: According to the histogram of per-class object counts, the nuScenes benchmark ( Caesar et al., 2020 ) focuses on the common classes in cyan (e.g., car and barrier ) and ignores rare ones in red (e.g., stroller and debris ). Moreover, the benchmark creates a superclass pedestrian by grouping multiple classes in green , including the common class adult and several rare ones (e.g., child and police-officer ). Such a grouping complicates the analysis of detection performance as pedestrian performance is dominated by adult . Right: Furthermore, the ignored superclass pushable-pullable also contains diverse objects such as shopping-cart , dolly , luggage and trash-can as shown in the top row. Arguably, AVs should detect rare classes as they affect AV behaviors. To gain a comprehensive analysis of performance, we report performance in the main paper primarily for three groups of classes ( Many , Medium , and Few shown in the left panel) and provide per-class performance in the appendix.
Figure 2: The nuScenes dataset defines a hierarchy for the annotated classes (Fig. 1 ). We mark common classes in white and rare classes in gold. The standard nuScenes benchmark makes two choices for dealing with rare classes: (1) ignoring them (e.g., stroller and pushable-pullable ), or (2) grouping them into super classes (e.g., adult , child , construction-worker , police-officer are grouped as pedestrian ). As the pedestrian class is dominated by adult (Fig. 1 ), the standard benchmarking protocol masks the challenge of detecting rare classes like child and police-officer .
Figure 3: We adopt a Multi-Modal Late-Fusion (MMLF) framework for LT3D by fusing RGB and LiDAR uni-modal detectors. We rigorously examine three fundamental design choices within MMLF (Fig. 4 ) and propose a simple method that fuses detections produced by a 2D RGB-detector and a 3D LiDAR-detector. Our method achieves 57.0 mAP on the nuScenes LT3D benchmark, significantly improving over the recent 3D detector CCF ( Wu et al., 2026 ) by 9.9 mAP and the LT3D dedicated approach FOMO-3D ( Yang et al., 2025 ) by 2.4 mAP (Table 1 ).
Figure 4: We examine three key components in multi-modal late-fusion (MMLF) from first principles: A. whether to train a 2D or a 3D RGB detector, B. whether to match uni-modal detections on the 2D image plane or in the 3D bird’s-eye-view (BEV), and C. how to optimally fuse these detections. We show that using 2D RGB detectors, matching on the 2D image plane, and probabilistically fusing score-calibrated detections achieves state-of-the-art LT3D performance (Table 3 and Table 4 ).
Figure 5: We exploit the semantic hierarchy defined in the nuScenes dataset (Fig. 2 ) to train detectors by predicting class labels at multiple levels for an object: the fine-grained label (e.g., child ), the coarse class (e.g., pedestrian ), and the root-level class object . That said, the classes in the hierarchy are no longer mutually exclusive. This allows us to use a sigmoid focal loss to learn a separate heatmap for each class.
Figure 6: The proposed method Multi-Modal Filtering (MMF) effectively removes high-scoring false-positive LiDAR detections. The green boxes are ground-truth strollers , while the blue boxes are stroller detections from the LiDAR-based detector CenterPoint ( Yin et al., 2021a ) ( left ) and RGB-based detector FCOS3D ( Wang et al., 2021 ) ( mid ). MMF removes LiDAR detections not within m meters of any RGB detection, shown in the red region of right , and keeps all other LiDAR detections.
Figure 7: We analyze the MMF approach. Rare classes are most often confused by the common class (in blue ) in each superclass: ( left ) Vehicle is dominated by car , ( mid ) Pedestrian is dominated by adult , and ( right ) Movable is dominated by barrier . Note that class confusions are quite reasonable. Car is often mistaken for truck . Similarly, truck , construction-vehicle and emergency-vehicle are most often mistaken for car . Bicycle and motorcycle are sometimes misclassified as car , presumably because they are sometimes spatially close to cars . Adults have similar appearance to police-officer and construction-worker , and they are often co-localized with child and stroller .
Figure 8: Our Multi-Modal Late-Fusion (MMLF) strategy takes 3D LiDAR and 2D RGB detections as input (Fig. 4 A), matches the detections on the image plane (Fig. 4 B), and probabilistically fuses score-calibrated detections towards the final 3D detections.
Method
Modality
All
Many
Medium
Few
FCOS3D ( Wang et al., 2021 ) (ICCVW’21)
C
20.9
39.0
23.3
2.9
BEVFormer ( Li et al., 2022b ) (ECCV’22)
C
27.3
52.3
31.6
1.4
PolarFormer ( Jiang et al., 2023 ) (AAAI’23)
C
28.0
54.0
31.6
2.2
DenseBEV ( Dähling et al., 2026 ) (WACV’26)
C
29.7
54.9
33.1
4.7
CenterPoint ( Yin et al., 2021a ) (CVPR’21)
L
39.2
76.4
43.1
3.5
TransFusion-L ( Bai et al., 2022 ) (CVPR’22)
L
38.5
68.5
42.8
8.4
Table 1: Benchmarking results on nuScenes . We denote the RGB and LiDAR modalities by C and L, respectively. Our simple late-fusion method MMF, which uses detections of the 3D RGB-based detector FCOS3D to filter detections of the 3D LiDAR-based detector CenterPoint (improved by using our grouping-free head and training by our hierarchy loss), rivals most RGB-only, LiDAR-only and multimodal 3D detectors. Furthermore, our final method MMLF, which probabilistically fuses 3D LiDAR and 2D RGB detections, performs the best among non-oracle methods.
Figure 9: We demonstrate how our multi-modal late-fusion (MMLF) approach improves LT3D by fusing 2D RGB detections (from Co-DINO) and 3D LiDAR detections (from CenterPoint). In the three examples, MMLF correctly relabels detections which are geometrically similar (w.r.t size and shape) in LiDAR but are visually distinct in RGB, such as bus -vs- truck , adult -vs- stroller , and adult -vs- child . Refer to Appendix N for more visualizations.
Method
Modality
All
Many
Medium
Few
FCOS3D
C
20.9
39.0
23.3
2.9
PointPillars (PP)
L
30.0
64.2
28.4
3.4
+ Hierarchy Loss
L
31.2
66.4
30.4
2.9
+ MMF(PP, FCOS3D)
C+L
35.8
66.2
41.0
4.4
CenterPoint (CP)
L
39.2
76.4
43.1
3.5
+ Hierarchy Loss
L
40.4
77.1
45.1
4.3
Table 2: Training with the semantic hierarchy improves both PointPillars ( Lang et al., 2019 ) and CenterPoint ( Yin et al., 2021a ) for LT3D by > 1 mAP over All classes. Further, MMF improves on Medium and Few classes by 1 ∼ 13 mAP, showing the benefit of late fusion for LT3D.
Method
Fusion in 3D
Fusion in 2D
All
Many
Medium
Few
All
Many
Medium
Few
CenterPoint
40.4
77.1
45.1
4.3
40.4
77.1
45.1
4.3
+ FCOS3D
42.9
76.6
48.7
8.1
42.6
75.0
49.4
7.7
+ BEVFormer
43.2
76.9
50.8
6.3
42.8
75.2
51.4
5.7
+ PolarFormer
42.8
76.8
50.0
6.1
42.6
75.1
51.1
5.6
+ YOLOV7
40.1
76.1
43.8
5.8
48.6
76.7
57.1
15.3
Table 3: Fusing uni-modal detections in 3D vs. on the 2D image plane . We evaluate the impact of fusing 3D LiDAR detections (from CenterPoint trained with our hierarchy loss) with 2D RGB vs. and 3D RGB detections, in 3D vs. on the image plane. We match and filter detections in 3D using center distance (MMF), and match and filter detections in the 2D image plane using IoU (MMLF). When fusing in 3D, we follow ( Wilson et al., 2020 ; Khurana et al., 2024 ) that inflate 2D detections to 3D using LiDAR points within their box frustums. When fusing on the 2D image plane, we project 3D detections to the image plane using provided sensor extrinsics. When fusing 3D RGB detections, similar results are achieved regardless of whether matching is performed in 3D or on the 2D image plane. Unsurprisingly, inflating 2D RGB detections for matching in 3D performs worse than matching 3D RGB detections in 3D. Yet, late-fusion between LiDAR-based 3D detections and RGB-based 2D detections in the image plane (bottom right panel) significantly improves performance for classes with medium and few examples by > 10 mAP. Refer to Appendix D for more results.
Method
Δ
All
Many
Medium
Few
CenterPoint
39.2
76.4
43.1
3.5
+ Hierarchy Loss
+1.2
40.4
77.1
45.1
4.3
+ MMLF w/ Co-DINO
+7.7
48.1
78.8
52.9
16.9
+ External Data
+4.5
52.6
79.8
58.1
23.5
+ Score Calibration
+2.3
54.9
80.2
60.7
27.1
+ Probabilistic Fusion
+2.1
57.0
80.7
64.0
29.2
Table 4: Adding our developed components progressively improves LT3D performance, confirming that (1) 2D RGB detectors are better suited for our MMLF, (2) matching projected 3D LiDAR detections on the 2D image plane outperforms matching 2D RGB detections inflated to 3D, and (3) score calibration prior and probabilistic fusion improve performance. A majority of performance improvements is attributed to fusing detections of the 2D RGB detector Co-DINO on the 2D image plane. Moreover, training with external data notably improves rare class accuracy by 6.6 mAP.
Figure 10: Training 2D detectors with more data (e.g., nuScenes + nuImages vs. nuScenes only) and using better 2D detectors (e.g., Co-DINO, GroundingDINO, DINO, compared with YOLOV7) improve performance not only on 2D detection but also on the 3D detection achieved by our MMLF.
Method
Modality
Time
Memory
mAP
FCOS3D ( Wang et al., 2021 )
C
89
4.2 GB
20.9
BEVFormer ( Li et al., 2022b )
C
327
6.4 GB
27.3
CenterPoint ( Yin et al., 2021a )
L
323
4.0 GB
40.4
FSHNet ( Liu et al., 2025a )
L
296
4.8 GB
41.6
CMT ( Yan et al., 2023 )
L+C
520
9.0 GB
44.4
TransFusion ( Bai et al., 2022 )
L+C
367
5.6 GB
39.8
Table 5: Comparison of inference efficiency with respect to wall-clock time (ms) and GPU consumption. We use NVIDIA A6000 GPUs to measure the computation cost. Compared with other multimodal 3D detectors, MMLF is not only competitive in inference efficiency, but also significantly better in performance (mAP).
Methods
Truck
Adult
Bicycle
Traffic cone
near
far
sparse
dense
near
far
sparse
dense
near
far
sparse
dense
near
far
sparse
dense
CenterPoint
65.6
35.6
30.1
71.8
87.4
57.8
74.5
89.6
54.7
3.4
16.4
67.6
75.7
N/A
56.3
83.7
FSHNet
68.5
38.5
34.0
74.4
88.3
59.3
76.1
90.8
62.1
7.8
24.3
72.1
78.7
N/A
61.1
85.1
CMT
65.8
42.5
39.1
69.9
88.6
59.7
76.1
91.3
70.0
19.6
43.2
76.7
80.1
N/A
63.8
84.8
CCF
67.2
42.9
39.5
71.1
88.8
60.8
77.1
91.7
72.2
19.7
44.7
78.2
80.8
N/A
64.8
85.8
MMLF
71.2
45.2
40.1
76.1
90.2
63.9
80.1
93.3
81.3
26.5
58.2
88.4
82.3
N/A
66.8
86.7
Table 6: Breakdown results on objects from common (in top panel) and rare (in bottom panel) classes, with respect to distance and LiDAR density. We define far-field and near-field objects when they are > 30m and < 30m away from the ego vehicle, respectively. To define dense or sparse LiDAR returns, we consider a per-class threshold to evenly divide the test instances of a class. We provide the results for all the classes in Table 19 of the Appendix. The results demonstrate that our method boosts performance across most common and rare classes, both far-field and near-field objects, and both objects with dense and sparse LiDAR returns.
Method
mAPH
Car
Adult
Truck
CV
Bicycle
MC
Child
CW
Stroller
PP
CenterPoint ( Yin et al., 2021a )
LCA=0
82.4
81.2
49.4
19.7
33.6
48.9
0.1
14.2
0.1
21.7
(original)
LCA=1
83.9
82.0
58.7
20.5
35.2
50.5
0.1
18.3
0.1
22.0
LCA=2
84.0
82.4
58.8
20.7
36.4
51.0
0.1
19.5
0.1
22.6
CenterPoint (w/ our grouping-free head)
LCA=0
88.1
86.3
62.7
24.5
48.5
62.8
0.1
22.2
4.3
32.7
LCA=1
89.0
87.1
71.6
26.7
50.2
64.7
0.1
29.4
4.5
32.9
LCA=2
89.1
87.5
71.7
26.8
51.1
65.2
0.1
30.5
4.8
33.4
Table 7: Diagnosis using the mAP H metric on selected classes . We analyze the LiDAR-based 3D detector CenterPoint ( Yin et al., 2021a ) and multi-modal detector CMT ( Yan et al., 2023 ) , with our proposed grouping-free detector head, hierarchy loss, and MMLF (using the 2D RGB detector Co-DINO ( Zong et al., 2023 ) ). Comparing the rows of LCA=0 for CenterPoint and CMT, we see that MMLF significantly improves on rare classes (in blue ), e.g., construction-vehicle (CV), bicycle, motorcycle (MC), construction-worker (CW), stroller, and pushable-pullable (PP). Moreover, performance increases significantly from LCA=0 to LCA=1 compared against LCA=1 to LCA=2, suggesting that objects from rare classes are often detected but misclassified as sibling classes.
Figure 11: Both our MMLF (columns 1-3) and TransFusion ( Bai et al., 2022 ) (columns 4 -5) fail on these cases. In the first and second row, the 2D RGB detector Co-DINO detects the heavily occluded cars but 3D LiDAR detector fails to detect them. As a result, our MMLF misses these cars because it throws away unmatched RGB detections which do not have accurate 3D information. In the third row, we see that, although both the LiDAR and RGB detectors fire on the object (whose ground-truth label is police-officer ), the LiDAR detector classifies it as adult and the RGB detector classifies it as construction-worker . As a result, the final label is incorrectly predicted. TransFusion also misclassifies this object, confusing it with adult . Refer to Appendix N for more visualizations.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
CenterPoint
Multi-Head
Car
Ped.
Barrier
TC
Truck
Bus
Trailer
CV
MC
Bicyc.
Original
✓
87.7
87.7
70.7
74.0
63.6
72.7
45.1
26.3
64.7
47.9
89.1
88.4
70.8
74.3
64.8
72.9
42.0
25.7
65.9
53.6
for LT3D
✓
82.4
—
62.0
60.1
49.4
55.7
28.9
19.7
48.9
33.6
88.1
—
72.4
72.7
62.7
70.8
40.2
24.5
62.8
48.5
Adult
PP
CW
Debris
Child
Stroller
PO
EV
PM
All
Original
✓
—
—
—
—
—
—
—
—
—
64.0
Appendix
Table 8: Our proposed grouping-free detector head architecture consistently outperforms grouping-based approaches on both the standard and LT3D benchmarks. We note that sub-optimal grouping strategies (such as those adopted for LT3D) may yield significantly diminished performance, whereas optimized grouping strategies (such as those adopted for the standard setup) have comparable performance to the grouping-free approach. Note, TC is traffic-cone , CV is construction vehicle , MC is motorcycle , PP is pushable-pullable , CW is construction-worker , and PO is police-officer . We highlight Medium and Few classes in blue .
Method
Hier.
Many
Medium
Few
All
CenterPoint (w/o Hier.)
n/a
76.4
43.1
3.5
39.2
CenterPoint w/ Hier.
(a)
77.1
45.1
4.3
40.4
(b)
76.4
45.0
5.3
40.5
(c)
76.5
45.2
5.2
40.6
(d)
74.5
43.5
5.6
39.5
Appendix
Table 9: Different variants achieve similar performance. We note that other methods do improve accuracy in the tail by sacrificing performance in the head, suggesting that hybrid approaches that apply different techniques for head-vs-tail classes may further improve accuracy. Unlike ( Li et al., 2020 ; Wu et al., 2019 ) which requires a strict label hierarchy, our approach is not limited to a hierarchy.
Method
All
Many
Medium
Few
FCOS3D ( 3D RGB Detector)
18.3
36.0
21.1
0.2
BEVFormer ( 3D RGB Detector)
23.0
40.8
28.2
2.1
PolarFormer ( 3D RGB Detector)
20.7
37.5
25.2
1.6
DenseBEV ( 3D RGB Detector)
24.4
41.2
30.1
3.7
YOLOV7 ( 2D RGB Detector)
42.3
61.1
46.3
22.0
DINO ( 2D RGB Detector)
48.9
67.6
52.9
28.7
Appendix
Table 10: Evaluating 2D detection performance . Simply projecting 3D detections from RGB-only 3D detectors to 2D image plane yields considerably lower 2D detection mAP across all classes. However, 2D RGB detectors achieve higher performance. Intuitively, 2D RGB detectors achieve better classification performance and predict tighter fitting bounding boxes than those derived by projecting 3D detections to the image plane. This explains why fusing detections from 2D RGB detectors yields better LT3D performance than detections from 3D RGB detectors. All detectors in this table are only trained on the nuScenes train-set. Refer to Table 11 for per-class results of ensembling the strong 2D RGB detectors with the 3D LiDAR detector CenterPoint.
Class name
w/ YOLOV7
w/ DINO
w/ GD
w/ Co-DINO
Personal Mobility
18.9
35.8
37.1
42.3
Emergency Vehicle
12.5
21.1
24.7
28.0
Police Officer
10.9
14.4
15.9
16.1
Stroller
27.5
36.9
40.8
45.4
Child
16.7
24.9
28.6
32.1
Debris
5.3
6.1
8.9
11.4
Appendix
Table 11: Using a stronger 2D detector improves 3D detection by MMLF. We compare YOLOV7, DINO, GroundingDINO, and Co-DINO as the 2D detector, applied along with the 3D LiDAR detector CenterPoint ( Yin et al., 2021a ) in our MMLF. Results demonstrate that using stronger 2D detectors, especially Co-DINO, yields significantly better 3D detection performance on all classes, particularly on medium and few classes. Refer to Fig. 1 for the distribution of these classes.
Method
All
Many
Medium
Few
CenterPoint w/ Hier.
40.4
77.1
45.1
4.3
MMLF(CenterPoint, Co-DINO)
57.0
80.7
64.0
29.2
BEVFusion-L
42.5
72.5
48.0
10.6
MMLF(BEVFusion-L, Co-DINO)
56.2
80.1
63.2
28.1
CMT-L
34.7
73.4
35.9
1.1
MMLF(CMT-L, Co-DINO)
57.1
81.2
63.8
29.2
Appendix
Table 12: Transformer-based 3D LiDAR detector . We evaluate the late-fusion performance of BEVFusion-L and CMT-L (LiDAR-only) with Co-DINO, and find that our late fusion strategy improves over BEVFusion-L by 13.7 mAP overall and CMT-L by 22.4 mAP. This suggests that our late-fusion approach generalizes across different 3D detector architectures.
Class name
BevFormer
Deepinteraction
CMT
MMLF (CenterPoint, Co-DINO)
( Li et al., 2022b )
( Yang et al., 2022 )
( Yan et al., 2023 )
(Ours)
Personal Mobility
0.7
2.5
4.1
42.3
Emergency Vehicle
0.2
1.5
6.9
28.0
Police Officer
0.9
0.0
7.3
16.1
Stroller
1.1
30.9
0.9
45.4
Child
3.9
6.0
4.7
32.1
Appendix
Table 13: Per-class performance on nuScenes. Our approach MMLF fuses detections from the LiDAR-based 3D detector CenterPoint and the RGB-based 2D detector Co-DINO, achieving the highest per-class AP on 16 out of 18 classes. It is worth noting that, compared to CMT, our approach improves construction worker by 29.5 AP, stroller by 44.5 AP, and pushable-pullable by 23.1 AP.
FCOS3D
BEVFormer
CP
TransFusion
DeepInteraction
CMT
MMLF(CP, Co-DINO)
( Wang et al., 2021 )
( Li et al., 2022b )
( Yin et al., 2021a )
( Bai et al., 2022 )
( Yang et al., 2022 )
( Yan et al., 2023 )
(Ours)
mAP
20.9
27.3
39.2
39.8
43.7
44.4
57.0
NDS
30.4
38.8
54.9
53.9
54.4
55.9
62.8
Appendix
Table 14: AP vs. NDS results . We compare methods reported in the main paper w.r.t nuScenes Detection Score (NDS) and mAP. We find that the methods show broadly consistent rankings on both metrics, with MMLF ranking first under both mAP and NDS.
Method
Modality
0-40%
40-60%
60-80%
80-100%
CenterPoint (CP)
L
84.1
90.9
92.8
96.2
CMT
C + L
84.3
91.6
93.1
96.1
MMLF(CP, Co-DINO)
C + L
83.2
90.4
92.5
95.9
Appendix
Table 15: Average recall for occluded targets at different visibility levels . We evaluate average recall at four visibility levels (defined by nuScenes). Perhaps unsurprisingly, detection accuracy drops as objects become more occluded. Note that visibility attributes are not predicted by our model and are manually annotated for the ground truth, making it difficult to measure precision.
Method
Modality
0-10m
10-20m
20-30m
Full Range
CenterPoint (CP)
L
45.5
40.1
26.7
40.4
CMT
C + L
42.0
43.2
31.4
44.4
MMLF(CP, Co-DINO)
C + L
57.7
61.5
39.3
57.0
Appendix
Table 16: Detection accuracy at different distance thresholds . We evaluate detection accuracy at three distance ranges. Detecting objects closer to the ego-vehicle is easier (due to higher LiDAR point density) than detecting objects further away.
Method
Modality
Vehicle
Pedestrian
Movable
Object
CenterPoint
L
85.5
95.5
90.7
89.3
CMT
C + L
86.4
91.2
87.2
87.9
Appendix
Table 17: Average recall between detectors . CenterPoint (LiDAR-only) achieves higher recall than CMT for the Pedestrian (LCA=1), Movable (LCA=1), and Object (LCA=2) superclasses.
Translation Noise NT
0.01m
0.02m
0.03m
0.04m
CMT
44.4
44.3
44.3
44.2
MMLF(CP, Co-DINO)
57.0
56.9
56.8
56.8
Rotation Noise NR
1 ∘
2 ∘
3 ∘
4 ∘
CMT
44.3
43.1
42.7
41.1
MMLF(CP, Co-DINO)
54.3
53.4
51.7
51.3
Translation Noise NT
0.01m
0.02m
0.03m
0.04m
Appendix
Table 18: Impact of calibration noise . We apply a small amount of additive Gaussian noise to the translation and rotation calibration matrices to measure the impact of imprecise calibration on late fusion. While translation noise has little effect on detection accuracy, rotation noise significantly impacts performance. As expected, combining both translation and rotation noise degrades accuracy further.
Methods
near
far
sparse
dense
CenterPoint
93.7
65.9
70.5
96.5
FSHNet
94.1
67.8
72.5
96.9
CMT
93.6
69.8
73.6
96.2
CCF
93.8
70.2
73.8
96.4
MMLF
93.5
71.4
73.7
96.1
Appendix
Table 19: Detailed evaluation results of various methods across 18 obstacle categories.
Method
Modality
All
Many
Medium
Few
FCOS3D
C
14.6
27.4
17.0
7.8
DenseBEV
C
26.1
35.8
28.9
19.6
CenterPoint (CP)
L
44.0
77.4
46.9
30.2
FSHNet
L
45.1
78.2
47.0
32.7
CCF
C + L
49.9
81.8
52.0
37.6
MMF(CP, FCOS3D)
C + L
48.4
79.0
51.4
35.3
Appendix
Table 20: Results on the Argoverse 2 dataset . We present results on AV2 evaluated at 50m. The RGB-only 3D detector FCOS3D ( Wang et al., 2021 ) achieves poor performance, likely due to inaccurate depth estimates. In contrast, the LiDAR-only detector CenterPoint (CP) ( Yin et al., 2021a ) achieves strong performance on all classes. Our multi-modal fusion approach significantly improves over CenterPoint, achieving a 14.6 mAP improvement averaged over all classes. Importantly, late-fusion with 2D detector Co-DINO ( Zong et al., 2023 ) performs best by a notable margin. These results on AV2 are consistent with those on nuScenes, demonstrating the general applicability of our approach.
Figure 12: Left: According to the histogram of per-class object counts (on the left ), classes in Argoverse 2.0 (AV2) follow a long tailed distribution. Following ( Liu et al., 2019a ) and nuScenes, we report performance for three groups of classes based on their cardinality (split by dotted lines): Many , Medium , and Few . As AV2 does not provide a class hierarchy, we construct one by referring to the nuScenes hierarchy (cf. Fig. 2 on the right).
Method
Car
Truck
Trailer
Bus
CV
Bicy.
MC
Ped.
TC
Barrier
mAP
NDS
DeepInteraction ( Yang et al., 2022 ) (NeurIPS’22)
88.5
64.4
44.6
79.2
30.1
67.8
79.0
88.9
80.0
76.4
69.9
72.6
FSHNet ( Liu et al., 2025a ) (CVPR’25)
88.7
61.4
47.8
79.3
26.3
60.5
76.7
89.3
78.6
72.3
68.1
71.7
DenseBEV ( Dähling et al., 2026 ) (WACV’26)
63.1
38.6
17.1
46.2
12.9
48.2
45.6
53.2
63.5
60.6
44.9
54.9
CenterPoint ( Yin et al., 2021a ) (CVPR’21)
87.7
61.6
43.7
73.4
28.4
49.5
65.9
87.1
75.7
71.3
66.6
64.4
MMLF(CenterPoint, Co-DINO)
88.5
68.1
49.4
79.2
40.0
77.3
80.7
89.9
82.3
75.6
73.1
74.6
CMT ( Yan et al., 2023 ) (ICCV’23)
88.6
65.1
46.5
76.4
36.9
66.7
76.2
88.1
80.1
77.9
70.3
72.9
Appendix
Table 21: 3D detection performance on the standard nuScenes benchmark. We evaluate the impact of late fusion on the standard 10-class validation benchmark. Although our work focuses on improving detection accuracy in the tail, our proposed approach also improves performance on the standard benchmark. Notably, fusing CenterPoint ( Yin et al., 2021a ) with Co-DINO ( Zong et al., 2023 ) improves mAP and NDS by 6.5 and 10.2 percentage points, respectively. Fusing CMT ( Yan et al., 2023 ) , a multimodal 3D detector, with Co-DINO yields improvements of 2.7 mAP and 1.9 NDS, improving all standard classes, with the largest gains on bicycle , motorcycle , truck , trailer , and bus . We posit that CMT already effectively uses multimodal training data for the common classes of the standard nuScenes benchmark, yielding diminishing returns with late fusion. CV denotes construction vehicle , MC denotes motorcycle , Ped. denotes pedestrian , and TC denotes traffic cone .
Figure 13: We visualize the output of our late-fusion method, which combines 3D LiDAR detections from CenterPoint and 2D RGB detections from Co-DINO. In all cases, we find that the 2D RGB detector is able to correct classification errors from the 3D LiDAR detector, improving overall performance. Importantly, score calibration and probabilistic ensembling increase the confidence of the final prediction.
Figure 14: Three examples demonstrate how our multi-modal late-fusion (MMLF) approach improves LT3D by ensembling 2D RGB detections (from Co-DINO) and 3D LiDAR detections (from CenterPoint ( Yin et al., 2021a ) ). In all examples, MMLF correctly relabels detections which are geometrically similar (w.r.t size and shape) in LiDAR but are visually distinct in RGB, such as bus -vs- truck , adult -vs- stroller , and adult -vs- child . This figure complements Fig. 9 in the main paper.
Figure 15: Both our MMLF method (column 1) and TransFusion ( Bai et al., 2022 ) (column 2) share the same failure cases. In the first and second row, the 2D RGB detector Co-DINO detects the heavily occluded cars but 3D LiDAR detector fails to detect them. As a result, our MMLF misses these cars because it throws away unmatched RGB detections which do not have accurate 3D information. In the third row, we see that, although both the LiDAR and RGB detectors fire on the object (whose ground-truth label is police-officer ), the LiDAR detector classifies it as adult and RGB detector classifies it as construction-worker . As a result, the final detection is incorrect w.r.t the predicted label. TransFusion also misclassifies this object as an adult . This figure complements Fig. 11 in the main paper.
In order to navigate complex traffic environments, self-driving vehicles must recognize many semantic classes pertaining to vulnerable road users or traffic control devices. However, many safety-critical objects (e.g., construction worker) appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone. Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge to improve generalization. We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection. Specifically, FOMO-3D exploits rich semantic and depth priors from OWLv2 and Metric3Dv2 within a two-stage detection paradigm that first generates proposals with a LiDAR-based branch and a novel camera-based branch, and refines them with attention especially to image features from OWL. Evaluations on real-world driving data show that using rich priors from vision foundation models with careful multi-modal fusion designs leads to large gains for long-tailed 3D detection. Project website is at https://waabi.ai/fomo3d/.
Long-tail distributions in driving datasets pose a fundamental challenge for 3D perception, as rare classes exhibit substantial intra-class diversity yet available samples cover this variation space only sparsely. Existing instance augmentation methods based on copy-paste or asset libraries improve rare-class exposure but are often limited in fine-grained diversity and scene-context placement. We propose VERIA, an image-first multimodal augmentation framework that synthesizes synchronized RGB--LiDAR instances using off-the-shelf foundation models and curates them with sequential semantic and geometric verification. This verification-centric design tends to select instances that better match real LiDAR statistics while spanning a wider range of intra-class variation. Stage-wise yield decomposition provides a log-based diagnostic of pipeline reliability. On nuScenes and Lyft, VERIA improves rare-class 3D object detection in both LiDAR-only and multimodal settings. Our code is available at https://sgvr.kaist.ac.kr/VERIA/.
Camera-only 3D object detection has emerged as a cost-effective and scalable alternative to LiDAR for autonomous driving, yet existing methods primarily prioritize overall performance while overlooking the severe long-tail imbalance inherent in real-world datasets. In practice, many rare but safety-critical categories such as children, strollers, or emergency vehicles are heavily underrepresented, leading to biased learning and degraded performance. This challenge is further exacerbated by pronounced inter-class ambiguity (e.g., visually similar subclasses) and substantial intra-class diversity (e.g., objects varying widely in appearance, scale, pose, or context), which together hinder reliable long-tail recognition. In this work, we introduce SemLT3D, a Semantic-Guided Expert Distillation framework designed to enrich the representation space for underrepresented classes through semantic priors. SemLT3D consists of: (1) a language-guided mixture-of-experts module that routes 3D queries to specialized experts according to their semantic affinity, enabling the model to better disentangle confusing classes and specialize on tail distributions; and (2) a semantic projection distillation pipeline that aligns 3D queries with CLIP-informed 2D semantics, producing more coherent and discriminative features across diverse visual manifestations. Although motivated by long-tail imbalance, the semantically structured learning in SemLT3D also improves robustness under broader appearance variations and challenging corner cases, offering a principled step toward more reliable camera-only 3D perception.
Hao Vo, Khoa Vo, Thinh Phan +5
AICV Lab, University of Arkansas, USA · University of Utah, USA · University of Houston, USA +1