FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detection

Anqi Joyce Yang; James Tu; Nikita Dvornik; Enxu Li; Raquel Urtasun

cs.CVMar 9, 2026

FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detection

Authors: Anqi Joyce Yang, James Tu, Nikita Dvornik, Enxu Li, Raquel Urtasun

Organizations: Waabi · University of Toronto

Abstract

In order to navigate complex traffic environments, self-driving vehicles must recognize many semantic classes pertaining to vulnerable road users or traffic control devices. However, many safety-critical objects (e.g., construction worker) appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone. Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge to improve generalization. We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection. Specifically, FOMO-3D exploits rich semantic and depth priors from OWLv2 and Metric3Dv2 within a two-stage detection paradigm that first generates proposals with a LiDAR-based branch and a novel camera-based branch, and refines them with attention especially to image features from OWL. Evaluations on real-world driving data show that using rich priors from vision foundation models with careful multi-modal fusion designs leads to large gains for long-tailed 3D detection. Project website is at https://waabi.ai/fomo3d/.

Figures & tables

Figure 1: Vision foundation models OWL (left) and Metric3D (middle) show remarkable zero-shot generalization capabilities for 2D object detection and monocular depth estimation. Our model FOMO-3D (right) incorporates these strong priors along with LiDAR for multi-modal 3D object detection.

Figure 2: Overview of FOMO-3D, which leverages vision foundation models OWL and Metric3D, and follows a two-stage paradigm with a multi-modal proposal stage and an attention-based refinement stage.

Figure 3: [Left] Lifting OWL camera proposals to 3D bounding boxes. We first unproject pixels inside the camera proposal into 3D using Metric3D depths, and then encode the points into a BEV feature map. Each OWL token subsequently attends to fused LiDAR and image BEV features sampled along the frustum. [Right] During supervision, camera proposals are only matched to ground truth boxes inside the object frustum.

Method	Modality	All	Many	Medium	Few
FCOS3D [ 36 ]	C	20.9	39.0	23.3	2.9
BEVFormer [ 38 ]	C	27.3	52.3	31.6	1.4
CenterPoint (Group-Free) [ 1 , 5 ]	L	39.2	76.4	43.1	3.5
CenterPoint (Group-Free, w/ Hier.) [ 1 , 5 ]	L	40.4	77.1	45.1	4.3
BEVFusion-L [ 2 ]	L	42.5	72.5	48.0	10.6
TransFusion [ 25 ]	L+C	39.8	73.9	41.2	9.8

Table 1: [nuScenes] Comparison with SOTA methods (measured by mAP). L denotes LiDAR, C denotes camera and ∗ is our re-implementation. Our method FOMO-3D improves LT3D performance across all object groups, with a significant 7.6 mAP gain in Few and a 2.0 mAP gain in Many .

Figure 4: Real-world class distribution on nuScenes and Highway . Both exhibit severe class imbalances.

Method	Car	Adult	Truck	CV	Bicycle	MC	Child	CW	Stroller	PP
MMF [ 5 ]	88.5	86.6	63.4	29.0	58.5	68.2	5.3	35.8	31.6	39.3
MMLF [ 24 ]	86.3	87.7	60.6	35.3	70.0	75.9	8.8	55.9	37.7	58.1
FOMO-3D	88.9	90.7	65.1	36.6	72.8	80.2	29.8	60.2	40.1	50.0

Table 2: [nuScenes] Class-specific mAP . CV = Construction Vehicle. MC = Motorcycle. CW = Construction Worker. PP = Pushable-Pullable. Medium and Few classes are in blue.

Figure 5: [Highway] Per-class mAP gains over the base LiDAR-only detector, for distance buckets [0, 50], [50, 200] and [200, 230] meters relatives to the SDV. FOMO-3D (no cam prop) corresponds to

M_{2}

in Table 4 .

Method	Mod.	Vehicle	Towed	Cone	Person	Cyclist
FOMO-L [ 1 , 45 ]	L	86.7	73.5	82.6	58.4	74.8
MMF ∗ [ 5 ]	L+C	86.3	73.3	82.6	58.3	75.0
MMLF ∗ [ 24 ]	L+C	86.6	73.4	83.0	58.3	76.2
FOMO-3D	L+C	87.2	74.2	88.9	68.7	79.3

Table 3: [Highway] mAP comparisons.

Figure 6: [Qualitative results] OWL successfully detects the child but has a false positive cone . The LiDAR-only model misclassifies the child as an adult. FOMO-3D fuses multi-modal information and foundation model priors to generate an accurate 3D bounding box of the child , while rejecting the false positive cone .

Method	Prop	Refine	Cam Model	All	Many	Medium	Few
$M_{1}$	L	L	-	42.3	76.3	45.1	10.6
$M_{2}$	L	L+C	OWL	53.4	80.1	59.0	24.7
$M_{3}$	L+C	L+C	DETR [ 28 ]	47.9	80.1	55.6	12.2
$M_{4}$	L+C	L+C	OWL	54.6	79.9	59.6	27.6

Table 4: [nuScenes] Ablations.

$mAP_{H}$	Method	Car	Adult	Truck	CV	Bicycle	MC	Child	CW	Stroller	PP
LCA0	MMF [ 5 ]	88.5	86.6	63.4	29.0	58.5	68.2	5.3	35.8	31.6	39.3
	MMLF [ 24 ]	86.3	87.7	60.6	35.3	70.0	75.9	8.8	55.9	37.7	58.1
	FOMO-3D	88.9	90.7	65.1	36.6	72.8	80.2	29.8	60.2	40.1	50.0
LCA1	MMF [ 5 ]	89.4	87.4	72.4	31.3	61.2	69.7	15.2	52.0	37.7	39.4
	MMLF [ 24 ]	86.8	88.3	68.5	37.3	70.4	77.1	16.2	66.0	51.5	58.2
	FOMO-3D	89.4	91.2	70.3	39.2	73.9	81.5	59.2	73.4	54.8	50.1

Table 5: [nuScenes] Class-specific hierarchical metrics . CV = Construction Vehicle. MC = Motorcycle. CW = Construction Worker. PP = Pushable-Pullable. Medium and Few classes are in blue.

Frustum Attention	All	Many	Medium	Few
	53.6	80.9	58.6	25.1
✓	54.6	79.9	59.6	27.6

Table 6: [nuScenes] Frustum attention ablation.

Frustum Attention	Vehicle	Towed	Cone	Person	Cyclist
	87.0	73.6	86.9	64.1	76.4
✓	87.2	74.2	88.9	68.7	79.3

Table 7: [Highway] Frustum attention ablation.

$N_{x}$	$N_{y}$	$N_{z}$	$\delta$	All	Many	Medium	Few
0	0	20	10	53.5	79.8	58.2	26.1
3	3	20	10	55.0	80.2	59.3	29.0
1	1	10	10	54.6	79.9	59.1	28.2
1	1	40	10	54.3	79.8	58.8	27.9
1	1	10	4	53.8	79.6	58.8	26.3
1	1	10	25	52.6	79.9	59.1	22.8

Table 8: [nuScenes] Sampling resolution ablation.

Method	Frustum-based Loss Constraint	All	Many	Medium	Few
$M_{1}$		54.0	80.3	59.5	25.8
$M_{2}$ (FOMO-3D)	✓	54.6	79.9	59.6	27.6

Table 9: [nuScenes] Ablation for frustum-based matching constraint in camera proposals.

Method	Depth Estimator	All	Many	Medium	Few
$M_{1}$	LiDAR	53.0	79.9	58.9	23.7
$M_{2}$	M3D Rescaled	53.5	79.9	58.9	25.2
$M_{3}$ (FOMO-3D)	M3D	54.6	79.9	59.6	27.6

Table 10: [nuScenes] LiDAR depth ablations.

Camera	LiDAR	Object	All	Many	Medium	Few
	✓	✓	41.0	76.1	45.3	6.7
✓		✓	52.0	80.0	58.3	23.4
✓	✓		54.1	81.0	60.1	24.8
✓	✓	✓	54.6	79.9	59.6	27.6

Table 11: [nuScenes] Refinement type ablations.

Method	Prop	Refine	Vehicle	Towed	Cone	Person	Cyclist
$M_{1}$	L	L	86.7	73.5	82.6	58.4	74.8
$M_{2}$	L	L+C	86.9	73.8	86.0	65.4	77.7
$M_{3}$	L+C	L+C	87.2	74.2	88.9	68.7	79.3

Table 12: [Highway] Ablations.

Method	Proposal Stage Fusion	Learned Depth	All	Many	Medium	Few
$M_{1}$	Feature-level		54.8	80.2	60.6	26.9
$M_{2}$	Feature-level	✓	53.3	79.6	59.0	24.7
$M_{3}$ (FOMO-3D)	Late		54.6	79.9	59.6	27.6

Table 13: [nuScenes] Explorations for alternative proposal-level fusion strategies.

Method	Proposal Stage Fusion	Learned Depth	Vehicle	Towed	Cone	Person	Cyclist
$M_{1}$	Feature-level		86.1	71.7	86.5	64.6	76.4
$M_{2}$	Feature-level	✓	87.0	73.7	87.0	64.2	74.7
$M_{3}$ (FOMO-3D)	Late		87.2	74.2	88.9	68.7	79.3

Table 14: [Highway] Explorations for alternative proposal-level fusion strategies.

Figure 7: [nuScenes] Qualitative example #1. In this example, we added OWL prompts for barriers to illustrate low-quality zero-shot barrier detections. As shown in the visualization, OWL produces many false positive barriers without capturing most of the true positive barriers. As a result we removed barrier prompting from the camera proposal branch. In addition, the LiDAR-only model misses small objects such as cones and construction workers, but OWL and FOMO-3D are able to detect them successfully.

Figure 8: [nuScenes] Qualitative example #2. In this example, OWL correctly detects and classifies an adult and an emergency vehicle. The LiDAR-only model misses the adult (because it is small and sparsely observed by LiDAR) and misclassifies the emergency vehicle as car due to lack of semantics. FOMO-3D is able to successfully detect the adult thanks to OWL, but fails to retain the original OWL emergency vehicle classification.

Figure 9: [nuScenes] Qualitative example #3. In this example, the LiDAR-only model fails to detect a cone and a construction vehicle, while FOMO-3D is able to detect and classify them successfully.

Figure 10: [nuScenes] Qualitative example #4. In this example, the LiDAR-only model fails to detect an adult and a construction worker and also outputs a false positive bicycle, while FOMO-3D is able to detect them successfully while rejecting the false positive.

Figure 11: [nuScenes] Qualitative example #5. This example shows a failure case of OWL and FOMO-3D. OWL produces many duplicated construction workers. While FOMO-3D lifts them to the correct construction worker size, it fails to reject some duplicated false positives in the final detections. This is a challenging case because these false positives are adjacent to true positives in the image. False positives can be generated in the camera proposal stage directly from OWL proposals. In the refinement stage, many proposals which project onto the same image region could pick up features describing construction workers, and multiple proposals can attend to the same image features to generate these false positives. Designing a more optimal false positive and duplicate removal method is a future direction to improve FOMO-3D.

Figure 12: [Highway] Qualitative example #1. Green denotes vehicle. Purple denotes towed object. Blue denotes cyclists. Orange denotes person. Yellow denotes cone. Opacity of the boxes reflects detection confidence (higher confidence corresponds to more solid lines). FP=False Positive. FOMO-3D is able to correct various errors in OWL and LiDAR-only detections, but still mis-classifies a truck as trailer.

Figure 13: [Highway] Qualitative example #2. Green denotes vehicle. Purple denotes towed object. Blue denotes cyclists. Orange denotes person. Yellow denotes cone. Opacity of the boxes reflects detection confidence (higher confidence corresponds to more solid lines). FP=False Positive. This example shows a very dense traffic scene on the highway. OWL has impressive zero-shot 2D detections, but it also outputs a few false positives. LiDAR-only model mis-classifies a vehicle as a cyclist, and also draws a false positive cone. FOMO-3D is able to correctly classify in most cases without having many false positives. All three detectors miss two cones on the right.

Figure 14: [Highway] Qualitative example #3. Green denotes vehicle. Purple denotes towed object. Blue denotes cyclists. Orange denotes person. Yellow denotes cone. Opacity of the boxes reflects detection confidence (higher confidence corresponds to more solid lines). FP=False Positive. Unlike OWL, FOMO-3D is able to utilize 3D LiDAR information and the training data to distinguish between vehicles and towed objects. FOMO-3D is able to effectively fuse 3D and 2D information and learn from the training data. It also outputs fewer false positives compared to the LiDAR-only model.

Figure 15: [Highway] Qualitative example #4. Green denotes vehicle. Purple denotes towed object. Blue denotes cyclists. Orange denotes person. Yellow denotes cone. Opacity of the boxes reflects detection confidence (higher confidence corresponds to more solid lines). FP=False Positive. In this example, the LiDAR-only model misses the person on the right, while OWL is able to detect it. With effective multi-modal fusion, FOMO-3D successfully detects the person.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

Long-Tailed 3D Detection via Multi-Modal Fusion
Dec 18, 2023Yechi Ma, Neehar Peri, Achal Dave +33D Object Detection Long-Tail Autonomous Driving
Department of Computer Science, Zhejiang University. · Computing & Mathematical Sciences, Caltech. · Toyota Research Institute. +2
Towards robust multimodal 3D object detection via visual foundation models
Sep 20, 2026Ziying Song, Lin Liu, Hongyu Pan +4Multimodal Robustness Vision Foundation Models
School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China · Horizon Robotics, Beijing 100000, China · University of Macau, Macao 999078, China +1
SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection
Apr 20, 2026Hao Vo, Khoa Vo, Thinh Phan +5Autonomous Driving Perception 3D Object Detection
AICV Lab, University of Arkansas, USA · University of Utah, USA · University of Houston, USA +1

Dec 18, 2023J/K move · Enter open · S save

Long-Tailed 3D Detection via Multi-Modal Fusion

Yechi Ma, Neehar Peri, Achal Dave, Wei Hua+2

Department of Computer Science, Zhejiang University. · Computing & Mathematical Sciences, Caltech. · Toyota Research Institute.+2

3D Object Detection Long-Tail Autonomous Driving Multimodal Object Detection Camera-LiDAR Fusion

Open paper AskPDF arXiv

Contemporary autonomous vehicle (AV) benchmarks have significantly advanced multimodal (LiDAR+RGB) 3D detection. However, despite the naturally long-tailed distribution of object classes, existing benchmarking protocols primarily focus on frequent categories (e.g., pedestrian and car), largely overlooking rare but safety-critical classes such as stroller and emergency vehicle. In practice, reliable detection of both common and rare classes is essential for safe autonomous driving. We formalize this problem as Long-Tailed 3D Detection (LT3D), where evaluation encompasses all annotated classes, including rare ones. To address LT3D, we introduce hierarchical losses that promote feature sharing across classes, diagnostic metrics that assign partial credit to semantically reasonable mistakes with respect to the semantic hierarchy (e.g., confusing a child with an adult), and a multimodal late-fusion (MMLF) framework to fuse detections. In particular, we show that rare-class accuracy benefits substantially from MMLF of independently trained uni-modal LiDAR and RGB detectors. Because of the modular design, unlike prevailing end-to-end trained multi-modal detectors that require paired LiDAR-RGB data, MMLF enables the use of advanced unimodal detectors that are trained on large-scale uni-modal datasets with sufficient data for rare classes. Lastly, we examine three fundamental design choices in MMLF, including the RGB detector representation (2D vs. 3D), cross-modal association (3D vs. image plane), and fusion strategy. We find that 2D RGB detectors recognize rare classes more reliably than 3D RGB detectors, image-plane association is more robust to depth estimation errors, and probabilistic score-calibrated fusion consistently yields the best performance. Extensive experiments on nuScenes and Argoverse2 demonstrate substantial improvements of MMLF, establishing a new state of the art.

Abstract

3D Object Detection

Explore similar work

Long-Tailed 3D Detection via Multi-Modal Fusion

Towards robust multimodal 3D object detection via visual foundation models

SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection