OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

Mariia Gladkova; Neehar Peri; Ishan Khatri; Deva Ramanan; Daniel Cremers

cs.CVAug 17, 2026

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

Authors: Mariia Gladkova, Neehar Peri, Ishan Khatri, Deva Ramanan, Daniel Cremers

Organizations: Technical University of Munich · Carnegie Mellon University · StackAV

Abstract

Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric. To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer. We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D's performance collapses from 18.6 to 5.4 AP when prompted with "a detailed high-resolution photo of a car" rather than "car"); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D's AP by 1.9 $\times$ on ScanNet). Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector's predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.

Figures & tables

Panel of Figure 1 — Figure 1: Open-Vocabulary Monocular 3D Detectors Struggle with Vocabulary, Not Geometry. All five detectors consistently localize a regular vehicle (mean 3D IoU $0.58$ , BEV insets) yet disagree on its category (top) . Prompt phrasing swings accuracy sharply for prompt-conditioned detectors but far less for our SigLIPv2-based remapping baseline (bottom left) . The target-aware protocol hides these semantic errors, inflating scores most where the dataset-level score is lowest. (bottom right) .

	Open Vocab.	Mono. Input	Text Prompt	Box Prompt	Point Prompt	Large Scale
Cube R-CNN [ 1 ]	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
OV-3Det [ 13 ]	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$	$\times$
ImOV3D [ 30 ]	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$	$\times$
OVM3D-Det [ 7 ]	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
OVMono3D [ 31 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$
DetAny3D [ 32 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Model Comparison. We summarize the input modality, prompting modes, and training scale of the seven detectors in our benchmark above. A check mark denotes support.

	SUNRGBD	ARKitScenes	Hypersim	ScanNet-200	KITTI	nuScenes	Argoverse 2
Cube R-CNN*	0.523	0.335	0.085	0.157	0.716	0.389	0.286
OVMono3D	0.586	0.389	0.129	0.218	0.729	0.396	0.279
DetAny3D	0.440	0.171	0.064	0.146	0.642	0.281	0.142
WildDet3D	0.640	0.323	0.117	0.389	0.597	0.304	0.326

Table 2: 3D Class-Agnostic Recall (

\uparrow

). No method dominates every dataset; recall varies far more across datasets (dense indoor scenes like Hypersim and ScanNet-200 are hardest) than across methods within a dataset. Green marks in-domain datasets (Omni3D); yellow marks unseen ones (green for DetAny3D, whose training corpus includes Argoverse 2 and ScanNet). We mark the best method per dataset in bold .

	SUNRGBD	ARKitScenes	Hypersim	ScanNet-200	KITTI	nuScenes	Argoverse 2
Cube R-CNN*	6.38	21.33	5.33	2.22	17.64	15.49	11.10
OVMono3D	7.18	28.31	6.95	2.44	15.96	13.81	2.26
DetAny3D	8.81	19.92	6.23	2.89	19.65	12.83	0.82
WildDet3D	5.92	23.24	6.43	2.23	19.92	18.00	4.82

Table 3: Standard mAP

{}_{\text{3D}}

Detection Evaluation (

\uparrow

). No method wins outright, with each detector leading on at most two of seven datasets. Green marks in-domain datasets (Omni3D); yellow marks unseen ones. Argoverse 2 and ScanNet are in the DetAny3D training corpus, so those results are not zero-shot. We mark the best method per dataset in bold .

Table 4: Cross-Dataset Transfer Costs Recognition, Not Geometry. We train a separate model per dataset and evaluate each in-domain and on out-of-domain transfer, across SUNRGBD (SU), ScanNet-200 (SC), and ARKitScenes (AR) on ( 4(a) ) 3D Class-Agnostic Recall and ( 4(b) ) standard mAP 3D . Recall shifts modestly between in- and out-of-domain models, while mAP 3D favors the in-domain model by as much as

5.5\times

. Green marks in-domain results; yellow marks cross-domain transfer. Higher values are better for both metrics, and we mark the best setting per dataset in bold .

Train	KITTI	nuScenes	Argoverse 2
KITTI	12.87	2.78	1.37
nuScenes	9.69	7.95	14.99

Table 5: Only nuScenes Pre-Training Transfers Outdoors. We train OVM3D-Det* on KITTI and on nuScenes and evaluate both on all three outdoor datasets. The nuScenes-trained model recovers

75\%

of in-domain KITTI mAP 3D , while the KITTI-trained model recovers only

35\%

on nuScenes. Green indicates in-domain; yellow indicates cross-domain transfer; bold denotes the best configuration.

Figure 2: Detectors Share the Same Confusions. We visualize confusion matrices over the $15$ ARKitScenes categories for Cube R-CNN*, OVM3D-Det*, DetAny3D, and OVMono3D. Each row is normalized by its ground-truth count over matched boxes, and the diagonal gives per-class recall. Furniture, appliances and storage absorb nearly all off-diagonal probability mass across all four detectors, while sink and bathtub stay cleanly separated. Best viewed in color and zoomed in.

Panel of Figure 3 — Figure 3: Failures Are Semantic, Not Geometric. Ground truth box alongside predictions from each of the five detectors. Boxes are well aligned to the object across nearly every detector, while the predicted labels disagree.

Figure 4: Confusion Concentrates Among Indoor Furniture. Predictions aggregated over all datasets and detectors, row-normalized by GT super-category.

Table 10

Encoder	Prompt usage	Tokenizer	ARKitScenes			KITTI			CV%
Encoder	Prompt usage	Tokenizer	Und.	Std.	Ovr.	Und.	Std.	Ovr.	$\downarrow$
SigLIPv2 *	Remapping	SentencePiece	26.8	27.7	23.3	19.0	19.4	17.7	8
CLIP *	Remapping	BPE	19.1	21.0	19.5	14.8	10.7	12.4	12
OVMono3D [ 31 ]	Grounding	BERT	26.4	22.2	21.3	11.3	11.2	8.4	20
WildDet3D [ 8 ]	Prompt Cond.	BPE	28.9	33.1	21.7	18.6	15.5	5.4	34
DetAny3D [ 32 ]	Grounding	BERT	16.6	14.3	13.8	12.3	10.8	7.8	28

Table 8: Prompt Robustness. We evaluate mAP 3D on a fixed 500-image subset, grouped into underdetailed (Und.), standard (Std.), and overdetailed (Ovr.) templates (5 variations per template). Encoders that only remap fixed detections vary least across prompt phrasings, while those that condition detections on the prompt vary far more. * marks the encoder used to remap Cube R-CNN detections (SigLIPv2 or CLIP). “Grounding” denotes GroundingDINO, and “Prompt Cond.” denotes SAM 3-style prompt-conditioned detection. CV% is the coefficient of variation across all 15 templates, averaged over both datasets (

\downarrow

is better).

Method	Dataset	# CLS	Full	Random	$10$ Class Split		Fav./
Method	Dataset	(GT)	Vocab.	( $\mu\pm\sigma$ )	Adverse	Favorable	Full
DetAny3D	SUNRGBD	82	8.81	8.7 $\pm$ 3.9	0.0	38.9	4.4 $\times$
DetAny3D	ScanNet-200	174	2.89	2.9 $\pm$ 1.7	0.0	22.2	7.7 $\times$
OVMono3D	SUNRGBD	82	7.18	7.2 $\pm$ 3.9	0.0	37.9	5.3 $\times$
OVMono3D	ScanNet-200	174	2.44	2.4 $\pm$ 1.7	0.0	21.2	8.7 $\times$

Table 9: Vocabulary Subset Selection Outweighs the Choice of Detector. For fixed predictions (model unchanged, having seen all classes) we report mAP 3D over random

10

-class subsets of the dataset vocabulary. “Full Vocab.” is the dataset-level AP. “Random” is the mean

\pm

std over

5{,}000

random

10

-class subsets. “Adverse” and “Favorable” are the worst and best possible

10

-class subsets, taken as the bottom- and top-

10

categories by per-category AP, and they bound the achievable range. “Fav./Full” is the favorable-bound inflation over the full-vocabulary number. #CLS (GT) counts categories with at least one ground-truth instance in the test set, fewer than the dataset-level

83

196

classes.

Figure 5: No Single Axis Predicts Another. We compare monocular 3D detectors along three benchmark axes and mAP 3D , min-max normalized across the four detectors. WildDet3D tops the in-domain axis yet ranks last on prompt robustness; Cube R-CNN* is weakest in-domain yet leads domain generalization and prompt robustness.

Method	Sem.	Geom.	Total
Cube R-CNN*	190.6	33.3	223.9
OVM3D-Det*	190.6	83.0	273.6
OVMono3D	300.8		300.8
DetAny3D	182.9	400.0	582.9

Table 10: End-to-End Latency (ms/image). We benchmark runtime on 1 H100 by stage. Cube R-CNN* is fastest at

223.9

ms,

2.6\times

ahead of DetAny3D.

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 89 papers

WildDet3D: Scaling Promptable 3D Detection in the Wild
Apr 9, 2026Weikai Huang, Jieyu Zhang, Sijun Li +153D Object Detection Monocular 3D Object Detection
Allen Institute for AI · University of Washington · Johns Hopkins University +2
Open-vocabulary 3D object detection with promptable segmentation
Sep 16, 2026Ömer Faruk Deniz, Mustafa Taha KoçyiğitLidar-Based 3D Object Detection Multi-View 3D Object Detection
Institute of Data Science and Artificial Intelligence, Boğaziçi University, South Campus, Bebek, 34342, Istanbul, Türkiye
RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection
Aug 10, 2026Zhihao Zhang, Gengwei Zhang, Tianlong Chen +13D Object Detection Monocular 3D Object Detection
Michigan State University · University of North Carolina at Chapel Hill

Apr 9, 2026J/K move · Enter open · S save

WildDet3D: Scaling Promptable 3D Detection in the Wild

Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia+14

Allen Institute for AI · University of Washington · Johns Hopkins University+2

3D Object Detection Monocular 3D Object Detection

Open paper AskPDF arXiv

Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).

Abstract

Open-Vocabulary Object Detection

Explore similar work

WildDet3D: Scaling Promptable 3D Detection in the Wild

Open-vocabulary 3D object detection with promptable segmentation

RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection