cs.CVApr 9, 2026
SaveWildDet3D: Scaling Promptable 3D Detection in the Wild
Organizations: Allen Institute for AI · University of Washington · Johns Hopkins University · Cornell University ·
Abstract
Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).
Figures & tables
Figure 1: Overview of WildDet3D. Given a single RGB image and an optional depth map, WildDet3D performs open-vocabulary monocular 3D object detection by accepting flexible prompt modalities—text queries, 2D point clicks, or 2D bounding boxes—and predicting full 3D bounding boxes for the specified objects. This unified framework enables interactive, open-world 3D perception across diverse scenes and thousands of object categories, supporting applications in mobile devices, robotics, and AR/VR. The model gracefully leverages additional geometric cues, i.e . , depth when available. To train WildDet3D for broad generalization, we also curate WildDet3D-Data, a large-scale in-the-wild dataset with approximately 1M human-verified samples spanning 13K categories.
Figure 2: Input modality comparison for generalized 3D detection. LiDAR point clouds lack reliable height information and full 6-DoF rotation cues. RGB images provide dense appearance features but suffer from inherent scale and occlusion ambiguity. By combining RGB with optional depth, our approach retains the rich visual semantics needed for open-vocabulary recognition while reducing metric scale ambiguity when geometric signals are available.
Figure 3: Overview of WildDet3D. Given an RGB image and optional depth input, dual-vision encoders (Image + RGBD) extract visual features and depth latents in parallel. The depth fusion module processes the depth latents generated by the RGBD encoder and combines it with image features from the image encoder via element-wise addition to produce enriched visual queries, which are then integrated with diverse input prompts via the promptable detector . The resulting outputs are passed through cascaded 2D and 3D Heads for open-vocabulary object detection, while the depth latents are separately decoded for depth estimation. Although our primary focus is 3D object detection, the auxiliary 2D detection and depth heads provide complementary supervision that improves overall performance.
Figure 4: Overview of the WildDet3D-Data pipeline. Given an image with 2D boxes or masks and a depth-derived point cloud, five complementary models generate candidate 3D bounding boxes, shown in different colors. These candidates are refined through translation and rotation optimization, followed by rule-based filtering. The filtered candidates then enter two parallel selection branches: a VLM filtering branch, which scores each candidate on six perceptual criteria and retains those whose scores exceed a threshold, and a human annotation branch, in which annotators select the best candidate and assess its quality.
Figure 5: Qualitative examples from WildDet3D-Data. Each pair shows 3D bounding box annotations overlaid on the input image with category labels (left) and the corresponding 3D bounding boxes rendered in the reconstructed point cloud (right). The dataset covers diverse settings including indoor scenes (desks, kitchens), outdoor environments (floating markets, streets), and animals in the wild, spanning a wide range of object categories, scales, and spatial layouts.
| Split | Source | Images | Ann. | Categories | Type | Scene | Max depth |
| Existing datasets | |||||||
| Omni3D [ 5 ] | KITTI, nuSc., SUNRGBD, etc. | 234K | 3M+ | 98 | Human | Driving, Furniture | 67 m |
| COCO-3D [ 61 ] | COCO | 18K | 92K | 80 | Synthetic | In-the-wild | 35 m |
| CA-1M [ 24 ] | ARKitScenes | 3,500 (videos) | 400K | Class-agnostic | Human | Indoor | 5 m |
| WildDet3D-Data | |||||||
| Train (Human) | COCO, LVIS, Obj365, V3Det | 102,979 | 229,934 | 12,064 | Human | In-the-wild | |
Table 1: WildDet3D-Data statistics. Human annotations are rated by crowd-source workers, while synthetic annotations are auto-selected by VLM scoring. Combined, the dataset spans 13.5K categories—a 138 increase over Omni3D’s 98 categories.
| Method | Training data | AP | AP | AP | AP |
| Text Prompt | |||||
| 3D-MOOD [ 59 ] | Omni3D | 2.4 | 2.1 | 2.6 | 2.3 |
| WildDet3D | Omni3D | 9.0 | 6.5 | 5.2 | 6.8 |
| WildDet3D w/ depth | Omni3D | 23.0 | 21.5 | 16.1 | 20.7 |
| WildDet3D | Omni3D, Others, WildDet3D-Data | 28.3 | 21.6 | 18.7 | 22.6 |
| WildDet3D w/ depth | Omni3D, Others, WildDet3D-Data | 47.4 | 40.7 | 37.2 | 41.6 |
Table 3: WildDet3D-Bench evaluation . We observe that (1) WildDet3D outperforms baseline models when trained on the same data ( i.e . , Omni3D), (2) our newly introduced WildDet3D-Data further improves performance by a significant margin (6.8 22.6, 8.4 24.8), and (3) incorporating depth input at test time nearly doubles performance (22.6 41.6, 24.8 47.2).
| Method | KITTI [ 17 ] | nuScenes [ 7 ] | SUNRGBD [ 46 ] | Hypersim [ 43 ] | ARKitScenes [ 4 ] | Objectron [ 1 ] | AP |
| Text Prompt | |||||||
| Cube R-CNN [ 5 ] | 32.6 | 30.1 | 15.3 | 7.5 | 41.7 | 50.8 | 23.3 |
| Uni-MODE* [ 28 ] | 29.2 | 36.0 | 23.0 | 8.1 | 48.0 | 66.1 | 28.2 |
| 3D-MOOD Swin-T [ 59 ] | 32.8 | 31.5 | 21.9 | 10.5 | 51.0 | 64.3 | 28.4 |
| 3D-MOOD Swin-B [ 59 ] | 31.4 | 35.8 | 23.8 | 9.1 | 53.9 | 67.9 | 30.0 |
| WildDet3D | 37.0 | 31.7 | 38.9 | 16.5 | 64.6 | 60.5 | 34.2 |
Table 4: Omni3D evaluation. Our model outperforms previous state-of-the-art methods in both text and box prompt settings. Incorporating depth input at test time further improves performance significantly.
| Argoverse 2 [ 55 ] | ScanNet [ 12 ] | |||||||||
| Method | AP | mATE | mASE | mAOE | ODS | AP | mATE | mASE | mAOE | ODS |
| Cube R-CNN [ 5 ] | 8.6 | 0.903 | 0.867 | 0.953 | 8.9 | 20.0 | 0.733 | 0.774 | 0.921 | 19.5 |
| 3D-MOOD Swin-T [ 59 ] | 14.8 | 0.782 | 0.697 | 0.612 | 22.5 | 27.3 | 0.630 | 0.726 | 0.650 | 30.2 |
| 3D-MOOD Swin-B [ 59 ] | 14.7 | 0.755 | 0.680 | 0.580 | 23.8 | 28.8 | 0.612 | 0.706 | 0.655 | 31.5 |
| WildDet3D | 43.4 | 0.714 | 0.645 | 0.526 | 40.3 | 56.5 | 0.601 | 0.720 | 0.437 | 48.9 |
| WildDet3D w/ depth | 43.4 | 0.701 | 0.645 | 0.526 | 40.4 | 57.6 | 0.589 | 0.707 | 0.422 | 50.2 |
Table 5: Zero-shot evaluation. ODS is the Open Detection Score [ 59 ] ; higher is better. mATE, mASE, mAOE denote mean translation, scale, and orientation errors (lower is better).
| Method | AP | AP | AP | AP |
| Box Prompt | ||||
| OVMono3D-LIFT [ 60 ] | 12.3 | 7.1 | 11.4 | 9.9 |
| DetAny3D [ 64 ] | 8.3 | 8.2 | 4.9 | 7.1 |
| WildDet3D | 8.1 | 6.3 | 8.5 | 7.5 |
| WildDet3D w/ depth | 26.2 | 31.1 | 24.6 | 27.7 |
Table 6: Stereo4D evaluation. AP is computed using center-distance matching. All models are evaluated in a zero-shot manner, i.e . , not trained on Stereo4D.
| Configuration | KITTI [ 17 ] | nuScenes [ 7 ] | SUNRGBD [ 46 ] | Hypersim [ 43 ] | ARKitScenes [ 4 ] | Objectron [ 1 ] | AP |
| Full model | 27.9 | 28.2 | 33.9 | 13.2 | 59.4 | 56.8 | 30.2 |
| w/o 3D confidence head | 28.0 | 27.9 | 32.1 | 13.0 | 58.2 | 56.9 | 29.4 ( 0.8) |
| w/o 2D head (3D only) | 18.3 | 15.6 | 5.1 | 9.7 | 28.5 | 10.9 | 11.1 ( 19.1) |
Table 7: Detection head architecture. Joint 2D+3D prediction is critical; the 3D confidence head provides complementary geometry-aware scoring. Evaluated in the oracle setting on Omni3D.
| Configuration | KITTI [ 17 ] | nuScenes [ 7 ] | SUNRGBD [ 46 ] | Hypersim [ 43 ] | ARKitScenes [ 4 ] | Objectron [ 1 ] | AP |
| Full model | 27.9 | 28.2 | 33.9 | 13.2 | 59.4 | 56.8 | 30.2 |
| w/o O2M matching | 23.2 | 23.9 | 30.8 | 12.2 | 56.8 | 53.5 | 27.7 ( 2.5) |
| w/o geometry loss | 28.3 | 27.7 | 28.6 | 11.1 | 57.0 | 56.4 | 28.5 ( 1.7) |
| w/o deep supervision | 28.1 | 28.3 | 32.4 | 12.6 | 58.7 | 56.6 | 29.9 ( 0.3) |
| w/o ignore-aware suppression | 28.2 | 29.4 | 33.2 | 13.0 | 59.2 | 56.4 | 30.0 ( 0.2) |
Table 8: Training objectives. One-to-many matching and geometry loss are the most impactful training signals; deep supervision and ignore-aware suppression provide smaller but consistent gains. Evaluated in the oracle setting on Omni3D.
Figure 7: Qualitative comparison on in-the-wild images (box prompts). Each block shows the same scene detected by three models, all prompted using 2D bounding boxes. From top to bottom: 2D box prompt visualizations (only box prompts are used, the text labels are for reference), ground truth 3D boxes, WildDet3D predictions, OVMono3D predictions, and DetAny3D predictions, with 2D overlays and corresponding 3D bounding boxes. WildDet3D produces more accurate 3D localization and tighter boxes across diverse scenarios, including animals, vehicles, indoor electronics, and small food items.
Figure 8: Qualitative comparison on in-the-wild images (text prompts). Each block shows the same scene detected by WildDet3D (top) and 3D-MOOD (bottom), prompted with text categories only.
Figure 9: Real-world deployment demos. Each row shows three frames from a different deployment platform, demonstrating WildDet3D across diverse interaction modes and environments.
Figure 10: WildDet3D-agent: referring expression localization. Results of 3D box outputs by WildDet3D compared to VST [ 57 ] and Qwen3-VL [ 56 ] . WildDet3D-Agent more reliably localizes the queried object.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Data | Epochs | LR decay ( / ) | Init |
| 1 | Omni3D | 12 | / | Scratch |
| 2 | Omni3D + Others + WildDet3D-Data (H+S) | 12 | / | Stage 1 |
| 3 | Omni3D + WildDet3D-Data (H) | 3 | / | Stage 2 |
Table 9: Training stage summary.
Figure 11: Annotation interface. Two example tasks from the WildDet3D-Data annotation interface. The top panel shows the reference image with the target 2D box highlighted in yellow and all candidate 3D boxes projected in color. Each candidate column (A–D, color-coded) shows a cropped 2D projection and an orthographic point cloud view. Annotators select the best candidate and rate its quality as good_fit , acceptable , unacceptable (3D) , or unacceptable (2D) .
Figure 12: Qualitative examples from WildDet3D-Data. Each pair shows 3D bounding box annotations overlaid on the input image with category labels (left) and the corresponding 3D bounding boxes rendered in the reconstructed point cloud (right).
Figure 13: Box-prompted comparison. Each block shows the same scene detected by three models, all prompted using 2D bounding boxes. From top to bottom: 2D box prompt visualizations (only box prompts are used, the text labels are for reference), ground truth 3D boxes, WildDet3D predictions, OVMono3D predictions, and DetAny3D predictions, with 2D overlays and corresponding 3D bounding boxes.
Figure 14: Text-prompted comparison. Each block shows the same scene detected by WildDet3D (top) and 3D-MOOD (bottom), prompted with text categories only.
More in this topic
3D Object Detection
3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.
See all 181 papers