OmniAct3D: Leveraging Foundation Geometry and Evidence-Grounded Reasoning for Panoramic 3D Detection
Authors: Runtong Wu, Fei Teng, Di Wen, Guoqiang Zhao, Kunyu Peng, Kailun Yang
Organizations: School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany
Accurate 3D detection is essential for mobile embodied agents, while Vision Foundation Models (VFMs) offer transferable visual and geometric priors. Yet existing VFM-based 3D detectors rely on narrow-view monocular images or discrete perspective views, limiting coherent surround perception; equirectangular projection (ERP) instead encodes a continuous 360 scene in a single image. Direct transfer remains difficult because ERP organizes geometry and visual information differently, making object-relevant cues hard to model, localize, and preserve. We propose OmniAct3D, a framework that adapts perspective-trained VFM detectors to ERP while preserving transferable VFM priors. To resolve geometric mismatch, the ERP-Ray Geometry Adapter (ERGA-Ray) models spherical viewing rays and periodic spatial structure. To localize evidence in scene-wide context, the Visual-Action Reasoning Chain (VARC) grounds each hypothesis in relevant panoramic evidence and converts it into a structured geometric action. To recover local cues lost under fixed token budgets, the Appearance-Guided Heading Expert (AGHE) re-encodes object regions at higher resolution for heading estimation. Experiments show that OmniAct3D improves over the previous best 3D detector by 2.96 NDS points on Spheriverse and over the unadapted VFM baseline by 24.87 mAP points on PanoMMOcc. With target-specific geometry adaptation, VARC retains 95--98% of the same-configuration mAP, indicating reusable object-level 3D reasoning across sensing configurations. The source code will be made publicly available at https://github.com/FeiT-FeiTeng/OmniAct3D.
Figures & tables
Fig. 1: Comparison of monocular, multi-view, and ERP 3D detection paradigms. Monocular methods observe a limited field of view [ 3 ] , while multi-view methods rely on discrete perspective observations [ 4 ] . In contrast, our method enables coherent 360∘ perception from a single ERP image.
Fig. 2: Overview of the feature flow in OmniAct3D . The frozen VFM-based detector extracts visual features Fvis and pretrained geometry memory M0 from the ERP image. ERGA-Ray transforms M0 into the ERP-adapted memory M , after which the detector produces B , Q , and C . VARC retrieves evidence V and predicts an action A to obtain B′ , while the AGHE updates its directed heading to produce the final prediction B^ .
Fig. 3: Qualitative 3D detection comparisons on Spheriverse [ 12 ] (top) and PanoMMOcc [ 13 ] (bottom). Columns show the Ground Truth (GT), OmniAct3D without VARC (W.V), full OmniAct3D (Ours), and representative baselines.
Fig. 4: Cumulative mAP and NDS are reported for Spheriverse [ 12 ] (a–d) and PanoMMOcc [ 13 ] (e–h) across planar-range thresholds (a,b,e,f) and overlap levels (c,d,g,h). Blue curves denote the ERGA-Ray+AGHE variant without VARC, whereas red curves denote full OmniAct3D.
Method
Backbone
Spheriverse mAP / NDS
PanoMMOcc mAP / NDS
No Adapt.
LingBot [ 46 ]
0.0301 / 0.0335
0.3410 / 0.2330
MLP
LingBot [ 46 ]
0.0980 / 0.1066
0.3730 / 0.2575
AdaptFormer [ 55 ]
LingBot [ 46 ]
0.0959 / 0.1060
0.3585 / 0.2458
Circular Conv [ 56 ]
LingBot [ 46 ]
0.0975 / 0.1071
0.3692 / 0.2522
Ray-FiLM [ 57 ]
LingBot [ 46 ]
0.0929 / 0.1051
0.3689 / 0.2534
PanoSwin-based [ 28 ]
LingBot [ 46 ]
0.0948 / 0.1056
0.3690 / 0.2552
TABLE III: Geometry adaptation and backbone comparisons.
Spheriverse
PanoMMOcc
Variant
mAP ↑
NDS ↑
mAP ↑
NDS ↑
Diag. mAP ↑
Fixed Evidence
0.1199
0.1274
0.5892
0.4034
0.1257
Unstruct. Action
0.1180
0.1266
0.5890
0.4037
0.1400
BBox State Only
0.1194
0.1271
0.5889
0.4015
0.1311
Full VARC
0.1887
0.1825
0.5897
0.4032
0.1768
TABLE V: VARC component ablation.
Dataset
Center Gain ↑
Δ S@2m ↑
Δ S@4m ↑
Positive-change Prec. ↑
Spheriverse
0.8019 m
+3.79%
+12.30%
86.79%
PanoMMOcc
0.5430 m
+13.97%
+7.94%
87.88%
TABLE VI: VARC gains on evidence-challenging targets.
Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.
Monocular 3D object detection methods increasingly rely on depth-guided transformers, but they operate entirely in the perspective image plane where spatial relationships are distorted by foreshortening. We propose MonoMIP, a dual-decoder framework that adds complementary 3D geometric reasoning through Bird's-Eye View (BEV) features constructed via Multi-Height Inverse Perspective Mapping. We perform IPM at K elevation planes spanning ground to rooftop, producing an ordered stack of height-conditioned metric BEV features that reduces above-ground misprojection and exposes depth-dependent appearance variation across planes. A BEV decoder with multi-scale deformable attention aggregates this multi-height evidence per query, while a depth-guided 3D decoder provides complementary perspective-space predictions. Without extra training data, MonoMIP attains 25.84% Moderate AP3D|R40 on KITTI val and 21.66% on the official test set for Car, outperforming recent published monocular methods on both the KITTI and Waymo benchmarks.
Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric. To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer. We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D's performance collapses from 18.6 to 5.4 AP when prompted with "a detailed high-resolution photo of a car" rather than "car"); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D's AP by 1.9 × on ScanNet). Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector's predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.
Mariia Gladkova, Neehar Peri, Ishan Khatri +2
Technical University of Munich · Carnegie Mellon University · StackAV