OmniAct3D: Leveraging Foundation Geometry and Evidence-Grounded Reasoning for Panoramic 3D Detection

Runtong Wu; Fei Teng; Di Wen; Guoqiang Zhao; Kunyu Peng; Kailun Yang

cs.CVOct 2, 2026

OmniAct3D: Leveraging Foundation Geometry and Evidence-Grounded Reasoning for Panoramic 3D Detection

Authors: Runtong Wu, Fei Teng, Di Wen, Guoqiang Zhao, Kunyu Peng, Kailun Yang

Organizations: School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany

Abstract

Accurate 3D detection is essential for mobile embodied agents, while Vision Foundation Models (VFMs) offer transferable visual and geometric priors. Yet existing VFM-based 3D detectors rely on narrow-view monocular images or discrete perspective views, limiting coherent surround perception; equirectangular projection (ERP) instead encodes a continuous 360 scene in a single image. Direct transfer remains difficult because ERP organizes geometry and visual information differently, making object-relevant cues hard to model, localize, and preserve. We propose OmniAct3D, a framework that adapts perspective-trained VFM detectors to ERP while preserving transferable VFM priors. To resolve geometric mismatch, the ERP-Ray Geometry Adapter (ERGA-Ray) models spherical viewing rays and periodic spatial structure. To localize evidence in scene-wide context, the Visual-Action Reasoning Chain (VARC) grounds each hypothesis in relevant panoramic evidence and converts it into a structured geometric action. To recover local cues lost under fixed token budgets, the Appearance-Guided Heading Expert (AGHE) re-encodes object regions at higher resolution for heading estimation. Experiments show that OmniAct3D improves over the previous best 3D detector by 2.96 NDS points on Spheriverse and over the unadapted VFM baseline by 24.87 mAP points on PanoMMOcc. With target-specific geometry adaptation, VARC retains 95--98% of the same-configuration mAP, indicating reusable object-level 3D reasoning across sensing configurations. The source code will be made publicly available at https://github.com/FeiT-FeiTeng/OmniAct3D.

Figures & tables

Fig. 1: Comparison of monocular, multi-view, and ERP 3D detection paradigms. Monocular methods observe a limited field of view [ 3 ] , while multi-view methods rely on discrete perspective observations [ 4 ] . In contrast, our method enables coherent $360^{\circ}$ perception from a single ERP image.

Fig. 2: Overview of the feature flow in OmniAct3D . The frozen VFM-based detector extracts visual features $F_{\mathrm{vis}}$ and pretrained geometry memory $M_{0}$ from the ERP image. ERGA-Ray transforms $M_{0}$ into the ERP-adapted memory $M$ , after which the detector produces $B$ , $Q$ , and $C$ . VARC retrieves evidence $V$ and predicts an action $A$ to obtain $B^{\prime}$ , while the AGHE updates its directed heading to produce the final prediction $\hat{B}$ .

Fig. 3: Qualitative 3D detection comparisons on Spheriverse [ 12 ] (top) and PanoMMOcc [ 13 ] (bottom). Columns show the Ground Truth (GT), OmniAct3D without VARC (W.V), full OmniAct3D (Ours), and representative baselines.

Fig. 4: Cumulative mAP and NDS are reported for Spheriverse [ 12 ] (a–d) and PanoMMOcc [ 13 ] (e–h) across planar-range thresholds (a,b,e,f) and overlap levels (c,d,g,h). Blue curves denote the ERGA-Ray+AGHE variant without VARC, whereas red curves denote full OmniAct3D.

Method	Backbone	Spheriverse mAP / NDS	PanoMMOcc mAP / NDS
No Adapt.	LingBot [ 46 ]	0.0301 / 0.0335	0.3410 / 0.2330
MLP	LingBot [ 46 ]	0.0980 / 0.1066	0.3730 / 0.2575
AdaptFormer [ 55 ]	LingBot [ 46 ]	0.0959 / 0.1060	0.3585 / 0.2458
Circular Conv [ 56 ]	LingBot [ 46 ]	0.0975 / 0.1071	0.3692 / 0.2522
Ray-FiLM [ 57 ]	LingBot [ 46 ]	0.0929 / 0.1051	0.3689 / 0.2534
PanoSwin-based [ 28 ]	LingBot [ 46 ]	0.0948 / 0.1056	0.3690 / 0.2552

TABLE III: Geometry adaptation and backbone comparisons.

	Spheriverse		PanoMMOcc
Variant	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	Diag. mAP $\uparrow$
Fixed Evidence	0.1199	0.1274	0.5892	0.4034	0.1257
Unstruct. Action	0.1180	0.1266	0.5890	0.4037	0.1400
BBox State Only	0.1194	0.1271	0.5889	0.4015	0.1311
Full VARC	0.1887	0.1825	0.5897	0.4032	0.1768

TABLE V: VARC component ablation.

Dataset	Center Gain $\uparrow$	$\Delta$ S@2m $\uparrow$	$\Delta$ S@4m $\uparrow$	Positive-change Prec. $\uparrow$
Spheriverse	0.8019 m	+3.79%	+12.30%	86.79%
PanoMMOcc	0.5430 m	+13.97%	+7.94%	87.88%

TABLE VI: VARC gains on evidence-challenging targets.

	Spheriverse Eval		PanoMMOcc Eval
VARC Train	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$
Spheriverse	0.1887	0.1825	0.5803	0.3833
PanoMMOcc	0.1800	0.1727	0.5897	0.4032

TABLE VII: Cross-configuration transfer of VARC.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 180 papers

Unified Panoramic Geometry Estimation via Multi-View Foundation Models
May 25, 2026Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer +43D Foundation Models 3D ViTs
ETH Zürich · Google · Athlence Sports +1
MonoMIP: Multi-Height Inverse Perspective Mapping for Monocular 3D Object Detection
Oct 13, 2026Emir Ulurak, Furkan KıraçMonocular 3D Object Detection Autonomous Driving
Ozyegin University, Istanbul, Turkey
OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection
Aug 17, 2026Mariia Gladkova, Neehar Peri, Ishan Khatri +2Open-Vocabulary Object Detection Monocular 3D Object Detection
Technical University of Munich · Carnegie Mellon University · StackAV

May 25, 2026J/K move · Enter open · S save

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger+3

ETH Zürich · Google · Athlence Sports+1

3D Foundation Models 3D ViTs Panoramic 3D Reconstruction 3D Reconstruction Depth Estimation Single-View 3D Reconstruction+1

Open paper AskPDF arXiv

Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.

Abstract

3D Object Detection

Explore similar work

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

MonoMIP: Multi-Height Inverse Perspective Mapping for Monocular 3D Object Detection

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection