Spherical Geometry-Aware Panoptic Scene Graph Generation for Equirectangular Panoramas

Panoramic images capture the full visual sphere in a frame, offering context unavailable to conventional cameras. Yet this completeness has an unavoidable geometric cost: the 2-sphere cannot be faithfully mapped to the plane, and every projection introduces distortions that challenge standard vision architectures. This survey traces panoramic scene understanding from projection-based adaptation and distortion-aware engineering to sphere-native modeling, reflecting increasing commitment to spherical geometry. Foundation models form a fourth family, geometry-aware tokenization, which adapts the input interface while reusing perspective-pretrained weights. We review these approaches across five task families: dense prediction, unified multi-task understanding, open-world perception, vision-language reasoning, and dynamic video analysis. Across tasks, the same shift toward spherical geometry recurs. In practice, however, the field has converged not on the strongest sphere-native operators, which are exactly rotation-equivariant but cannot reuse perspective-pretrained backbones and thus have not scaled, but on a compatibility-preserving middle ground combining moderate geometric awareness with large pretrained models. This commitment is uneven: deepest in dense prediction and shallowest in dynamic perception, where methods are spatially sphere-aware yet temporally planar. Foundation-model adaptation has advanced panoramic depth fastest, while layout, surface-normal, and video-level understanding remain largely unexplored. No panoramic foundation model has yet been pretrained on spherical data. We identify five evaluation gaps: spherical-area-weighted metrics, seam-consistency tests, polar-robustness stratification, cross-projection generalization, and standardized open-world protocols. We conclude with a six-point roadmap toward general-purpose panoramic intelligence.

Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.

Panoramic 3D reconstruction enables the holistic recovery of complete 360 $^\circ$ scene geometry, offering significant benefits for robotics and AR/VR applications. However, directly estimating geometry from ERP images is inherently challenged by spherical distortions. To reduce spherical projection distortions, a common strategy is to decompose panoramas into distortion-free perspective views, but independently processing these views often introduces cross-view geometric inconsistencies. To address these challenges, we introduce GeoPano, a feed-forward framework for single-view panoramic 3D reconstruction that recovers a geometrically accurate and consistent point cloud from a single panorama. Our approach is driven by two key innovations: First, we propose a panoramic-aware attention aggregator, featuring a streamlined attention architecture equipped with a novel Pano-Attention mechanism, which explicitly exploits the intrinsic spherical geometry of the panorama to guide cross-view interactions, thereby suppressing geometric inconsistencies. Furthermore, we formulate a hybrid supervision strategy to jointly enforce fine-grained local details in perspective space and global structural coherence in spherical space. Experiments demonstrate that our method significantly outperforms previous SOTA approaches in reconstruction accuracy, geometric consistency, and visual fidelity.