RobotPan: A 360$^\circ$ Surround-View Robotic Vision System for Embodied Perception

Jiahao Ma; Qiang Zhang; Peiran Liu; Zeran Su; Pihai Sun; Gang Han; Wen Zhao; Wei Cui

cs.ROApr 15, 2026

RobotPan: A 360 $^\circ$ Surround-View Robotic Vision System for Embodied Perception

Authors: Jiahao Ma, Qiang Zhang, Peiran Liu, Zeran Su, Pihai Sun, Gang Han, Wen Zhao, Wei Cui, +6 more

Abstract

Surround-view perception is increasingly important for robotic navigation and loco-manipulation, especially in human-in-the-loop settings such as teleoperation, data collection, and emergency takeover. However, current robotic visual interfaces are often limited to narrow forward-facing views, or, when multiple on-board cameras are available, require cumbersome manual switching that interrupts the operator's workflow. Both configurations suffer from motion-induced jitter that causes simulator sickness in head-mounted displays. We introduce a surround-view robotic vision system that combines six cameras with LiDAR to provide full 360 $^\circ$ visual coverage, while meeting the geometric and real-time constraints of embodied deployment. We further present RobotPan, a feed-forward framework that predicts metric-scaled and compact 3D Gaussians from calibrated sparse-view inputs for real-time rendering, reconstruction, and streaming. RobotPan lifts multi-view features into a unified spherical coordinate representation and decodes Gaussians using hierarchical spherical voxel priors, allocating fine resolution near the robot and coarser resolution at larger radii to reduce computational redundancy without sacrificing fidelity. To support long sequences, our online fusion updates dynamic content while preventing unbounded growth in static regions by selectively updating appearance. Finally, we release a multi-sensor dataset tailored to 360 $^\circ$ novel view synthesis and metric 3D reconstruction for robotics, covering navigation, manipulation, and locomotion on real platforms. Experiments show that RobotPan achieves competitive quality against prior feed-forward reconstruction and view-synthesis methods while producing substantially fewer Gaussians, enabling practical real-time embodied deployment.

Figures & tables

Fig. 1: System of RobotPan: surround-view robotic vision system for real-time embodied perception which is deployed on the Tiangong 3.0 humanoid platform. Our system combines six cameras and LiDAR to provide full $360^{\circ}$ visual coverage for embodied robot operation. From calibrated sparse multi-view observations, RobotPan predicts metric-scaled and compact 3D Gaussians, enabling real-time surround-view rendering, novel view synthesis, depth estimation, and sparse-view dense reconstruction. By jointly supporting geometric consistency, compact representation, and real-time performance, the system serves as a practical robot visual interface for teleoperation, navigation, and loco-manipulation.

Fig. 2: Pipeline of RobotPan. Multi-view images are encoded by a transformer to predict per-view depth and features. The reconstructed 3D points are converted to robot-centric spherical coordinates, voxelized into hierarchical spherical cells, and aggregated into anchor features, which are finally decoded into compact 3D Gaussian parameters for real-time rendering and reconstruction.

Fig. 3: Multi-view consistent dynamic region identification via range-image fusion. We segment dynamic regions per view, reconstruct a shared 3D point cloud, project points to spherical coordinates to form a panoramic range image, and fuse multi-view results in the range-image domain to mitigate missed detections, yielding a robust dynamic-region mask for dynamic/static splitting.

Fig. 4: Tiangong 3.0 robot head and its sensor layout. Left: front and side views of the robot head. Right: orthographic views of the head-mounted sensing system, showing the arrangement of six RGB cameras and one LiDAR with annotated dimensions and viewing directions.

Fig. 5: Sensor layout and data acquisition. Left and middle : side and isometric views of our surround-view rig with six RGB cameras and a 40-beam LiDAR. Right : wearable data collection setup with sensor height matched to the humanoid robot.

Method	Acc. $\downarrow$		Comp. $\downarrow$		Overall $\downarrow$
Method	w/o p.	w/ p.	w/o p.	w/ p.	w/o p.	w/ p.
Dust3R [ 1 ]	0.783	0.634	0.525	0.418	0.654	0.521
Fast3R [ 9 ]	0.716	0.562	0.482	0.350	0.602	0.460
FLARE [ 12 ]	0.660	0.518	0.421	0.298	0.549	0.404
VGGT [ 8 ]	0.611	0.479	0.388	0.245	0.496	0.362
Pi3 [ 13 ]	0.580	0.440	0.353	0.221	0.460	0.332

TABLE I: Point map estimation results on the proposed dataset under sparse-view input, evaluated with and without camera poses.

Method	DTU			ETH3D
Method	Acc. $\downarrow$	Comp. $\downarrow$	Overall $\downarrow$	Acc. $\downarrow$	Comp. $\downarrow$	Overall $\downarrow$
Dust3R [ 1 ]	3.8562	3.1219	3.4891	0.4856	0.7114	0.5984
MASt3R [ 2 ]	4.2380	3.2695	3.7537	0.3417	0.3626	0.3522
Spann3R [ 4 ]	4.3097	4.5573	4.4335	1.1589	0.8005	0.9797
VGGT [ 8 ]	3.5042	2.7254	3.1152	0.3127	0.2946	0.3456
Pi3 [ 13 ]	3.0213	2.1322	3.0785	0.1341	0.1169	0.1672

TABLE II: Sparse-view reconstruction on DTU and ETH3D.

Fig. 6: Qualitative comparison of feed-forward 3D reconstruction. For each example, we show the input images, the LiDAR point cloud as ground truth, and the reconstructed geometry from different methods. Our method recovers more complete structures with sharper details.

Fig. 7: Qualitative comparison of monocular depth prediction. For each example, we show the input image, predicted depth maps from different methods, and zoomed-in regions. Our method produces clearer boundaries and more consistent geometries across diverse scenes.

Method	Paradigm	#Gaussians $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
pixelSplat [ 26 ]	Pixel-wise	3,783K	17.79	0.447	0.443
MVSplat [ 27 ]	Pixel-wise	1,261K	18.57	0.614	0.424
FLARE [ 12 ]	Pixel-wise	1,261K	18.29	0.625	0.393
DepthSplat [ 34 ]	Pixel-wise	1,261K	22.97	0.787	0.200
Ours	Spherical voxel-wise	327K	24.70	0.811	0.197

TABLE III: Comparison with state-of-the-art generalized novel view synthesis methods on the proposed dataset. Our method achieves the best overall performance with the fewest number of Gaussians.

Method	DL3DV-Benchmarks			RealEstate10K
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
pixelSplat [ 26 ]	16.55	0.456	0.480	25.89	0.858	0.142
MVSplat [ 27 ]	18.13	0.559	0.393	26.39	0.869	0.128
FLARE [ 12 ]	18.89	0.591	0.352	27.39	0.873	0.107
DepthSplat [ 34 ]	19.24	0.620	0.322	27.47	0.889	0.114
Ours	20.13	0.616	0.318	27.43	0.896	0.109

TABLE IV: Quantitative comparison with state-of-the-art novel view synthesis methods on the DL3DV-Benchmarks and RealEstate10K datasets.

Fig. 8: Qualitative comparison of novel view synthesis results against state-of-the-art methods. The first column shows the selected input reference views, and the remaining columns present the rendered novel views from different methods alongside the ground truth. Across challenging scenes, our method produces sharper structures, more faithful geometry, and fewer visual artifacts than competing approaches.

Fig. 9: Qualitative comparison on DL3DV-Benchmarks and RealEstate10K. We show sparse input views and compare novel-view renderings from different methods against the ground truth. Our method preserves finer structures and cleaner boundaries.

Category	Method	PSNR $\uparrow$	Train $\downarrow$	Render $\uparrow$	Storage $\downarrow$
		(dB)	(s)	(FPS)	(MB)
Offline Training	Kplanes [ 60 ]	23.17	52	0.15	1.5
	4DGS [ 61 ]	24.55	9.8	30	0.3
	Spacetime-GS [ 62 ]	25.41	51	145	0.8
Online Training	StreamRF [ 63 ]	24.09	15.50	8.3	31.4
	3DGStream [ 54 ]	26.11	12.25	215	7.8

TABLE V: Quantitative comparison of offline and online training methods on the proposed multi-view camera dynamic scene dataset.

Paradigm	#Gaussians	Storage (MB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Pixel-wise	1261K	378	21.43	0.713	0.341
Voxel-wise	439K	130	23.12	0.791	0.236
Spherical voxel-wise	327K	97	24.70	0.811	0.197

TABLE VI: Ablation of Gaussian prediction paradigms on the proposed dataset. The default spherical voxel-wise prediction achieves the best trade-off between compactness and rendering quality.

Configuration	#Gaussians	Storage (MB) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Naive Concatenation (w/o MLP)	65,400K	19,400	24.66	0.806	0.201
w/o Range-Image Fusion	3,094K	1,371	24.71	0.812	0.213
w/o Tiny-MLP Refinement	3,023K	872	24.32	0.739	0.241
Full Model	3,023K	1,296	25.12	0.832	0.184

TABLE VII: Ablation study of key design choices on the proposed dataset for streaming novel view synthesis over 200 evaluation frames.

Explore similar work

Apr 13, 2026

Ψ-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer

Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.

Xuan Yu, Yuxuan Xie, Changjian Jiang +4

Department of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, P. R. China

3D Reconstruction Robotic Perception 3D Gaussian Splatting Panoptic Segmentation

Ask PDF

Apr 8, 2026

Genie Sim PanoRecon: Fast Immersive Scene Generation from Single-View Panorama

We present Genie Sim PanoRecon, a feed-forward Gaussian-splatting pipeline that delivers high-fidelity, low-cost 3D scenes for robotic manipulation simulation. The panorama input is decomposed into six non-overlapping cube-map faces, processed in parallel, and seamlessly reassembled. To guarantee geometric consistency across views, we devise a depth-aware fusion strategy coupled with a training-free depth-injection module that steers the monocular feed-forward network to generate coherent 3D Gaussians. The whole system reconstructs photo-realistic scenes in seconds and has been integrated into Genie Sim - a LLM-driven simulation platform for embodied synthetic data generation and evaluation - to provide scalable backgrounds for manipulation tasks. For code details, please refer to: https://github.com/AgibotTech/genie_sim/tree/main/source/geniesim_world.

Zhijun Li, Yongxin Su, Di Yang +4

3D Scene Generation Panoramic 3D Reconstruction Feed-Forward 3D Reconstruction 3D Gaussian Splatting Synthetic Data Generation for Robotics

Ask PDF

May 13, 2026

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World

Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

Changpeng Wang, Xin Lin, Junhan Liu +5

1Zhejiang University · University of California, San Diego · University of California, Irvine +1

3D Spatial Reasoning 3D Scene Understanding 3D VLMs

Ask PDF

Abstract

Explore similar work

Ψ-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer

Genie Sim PanoRecon: Fast Immersive Scene Generation from Single-View Panorama

PanoWorld: Towards Spatial Supersensing in 360∘^\circ∘ Panorama World

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World