Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots
Authors: Guoqiang Zhao, Zhe Yang, Sheng Wu, Fei Teng, Mengfei Duan, Yuanfan Zheng, Kai Luo, Kailun Yang
Organizations: The authors are with the School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China · The author is also with the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China.
Panoramic imagery provides holistic 360° visual coverage for environmental perception in quadruped robots. However, existing occupancy prediction methods are primarily designed for wheeled autonomous driving and rely heavily on RGB cues, which limits their robustness in complex, dynamically changing environments. To bridge this gap, we introduce PanoMMOcc, the first real-world panoramic multimodal occupancy dataset for quadruped robots, comprising four sensing modalities collected across diverse scenes. We further propose VoxelHound, a panoramic multimodal occupancy perception framework tailored to legged locomotion and spherical imaging. VoxelHound incorporates a Vertical Jitter Compensation (VJC) module to mitigate severe viewpoint perturbations caused by body pitch and roll during locomotion, enabling more consistent spatial reasoning, and a Multimodal Information Prompt Fusion (MIPF) module to effectively integrate panoramic visual cues with auxiliary modalities for enhanced volumetric occupancy prediction. We also establish a comprehensive benchmark on PanoMMOcc and provide detailed dataset analyses to enable systematic evaluation in challenging embodied perception scenarios. Extensive experiments demonstrate that VoxelHound achieves state-of-the-art performance on PanoMMOcc, with a +4.16 gain in mIoU. The dataset and code will be publicly released to facilitate future research on panoramic multimodal 3D perception for embodied robotic systems at https://github.com/SXDR/PanoMMOcc.
Figures & tables
Fig. 1: Comparison of vehicle-based and quadruped-based multimodal occupancy perception. Quadruped-based perception provides seamless 360° panoramic sensing, where VJC mitigates motion-induced viewpoint changes and MIPF performs efficient multimodal information fusion.
Fig. 2: Overview of the proposed quadruped-based panoramic multimodal occupancy dataset, integrating panoramic RGB, thermal, polarization, and LiDAR sensing with corresponding 3D occupancy annotations across six scene types, ranging from structured to unstructured environments.
Data
Domain
Annotation
Dataset
Year
Scene
Camera
Modality
Plat.
Move.
Lighting
Classes
Voxel Grid
Frames
Indoor scene completion
MonoScene-NYUv2 [ 62 ]
2022
Indoor
Pinhole
C+D
11
240×240×144
1.4K
Occ-ScanNet [ 83 ]
2024
Indoor
Pinhole
C+D
11
60×60×36
65.5K
EmbodiedOcc-ScanNet [ 17 ]
2025
Indoor
Pinhole
C+D
11
-
20.2K
TABLE I: Typical datasets for Semantic Occupancy Prediction. Abbreviations: (Car), (Human), (Quadruped Robot), (Internet images/videos), (Wheels), (Gait), (Stationary), (Day), (Night), (Inapplicable), Plat. (Platform), Move. (Movement), n.a. (Not applicable), C (Camera), D (Depth), E (Event), L (LiDAR), R (Radar), T (Thermal), and P (Polarization).
Fig. 3: Semantic class distribution of occupied voxels in the PanoMMOcc.
Fig. 4: Overview of the proposed VoxelHound. The camera branch receives panoramic RGB, thermal, and polarization images, while the LiDAR branch processes raw point clouds. Features from independent encoders are projected into BEV space for multimodal fusion.
Fig. 5: Overview of the proposed modules. a) Vertical Jitter Compensation (VJC) module mitigates image distortions caused by quadruped locomotion. b) Multimodal Information Prompt Fusion (MIPF) module for effective multimodal feature fusion.
#
Modality
Day
Night
mIoU ↑
VoxelHound
C
6.47
3.46
C+T+P
6.85
4.07
C+L
22.56
19.17
C+L+T+P
23.34
18.68
TABLE III: Results on different lighting conditions on PanoMMOcc.
#
VJC
MIPF
IoU ↑
mIoU ↑
1
46.03
22.74
2
✓
45.96
22.97
3
✓
46.06
22.88
Ours
✓
✓
46.08
23.34
TABLE IV: Ablation on different components in VoxelHound.
Fig. 6: Qualitative comparison on PanoMMOcc. From left to right: thermal image, panoramic image, polarization image, GT occupancy, VoxelHound (ours), EFFOcc [ 93 ] , and MonoScene [ 62 ] . VoxelHound produces more complete scene structures and preserves finer semantic details in complex regions.
Fig. 7: FOV-aware evaluation of multimodal occupancy perception. (a) Comparison between C and the corresponding multimodal inputs within the thermal, polarization, and union FOVs; (b-c) directional mIoU over all 3D voxel space, partitioned by platform-centered azimuth into six 60∘ and eight 45∘ sectors, respectively, with forward at the top.
Cpd
Cnh
Params. (M) ↓
Mem. (MB) ↓
mIoU ↑
8
4
43.50
386.97
23.13
8
23.34
16
4
43.51
387.02
22.91
8
22.94
32
4
43.54
387.11
22.67
8
22.48
TABLE V: Ablation study of prompt channel dimensions and attention heads in the MIPF module.
Chd
Params. (M) ↓
Mem. (MB) ↓
mIoU ↑
1
43.46
386.83
23.12
4
43.46
386.84
23.01
8
43.47
386.84
22.90
16
43.47
386.86
23.11
32
43.48
386.89
22.88
64
43.50
386.97
23.34
TABLE VI: Ablation study of different hidden channel dimensions for the VJC module.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. A.1: Qualitative visualization of LiDAR–camera extrinsic calibration. LiDAR point clouds are projected onto different camera views using the calibrated extrinsic parameters: (a) equirectangularly unwrapped PAL image, (b) thermal image, (c) polarization image, and (d) original PAL image. The consistent alignment between projected points and image structures demonstrates the accuracy of the calibration.
Fig. A.2: Qualitative comparison on PanoMMOcc under different environmental conditions. From left to right: thermal image, panoramic image, polarization image, GT occupancy, VoxelHound (ours), EFFOcc [ 93 ] , and MonoScene [ 62 ] . From top to bottom: overexposed daytime, nighttime, and rural scenes. VoxelHound yields more coherent occupancy structures and more consistent semantic predictions under challenging conditions.
Resolution
Size
Params. (M) ↓
GFLOPs ↓
Mem. (MB) ↓
FPS ↑
mIoU ↑
64×64×16
0.4m
43.50
84.47
386.97
11.03
23.34
128×128×32
0.2m
43.61
130.62
393.99
10.57
18.28
Appendix
TABLE B.1: Effect of voxel resolution under a fixed perception range.
Fig. B.1: Example of gait-induced jitter in PanoMMOcc. (a) Panoramic image with motion blur caused by gait-induced jitter during data acquisition. (b) Vertical acceleration measured by the IMU integrated in the LiDAR during locomotion. (c) Downsampled vertical acceleration revealing the jitter pattern.
Fig. C.1: Failure case in an extremely dark nighttime scene. Due to the sparsity of long-range LiDAR observations, the predictions near the boundaries become sparse.
School of Artificial Intelligence and Robotics, Hunan University, China · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China