More than the Sum: Panorama-Language Models for Adverse Omni-Scenes

Weijia Fan; Ruiping Liu; Jiale Wei; Yufan Chen; Junwei Zheng; Zichao Zeng; Jiaming Zhang; Qiufu Li

cs.CVMar 10, 2026

More than the Sum: Panorama-Language Models for Adverse Omni-Scenes

Authors: Weijia Fan, Ruiping Liu, Jiale Wei, Yufan Chen, Junwei Zheng, Zichao Zeng, Jiaming Zhang, Qiufu Li,

Organizations: Karlsruhe Institute of Technology · Shenzhen University · UCL · Hunan University

Abstract

Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding. Yet, such multi-view perception overlooks the holistic spatial and contextual relationships that a single panorama inherently preserves. In this work, we introduce the Panorama-Language Modeling (PLM)paradigm, a unified $360^\circ$ vision-language reasoning that is more than the sum of its pinhole counterparts. Besides, we present PanoVQA, a large-scale panoramic VQA dataset that involves adverse omni-scenes, enabling comprehensive reasoning under object occlusions and driving accidents. To establish a foundation for PLM, we develop a plug-and-play panoramic sparse attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining. Extensive experiments demonstrate that our PLM achieves superior robustness and holistic reasoning under challenging omni-scenes, yielding understanding greater than the sum of its narrow parts. Project page: https://github.com/InSAI-Lab/PanoVQA.

Figures & tables

Figure 1: 1-Pano (41.42 $\%$ ) outperforms 6-Cam (40.22 $\%$ ) on PanoVQA-mini. The panorama’s seamless $360^{\circ}$ context is key for spatial awareness. As shown, the 6-cam model fails the query, e.g., misidentifying the direction. In contrast, the 1-Pano model leverages the full context to, e.g., correctly locate the object, matching the GT. More examples can be found in the supplementary.

Figure 2: Panorama generation overview. Following [ 43 ] , we center a viewing sphere at $O$ . For each camera, an image plane $I_{i}$ is tangent to a concentric sphere at $M_{i}$ on its optical axis. For each panorama pixel $P_{360_{jk}}$ , we cast a ray $l_{jk}$ from $O$ , intersect it with the tangent planes, and sample the color at the projected image coordinate $N_{jk}$ . Overlaps are resolved by a fixed camera order (“first hit wins”), ensuring consistency without feature matching.

Data.	ID	QA Task Description	#Samples
Normal Scenes	N1	Scene Captioning	118,101
	N2	Object Identification	144,927
	N3	O2O Spatial Relationships	117,174
	N4	E2O Spatial Relationships	128,793
Occluded Scenes	O1	Occlusion Relationships	698
	O2	Actions of Occluded Objects	252

Table 2: Distribution of 12 QA types on PanoVQA. Tasks are grouped by their three source datasets: NuScenes (normal scenarios), BlendPASS (occlusion reasoning), and DeepAccident (collision accident), along with their respective sample counts.

Dataset	Samples		Average Word Length
Dataset	Train	Val	Question	Answer
NuScenes [ 32 ]	419,336	89,659	15.6	34.8
BlendPASS [ 5 ]	–	1,292	17.1	51.0
DeepAccident [ 41 ]	119,299	24,328	24.5	41.4
PanoVQA (Ours)	538,635	115,279	19.07	42.4

Table 3: Comparison of dataset statistics. PanoVQA introduces over 650K total samples, offering a larger scale and comprehensive question-answer lengths compared to existing benchmarks.

Dataset	Question	Answer
Dataset	Quality	Correctness	Actionability	Fluency
PanoVQA-N	4.73	4.50	4.31	4.84
PanoVQA-O	4.21	4.04	3.52	4.66
PanoVQA-D	4.80	4.72	4.64	4.91

Table 4: Data quality verified by the human-GPT evaluation scores. The higher the better, up to 5.

Panel of Figure 3 — Figure 3: Left : Structure of our proposed attention block with SWA and PSA. Right : The visualization of attention masks for Sliding Window Attention (SWA), Simplified Sparse Attention (SSA), and Panoramic Sparse Attention (PSA), respectively.

Model	PanoVQA-N				PanoVQA-O			PanoVQA-D					Avg.
Model	N1	N2	N3	N4	O1	O2	O3	D1	D2	D3	D4	D5	Avg.
Zero-shot
Chameleon-7B [ 54 ]	9.87	18.04	12.60	10.62	6.98	33.04	38.96	8.71	23.65	35.01	30.76	11.69	19.99
LLaVA-NEXT-7B [ 27 ]	11.19	17.15	15.06	12.13	10.54	40.47	46.91	8.93	20.66	36.73	23.33	18.57	21.81
LLaVA-OneVision-7B [ 20 ]	12.54	19.61	19.83	14.49	11.97	40.66	43.25	12.05	25.88	47.09	34.75	43.00	27.09
Qwen2.5-VL-7B-Ins. [ 3 ]	13.65	18.53	22.08	14.72	15.13	35.89	44.68	12.86	30.97	48.86	36.93	62.90	29.77

Table 5: Results for all models on the PanoVQA benchmark. ‘-’ means not available. The metric is the normalized GPT-score. All experiments are conducted on the PanoVQA.

Model	Training				#Param	Avg
Model	ViT	MLP	Module	LLM	#Param	Avg
3B-Lora	✓	✓	-	$\times$	259.21M	28.55
Ablation experiments
+ SSA	$\times$	✓	✓	$\times$	85.09M	29.80
+ PSA	$\times$	✓	✓	$\times$	95.56M	32.14
+ PSA	$\times$	✓	✓	✓	3196.25M	41.49

Table 6: Ablation study of components in PanoLM. SSA denotes Simplified Sparse Attention, where we use an MLP to replace our gate network. Experiments are conducted in PanoVQA-mini.

Figure 4: Analysis of the performance-parameter trade-off. Left : Impact of varying bottle dimensions. Right : Impact of varying the selected Top-K.

Figure 5: Scaling law study on PanoVQA-mini.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Attention visualization of our proposed PSA. PSA filters uninformative regions like “sky” and distant backgrounds, while concentrating on the critical navigation areas (e.g., the road and surrounding vehicles) across the 360-degree panoramic view.

Model	Avg(N)	Avg(O)	Avg(D)	Avg
LLaVA-Next-7B	15.36	34.14	26.25	24.59
LLaVA-Next-7B + PSA	16.05	33.57	28.53	25.63

Appendix

Table 7: compatibility of PSA on LLaVA-Next-7B

Figure 7: Distribution of QA samples in PanoVQA. The dataset features disparate scales to mimic real-world distributions: PanoVQA-N (left) offers a large-scale foundation (

\sim

510k), PanoVQA-O (middle) introduces a challenging long-tailed setting (

\sim

1.3k), and PanoVQA-D (right) covers safety-critical events (

\sim

143k).

Panel of Figure 8 — Figure 8: Hyperparameters for Qwen2.5-VL Baseline.

Model	Input Strategy	Metrics ( $\uparrow$ )
Model	Input Strategy	Avg(N)	Avg(O)	Avg(D)	Avg
Qwen2.5-VL-3B	Multi-view (6 cam)	16.82	32.40	34.92	28.26
Qwen2.5-VL-3B	Uni-view ( ${2\times}3$ grid)	15.69	31.27	30.39	25.71
	Panoramic (1 pano)	16.86	30.44	31.25	26.25
Qwen2.5-VL-3B-SFT	Multi-view (6 cam)	26.33	39.88	54.45	40.22
	Uni-view ( ${2\times}3$ grid)	26.04	40.52	50.95	40.04

Appendix

Table 8: Comparison of multi-view (6 cameras) and panoramic (1 camera) modeling across Normal (N), Occluded (O), and Accidental (D) scenes.

Panel of Figure 9 — Figure 9: Prompt used in PanoVQA generation. Using PanoVQA-N as an example.

Panel of Figure 10 — Figure 10: Prompt used in evaluation.

Figure 11: Qualitative comparison on PanoVQA-N . The panoramic model (1-Pano) correctly identifies the spatial location (“front”) and visibility of the pedestrian, whereas the multi-view model (6-Cam) hallucinates a “front left” direction due to fragmented spatial context.

Model	Avg(N)	Avg(O)	Avg(D)	Avg
Gemini-3-Pro	18.04	20.86	37.32	26.78
GPT-5	22.02	40.55	51.64	38.99
Claude-Sonnet-4.5	19.29	41.73	52.76	38.84

Appendix

Table 9: Proprietary results on PanoVQA-mini.

Figure 12: Qualitative comparison on PanoVQA-O . Facing a cluster of bicycles, the 1-Pano model benefits from the unified view to propose a coherent defensive maneuver, demonstrating the importance of seamless context for planning.

Figure 13: Qualitative comparison on PanoVQA-D . Both models accurately predict the severity and type of collision. This confirms that the panoramic representation retains critical visual details necessary for complex accident reasoning.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation
Sep 28, 2026Zhen Wang, Changpeng Wang, Zhe Liu +5Vision-Language Navigation VLMs for Robotics
Zhejiang University · The University of Hong Kong
OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning
Jun 29, 2026Haocong He, Chenfei Liao, Zichen Wen +13Visual Spatial Reasoning Multimodal Large Language Models
SJTU · HKUST(GZ) · Shanghai AI Lab +6
PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding
Dec 24, 2025Seongmin Jung, Seongho Choi, Gunwoo Jeon +23D Scene Representation 3D Visual Grounding
Seoul National University, South Korea · Robotics Lab, Hyundai Motor Company, South Korea · Pohang University of Science and Technology (POSTECH), South Korea

Sep 28, 2026J/K move · Enter open · S save

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation

Zhen Wang, Changpeng Wang, Zhe Liu, Zhangyang Qi+4

Zhejiang University · The University of Hong Kong

Vision-Language Navigation VLMs for Robotics

Open paper AskPDF arXiv

Recent vision-language models (VLMs) have advanced vision-and-language navigation (VLN), enabling models to predict navigation actions from visual observations and language instructions. In this work, we explore VLN with panoramic observations and introduce PanoVLN. The motivation is straightforward: more complete visual context should enable better-informed navigation decisions. For example, a panorama can reveal a passage outside a perspective camera's field of view, allowing the model to identify the intended route without additional exploration. However, we find that simply replacing perspective images with panoramas yields only limited gains. Our diagnosis suggests that fully exploiting wider visibility requires modifications to action prediction, training supervision, and visual representation. First, wider visibility supports longer-horizon action planning. We make the model predict longer action sequences, enabling larger turns and subsequent movement from a single panorama. Specifically, we introduce a confidence-guided execution (CGE) strategy that dynamically determines how many predicted actions to execute before replanning. Second, wider visibility also brings more complex route choices. We therefore construct training routes with frequent branching points and clear instructions to provide targeted supervision for route selection. Third, panoramic navigation requires understanding spatial relationships across viewing directions, beyond recognizing individual landmarks. We combine semantic and geometric features from RGB panoramas to capture both scene content and spatial layout without adding visual tokens. With a 4B backbone and RGB-only input, PanoVLN surpasses the previous SOTA by 11.9% and 8.7% in success rate on R2R-CE and RxR-CE Val-Unseen. Real-world experiments on a quadruped further demonstrate faster navigation with fewer pauses than prior VLN methods.

Abstract

Vision-Language Models

Explore similar work

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation

OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning

PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding