Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason about objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline that leverages panoramic imagery for large-scale and diverse multi-choice VQA synthesis, where full-scene coverage enables flexible view framing while preserving global scene awareness. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, multiple MLLMs, upon empowered by OpenView, can consistently boost their performance, uplifted from 48.3% to 62.3% on average. Code, benchmark, and data are available at https://github.com/q1xiangchen/OpenView.
Figures & tables
Figure 1: Examples of out-of-view visual question answering (VQA). OpenView synthesizes multi-choice out-of-view VQA data, featuring both contextual and directional question types. Empowered with OpenView, several MLLMs can largely improve their performance (mid right vs. bottom right) on out-of-view VQA, making their capability closer to human performance (top right).
Figure 2: Overview of the OpenView pipeline. Top: Panorama Annotation (Section 3.1 ) includes Stage 1, which collects, samples, and filters panoramic images, and Stage 2, which produces a spatial-grounded caption per local patch and a comprehensive summary per panorama. Bottom: Multi-choice VQA Creation (Section 3.2 ) includes Stage 3, which generates multi-choice VQA proposals via view framing, question formulation, and answer elaboration for contextual and directional OOV tasks, and Stage 4, which improves the synthesis quality through format refinement, confidence-based filtering, and an augmentation to shuffle options and jitter views. Right: Our generated VQAs cover diverse scene categories, both functional and environmental.
Figure 3: An analysis on OpenView-Dataset. Question patterns (a) exhibit high linguistic consistency, which guides the model to focus on OOV understanding. The option word cloud (b) reveals diverse object and scene terms, whereas the rationale word cloud (c) highlights recurring reasoning phrases commonly-used across questions. Scene category distribution (d) across indoor and outdoor environments, illustrating the broad coverage of real-world scenarios represented in the collection.
Model
Rank
Contextual
Directional
Overall
Choice
Rationale
Joint
Choice
Rationale
Joint
Choice
Rationale
Joint
Human-level
–
83.02
-
-
89.58
-
-
86.00
-
-
† GPT-4o [ 21 ]
–
64.14
-
-
41.74
-
-
52.99
-
-
† GPT-5 [ 46 ]
–
63.79
-
-
44.52
-
-
54.20
-
-
Proprietary Models
GPT-4o-mini [ 39 ]
9
63.10
93.99
59.31
48.35
87.05
42.09
55.76
90.99
50.74
Table 1: Evaluation on OpenView-Bench. Results are reported as percentages for choice, rationale, and joint accuracy across contextual and directional tasks. Joint accuracy columns are shaded as the primary comparison metric, with models ranked by the overall score. The best results among all models and within open-source models are highlighted in red and light red , respectively. † indicates description-based test setting; ‡ indicates that thinking mode is enabled.
Figure 4: Left: Breakdown of error cases across models. Right: An example test case. Baseline MLLMs misinterpret the scene, while fine-tuned models correctly infer the unseen activity.
Model
Contextual
Directional
Overall
Choice
Rationale
Joint
Choice
Rationale
Joint
Choice
Rationale
Joint
% gain
LLaVA-NeXT-Mistral-7B
42.07
39.34
16.55
29.04
29.94
8.69
35.58
35.52
12.64
↑ 332%
+ OpenView-Dataset
71.55
90.84
65.00
49.57
89.12
44.18
60.61
90.14
54.63
InternVL3.5-8B
56.90
90.00
51.21
45.39
79.69
36.17
51.17
85.45
43.72
↑ 18%
+ OpenView-Dataset
66.03
90.86
59.99
48.00
89.13
42.78
57.06
90.14
51.43
Qwen2.5-VL-7B-Instruct
61.03
81.36
49.65
45.39
76.63
34.78
53.25
79.35
42.25
↑ 39%
Table 2: OOV performance boosted by fine-tuning using OpenView-Dataset. Results are reported as percentages for choice, rationale, and joint accuracy across contextual and directional tasks. The final column reports the relative improvement in overall joint accuracy. After supervised fine-tuning, four open-source models show consistent joint improvements, with gains highlighted in blue and minor rationale drops indicated in gray .
Table 7
Appendix figures & tables46 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Images
# Videos
Sampling Strategy
Matterport3D [ 6 ]
2,295
–
All frames
Mapillary Metropolis [ 35 ]
2,994
–
Random sample 10%
360Loc [ 19 ]
–
14
Frame step: 10
360+x [ 7 ]
–
232
5 frames per video
360-1M [ 52 ]
–
2,792
5 frames per video
Appendix
Table 5: Details of source datasets for panoramic images and videos.
Figure 5: Additional statistical analysis of OpenView-Dataset. Word length distributions are shown for questions, options, and rationales.
Stage/Process
Avg. Input Tokens
Avg. Output Tokens
Time (s)
S1/Prompt-based Filtering
2,442
75
4.74
S2/Patch Analysis
1,205
121
1.21
S2/Panorama Analysis
5,278
101
4.65
S3/Proposal Generation
5,538
1,669
44.89
S4/Format Refine
459
99
2.12
Appendix
Table 6: Computational cost of one forward pass for each stage in OpenView. We round the average token counts to integers and report the average runtime on two A100 80GB GPUs in seconds.
Figure 6: Distribution of scene categories across VQAs in the OpenView-Bench.
Figure 7: VQA validation interface. Each annotator first examines the reference projection and the VQA proposal displayed in the right panel, while consulting the full panorama in the top-left for additional context. They then review the rationale in the Answer Reasoning section and revise the question accordingly. Any modification of the view metadata is immediately reflected in the live projection grid. The interface provides default fields along with editable input boxes to support efficient correction and refinement.
Task Type
Acc. (majority)
κ (majority)
κ (Human1)
κ (Human2)
κ (Human3)
Fleiss’ κ (all)
Contextual
90.12
0.648
0.344
0.648
0.613
0.611
Directional
82.61
0.425
0.497
0.425
0.425
0.618
Overall
86.67
0.417
0.439
0.532
0.517
0.616
Appendix
Table 7: Agreement between GPT5-mini judge and human.
Model
Rank
Contextual
Directional
Overall
GPT-5
2
70.86
57.04
63.98
† GPT-5
–
63.79 ( −7.07 )
44.52 ( −12.52 )
54.20 ( −9.78 )
MiniCPM-V 4.5
3
68.79
57.04
62.94
† MiniCPM-V 4.5
–
38.79 ( −30.00 )
32.17 ( −24.87 )
35.50 ( −27.44 )
GPT-4o
6
71.72
55.83
63.81
† GPT-4o
–
64.14 ( −7.58 )
41.74 ( −14.09 )
52.99 ( −10.82 )
Appendix
Table 8: Performance under the description-based setting. We report choice accuracy. † denotes replacing images with object descriptions, with percentage-point drops shown in parentheses. Ranks follow Table 1 .
Model
Contextual
Directional
Overall
Choice
Rationale
Joint
Choice
Rationale
Joint
Choice
Rationale
Joint
Original questions
GPT-5
70.86
91.73
65.00
57.04
95.43
54.43
63.98
93.37
59.74
GPT-4o
71.72
83.89
60.17
55.83
89.41
49.91
63.81
86.30
55.06
Qwen3-VL-8B-Instruct
58.10
94.96
55.17
48.35
90.65
43.83
53.25
93.01
49.53
Qwen3-VL-8B + OpenView-Dataset
77.07
93.29
71.90
57.22
89.36
51.13
67.19
91.62
61.56
Appendix
Table 9: Performance on rephrased questions. We compare model performance on the original questions and their DeepSeek-V4-Flash rephrasings. Choice, rationale, and joint accuracy are reported as percentages; joint accuracy columns are shaded .
Model
Contextual
Directional
Overall
Choice
Rationale
Joint
Choice
Rationale
Joint
Choice
Rationale
Joint
Qwen3-VL-8B (zero-shot)
58.10
94.96
55.17
48.35
90.65
43.83
53.25
93.01
49.53
+ 2-shot
65.34
98.15
64.14
50.96
92.15
46.96
58.18
95.54
55.58
+ 4-shot
66.03
98.43
65.00
51.48
94.26
48.52
58.79
96.61
56.80
+ CoT
65.17
99.74
65.00
52.00
96.66
50.26
58.61
98.38
57.66
InternVL3.5-8B (zero-shot)
56.90
90.00
51.21
45.39
79.69
36.17
51.17
85.45
43.72
Appendix
Table 10: Effects of prompting strategies. We compare zero-shot performance with 2-shot, 4-shot, and CoT prompting for three open-source MLLMs. Choice, rationale, and joint accuracy are reported as percentages; joint accuracy columns are shaded .
Model
Contextual
Directional
Overall
Choice
Rationale
Joint
Choice
Rationale
Joint
Choice
Rationale
Joint
Qwen2.5-VL-3B-Instruct
57.93
13.99
8.10
44.70
12.06
5.39
51.34
13.15
6.75
MindCube-RL (3B)
48.97
34.51
16.90
38.78
5.38
2.09
43.90
21.70
9.52
Appendix
Table 11: MindCube-RL on OpenView-Bench. We compare MindCube-RL with its Qwen2.5-VL-3B-Instruct base model. Choice, rationale, and joint accuracy are reported as percentages; joint accuracy columns are shaded .
Figure 8: Response distribution across models on directional ( left ) and contextual ( right ) tasks. Each stacked bar shows the percentage of correct predictions and error cases, with error subtypes grouped into perception limitations (hallucination, missed evidence) and reasoning limitations (option analysis error, logic error).
Model
Civic
Commercial
Culture
Education
Heritage
Hospitality
Nature
Residential
Rural
Transport
Workplace
GPT-5.4
72.27
62.81
52.78
62.60
57.32
68.87
64.65
51.19
53.70
76.08
73.45
GPT-5
57.98
61.15
54.63
66.67
50.00
66.04
62.63
54.76
49.07
66.30
64.61
MiniCPM-V 4.5
61.35
56.20
57.40
51.22
63.41
55.66
67.68
51.19
41.67
69.56
61.06
Qwen2.5-VL-32B-Instruct
53.78
54.55
41.67
61.79
57.32
61.32
61.62
47.62
56.48
67.39
61.06
Qwen3-VL-8B-Instruct
47.90
51.24
41.67
54.47
45.12
52.83
62.63
32.14
36.12
60.87
56.63
Qwen3-VL-8B-Instruct (SFT)
62.19
66.94
54.63
55.28
63.41
60.38
72.73
54.76
62.96
70.65
54.87
Appendix
Table 12: Joint accuracy across scene categories. Performance of five representative MLLMs and our fine-tuned model on OpenView-Bench. The best result in each category is highlighted in bold.
# Panorama
# Training samples
InternVL3.5-8B
Qwen2.5-VL-7B-Instruct
Choice
Rationale
Joint
Choice
Rationale
Joint
1k
9,596
52.99
85.13
45.11
61.90
85.03
52.63
3k
29,028
55.50
87.68
48.66
61.04
87.66
53.51
5k
48,296
54.98
89.45
49.18
64.87
89.30
57.93
10k
96,470
57.92
86.85
50.30
63.90
89.84
57.41
16k
158,352
57.06
90.14
51.43
64.42
90.99
58.62
Appendix
Table 13: Ablation study on data scaling. We report the number of training samples at different scales, together with the overall accuracies on the contextual and directional tasks.
Figure 9: Outputs of top-performing MLLMs and two fine-tuned models on an example contextual OOV question in a nature scene. While base models tend to rely heavily on the shadow of the person holding the 360-degree camera to infer possible activities, they fail to recognize the camera rig and show limited understanding of the surrounding environment. In contrast, our fine-tuned models capture a broader scene context and the natural surroundings, enabling more accurate and grounded reasoning about the correct answer.
Figure 10: Outputs of top-performing MLLMs and two fine-tuned models on an example directional OOV question in a residential scene. Most base models predict with high confidence that slightly tilts the camera upward would not reveal any rooftop of a nearby building. In contrast, our fine-tuned models consider both options as plausible, demonstrating a more nuanced and fine-grained visual understanding and better uncertainty awareness in OOV understanding.
Figure 11: Outputs of top-performing MLLMs and two fine-tuned models on an example contextual OOV question in a transport scene. Most of the base models identify the building but incorrectly infer office furniture, revealing limited understanding of the broader street scene contents. In contrast, our fine-tuned models are able to correctly interpret the scene as an urban transport environment and infer plausible items such as cycling gear, demonstrating stronger context reasoning in OOV settings.
Figure 12: Example of a contextual question on a rural scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 13: Example of a contextual question on a culture scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 14: Example of a contextual question on an education scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 15: Example of a contextual question on a hospitality scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 16: Example of a contextual question on a workplace scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 17: Example of a directional question on a residential scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 18: Example of a directional question on a transport scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 19: Example of a directional question on a heritage scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 20: Example of a directional question on a nature scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 21: Example of a directional question on a civic scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 22: Example of a directional question on a commercial scene. The corresponding perspective projection is outlined in red and the correct answer is highlighted in green.
Figure 23: Example of outdoor Image&text-conditioned panorama generation results. We employ Qwen3-VL-8B-Instuct and its fine-tuned variant as assistants for adjacent-view description generation. Conditioned views are bounded in red. Objects are colorized to highlight repetitive or salient elements. For fair OOV comparison, we show generated descriptions for the left and right 90 ∘ views, which are non-overlapping yet closest to the conditioned image. The fine-tuned model provides more informative captions than base model, it also offers improved guidance for rear-facing views.
Figure 24: Example of indoor Image&text-conditioned panorama generation results. Conditioned views are bounded in red. Objects are colorized to highlight repetitive or salient elements.
Figure 25: Example of indoor Image&text-conditioned panorama generation results. Conditioned views are bounded in red. Objects are colorized to highlight repetitive or salient elements.
Figure 26: Example of indoor Image&text-conditioned panorama generation results. Conditioned views are bounded in red. Objects are colorized to highlight repetitive or salient elements.
Figure 27: Example of indoor Image&text-conditioned panorama generation results. Conditioned views are bounded in red. Objects are colorized to highlight repetitive or salient elements.
Figure 28: Example of indoor Image&text-conditioned panorama generation results. Conditioned views are bounded in red. Objects are colorized to highlight repetitive or salient elements.
Table 14: The prompt template for prompt-based filtering in the Data Preprocessing stage.
Table 15: The prompt template for perspective-projected view analysis in the Visual Analyzer stage.
Table 16: The prompt template for panorama analysis in the Visual Analyzer stage.
Table 17: The base system prompt template (Part 1) in the Proposal Generator stage.
Table 18: The base system prompt template (Part 2) in the Proposal Generator stage.
Table 19: The base system prompt template (Part 3) in the Proposal Generator stage.
Table 20: The system prompt template for contextual task design in the Proposal Generator stage.
Table 21: The system prompt template for directional task design in the Proposal Generator stage.
Table 22: The user prompt template in the Proposal Generator stage.
Table 23: The prompt template for quality control in the Proposal Refiner stage.
Table 24: The user prompt template for inference in OpenView-Bench.
Table 25: The user prompt template for rationale correctness evaluation in OpenView-Bench.
Table 26: The prompt template for generating textual descriptions from a single view, serving as a semantic support for outpainting.
Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding. Yet, such multi-view perception overlooks the holistic spatial and contextual relationships that a single panorama inherently preserves. In this work, we introduce the Panorama-Language Modeling (PLM)paradigm, a unified 360∘ vision-language reasoning that is more than the sum of its pinhole counterparts. Besides, we present PanoVQA, a large-scale panoramic VQA dataset that involves adverse omni-scenes, enabling comprehensive reasoning under object occlusions and driving accidents. To establish a foundation for PLM, we develop a plug-and-play panoramic sparse attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining. Extensive experiments demonstrate that our PLM achieves superior robustness and holistic reasoning under challenging omni-scenes, yielding understanding greater than the sum of its narrow parts. Project page: https://github.com/InSAI-Lab/PanoVQA.
Weijia Fan, Ruiping Liu, Jiale Wei +7
Karlsruhe Institute of Technology · Shenzhen University · UCL +1
Vision-language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition, largely because natural image datasets provide limited supervision for low-level visual skills. Can targeted synthetic supervision address these weaknesses without reference images or manual annotation? To investigate this, we introduce VisionFoundry, an automated pipeline that takes only a task name as input, uses LLMs to synthesize paired questions, answers, and text-to-image (T2I) prompts, generates images with T2I models, and filters samples via multimodal verification. With VisionFoundry, we construct VisionFoundry-10k, a synthetic VQA dataset spanning 10 perception tasks. Finetuning on VisionFoundry-10k consistently improves perception benchmarks across three open-source backbones (e.g., +6.7% on MMVP-pair and +10.5% on CV-Bench-3D for Qwen2.5-VL-3B-Instruct) while preserving broader capabilities and showing positive data scaling. The same synthetic supervision also yields consistent gains under reinforcement learning (RL) across all three backbones, and the framework remains effective under open-source synthesis and self-verification. Our findings demonstrate that automated synthetic supervision offers an effective and scalable path toward systematic VLM training.
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences