Recent advances in Vision-Language Models (VLMs) have demonstrated impressive multimodal understanding in general domains. However, their applicability to decision-oriented domains such as hospitality remains largely unexplored. In this work, we investigate how well VLMs can perform visual question answering (VQA) about hotel and facility images that are central to consumer decision-making. While many existing VQA benchmarks focus on factual correctness, they rarely capture what information users actually find useful. To address this, we first introduce Informativeness as a formal framework to quantify how much hospitality-relevant information an image-question pair provides. Guided by this framework, we construct a new hospitality-specific VQA dataset that covers various facility types, where questions are specifically designed to reflect key user information needs. Using this benchmark, we conduct experiments with several state-of-the-art VLMs, revealing that VLMs are not intrinsically decision-aware-key visual signals remain underutilized, and reliable informativeness reasoning emerges only after modest domain-specific finetuning.
Figures & tables
Figure 1: Comparison between general VQA (top) and decision-oriented Hospitality-VQA (bottom).
Figure 2: Bad vs. Good examples for each informativeness dimension. Bad images lack decision-relevant visual cues—resulting in low spatial legibility, weak activity affordance, obstructed or unbalanced contextual openness, or incomplete geometric completeness. Good images exhibit high spatial legibility, clear activity affordances, well-balanced contextual openness, and strong geometric completeness, enabling more reliable assessment of hospitality informativeness.
Figure 4: Dataset statistics of Hospitality-VQA. (a) Distribution of main facility categories. (b–e) Distributions of the four informativeness axes, reflecting characteristic properties of professionally curated hospitality listing images.
Model
Facility
Informativeness
Main
Main&Sub
SL
AA
CO
GC
Room
View
Gemini 2.5 Pro
90.66
75.00
11.51
9.43
46.81
7.35
80.65
50.00
GPT-5
92.33
82.55
46.76
18.87
31.91
20.59
83.87
64.10
GPT-4o-mini
92.33
84.91
97.12
38.21
56.03
8.82
70.97
79.49
GLM-4.1V-9B-Thinking
93.66
79.25
89.21
35.85
57.45
16.18
61.29
57.69
LLaVA-NeXT-7B
73.33
53.77
94.24
8.02
19.86
5.88
25.81
79.49
Table 2: Comparison of VLM performance across facility types and informativeness categories. Best in each column is in bold and second-best is underlined .
Task
3B ( Δ Acc)
7B ( Δ Acc)
Main
+22.00
+13.34
Main&Sub
+36.79
+21.22
SL
+26.61
+53.24
AA
+23.58
+18.40
CO
+17.73
+18.43
GC
+25.00
+26.47
Table 3: Absolute accuracy gains (%) from domain adaptation via LoRA fine-tuning for Qwen2.5-VL models (Finetuned − Base).
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks often target predefined capabilities, such as multi-hop retrieval or long-form synthesis, whereas users ask photo-grounded questions spanning a long tail of everyday scenarios. Despite advances in VLMs, users on Xiaohongshu, a mainstream Chinese image-sharing platform, continue to turn to other people for help with everyday visual questions. Motivated by this behaviour, we curate NoteVQA from these questions, yielding 252 items across 12 topical categories and 7 user intents. Each item includes a concise reference distilled from expert community responses and a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence. We evaluate both short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs. Across 9 frontier VLMs, the highest short-answer accuracy is 52.8%, while adding agentic search to Qwen3.5-397B-A17B improves accuracy by only 2.0%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap in content quality. These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.
Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.
L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5
The reliability of VLM-as-a-Judge is critical for the automatic evaluation of vision-language models (VLMs). Despite recent progress, our analysis reveals that VLM-as-a-Judge often pays limited attention to the image when making decisions. Instead, they often blindly favor the more informative answer, even when they can recognize it conflicts with the image content. We call this problem informativeness bias, which significantly undermines judge reliability. To address it, we propose BIRCH (Balanced Informativeness and CoRrectness with a Truthful AnCHor), a judging paradigm that first corrects inconsistencies with the image content in candidate answers, and then compares the answers against this corrected version. This shifts the judge's focus from informativeness to image-grounded correctness. Experiments on multiple models and benchmarks show that BIRCH reduces informativeness bias by up to 17%, resulting in performance gains of up to 9.8%. Our work reveals an overlooked but fundamental flaw in current VLM-as-a-Judge systems and highlights the need for more principled designs.