Visual Input and Its Framing Affect Undermine the Stability of Attribute-based Descriptions Produced by Large Vision-Language Models

Xiaomeng Wang; Martha Larson; Ping Mei Vogels; Zhengyu Zhao

cs.CVSep 16, 2026

Visual Input and Its Framing Affect Undermine the Stability of Attribute-based Descriptions Produced by Large Vision-Language Models

Authors: Xiaomeng Wang, Martha Larson, Ping Mei Vogels, Zhengyu Zhao

Organizations: Radboud Univeristy · Xi’an Jiaotong University

Abstract

Large vision-language models (LVLMs) are commonly prompted with text alone or together with an image. In this paper, we find a new aspect of LVLM vulnerability in its response robustness: adding an image can unexpectedly affect responses even when text prompts ask about a concept rather than the instance of the concept depicted in the image. For example, adding an image of an animal can change the generated attribute descriptions of its breed, and framing of the image further affects those changes. We study this new problem on seven LVLMs with image sets covering 30 animal breeds and eight food categories, using two types of framings that focus on the subject (subject-focused) or place it in a broader context (subject-in-situation). We draw the following main findings: (1) adding images increases physical-term proportions for animals (16.7% $\to$ 32.8%) and decreases sensory-term proportions for food (83.6% $\to$ 73.2%) among the most frequent terms, averaged over seven models and two framings on correctly recognized images. (2) attribute-term distributions differ significantly between the two framing types for 41.4% of animal and 71.4% of food model--concept pairs, with generally higher physical-term proportions for animals and sensory-term proportions for food under subject-focused than subject-in-situation framing.

Figures & tables

Figure 1: When concept-relevant prompts ask about the same concept, responses in the text+image setting shift towards more physical attribute terms for the animal example and fewer sensory attribute terms for the food example compared with the text-only setting. How the depicted subject is framed (subject-focused or subject-in-situation) further affects these shifts.

Figure 2: Percentage of physical (animal) and sensory (food) terms among the top-

K

most frequent attribute terms in the attribute-term distributions under text-only and text+image (subject-focused and subject-in-situation) settings. The upper and lower panels share model names, x-axis tick values, and x-axis labels. Visual input shifts attribute descriptions of the target concept, generally toward more physical terms for animal breeds and fewer sensory terms for food categories.

Figure 3: Per-concept TVD between subject-focused and subject-in-situation attribute-term distributions under concept-relevant prompts. Rows represent models and columns represent concepts. Asterisks mark statistically significant differences (image-block permutation test, $p<0.05$ ). Row-end counts show significant/total concepts. Across all models, visual framing significantly affects attribute descriptions of a subset of concepts, even when the concepts are correctly recognized.

Figure 4: Text+image setting across framing types: significant and non-significant cases for both Qwen2.5-VL-3B and GPT-4o-mini. Significant cases tend to show clearer contrasts in subject presentation and context between two framing types.

Figure 5: Image-level pairwise TVD within and across framing types under concept-relevant prompts. Each point represents the mean pairwise TVD for one concept within subject-focused images, within subject-in-situation images, or across framing types. Lines connect the three values for the same concept. The upper and lower panels share model names. Visual framing can be an important source of variation in attribute descriptions of the concept, but its contribution varies across models and concepts.

Figure 6: Text+image settings: Percentages of physical (animal) and sensory (food) terms among the top-5 directional terms in the keyness list for each framing type. Bars show the number of concepts in each percentage range. The upper and lower panels share model names and x-axis tick values. Subject-focused images generally yield more physical terms for animal breeds and sensory terms for food categories than subject-in-situation images.

Figure 7: Example Qwen2.5-VL-3B outputs for two British Shorthair images with two framings, each queried with paired concept- and image-relevant prompts. Image-relevant prompts shift descriptions toward image-specific details of depicted content.

Figure 8: Per-concept TVD between subject-focused and subject-in-situation attribute-term distributions under image-relevant prompts. Wheaten Terrier for Qwen2.5-VL-3B is the only non-significant case. Large framing-related variation is observed under image-relevant prompts, which can appropriately reflect image differences in depicted content.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Examples of two visual framing types for animal breeds. Subject-focused images emphasize the target animal itself, while subject-in-situation images present the target animal within an activity, interaction, or environment.

Figure 10: Examples of two visual framing types for food categories. Subject-focused images emphasize the target food itself, while subject-in-situation images present the target food within a context.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 738 papers

Apr 30, 2026

Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models

When the visual style of text is considered, a wide variety can be observed in font, color, and size. However, when a word is read, its meaning is independent of the style in which it has been written or rendered. In this paper, we investigate whether, and how, the style in which a word is visualized in an image impacts the description that a Large Visual Language Model (LVLM) provides for the concept to which that word refers. Specifically, we investigate how functional text styles (readability-oriented, e.g., black sans-serif) versus decorative styles (display-oriented, e.g., colored cursive/script) affect LVLMs' descriptions of a concept in terms of the attributes of that concept. Our experiments study the situation in which the LVLM is able to correctly identify the concept referred to by a visual text, i.e., by a word or words rendered as an image, and in which the visual text style should not influence the attribute-based description that the LVLM produces. Our experimental results reveal that even when the concept is correctly identified, text style influences the model's attribute-based descriptions of the concept. Our findings demonstrate non-trivial style leakage from text style into semantic inference and motivate style-aware evaluation and mitigation for LVLM-based multimedia systems.

Xiaomeng Wang, Martha Larson, Zhengyu Zhao

Radboud University · Nijmegen, Netherlands · Xi’an Jiaotong University +1

VLM Evaluation Vision-Language Models Large Vision-Language Models Document Image Understanding Text Generation

Ask PDF

May 26, 2026

Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery

Visual inputs are often assumed to improve language understanding in multimodal models. We examine this assumption by asking whether vision-language models (VLMs) can distinguish useful visual evidence from incidental image context in lexical judgments. We use human concreteness and imagery ratings because they span words with varying expected visual relevance, from abstract and low-imagery words to concrete and high-imagery words. We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings, most sharply when visual evidence is least relevant. Through probing and canonical correlation analysis, complemented by an attribution case study, we find that real-image contexts are associated with representational shifts and greater sensitivity to spurious visual cues, coinciding with weaker recoverability of the targeted lexical properties. We further show that instructing models to focus solely on textual content at inference time can reduce this degradation, with the clearest gains on these vulnerable subsets. Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.

Yifan Jiang, Ruoxi Ning, Sheng Yao +1

University of Waterloo · Vector Institute

VLM Evaluation Vision-Language Models VLM Interpretability Visual Evidence Sensitivity in VLMs

Ask PDF

Mar 19, 2026

Tinted Frames: Question Framing Blinds Vision-Language Models

Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.

Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2

University of British Columbia · Vector Institute for AI · University of California, Berkeley

Vision-Language Models VLM Robustness Visual Evidence Sensitivity in VLMs Reasoning for VLMs Vision-Language Grounding

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models

Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery

Tinted Frames: Question Framing Blinds Vision-Language Models