Visual Input and Its Framing Affect Undermine the Stability of Attribute-based Descriptions Produced by Large Vision-Language Models

Xiaomeng Wang; Martha Larson; Ping Mei Vogels; Zhengyu Zhao

cs.CVSep 16, 2026

Visual Input and Its Framing Affect Undermine the Stability of Attribute-based Descriptions Produced by Large Vision-Language Models

Authors: Xiaomeng Wang, Martha Larson, Ping Mei Vogels, Zhengyu Zhao

Organizations: Radboud Univeristy · Xi’an Jiaotong University

Abstract

Large vision-language models (LVLMs) are commonly prompted with text alone or together with an image. In this paper, we find a new aspect of LVLM vulnerability in its response robustness: adding an image can unexpectedly affect responses even when text prompts ask about a concept rather than the instance of the concept depicted in the image. For example, adding an image of an animal can change the generated attribute descriptions of its breed, and framing of the image further affects those changes. We study this new problem on seven LVLMs with image sets covering 30 animal breeds and eight food categories, using two types of framings that focus on the subject (subject-focused) or place it in a broader context (subject-in-situation). We draw the following main findings: (1) adding images increases physical-term proportions for animals (16.7% $\to$ 32.8%) and decreases sensory-term proportions for food (83.6% $\to$ 73.2%) among the most frequent terms, averaged over seven models and two framings on correctly recognized images. (2) attribute-term distributions differ significantly between the two framing types for 41.4% of animal and 71.4% of food model--concept pairs, with generally higher physical-term proportions for animals and sensory-term proportions for food under subject-focused than subject-in-situation framing.

Figures & tables

Figure 1: When concept-relevant prompts ask about the same concept, responses in the text+image setting shift towards more physical attribute terms for the animal example and fewer sensory attribute terms for the food example compared with the text-only setting. How the depicted subject is framed (subject-focused or subject-in-situation) further affects these shifts.

Figure 2: Percentage of physical (animal) and sensory (food) terms among the top-

K

most frequent attribute terms in the attribute-term distributions under text-only and text+image (subject-focused and subject-in-situation) settings. The upper and lower panels share model names, x-axis tick values, and x-axis labels. Visual input shifts attribute descriptions of the target concept, generally toward more physical terms for animal breeds and fewer sensory terms for food categories.

Figure 3: Per-concept TVD between subject-focused and subject-in-situation attribute-term distributions under concept-relevant prompts. Rows represent models and columns represent concepts. Asterisks mark statistically significant differences (image-block permutation test, $p<0.05$ ). Row-end counts show significant/total concepts. Across all models, visual framing significantly affects attribute descriptions of a subset of concepts, even when the concepts are correctly recognized.

Figure 4: Text+image setting across framing types: significant and non-significant cases for both Qwen2.5-VL-3B and GPT-4o-mini. Significant cases tend to show clearer contrasts in subject presentation and context between two framing types.

Figure 5: Image-level pairwise TVD within and across framing types under concept-relevant prompts. Each point represents the mean pairwise TVD for one concept within subject-focused images, within subject-in-situation images, or across framing types. Lines connect the three values for the same concept. The upper and lower panels share model names. Visual framing can be an important source of variation in attribute descriptions of the concept, but its contribution varies across models and concepts.

Figure 6: Text+image settings: Percentages of physical (animal) and sensory (food) terms among the top-5 directional terms in the keyness list for each framing type. Bars show the number of concepts in each percentage range. The upper and lower panels share model names and x-axis tick values. Subject-focused images generally yield more physical terms for animal breeds and sensory terms for food categories than subject-in-situation images.

Figure 7: Example Qwen2.5-VL-3B outputs for two British Shorthair images with two framings, each queried with paired concept- and image-relevant prompts. Image-relevant prompts shift descriptions toward image-specific details of depicted content.

Figure 8: Per-concept TVD between subject-focused and subject-in-situation attribute-term distributions under image-relevant prompts. Wheaten Terrier for Qwen2.5-VL-3B is the only non-significant case. Large framing-related variation is observed under image-relevant prompts, which can appropriately reflect image differences in depicted content.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Examples of two visual framing types for animal breeds. Subject-focused images emphasize the target animal itself, while subject-in-situation images present the target animal within an activity, interaction, or environment.

Figure 10: Examples of two visual framing types for food categories. Subject-focused images emphasize the target food itself, while subject-in-situation images present the target food within a context.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 738 papers

Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models
Apr 30, 2026Xiaomeng Wang, Martha Larson, Zhengyu ZhaoVLM Evaluation Vision-Language Models
Radboud University · Nijmegen, Netherlands · Xi’an Jiaotong University +1
Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
May 26, 2026Yifan Jiang, Ruoxi Ning, Sheng Yao +1VLM Evaluation Vision-Language Models
University of Waterloo · Vector Institute
Tinted Frames: Question Framing Blinds Vision-Language Models
Mar 19, 2026Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2Vision-Language Models VLM Robustness
University of British Columbia · Vector Institute for AI · University of California, Berkeley

Apr 30, 2026J/K move · Enter open · S save

Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models

Xiaomeng Wang, Martha Larson, Zhengyu Zhao

Radboud University · Nijmegen, Netherlands · Xi’an Jiaotong University+1

VLM Evaluation Vision-Language Models Large Vision-Language Models Document Image Understanding Text Generation

Open paper AskPDF arXiv

When the visual style of text is considered, a wide variety can be observed in font, color, and size. However, when a word is read, its meaning is independent of the style in which it has been written or rendered. In this paper, we investigate whether, and how, the style in which a word is visualized in an image impacts the description that a Large Visual Language Model (LVLM) provides for the concept to which that word refers. Specifically, we investigate how functional text styles (readability-oriented, e.g., black sans-serif) versus decorative styles (display-oriented, e.g., colored cursive/script) affect LVLMs' descriptions of a concept in terms of the attributes of that concept. Our experiments study the situation in which the LVLM is able to correctly identify the concept referred to by a visual text, i.e., by a word or words rendered as an image, and in which the visual text style should not influence the attribute-based description that the LVLM produces. Our experimental results reveal that even when the concept is correctly identified, text style influences the model's attribute-based descriptions of the concept. Our findings demonstrate non-trivial style leakage from text style into semantic inference and motivate style-aware evaluation and mitigation for LVLM-based multimedia systems.

Abstract

Vision-Language Models

Explore similar work

Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models

Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery

Tinted Frames: Question Framing Blinds Vision-Language Models