Organizations: National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan · University of Tsukuba, Tsukuba, Japan · University of Technology Nuremberg, Nuremberg, Germany · University of Oxford, Oxford, UK
Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm
Figures & tables
Figure 1: Subtle visual styling biases VLM outputs. Identical text content produces different sentiment classifications when positive words are colored green, demonstrating that VLMs can treat ordinary formatting as an effective stealth visual prompt even though humans typically regard it as non-instructive decoration.
Figure 2: Examples of the generated visual stimuli. (a) A mixed-sentiment text used in the Color Axis experiment. (b) A structurally separated text. (c) A stimulus from the Saliency Competition (Contrast Axis), where the semantically incorrect decoy (“twenty-miles”) is made visually salient with high contrast.
Model
Max Pos. ↑
Max Neg. ↓
Range
IDEFICS2-8B
+0.160
-0.360
0.520
LLaVA-Mistral-7B
+0.030
-0.010
0.040
LLaVA-Vicuna-7B
+0.060
-0.060
0.120
Qwen2-VL-7B
+0.420
-0.480
0.900
Table 1: Short-sentence Sentiment Set: worst-case sentiment bias induced by color styling. Max Pos. ( ↑ ) and Max Neg. ( ↓ ) are measured relative to the all-black baseline. Range is Max Pos. − Max Neg., summarizing worst-case susceptibility across color conditions.
Figure 3: Short-sentence Sentiment Set: average sentiment bias vs. baseline for four VLMs. The y-axis is the change in sentiment score induced by the color prompt.
Model
Color Bias Range
Positional Strategy
Adherence
IDEFICS2-8B
0.780
Primacy
93%
LLaVA-Mistral-7B
0.000
Recency
100%
LLaVA-Vicuna-7B
0.160
Primacy
97%
Qwen2-VL-7B
0.000
Recency
100%
Table 2: Long-sentence Sentiment Set: positional heuristics under structured discourse. Positional Strategy indicates whether predictions follow the first half (primacy) or second half (recency). Adherence is the fraction of samples consistent with that strategy. Color Bias Range summarizes the residual color-induced variation across color conditions in this structured setting.
Figure 4: CLIP representation probe: hue-dependent semantic projections. We sweep text hue and project CLIP image embeddings of six rendered probe words ( warm , cold , safe , dangerous , good , bad ) onto four text-defined semantic axes: safety (safe–dangerous), valence (good–bad), temperature (warm–cold), and emotion (happy–sad). The emotion axis is included as an additional example where hue-dependent shifts are visually apparent; it is defined by the happy – sad text-embedding difference, while the plotted curves correspond to projections of the same six rendered probe words.
Model
Baseline F1
Answer Salient F1
Decoy Salient F1
IDEFICS2-8B
0.054
0.064 (+18.5%)
0.052 (-3.7%)
LLaVA-Mistral-7B
0.053
0.058 (+9.4%)
0.052 (-1.9%)
LLaVA-Vicuna-7B
0.056
0.061 (+8.9%)
0.055 (-1.8%)
Qwen2-VL-7B
0.745
0.828 (+11.1%)
0.738 (-0.9%)
Table 3: VQA Stealth Set: average F1 in Saliency Competition, aggregated over the six low-contrast grayscale levels. Columns differ only in which span is rendered in high-contrast black (none / answer / decoy).
Model
1
16
64
128
192
240
IDEFICS2-8B
24%
23%
24%
27%
32%
36%
LLaVA-Mistral-7B
27%
25%
26%
24%
24%
27%
LLaVA-Vicuna-7B
19%
19%
20%
20%
22%
25%
Qwen2-VL-7B
4%
4%
4%
4%
5%
6%
Table 4: VQA Stealth Set: Induced Error Rate (%) in the Decoy Salient condition at the six low-contrast grayscale levels used in our sweep (higher value ⇒ lower visibility).
Figure 5: VQA Stealth Set: Induced Error Rate as a function of the grayscale level used for the non-salient context text (higher ⇒ closer to white and lower contrast).
Figure 6: Model-dependent readability transitions measured by a VLM-based OCR proxy. OCR accuracy is plotted as a function of text–background contrast ( ΔE ) for multiple font sizes. The transition from low to high accuracy occurs over a relatively narrow range and varies by model and font size.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.