Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm

Nurbüke Teker; Rui Xiao; Zeynep Akata; Shuchen Wu

cs.CVOct 8, 2026

Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm

Authors: Nurbüke Teker, Rui Xiao, Zeynep Akata, Shuchen Wu

Organizations: Technical University of Munich · Munich Center for Machine Learning · University of Washington · Allen Institute

Abstract

When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image. But to act means to choose: saying one thing means choosing to respond not say another. This becomes an issue when the image contains multiple features that equally relate to the question at hand: how would VLMs choose to respond? This problem is precisely studied by the classical Stroop task in psychology: among a set of deliberately designed images whose text and color conflicts (e.g., \textcolor{blue}{RED}), participants are instructed to report the color. We adapt this psychological paradigm to a benchmark and prompted a range of contrastive and generative VLMs on their behavior. Across models, we observe a persistent bias toward the feature of text in Stroop images: when text and color disagree, models preferentially generate responses centered to the text rather than color. We then opened the black box and examined the source of this bias and why certain features influences the model's behavior more than others. In the embedding space, we found that textual cues are represented more strongly and separably than color cues. This representational asymmetry predicts intervention success: steering models toward text-related directions is substantially easier than steering them toward color-related directions. Using a classical psychological paradigm, we demonstrate, at both behavioral and representational levels, that distinct features within the same image can exert unequal influence on VLM behavior. Our findings suggest that VLMs assign different degrees of saliency to different features within the same image, highlighting the need to understand how cue-specific saliency shapes their performance in widely used downstream applications.

Figures & tables

Figure 1: Studying feature competition in VLMs with the Stroop task. (A) Example congruent stimuli, in which word identity matches ink color, and incongruent stimuli, in which the two conflict. An image encoder $f(I)$ and text encoder $g(T)$ map images and candidate prompts into a shared embedding space, where alignment is measured by cosine similarity $s(I,T)$ . (B) Each image is scored against two prompt families: word-oriented (e.g., “The text says blue ”) and ink-oriented (e.g., “The text is written in red”). (C) On congruent stimuli, both prompt families score correctly. On incongruent stimuli, word-oriented prompts still score correctly, but ink-oriented prompts do not — the model aligns with the written word rather than the actual ink color.

Figure 2: Word bias persists under visual variations. We test whether contrastive vlms’s word bias persist varying different factors influencing conflicting visual features for incongruent stimuli: the font size (top left), the font weight (top right), the contrast (bottom left), and semantic meaning (bottom right). For every image, CLIP scores prompts of the form “The word is written in {ink} color”, one per candidate color, and we take the highest-scoring prompt as its answer. We then check what that answer names: the actual ink color ( ink bias ), the color written in the image ( word bias ), or neither ( other ). The bottom row shows how often each outcome occurs in each condition, aggregated over all colors.

Figure 3: Evaluating the Stroop Paradigm on Generative VLMs (A) The task setup. We show each model an incongruent Stroop image and ask: “What color is the word in this image?” For Qwen2-VL-7B-Instruct we also test two variants of this prompt, to check how much the phrasing matters. (B) Example answers from models and their categories. Each answer is categorized into one of four types: answer containing only the actual ink color ( Ink-only ), only the word written ( Word-only ), both ( Both ), or neither ( Neither ). (C) Distribution of answer types across all generative vlms. Most models mostly give Word-only answers and they report the written word rather than the ink color. BLIP-2 often names both cues in one answer, LLaVA is the only model that reports the ink color more often than the word, and Qwen2-VL’s answers change substantially with the prompt phrasing. Generative VLMs mostly answer with the written word, not the ink color.

Figure 4: Feature Saliency is Represented in the Image Embedding Space Top: We calculate representation dissimilarity matrix among image embeddings of images that only contain colors ink-only ; images that only contain words ( word-only ) images that contained colored words ( word+ink ). Bottom: resulting delta RDMs showing that the feature of word shape on top of colored feature makes image embeddings more dissimilar in than adding ink color feature on top of word shape.

Figure 5: Model Steering Pipeline A Stroop stimulus (e.g., the word “ RED ” in red ink) is encoded by CLIP into an embedding $E$ . We extract source ( $\mu_{\text{src}}$ ) and target ( $\mu_{\text{tgt}}$ ) encoding vectors using the chunk-based interpretability method by by aggregating embeddings from all images containing the concept (for colors, all items written in that ink color; for words, all items containing that word). We then apply the intervention on the embedding vector $E$ as $E^{\prime}=E-\mu_{\text{src}}+\mu_{\text{tgt}}$ , and then re-evaluated the cosine similarity with source and target prompts.

Panel of Table 1 — Table 1: Illustrative FLUX-generated conflict cases. CLIP’s decision reflects either the textual or visual cue, depending on saliency.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Color	red	blue	green	yellow	orange	purple	brown	pink	gray	black
Hex	fc0404	0c0cfc	048404	fcfc17	fca404	8c178c	8c4414	fcc4cc	848484	000000

Appendix

Table 2: Fixed sRGB hex codes for ink colors.

Experiment type	#stimuli	#models	Total evaluations
Base Stroop (cong+incong)	100	7	700
Font size (4 $\times$ 100)	400	2 (CLIP+SigLIP-2)	800
Font weight (4 $\times$ 100)	400	2	800
Contrast (4 $\times$ 100)	400	2	800
Pseudowords	500	2	1000
Qwen2-VL (2 prompts)	100	1	200

Appendix

Table 3: Scale of evaluation datapoints across experiments (updated with Qwen2-VL).

Manipulation	Description	Purpose
Font size	48, 72, 90, 108 pt	Visual salience / overflow check
Font weight	Bold, Normal, Light, Narrow	Readability vs. ink reliance
Tone variation	10 brightness/saturation levels	Perceptual robustness
Contrast	High / Medium / Low / Same (text color equals background)	Visibility thresholding
Pseudowords	Word replaced by nonwords (e.g., GNEK , GRIV , GRUP , …)	Remove lexical semantics

Appendix

Table 4: Visual manipulations applied to Stroop stimuli.

Figure 6: Examples from the dataset showing congruent and incongruent stimuli.

Prompt family	Congruent (n=10)	Incongruent (n=90)
Word-oriented ( “The text says X” )	10/10 (100%)	88/90 ( 97.8% )
Ink-oriented ( “Written in X color” )	10/10 (100%)	18/90 ( 20.0% )

Appendix

Table 5: Family-specific accuracies. Reporting both views avoids conflating the decision rule (winner-takes-all) with family-specific correctness.

Manipulation	Setting	Text match	Color match	Notes
Font size	48 pt	97%	13%	Readable; text dominates
	72 pt	95%	15%
	90 pt	89%	21%
	108 pt	77%	33%	Overflow reduces readability
Font weight	Bold	89%	21%	Minor effect on color
	Normal	88%	22%

Appendix

Table 6: Effect of visual manipulations (aggregated across colors). Values mirror the panels in Figs. 7 – 11 ; tone-variation exemplars are in Fig. 10 .

Figure 7: Font size exemplars and aggregate trend (Table 6 ).

Figure 8: Font weight exemplars and aggregate trend (Table 6 ).

Figure 9: Contrast manipulation exemplars; see Table 6 .

Figure 11: Pseudoword exemplars; no lexical cue leads to color-based decisions.

Category	Count	Accuracy (%)
Color Match (ink-based)	389	77.8
Form–Color Aligned	49	9.8
Text-Biased (form-based)	28	5.6
Other Errors	34	6.8
Total	500	100

Appendix

Table 7: Aggregated pseudoword results across all colors (

N=500

). No true “congruent” cases exist; Form–Color Aligned refers to coincidental alignment of pseudoword form and ink color.

Condition	Text Accuracy (%)	Color Accuracy (%)
Congruent	100.0	100.0
Incongruent	100.0	5.6

Appendix

Table 8: SigLIP-2 behavioral accuracy on congruent and incongruent Stroop stimuli.

Font Size	Text Accuracy (%)	Color Accuracy (%)
48 pt	100.0	15.0
72 pt	100.0	16.0
90 pt	100.0	16.0
108 pt	100.0	16.0

Appendix

Table 9: Accuracy by font size for SigLIP-2. Unlike CLIP, larger sizes did not weaken text dominance.

Style	Text Accuracy (%)	Color Accuracy (%)
Light	100.0	6.7
Normal	100.0	6.7
Bold	100.0	7.8
Narrow	100.0	6.7

Appendix

Table 10: Accuracy by font weight. SigLIP-2 remained insensitive to style; text was always dominant.

Contrast	Text Win (%)	Color Win (%)	Tie (%)
High	55.6	11.1	33.3
Medium	44.4	11.1	44.4
Low	22.2	66.7	11.1
Same	33.3	11.1	55.6

Appendix

Table 11: Preference breakdown for contrast manipulations. SigLIP-2 showed indecision (ties) at medium/same levels, unlike CLIP which switched fully to color when text was unreadable.

Outcome	Count (%)
Ink-oriented prompt wins	335 (67%)
Word-oriented prompt wins	139 (28%)
Tie	26 (5%)

Appendix

Table 12: SigLIP-2 on pseudoword stimuli. Unlike CLIP, which often clung to text shapes, SigLIP-2 switched reliably to color.

Model	Backbone	Params	Training Objective	Source / Release
CLIP	ViT-B/32 + Transformer text encoder	$\sim$ 150M	Contrastive (softmax)	Radford et al. (2021)
SigLIP-2	ViT-B/16 (base) + text encoder	$\sim$ 150M	Contrastive (sigmoid)	Google, 2024 release
BLIP-2	ViT-g + FlanT5-XL	$\sim$ 13B	Q-former + pretrain + instruction tuning	Li et al. (2022)
InstructBLIP	ViT-g + Vicuna-7B	$\sim$ 7B	Instruction-tuned generation	Dai et al. (2023)
Kosmos-2	Multi-stream transformer (1B)	$\sim$ 1B	Multimodal pretrain (text+image)	Peng et al. (2024)
LLaVA (v1.6)	CLIP ViT-L/14 + Vicuna-7B	$\sim$ 7B	Instruction-tuned generation	Liu et al. (2023)

Appendix

Table 13: Model configurations used in this study. Parameter counts are approximate; all models are publicly available.

Condition	Main prompt (%)	Alt prompt (%)
Congruent (n=10)	100.0	100.0
Incongruent (n=90)	31.1	60.0

Appendix

Table 14: Qwen2-VL-7B-Instruct Stroop results under two prompts. Prompt choice strongly influences incongruent ink accuracy.

Prompt	Ink Preference (%)
p1	58.0
p2	88.0
p3	48.0
p4	45.0
p5	83.0
p6	37.0

Appendix

Table 15: Ink accuracy of Qwen2-VL-7B-Instruct under nine alternative prompt formulations (p1–p9).

Figure 12: Unguided similarity test: the image embedding (e.g., “RED” written in blue) is compared to the concept anchors red and blue in text space; the closer anchor indicates the dominant modality.

Figure 13: Design for modality-isolating RDM comparisons. $\Delta$ Word/Shape asks “what changes when text is added?”, $\Delta$ Ink/Color asks “what changes when color is added?”

Panel of Figure 14 — Figure 14: RDMs for grayscale vs. colorful Stroop images.

Panel of Figure 15 — Figure 15: RDMs for solid-color vs. colorful text images.

Figure 16: Mean differential dissimilarity change when adding word form (shape) vs. color. Adding information about the word form in the image on average changes dissimilarity more strongly than adding the color to the words.

Stimulus	Ground Truth (Ink)	Top-1 Text	Top-1 Color	Bias
“RED” in blue ink	Blue	Red (0.89)	Blue (0.12)	Textual
“GREEN” in pink ink	Pink	Green (0.76)	Pink (0.24)	Mixed
“BLACK” in yellow ink	Yellow	Black (0.82)	Yellow (0.10)	Textual

Appendix

Table 16: Example CLIP similarity outputs on incongruent Stroop stimuli (cosine scores in parentheses).

Panel of Figure 17 — Figure 17: Subpopulation-based intervention pipeline (RED $\to$ BLUE). After editing we always re-normalize embeddings.

Intervention	Success rate	Mean $\Delta$
Color	36.7%	$-0.0017$
Text	100.0%	$+0.0934$
Combined	100.0%	$+0.1172$

Appendix

Table 17: Success and shift for subpopulation-based interventions

Source $\to$ Target	Color norm	Text norm	Notes
red $\to$ blue	2.53	5.38
red $\to$ pink	3.60	6.38
orange $\to$ brown	2.09	6.61	(min color)
pink $\to$ green	3.86	6.49	(max color)
Mean	2.99	6.36

Appendix

Table 18: Chunk

\ell_{2}

norms (color vs. text); mean across pairs shows

\sim

2.1

\times

stronger text directions

Variant	Template	Success Rate
color	The text is written in {color} color	36.7%
just	{color}	41.1%
pure	The text is purely {color}	41.1%
long	The color of the text is purely {color}	38.9%

Appendix

Table 19: Effect of prompt variants on color-only intervention success rate

Metric	Value
Mean $\Delta$ (Combined vs. Text-only)	$+0.0069$
Cases with $\|\Delta\|<0.01$	60%
Mean color-chunk norm	3.04

Appendix

Table 20: Contribution of color chunk within combined edits

Panel of Table 21 — Table 21: FLUX-generated multimodal conflict examples. CLIP’s decision reflects either the textual or visual cue depending on saliency.

Representation Geometry in VLMs

See all 21 papers

Aug 14, 2026

Seeing Red, Thinking Bad: Color Bias in Vision Language Models

Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm

Kohsuke Ide, Ryousuke Yamada, Yoshihiro Fukuhara +2

National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan · University of Tsukuba, Tsukuba, Japan · University of Technology Nuremberg, Nuremberg, Germany +1

VLM Robustness VLM Interpretability Adversarial Attacks on VLMs

Ask PDF

Oct 14, 2026

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.

Josh Barua, Kyle Mahowald, William Rudman

The University of Texas at Austin

Vision-Language Models Modality Gap in VLMs

Ask PDF

Jul 21, 2026

Linguistic Context Recodes Visual Representations in Vision-Language Models

Goal-directed visual processing is a hallmark of human visual intelligence, resulting in representations that support downstream tasks such as categorization or search. Though vision-language models (VLMs) are often faced with these same tasks, their ability to recode visual representations when presented with goal-directed language remains poorly characterized. Indeed, prior work largely treats visual representations in VLMs as static repositories of visual information that are manipulated by language representations. In the present work, we provide evidence for two concrete instances of language-induced recoding of visual representations. First, we identify an abstract reference representation that denotes which objects are goal-relevant under a natural language prompt. We extract contrastive steering vectors corresponding to this reference representation and demonstrate that they are causally implicated in model predictions. These reference representations are abstract in that they generalize to different objects, different task contexts, and even from synthetic to naturalistic images. Second, we demonstrate language-induced attribute modulation: later layers selectively amplify goal-relevant attributes in visual representations of objects. We demonstrate this phenomenon across a range of different prompts. Finally, we provide a causal intervention that demonstrates that attribute modulation mediates a VLM's response distribution. Together, our results support a more dynamic account of cross-modality processing in VLMs -- rather than vision tokens serving as static repositories of information, they are modulated to support queries articulated in language.

Brian Song, Michael A. Lepori, Ellie Pavlick

Brown University

Vision-Language Models VLM Interpretability

Ask PDF

Abstract

Representation Geometry in VLMs

Explore similar work

Seeing Red, Thinking Bad: Color Bias in Vision Language Models

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Linguistic Context Recodes Visual Representations in Vision-Language Models