Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

Md Tanvirul Alam

cs.CVApr 13, 2026

Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

Authors: Md Tanvirul Alam

Organizations: Rochester Institute of Technology, Rochester, NY, USA

Abstract

Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap. Prompt interventions support this mechanism: neutral alias prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and dataset available at https://maveryn.github.io/vlm-fix/.

Figures & tables

Panel of Figure 1 — Figure 1: Tic-Tac-Toe example from VLM-Fix. Left: base, tag-alias, and tag-semantic prompt variants. Right-top: terminal board state. Right-bottom: accuracy trend across prompt variants. Blue marks standard-rule assignments, red marks inverse (misère) assignments, and green marks the added semantic definition in the tag-semantic prompt.

Game	Standard rule	Inverse rule
Tic-Tac-Toe	A player who gets 3-in-a-row wins.	A player who gets 3-in-a-row loses.
Connect Four	A player who gets 4-in-a-row wins.	A player who gets 4-in-a-row loses.
Reversi	The player with more pieces wins.	The player with fewer pieces wins.
Dots and Boxes	The player who claims more boxes wins.	The player who claims fewer boxes wins.

Table 1: Standard and inverse rule conditions in VLM-Fix.

Panel of Figure 2 — Figure 2: Compact Reversi example from VLM-Fix. Left: the three rendering variants, shown in order as the base, glyph, and checkerboard renderings used in the benchmark. Right: the three prompt variants that preserve the same underlying inverse-rule state while changing only semantic framing.

	Tic-Tac-Toe		Reversi		Connect Four		Dots and Boxes		Average
Model	Std	Inv	Std	Inv	Std	Inv	Std	Inv	Std	Inv
GPT-4.1	86.0	76.0	85.7	75.0	69.0	41.3	88.3	60.3	82.2	63.2
GPT-5.2	93.7	86.0	95.7	91.0	69.0	53.0	93.7	75.7	88.0	76.4
Sonnet-4.0	51.0	51.7	76.0	50.3	63.3	65.7	95.0	92.3	71.3	65.0
Sonnet-4.5	50.0	56.0	85.7	67.0	53.0	86.7	78.3	77.0	66.8	71.7
Qwen2.5-VL-3B	54.2	49.9	57.2	50.0	51.7	50.6	51.7	44.2	53.7	48.7

Table 2: VLM-Fix results under the baseline setting ( Base ; canonical rendering; direct answering). Each cell reports accuracy (%) under the standard or inverse rule, aggregated over winner/loser question targets and image-first/text-first orders.

Config	Tic-Tac-Toe		Reversi		Connect Four		Dots and Boxes		Average
	Std	Inv	Std	Inv	Std	Inv	Std	Inv	Std	Inv
Base	65.7	55.5	71.9	51.8	57.3	52.9	73.8	50.0	67.1	52.5
Glyph	65.2	54.6	74.1	53.3	61.8	56.9	70.3	54.9	67.8	54.9
Checkerboard	63.7	53.9	67.1	52.6	58.4	52.9	74.5	52.3	65.9	52.9
Alias	64.8	63.9	68.3	66.6	57.5	57.9	70.9	63.9	65.4	63.1
SemAlias	64.6	53.0	71.6	54.7	58.3	53.5	70.3	52.9	66.2	53.5

Table 3: Average effect of input interventions on VLM-Fix direct results, aggregated over the 14 models. Columns report standard and inverse accuracy (%) for each game and the macro-average across games.

Setting	Tic-Tac-Toe		Reversi		Connect Four		Dots and Boxes		Average
	Std	Inv	Std	Inv	Std	Inv	Std	Inv	Std	Inv
Direct (Base)	65.7	55.5	71.9	51.8	57.3	52.9	73.8	50.0	67.1	52.5
CoT (Base)	85.6	76.4	91.9	74.1	83.3	79.2	79.2	66.5	85.0	74.1
CoT (Alias)	84.2	84.8	91.3	90.6	84.4	83.9	79.4	77.3	84.8	84.1
CoT (SemAlias)	84.0	80.8	92.3	77.0	84.4	80.3	81.3	71.0	85.5	77.3
Descriptive	65.5	64.4	78.0	74.1	59.1	56.0	81.3	80.7	71.0	68.8

Table 4: Boundary-condition results on VLM-Fix. Columns report standard and inverse accuracy (%) for each game and the macro-average across games.

Panel of Figure 3 — Figure 3: Representative Animals example from VLMBias . Left: the Base and Flip images. Right: the corresponding Base and Alias prompts.

Config	Animals		Logos		Flags		Game Boards		Overall
	Acc	Bias	Acc	Bias	Acc	Bias	Acc	Bias	Acc	Bias
Base	3.6	93.2	13.9	71.4	25.9	55.1	11.4	66.6	11.6	76.7
Flip	5.7	88.9	16.6	60.2	24.2	55.4	13.7	61.1	13.3	70.9
Alias	8.5	83.8	14.9	65.6	30.6	51.8	14.1	58.0	15.0	69.5
Flip+Alias	22.2	67.5	15.7	56.0	29.0	49.8	16.2	51.1	20.7	58.9

Table 5: Aggregate VLMBias results across the four evaluation subsets, averaged over the 14 models using the canonical row-level summary.

Figure 4: VLM-Fix splits (D1–D3) for Qwen2.5-VL-7B (left) and Qwen2.5-VL-3B (right): D1/D2 report same-rule and opposite-rule evaluation, and D3 reports held-out standard/inverse evaluation on Connect Four and Dots and Boxes.

Figure 5: Synthetic leg-count transfer for Qwen2.5-VL-7B (left) and Qwen2.5-VL-3B (right), comparing Base, SFT, and RLVR on VLMBias and VLMBias Counting (Unbiased) (Animals vs Other).

Figure 6: Layerwise activation-steering results on VLM-Fix for Qwen2.5-VL-7B across Tic-Tac-Toe, Reversi, Connect Four, and Dots and Boxes (left to right).

Figure 7: Layerwise SFT

\rightarrow

Base activation-steering results on VLMBias Animals for Qwen2.5-VL-7B, Qwen2.5-VL-3B, Molmo2-4B, and InternVL3.5-4B (left to right).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Representative VLM-Fix inputs across four games. In each column, the top three images show rendering variants in the order Base $\rightarrow$ Glyph $\rightarrow$ Checkerboard . Below the images, we show the three prompt variants used in this work: Base , Alias , and SemAlias .

Panel of Figure 9 — Figure 9: Representative examples from the four VLMBias subsets. Top row: Base images. Second row: Flip images. Third row: Base prompts. Fourth row: Alias prompts.

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	86.0/76.0	84.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /60.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	77.3 ${}_{\scriptscriptstyle(\!\sim\!0.01)}$ /61.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	88.0 ${}_{\scriptscriptstyle(\!\sim\!0.42)}$ /85.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	87.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /21.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	93.7/86.0	82.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /70.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	89.0 ${}_{\scriptscriptstyle(\!\sim\!0.20)}$ /70.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	94.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /88.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	95.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /86.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	51.0/51.7	62.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /52.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	54.3 ${}_{\scriptscriptstyle(\!\sim\!0.10)}$ /50.0 ${}_{\scriptscriptstyle(\!\sim\!0.75)}$	57.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /57.7 ${}_{\scriptscriptstyle(\!\sim\!0.66)}$	59.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /50.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.5	50.0/56.0	61.0 ${}_{\scriptscriptstyle(\!\sim\!0.01)}$ /58.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	50.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /52.7 ${}_{\scriptscriptstyle(\!\sim\!0.26)}$	54.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /52.0 ${}_{\scriptscriptstyle(\!\sim\!0.18)}$	52.3 ${}_{\scriptscriptstyle(\!\sim\!0.06)}$ /52.7 ${}_{\scriptscriptstyle(\!\sim\!0.25)}$
Qwen2.5-VL-3B	54.2/49.9	51.7 ${}_{\scriptscriptstyle(\!\sim\!0.60)}$ /48.9 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	53.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /50.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	52.0 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$ /51.9 ${}_{\scriptscriptstyle(\!\sim\!0.37)}$	52.3 ${}_{\scriptscriptstyle(\!\sim\!0.08)}$ /51.2 ${}_{\scriptscriptstyle(\!\sim\!0.73)}$

Appendix

Table 6: VLM-Fix direct results on Tic-Tac-Toe. Cells report standard/inverse accuracy (%) ; subscripts in non-baseline columns give Holm-adjusted paired McNemar

p

-values versus Base for the same rule. Within each row, red and blue mark the largest and smallest

(\mathrm{Standard}-\mathrm{Inverse})

gaps.

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	85.7/75.0	87.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /77.7 ${}_{\scriptscriptstyle(\!\sim\!0.72)}$	86.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /79.3 ${}_{\scriptscriptstyle(\!\sim\!0.71)}$	87.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /80.0 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$	86.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /81.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	95.7/91.0	92.0 ${}_{\scriptscriptstyle(\!\sim\!0.26)}$ /84.3 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	90.3 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$ /86.7 ${}_{\scriptscriptstyle(\!\sim\!0.18)}$	96.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /93.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	95.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /88.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	76.0/50.3	80.7 ${}_{\scriptscriptstyle(\!\sim\!0.32)}$ /55.3 ${}_{\scriptscriptstyle(\!\sim\!0.64)}$	69.3 ${}_{\scriptscriptstyle(\!\sim\!0.28)}$ /51.3 ${}_{\scriptscriptstyle(\!\sim\!0.91)}$	84.7 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$ /71.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	84.7 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$ /78.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.5	85.7/67.0	86.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /81.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	79.0 ${}_{\scriptscriptstyle(\!\sim\!0.44)}$ /69.7 ${}_{\scriptscriptstyle(\!\sim\!0.40)}$	89.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /82.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	86.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /75.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Qwen2.5-VL-3B	57.2/50.0	57.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /43.1 ${}_{\scriptscriptstyle(\!<\!0.01)}$	56.4 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /50.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	58.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /55.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	54.5 ${}_{\scriptscriptstyle(\!\sim\!0.07)}$ /55.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 7: VLM-Fix direct results on Reversi (same format as Table 6 ).

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	69.0/41.3	74.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /48.3 ${}_{\scriptscriptstyle(\!\sim\!0.33)}$	73.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /52.0 ${}_{\scriptscriptstyle(\!\sim\!0.05)}$	69.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /62.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	69.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /36.3 ${}_{\scriptscriptstyle(\!\sim\!0.35)}$
GPT-5.2	69.0/53.0	71.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /65.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	69.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /53.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	68.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /56.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	71.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /51.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	63.3/65.7	56.0 ${}_{\scriptscriptstyle(\!\sim\!0.34)}$ /69.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	60.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /68.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	65.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /78.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	68.3 ${}_{\scriptscriptstyle(\!\sim\!0.63)}$ /82.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.5	53.0/86.7	74.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /87.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	67.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /72.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	60.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /77.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	62.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /92.7 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$
Qwen2.5-VL-3B	51.7/50.6	52.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /49.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	49.4 ${}_{\scriptscriptstyle(\!\sim\!0.13)}$ /50.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	50.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /49.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	50.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /47.7 ${}_{\scriptscriptstyle(\!\sim\!0.62)}$

Appendix

Table 8: VLM-Fix direct results on Connect Four (same format as Table 6 ).

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	88.3/60.3	80.3 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$ /60.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	90.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /71.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	89.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /81.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	85.7 ${}_{\scriptscriptstyle(\!\sim\!0.91)}$ /72.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	93.7/75.7	91.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /74.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	90.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /74.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	94.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /76.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	92.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /73.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	95.0/92.3	97.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /96.0 ${}_{\scriptscriptstyle(\!\sim\!0.21)}$	98.3 ${}_{\scriptscriptstyle(\!\sim\!0.10)}$ /95.7 ${}_{\scriptscriptstyle(\!\sim\!0.30)}$	94.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /96.0 ${}_{\scriptscriptstyle(\!\sim\!0.21)}$	96.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /95.0 ${}_{\scriptscriptstyle(\!\sim\!0.30)}$
Sonnet-4.5	78.3/77.0	97.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /94.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	83.0 ${}_{\scriptscriptstyle(\!\sim\!0.78)}$ /67.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	80.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /64.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	77.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /68.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Qwen2.5-VL-3B	51.7/44.2	51.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /47.8 ${}_{\scriptscriptstyle(\!\sim\!0.37)}$	57.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /41.5 ${}_{\scriptscriptstyle(\!\sim\!0.37)}$	51.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /51.4 ${}_{\scriptscriptstyle(\!<\!0.01)}$	51.8 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /46.0 ${}_{\scriptscriptstyle(\!\sim\!0.58)}$

Appendix

Table 9: VLM-Fix direct results on Dots and Boxes (same format as Table 6 ).

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	97.0/94.0	–	–	95.7 ${}_{\scriptscriptstyle(\!\sim\!0.69)}$ /96.7 ${}_{\scriptscriptstyle(\!\sim\!0.37)}$	95.0 ${}_{\scriptscriptstyle(\!\sim\!0.58)}$ /97.0 ${}_{\scriptscriptstyle(\!\sim\!0.37)}$
GPT-5.2	100.0/99.7	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	95.0/96.7	–	–	97.7 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$ /97.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	97.7 ${}_{\scriptscriptstyle(\!\sim\!0.08)}$ /96.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.5	96.7/95.3	–	–	95.3 ${}_{\scriptscriptstyle(\!\sim\!0.58)}$ /96.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	99.0 ${}_{\scriptscriptstyle(\!\sim\!0.26)}$ /96.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Qwen2.5-VL-3B	59.8/47.6	57.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /49.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	59.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /46.8 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	59.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /57.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$	58.8 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /54.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 10: VLM-Fix CoT results on Tic-Tac-Toe. API rows are included for the Base, Alias, and SemAlias CoT conditions; Glyph and Checkerboard CoT entries are unavailable for API models. Cells report standard/inverse accuracy (%) ; subscripts in non-baseline columns give Holm-adjusted paired McNemar

p

-values versus Base for the same rule. Within each row, red and blue mark the largest and smallest

(\mathrm{Standard}-\mathrm{Inverse})

gaps.

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	98.0/96.0	–	–	96.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /96.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	97.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /97.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
GPT-5.2	100.0/100.0	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	99.3/99.3	–	–	99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.5	100.0/100.0	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Qwen2.5-VL-3B	70.7/33.8	67.6 ${}_{\scriptscriptstyle(\!\sim\!0.25)}$ /36.9 ${}_{\scriptscriptstyle(\!\sim\!0.32)}$	64.1 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /31.5 ${}_{\scriptscriptstyle(\!\sim\!0.32)}$	72.3 ${}_{\scriptscriptstyle(\!\sim\!0.54)}$ /53.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	76.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /41.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 11: VLM-Fix CoT results on Reversi.

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	98.7/99.0	–	–	98.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	99.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /94.7 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$
GPT-5.2	99.0/99.0	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	99.7/99.7	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.5	100.0/98.7	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!0.50)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Qwen2.5-VL-3B	43.5/42.0	55.9 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /46.0 ${}_{\scriptscriptstyle(\!\sim\!0.10)}$	40.0 ${}_{\scriptscriptstyle(\!\sim\!0.13)}$ /33.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$	48.2 ${}_{\scriptscriptstyle(\!\sim\!0.06)}$ /49.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$	49.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /47.8 ${}_{\scriptscriptstyle(\!\sim\!0.01)}$

Appendix

Table 12: VLM-Fix CoT results on Connect Four.

Model	Base	Glyph	Checkerboard	Alias	SemAlias
	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)	Std/Inv (Acc)
GPT-4.1	85.7/81.7	–	–	88.0 ${}_{\scriptscriptstyle(\!\sim\!0.76)}$ /85.0 ${}_{\scriptscriptstyle(\!\sim\!0.59)}$	90.7 ${}_{\scriptscriptstyle(\!\sim\!0.13)}$ /86.0 ${}_{\scriptscriptstyle(\!\sim\!0.44)}$
GPT-5.2	100.0/100.0	–	–	99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.0	100.0/99.7	–	–	99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /99.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Sonnet-4.5	100.0/100.0	–	–	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /100.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$
Qwen2.5-VL-3B	52.1/49.0	52.5 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /42.0 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$	50.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$ /45.8 ${}_{\scriptscriptstyle(\!\sim\!0.23)}$	57.8 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$ /45.5 ${}_{\scriptscriptstyle(\!\sim\!0.26)}$	59.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$ /47.8 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$

Appendix

Table 13: VLM-Fix CoT results on Dots & Boxes.

Model	Tic-Tac-Toe	Reversi	Connect Four	Dots and Boxes	Average
GPT-4.1	85.0/87.0	91.0/92.0	72.0/60.3	90.7/91.0	84.7/82.6
GPT-5.2	91.7/91.7	94.3/94.0	73.0/67.0	96.3/88.3	88.8/85.2
Sonnet-4.0	70.0/58.7	70.3/50.7	64.3/53.7	95.7/96.0	75.1/64.8
Sonnet-4.5	61.3/54.7	88.0/52.0	71.0/56.0	99.7/98.3	80.0/65.2
Qwen2.5-VL-3B	52.3/52.3	72.8/72.9	49.7/49.6	48.7/48.8	55.9/55.9
Qwen2.5-VL-7B	62.2/62.3	77.2/77.2	55.9/55.8	85.8/85.7	70.2/70.2

Appendix

Table 14: VLM-Fix Direct Results under Descriptive Prompting (Canonical Rendering). Each cell reports Standard/Inverse accuracy (%) for the corresponding game. The Average column is the per-model mean across the four games.

Model	Tic-Tac-Toe	Reversi	Connect Four	Dots and Boxes	Average
GPT-4.1	96.3/57.7	90.0/91.3	88.3/40.0	93.0/95.7	91.9/71.2
GPT-5.2	98.3/93.7	95.3/95.7	87.7/82.0	97.7/90.3	94.8/90.4
Sonnet-4.0	86.0/71.7	93.0/93.0	82.3/80.3	100.0/98.3	90.3/85.8
Sonnet-4.5	77.7/67.3	99.0/95.0	66.7/71.0	100.0/99.7	85.8/83.2
Qwen2.5-VL-3B	51.5/50.0	50.3/50.0	49.8/50.2	57.2/42.5	52.2/48.2
Qwen2.5-VL-7B	63.0/45.5	70.3/36.7	70.2/49.8	70.5/34.2	68.5/41.5

Appendix

Table 15: VLM-Fix text-only direct results (canonical text board and standard prompt). API models are aligned to the same reduced 300-state winner-only seed subset used in their direct API runs. Each cell reports Standard/Inverse accuracy (%) for the corresponding game. The Average column is the per-model mean across the four games.

Model	Base		Glyph		Checkerboard		Alias		SemAlias		Descriptive
	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first
Qwen2.5-VL-3B	53.5/49.3	53.8/48.1	53.6/47.6	53.0/47.1	54.1/49.2	54.2/46.6	55.0/54.2	50.9/50.2	53.2/52.2	51.3/48.2	56.1/56.2	55.6/55.5
Qwen2.5-VL-7B	61.2/44.2	65.0/39.3	59.6/46.2	68.7/44.7	61.1/43.3	60.8/44.2	60.8/59.4	60.0/59.1	60.4/44.5	57.0/44.9	72.2/72.2	68.3/68.2
InternVL3.5-4B	54.5/42.8	68.9/37.3	56.7/44.8	66.1/48.2	52.5/46.5	61.1/44.1	56.0/56.6	66.0/52.4	56.7/43.2	64.2/45.3	58.5/58.5	69.4/69.3
InternVL3.5-8B	61.8/52.8	70.2/51.0	63.4/50.9	70.1/54.2	60.7/51.5	68.2/50.8	59.2/53.8	65.4/58.8	64.2/49.4	66.5/46.2	67.5/67.5	72.1/72.1
InternVL3.5-14B	56.4/48.1	61.3/55.1	61.3/53.2	70.3/58.5	56.7/48.8	61.5/51.8	54.0/58.8	61.6/67.4	54.4/51.6	62.0/53.5	59.4/59.5	65.8/65.8

Appendix

Table 16: VLM-Fix order marginals for direct prompting, averaged over the four games and reported for the 10 open-weight models only. Each cell reports standard/inverse accuracy (%) for a fixed condition under image-first versus text-first input order.

Model	Base		Glyph		Checkerboard		Alias		SemAlias
	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first	Img-first	Text-first
Qwen2.5-VL-3B	57.2/44.3	55.8/41.9	58.5/43.9	58.1/43.4	56.5/42.4	50.5/36.4	59.6/52.0	59.2/51.0	63.0/48.4	59.6/47.3
Qwen2.5-VL-7B	72.2/30.2	72.4/34.1	71.7/36.6	73.8/42.2	70.2/33.3	71.1/37.5	75.0/71.0	70.6/72.0	71.0/50.4	72.0/49.4
InternVL3.5-4B	88.0/65.8	85.0/70.1	88.6/77.2	88.0/76.8	87.8/73.0	86.6/72.6	85.2/84.9	84.9/86.2	87.7/71.1	83.5/71.2
InternVL3.5-8B	88.8/83.5	87.4/79.0	89.2/86.4	88.0/85.3	89.5/82.4	88.4/78.5	88.2/85.4	89.7/90.5	87.4/78.5	91.8/82.0
InternVL3.5-14B	78.6/71.8	76.7/74.9	80.4/77.1	78.2/75.4	77.1/75.6	76.4/72.7	75.2/75.1	73.6/78.2	75.8/72.1	73.6/64.8

Appendix

Table 17: VLM-Fix order marginals for CoT prompting, averaged over the four games and reported for the 10 open-weight models only. Each cell reports standard/inverse accuracy (%) for a fixed condition under image-first versus text-first input order.

	Accuracy $\uparrow$				Bias $\downarrow$
Model	Base	Flip	Alias	Flip+Alias	Base	Flip	Alias	Flip+Alias
GPT-4.1	9.9	10.5 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	12.0 ${}_{\scriptscriptstyle(\!\sim\!0.09)}$	10.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	81.0	71.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	73.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	76.1 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	20.2	21.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	26.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$	32.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	76.8	70.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	68.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	54.4 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.0	6.2	6.5 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	7.2 ${}_{\scriptscriptstyle(\!\sim\!0.35)}$	9.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	89.3	84.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	87.5 ${}_{\scriptscriptstyle(\!\sim\!0.08)}$	79.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.5	6.9	8.6 ${}_{\scriptscriptstyle(\!\sim\!0.07)}$	9.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	17.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	92.3	87.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$	85.9 ${}_{\scriptscriptstyle(\!<\!0.01)}$	77.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Qwen2.5-VL-3B	7.4	7.4 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	8.3 ${}_{\scriptscriptstyle(\!\sim\!0.42)}$	10.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	66.4	63.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	59.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$	53.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 18: VLMBias results averaged across the four subsets (Animals, Flags, Game Boards, Logos). Accuracy and Bias are shown side-by-side for four input configurations. Cells report percentages; for non-baseline columns, subscripts show Holm-adjusted paired McNemar

p

-values versus Base (within the same metric). Within each model row, blue marks the best value and red marks the worst value.

	Accuracy $\uparrow$				Bias $\downarrow$
Model	Base	Flip	Alias	Flip+Alias	Base	Flip	Alias	Flip+Alias
GPT-4.1	9.9	2.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	14.1 ${}_{\scriptscriptstyle(\!\sim\!0.07)}$	5.9 ${}_{\scriptscriptstyle(\!\sim\!0.07)}$	72.5	65.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$	58.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	76.6 ${}_{\scriptscriptstyle(\!\sim\!0.17)}$
GPT-5.2	4.6	9.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$	17.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	39.9 ${}_{\scriptscriptstyle(\!<\!0.01)}$	94.1	89.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	79.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$	53.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.0	0.0	0.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	0.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	3.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	99.8	97.4 ${}_{\scriptscriptstyle(\!<\!0.01)}$	98.5 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	92.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.5	0.0	0.5 ${}_{\scriptscriptstyle(\!\sim\!0.50)}$	2.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	17.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	100.0	97.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	95.1 ${}_{\scriptscriptstyle(\!<\!0.01)}$	79.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Qwen2.5-VL-3B	0.0	0.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	0.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	1.5 ${}_{\scriptscriptstyle(\!\sim\!0.05)}$	96.7	97.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	96.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	90.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 19: VLMBias results for the Animals subset (same metrics and annotations as the VLMBias tables below).

	Accuracy $\uparrow$				Bias $\downarrow$
Model	Base	Flip	Alias	Flip+Alias	Base	Flip	Alias	Flip+Alias
GPT-4.1	13.3	9.2 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	13.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	8.3 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	83.8	90.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	85.0 ${}_{\scriptscriptstyle(\!\sim\!0.75)}$	91.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	22.9	24.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	26.7 ${}_{\scriptscriptstyle(\!\sim\!0.20)}$	31.2 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$	73.8	64.6 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$	68.8 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	52.5 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.0	25.0	22.9 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	30.0 ${}_{\scriptscriptstyle(\!\sim\!0.21)}$	28.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	74.2	74.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	70.0 ${}_{\scriptscriptstyle(\!\sim\!0.39)}$	67.1 ${}_{\scriptscriptstyle(\!\sim\!0.07)}$
Sonnet-4.5	24.2	20.0 ${}_{\scriptscriptstyle(\!\sim\!0.98)}$	27.1 ${}_{\scriptscriptstyle(\!\sim\!0.98)}$	26.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	73.8	74.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	69.6 ${}_{\scriptscriptstyle(\!\sim\!0.21)}$	66.7 ${}_{\scriptscriptstyle(\!\sim\!0.16)}$
Qwen2.5-VL-3B	24.2	22.9 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	24.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	30.8 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$	35.4	33.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	32.5 ${}_{\scriptscriptstyle(\!\sim\!0.84)}$	29.2 ${}_{\scriptscriptstyle(\!\sim\!0.24)}$

Appendix

Table 20: VLMBias results for the Flags subset (same format as Table 19 ).

	Accuracy $\uparrow$				Bias $\downarrow$
Model	Base	Flip	Alias	Flip+Alias	Base	Flip	Alias	Flip+Alias
GPT-4.1	0.0	0.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	0.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	0.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	97.6	86.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$	97.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	88.1 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	35.7	41.1 ${}_{\scriptscriptstyle(\!\sim\!0.73)}$	39.9 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	35.1 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	56.5	49.4 ${}_{\scriptscriptstyle(\!\sim\!0.20)}$	47.6 ${}_{\scriptscriptstyle(\!\sim\!0.08)}$	50.0 ${}_{\scriptscriptstyle(\!\sim\!0.20)}$
Sonnet-4.0	14.9	16.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	16.1 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	24.4 ${}_{\scriptscriptstyle(\!\sim\!0.08)}$	75.6	74.4 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	74.4 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	61.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.5	20.8	28.0 ${}_{\scriptscriptstyle(\!\sim\!0.13)}$	28.6 ${}_{\scriptscriptstyle(\!\sim\!0.06)}$	30.4 ${}_{\scriptscriptstyle(\!\sim\!0.10)}$	78.6	69.0 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$	70.2 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$	65.5 ${}_{\scriptscriptstyle(\!\sim\!0.02)}$
Qwen2.5-VL-3B	6.5	8.9 ${}_{\scriptscriptstyle(\!\sim\!0.91)}$	17.9 ${}_{\scriptscriptstyle(\!<\!0.01)}$	22.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	50.0	48.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	26.2 ${}_{\scriptscriptstyle(\!<\!0.01)}$	19.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 21: VLMBias results for the Game Boards subset (same format as Table 19 ).

	Accuracy $\uparrow$				Bias $\downarrow$
Model	Base	Flip	Alias	Flip+Alias	Base	Flip	Alias	Flip+Alias
GPT-4.1	12.1	26.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	13.3 ${}_{\scriptscriptstyle(\!\sim\!0.92)}$	22.7 ${}_{\scriptscriptstyle(\!<\!0.01)}$	83.8	63.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	74.9 ${}_{\scriptscriptstyle(\!<\!0.01)}$	61.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$
GPT-5.2	33.1	26.3 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	32.6 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	20.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	63.8	58.2 ${}_{\scriptscriptstyle(\!\sim\!0.03)}$	63.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	58.7 ${}_{\scriptscriptstyle(\!\sim\!0.04)}$
Sonnet-4.0	0.0	1.2 ${}_{\scriptscriptstyle(\!\sim\!0.38)}$	0.0 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	1.2 ${}_{\scriptscriptstyle(\!\sim\!0.38)}$	89.6	75.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$	88.4 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	77.1 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Sonnet-4.5	0.5	4.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	0.7 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	6.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	98.3	88.4 ${}_{\scriptscriptstyle(\!<\!0.01)}$	89.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	84.3 ${}_{\scriptscriptstyle(\!<\!0.01)}$
Qwen2.5-VL-3B	7.7	7.2 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	6.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	6.3 ${}_{\scriptscriptstyle(\!\sim\!1.00)}$	51.0	43.0 ${}_{\scriptscriptstyle(\!<\!0.01)}$	39.6 ${}_{\scriptscriptstyle(\!<\!0.01)}$	33.8 ${}_{\scriptscriptstyle(\!<\!0.01)}$

Appendix

Table 22: VLMBias results for the Logos subset (same format as Table 19 ).

Panel of Figure 10 — Figure 10: Additional synthetic glyph examples from the procedurally rendered leg-counting training set. Top: bird categories. Bottom: quadruped-style animal categories.

Figure 11: Post-training performance on the three VLM-Fix transfer splits (D1–D3) for additional models with available post-training results: Molmo2-4B (left) and InternVL3.5-4B (right). For D1 and D2, we report both held-out standard-rule and inverse-rule evaluation under the original image and base prompt; for D3, we report held-out standard and inverse evaluation on Connect Four and Dots and Boxes.

Figure 12: Post-training transfer from the synthetic leg-counting dataset for additional models, comparing Base and SFT for Molmo2-4B (left) and InternVL3.5-4B (right). Each panel reports accuracy on VLMBias (Animals vs Other) and VLMBias Counting (Unbiased) (Animals vs Other).

Figure 13: Qwen2.5-VL-3B layerwise activation steering on VLM-Fix across Tic-Tac-Toe, Reversi, Connect Four, and Dots and Boxes.

Figure 14: Molmo2-4B layerwise activation steering on VLM-Fix across Tic-Tac-Toe, Reversi, Connect Four, and Dots and Boxes.

Figure 15: Molmo2-8B layerwise activation steering on VLM-Fix across Tic-Tac-Toe, Reversi, Connect Four, and Dots and Boxes.

Figure 16: InternVL3.5-4B layerwise activation steering on VLM-Fix across Tic-Tac-Toe, Reversi, Connect Four, and Dots and Boxes.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Apr 10, 2026

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of

ρ=

0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.

Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2

Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.

Vision-Language Grounding Visual Evidence Sensitivity in VLMs Faithfulness in VLMs VLM Interpretability

Ask PDF

Feb 6, 2026

Same Answer, Different Representations: Hidden instability in VLMs

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6

Sapienza University of Rome · University of Edinburgh · University of Oxford +5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness Representation Geometry in VLMs

Ask PDF

Apr 19, 2026

More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage

Vision-Language Models (VLMs) excel at photorealistic generation, yet often struggle to represent abstract meaning such as idiomatic interpretations of noun compounds. To study whether high visual fidelity interferes with idiomatic compositionality under visual abstraction, we introduce DIVA, a controlled benchmark that replaces high-fidelity visual detail with schematic iconicity by generating paired, sense-anchored visualizations for literal and idiomatic readings. We further propose Semantic Alignment Gap (

Δ

), an architecture-agnostic metric that quantifies divergence between literal and idiomatic visual grounding. We additionally introduce a directional signed bias

b(t)

to separately measure the direction and strength of literal preference. Evaluating 8 recent VLMs, we reveal a consistent Literal Superiority Bias: model scale alone does not resolve literal preference, and increased visual fidelity is associated with weaker symbolic alignment, suggesting cognitive interference from hyper-realistic imagery. Our findings suggest that improving compositional understanding requires iconographic abstraction of visual input and anchoring interpretation and generation in intended meaning.

Wei He

IDSAI, University of Exeter Exeter, UK

VLM Evaluation Figurative Language Understanding Vision-Language Models VLM Interpretability Vision-Language Grounding

Ask PDF

Abstract

VLM Reasoning

Explore similar work

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Same Answer, Different Representations: Hidden instability in VLMs

More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage