CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models

Souptik Kumar Majumdar; Fabian Kögel; Andreas Bulling

cs.CVAug 21, 2026

CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models

Authors: Souptik Kumar Majumdar, Fabian Kögel, Andreas Bulling

Organizations: Universität Stuttgart 70569, Stuttgart Germany

Abstract

Linear probes and activation steering have uncovered that vision-language models (VLMs) internally represent mental states such as agents' beliefs, knowledge, and intentions. However, it is unclear whether and how these representations are used by downstream predictions along these axes. To close this gap, we introduce Cross-Axis Routing Diagnostic (CARD), which steers activations along one axis while measuring the response of a different axis's prediction. Applied to open-weight VLMs on Relay Chain -- a new cooperative grid-world benchmark we propose -- we diagnose a critical routing failure: models fail to incorporate belief representations into their next action prediction, effectively leaving valuable information about their partners unused.

Figures & tables

Figure 1: Overview of CARD. For each axis

a\in\{\text{belief, intent, knowledge}\}

a linear probe on paired True-Belief / False-Belief activations at layer

L

yields a unit-norm direction

v_{a}

. At inference time, we hook the residual stream at the same layer and add

\alpha v_{a}

(steering axis

a

\alpha\in\{-10,+10\}

), then read out the answer head for a different question axis

b\neq a

. A causally-routed feature moves the cross-axis read-out; an encoded-but-unread feature does not. Within-axis cells (

a=b

, diagonal) serve as positive controls.

Figure 2: Relay Chain four-tuple discrimination layout. Panel elements: protagonist

a_{1}

(red, lever-operator) with field-of-view disk (radius

5

); partner

a_{2}

(blue, lever-operator); traveller

a_{3}

(green); gates (green when open, dark-red when closed); levers

L

; goal

G

. Top row: success (event occurs). Bottom row: counter (lever-operator never releases; gate stays closed). Left column: True-Belief (TB) –

a_{1}

’s lever close enough to perceive the crossing. Right column: False-Belief (FB) – lever beyond

a_{1}

’s view.

Model	Probe acc.	TB-acc.	FB-acc.
Qwen2-VL	$81.5\pm 1.8$	$100.0$	$20.0$
Gemma-4	$94.5\pm 0.6$	$79.3$	$30.7$
LLaVA-NeXT	$78.1\pm 0.9$	$80.0$	$20.0$
InternVL2.5	$93.3\pm 0.5$	$89.3$	$2.0$

Table 1: Positive control: linear-probe accuracy (mean

\pm

std across three seeds; chance

=50\%

) and behavioural TB/FB accuracy on the belief question.

Qwen2-VL
Probe	Belief Q (%)	Intent Q (%)	Epist. Q (%)
belief	$15.3{\pm 6.2}$	$0.0$	$29.2{\pm 6.9}$
intent	$2.1{\pm 2.8}$	$0.0$	$0.0$
knowledge	$3.5{\pm 3.5}$	$0.0$	$0.0$
Gemma-4
belief	$33.3{\pm 7.6}$	$4.9{\pm 3.5}$	$29.9{\pm 7.6}$

Table 2: Per-scenario flip rate on

a_{1}

: of

144

scenarios, the fraction whose answer differs between

\alpha{=}{+}10

and

\alpha{=}{-}10

steering (i.e., whether steering moves the model at all , before changes average out). In the intent column (middle),

9

12

(VLM, probe) entries are exactly

0

; only three are above zero (Gemma-4 and InternVL2.5 on the belief probe, InternVL2.5 on the knowledge probe;

{\leq}15.3\%

), with direction-balanced flips that cancel in net accuracy ( Table 7 in the Appendix). The belief and knowledge columns flip many scenarios under the same hook.

Figure 3: Steering improves accuracy on the belief and knowledge predictions, but never on the intent answer prediction, averaged across the three protagonists (

a_{1}

a_{2}

a_{3}

). For each (VLM, question axis) we report the mean TB/FB accuracy with no steering (

\alpha{=}0

, grey) and with the best-case steering (

\alpha\in\{-10,+10\}

across all probe directions, in VLM color).

\Delta

on top is best

-

baseline. Belief and knowledge improve by

+2.7

+10.6

pp; the intent

\Delta

is essentially

0

on every VLM. The identical intent baseline (

\sim 66\%

) across VLMs reflects the constant-prior signature. Per-protagonist breakdown in Figure 6 in the Appendix.

Figure 4: CARD intent column on lever-operator $a_{1}$ is uniformly null (red box) across four VLMs. Rows: probe axis being steered. Columns: question axis being asked. Cell value: $|\text{gap}(\alpha{=}+10)-\text{gap}(\alpha{=}-10)|$ in pp, mean $\pm$ std across three seeds. Non-intent columns shift by tens of pp under the same probe; the intent column does not move. Same steering hook, same scenarios, same protagonist – only the intent prediction fails to read from the belief direction.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Protagonist	Role	Success	Counter	Total
$a_{1}$	lever-operator	$100$	$44$	$144$
$a_{2}$	lever-operator	$100$	$50$	$150$
$a_{3}$	traveller	$100$	$50$	$150$
Total		$\mathbf{300}$	$\mathbf{144}$	$\mathbf{444}$

Appendix

Table 3: Relay Chain dataset breakdown by protagonist and world-state outcome. All three axes (belief, intent, knowledge) share the same scenario base; each entry is a multi-frame trajectory with

4

–

8

keyframes.

Figure 5: Relay Chain scenario layouts for each protagonist condition (rows:

a_{1}

and

a_{2}

lever-holders,

a_{3}

traveller) under True-Belief and False-Belief layouts (columns). The protagonist is highlighted with a red outline; the shaded disk shows their field of view (radius

5

cells; anything outside is fog of war and not perceptible). In TB the protagonist stands next to its gate-controlling lever or just before the gate ahead; in FB it stands at the far lever or in a corner, so the field-of-view disk shifts with it and the cooperative event falls outside. All panels show the success block (both gates open) so the contrast is purely perceptual.

Axis	$t{=}0.25$	$t{=}0.5$	$t{=}0.75$	$t{=}1.0$	peak
belief	$69.9$	$69.1$	$69.3$	$69.3$	$100.0$
intent	$95.6$	$95.1$	$94.7$	$94.3$	$100.0$
knowledge	$91.9$	$91.2$	$90.8$	$90.4$	$100.0$

Appendix

Table 4: Linear-probe held-out accuracy (%) as a function of trajectory fraction

t

on Qwen2-VL, for the three cooperative-ToM axes (belief, intent, knowledge). “Peak” = best (layer, head) probe; the remaining columns are means across all (layer, head) cells. The peak is saturated at

100\%

from

t{=}0.25

onward. Mean accuracy decreases very slightly (

\sim 1

pp) with

t

: while the best head remains perfectly recoverable, fewer heads on average carry a clean axis-specific signal as later tokens accumulate downstream context. The relevant headline is the saturated peak: the feature is present from early in the trajectory.

Variant	Family	Description
V0	cost	baseline; canonical intent question, no prefix
V1	cost	implicit consequences: “if released too early, gate closes on traveller; team fails.”
V2	cost	explicit numerical cost: release $=$ cost $2.0$ if not crossed, hold $=$ cost $1.0$ if crossed
V3	cost	utility framing: “what action MAXIMISES team success?”
V4	CoT	step-by-step: perception $\to$ belief $\to$ cost $\to$ action
V5	CoT	high-stakes framing: “a wrong release may kill the team”

Appendix

Table 5: The

12

T-A prompt variants used to evaluate prompt-resistance. Each variant prepends a prefix to the canonical intent question; in-context variants additionally include worked examples.

VLM	Axis	Lever $a_{1}$	Lever $a_{2}$	Trav. $a_{3}$
Qwen2-VL	belief	100/0/100/100	100/50/100/100	100/0/100/100
	intent	0/100/100/100	0/100/100/100	53/47/100/100
	know.	100/100/48/100	100/69/72/100	53/100/44/98
Gemma-4	belief	100/0/100/100	69/52/100/100	100/4/100/100
	intent	0/100/100/100	0/100/100/100	100/0/100/100
	know.	100/100/59/100	100/100/58/100	0/100/44/100

Appendix

Table 6: Full D2 four-tuple discrimination matrix (success-TB / success-FB / counter-TB / counter-FB) for every (VLM, axis, protagonist).

VLM	Protag.	Probe $=$ belief	Probe $=$ intent	Probe $=$ knowledge
Qwen2-VL	$a_{1}$	$+0.0$	$+0.0$	$+0.0$
	$a_{2}$	$+0.0$	$+0.0$	$+0.0$
	$a_{3}$	$+4.0\pm 2.9$	$+0.0$	$+0.0$
Gemma-4	$a_{1}$	$+0.0$	$+0.0$	$+0.0$
	$a_{2}$	$+0.0$	$+0.0$	$+0.0$
	$a_{3}$	$+0.0$	$+0.0$	$+0.0$

Appendix

Table 7: Best-

\alpha

intent accuracy gain (pp) over the no-steering baseline (

\alpha{=}0

) for every (VLM, protagonist, probe direction) cell, reported as mean

\pm

std across three seeds (seeds

\{42,43,44\}

; the

\pm

is dropped when the std rounds to

0.0

). Most cells are exactly

0

on every seed; across the full multi-seed sweep,

95/108

cells are exactly

0

and the largest gain anywhere is

+12.00

%. Compare with belief and knowledge accuracies, which improve by

+2.7

+10.6

pp on the across-protagonist mean under the same hooks ( Figure 3 ): the intent prediction does not use the belief direction usefully on any cell.

VLM	Probe	Question	$\Delta_{\rm gap}$ (pp)	$\Delta_{\rm acc}$ (pp)
Qwen2-VL	belief	belief	$+30.56$	$-15.28$
	belief	intent	$+0.00$	$+0.00$
	belief	knowledge	$+22.22$	$+11.11$
	intent	belief	$-4.17$	$+2.08$
	intent	intent	$+0.00$	$+0.00$
	intent	knowledge	$+0.00$	$+0.00$

Appendix

Table 8: CARD per-cell breakdown on lever-holder

a_{1}

, reporting both the gap shift

\Delta_{\rm gap}={\rm gap}_{B}(\alpha{=}{+}10)-{\rm gap}_{B}(\alpha{=}{-}10)

and the mean-accuracy shift

\Delta_{\rm acc}=\overline{\rm acc}_{B}(\alpha{=}{+}10)-\overline{\rm acc}_{B}(\alpha{=}{-}10)

, where

\overline{\rm acc}_{B}=\tfrac{1}{2}({\rm TB}_{B}+{\rm FB}_{B})

. Bold rows are the intent-question column:

9

of the

12

cells are

\mathbf{0.00}

on both metrics across the four VLMs (Qwen2-VL and LLaVA-NeXT all three probes; Gemma-4 and InternVL2.5 under intent and knowledge probes), and the remaining three (Gemma-4 and InternVL2.5 under belief steering, InternVL2.5 under knowledge steering) shift by at most

|\Delta_{\rm gap}|{=}25.0

pp /

|\Delta_{\rm acc}|{=}13.9

pp, ruling out both belief-conditional modulation (

\Delta_{\rm gap}

) and any uniform answer-flip (

\Delta_{\rm acc}

). Sibling columns (belief, knowledge) show non-trivial shifts under the same probe directions.

Figure 6: Per-protagonist breakdown of Figure 3 .

4\times 3

grid: rows are VLMs, columns are protagonists (

a_{1}

a_{2}

lever-holders;

a_{3}

traveller). Each cell shows mean TB/FB accuracy at

\alpha{=}0

(grey) vs the best

\alpha\in\{-10,+10\}

across probe directions (VLM color);

\Delta

on top is best

-

baseline. The intent column

\Delta

is essentially

0

in every (VLM, protagonist) cell (

12/12

), while sibling answer heads (belief, knowledge) move on the protagonists that have headroom. The routing failure is therefore not protagonist-specific.

Figure 7: Qwen2-VL-7B – full CARD heatmap per protagonist. Red box = intent column.

Figure 8: Gemma-4-E4B – full CARD heatmap per protagonist. Red box = intent column.

Figure 9: LLaVA-NeXT-Video-7B – full CARD heatmap per protagonist. Red box = intent column.

Figure 10: InternVL2.5-8B – full CARD heatmap per protagonist. Red box = intent column.

Figure 11: T-A prompt-resistance test. Each row is one of $12$ prompt rewrites we tried (cost framing, chain-of-thought, perception-first, rule-based, $1$ / $2$ -shot examples. Each column is one (VLM, protagonist) cell. Cell colour encodes the TB $-$ FB accuracy gap – warm = model is more accurate when the protagonist can see the event, cold = the reverse. A star marks cells that cleared our pre-registered bar for “genuinely belief-conditional” (lever-holder, TB and FB accuracy both above $50\%$ ). Only $5$ cells star out of $192$ , and only one points the right way (TB $>$ FB), with a margin an order of magnitude smaller than the same models’ gap on the canonical belief question – no prompt rewrite recovers belief-conditional action.

Figure 12: Projection-removal

\alpha

sweep.

x

-axis:

\alpha\in\{0.5,1,2,5\}\sigma

removal of the belief direction from the residual stream.

y

-axis: intent TB

-

FB gap. Three lines per VLM:

a_{1}

a_{2}

a_{3}

. The intent gap is constant across

\alpha

on every VLM and every protagonist.

VLM	layers	heads	head_dim	hidden	rigidity
Qwen2-VL-7B	28	28	128	3584	hardest
LLaVA-NeXT-7B	32	32	128	4096	middle
Gemma-4-E4B	34	8	320	2560	softest
InternVL2.5-8B	32	32	128	4096	intermediate

Appendix

Table 9: Text-decoder architecture specs and empirical rigidity ordering of the four VLMs. Rigidity = resistance of the intent constant prior to prompt perturbation in the T-A sweep: Qwen2-VL is hardest (no V0–V11 cell breaks

30/100

a_{1}

); LLaVA-NeXT is intermediate ; Gemma-4 is softest (V2 cost-explicit breaks the prior on

a_{2}

70/86

, though not toward belief-conditional behaviour). The ordering matches loosely with head dimensionality (Gemma’s

320

vs.

128

for the others), but the dataset (

4

vendors) is too small to draw architectural conclusions; a quantitative regression of rigidity against pretraining mix and RLHF intensity is left to future work.

VLM	Question	Baseline	P=count	P=shape	P=colour	best
Qwen2-VL	count	$86$	$+6$	$+2$	$+6$	$\mathbf{+6}$
	shape	$100$	$+0$	$+0$	$+0$	$+0$ (ceil)
	colour	$100$	$+0$	$+0$	$+0$	$+0$ (ceil)
Gemma-4	count	$20$	$+26$	$+60$	$+42$	$\mathbf{+60}$
	shape	$30$	$+24$	$+58$	$+52$	$\mathbf{+58}$
	colour	$52$	$+16$	$+26$	$+22$	$\mathbf{+26}$

Appendix

Table 10: CARD positive control on CLEVR (single image,

100

scenarios per axis, seed

43

). “Baseline” = mean accuracy at

\alpha{=}0

on the corresponding within-axis cell; each probe column = best-

\alpha\in\pm 10

accuracy gain. Bold = the largest non-zero

\Delta_{\rm acc}

per (VLM, question). At least one cross-axis cell shows

\geq+14

pp gain on every VLM that has headroom (Qwen and InternVL are at

\geq 96\%

on shape/colour; Gemma and LLaVA are well below ceiling on every axis and steering produces

+6

+60

pp gains). The steering hook is functional; the Relay Chain intent null in F3 is not a hook-broken artefact.

VLM	Axis	True-label acc	Random-label acc	Selectivity
Qwen2-VL	belief	$98.1$	$47.6$	$+50.5$
	intent	$98.0$	$47.4$	$+50.6$
	knowledge	$98.0$	$47.9$	$+50.1$
Gemma-4	belief	$100.0$	$50.7$	$+49.3$
	intent	$100.0$	$52.0$	$+48.0$
	knowledge	$100.0$	$50.5$	$+49.5$

Appendix

Table 11: Random-label probe control ( Hewitt and Liang, 2019 ) . For each (VLM, axis), we retrain logistic-regression probes on the same paired activations with randomly permuted labels and report mean held-out accuracy across the sampled (layer, head) cells. True-label probes attain

91.6

–

100\%

; random-label probes are at chance (

47.4

–

52.0\%

), giving selectivity

\geq 43.0

pp on every axis and every VLM. The top single-cell random-label accuracy across all sampled cells is

56.0\%

, well below the true-label minimum of

91.6\%

Figure 13: CS-CARD-T on Qwen2-VL: intent

|\Delta_{\rm acc}|

vs steering target position, one line per probe source. All nine (source, position) entries sit at

\leq 0.3

pp.

VLM	overall orig (pp)	overall flip (pp)	sign-preserved?
Qwen2-VL	+86.5	+82.4	yes (oracle)
Gemma-4	+73.0	+85.1	yes (oracle)
LLaVA-NeXT	+63.5	+35.1	weakened (drift)

Appendix

Table 12: D1 polarity-flip diagnostic. Original and flipped question polarity TB

-

FB gap on the canonical

148

hold-out. An oracle-reasoning model preserves the sign and magnitude under flip; a constant-prior model collapses to a smaller or sign-flipped gap.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
Sep 21, 2026Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak +4Vision-Language Models VLM Interpretability
University of Technology Nuremberg. · ETH Zürich. · Inria, Ecole Normale Sup´erieure, CNRS, PSL Research University.
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University
The Linear Representation Hypothesis for Vision-Language-Action Models
Sep 25, 2026Minseok Jeong, Hyewon Choi, Hiroyasu Tsukamoto +1Vision-Language-Action Models
KAIST · University of Illinois Urbana–Champaign

Sep 21, 2026J/K move · Enter open · S save

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior

Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak, Marco Hutter+3

University of Technology Nuremberg. · ETH Zürich. · Inria, Ecole Normale Sup´erieure, CNRS, PSL Research University.

Vision-Language Models VLM Interpretability Vision-Language Navigation Activation Steering

Open paper AskPDF arXiv

Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While this fusion of language instructions and visual observations allows multimodal reasoning, it obscures how information is routed across modalities or what mechanisms drive navigation decisions. Thus, it remains unclear whether VLN models ground their predictions in relevant semantic cues or can track task progress. In this work, we study the interpretability and steerability of VLN models. We use intervention-based metrics that measure how visual observations, instructions, and visual memory causally influence navigation decisions. Our results show that these navigation policies are sensitive to all input modalities and do not depend on a single one. We further show that these agents encode navigation progress and retain semantic structure from their VLM backbones, enabling concept-level steering through internal activations. Finally, we extract activation vectors for abstract behaviors to transfer them zero-shot to out-of-distribution real-world scenarios, improving performance without additional fine-tuning.

Abstract

VLM Reasoning

Explore similar work

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

The Linear Representation Hypothesis for Vision-Language-Action Models