From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making

Davide Testa; Hugh Mee Wong; Alessandro Lenci; Bernardo Magnini; Albert Gatt

cs.CLSep 4, 2026

From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making

Authors: Davide Testa, Hugh Mee Wong, Alessandro Lenci, Bernardo Magnini, Albert Gatt

Organizations: Fondazione Bruno Kessler (FBK) - Trento, Italy · Università di Roma La Sapienza - Rome, Italy · Utrecht University - Utrecht, The Netherlands · University of Pisa - Pisa, Italy

Abstract

Vision-Language Models are commonly evaluated through their final predictions, but understanding whether these decisions are grounded in visual evidence requires tracing how visual information contributes to language-based decisions. With this purpose in mind, we investigate cross-modal information flow in a video-based generative multiple-choice-like setting by applying a layer-wise causal intervention on video-text attention pathways. We target spatial, causal, and temporal visual reasoning. Our results show that visual information is mainly integrated while the model processes the candidate answer options, which serve as the primary textual grounding sites for the final decision. We further show that nouns play an important role as semantic anchors during multimodal enrichment, while verbs are more relevant when temporal relations are processed. Finally, we identify a distinct pattern in temporal reasoning, suggesting that VLMs struggle to reconstruct sequential information across video frames, but we remark that such fragility may also reflect linguistic biases associated with specific temporal expressions used for defining the relation between events within a scene.

Figures & tables

Figure 1: Overview of the Attention Knockout approach with prompt structure and region-based masking.

Dataset	Option type	Sentence
Spatial	Caption	At the end of the scene, the little girl walks on a carpet .
Spatial	Foil	At the end of the scene, the little girl walks on a path .
Temporal	Caption	Right after cutting the pizza, the girl hands it to a person in front of her.
Temporal	Foil	Before cutting the pizza, the girl hands it to a person in front of her.
Causal	Caption	The girl is upset because of a cockroach .
Causal	Foil	The girl is upset because of a loud noise .

Table 1: Caption–foil examples from MAIA subsets (English translations; originals in Table 5 , Appendix B ).

Model	Dataset	Acc ${}_{\text{ A / B}}$	Acc ${}_{\text{ caption / foil}}$
Qwen2.5-VL	Spatial	0.69	0.67
	Temporal	0.77	0.43
	Causal	0.84	0.81
LLaVA-OneVision	Spatial	0.69	0.68
	Temporal	0.73	0.44
	Causal	0.83	0.81

Table 2: Behavioral performance on the caption–foil task for the three MAIA subsets, under two scoring schemes: A/B labels vs. full candidate sentences.

Figure 2: Region-based attention knockout results. Rows indicate the textual prompt region prevented from attending to visual tokens (i.e., Vision $\rightarrow$ Region ). Colors show the layer-wise caption–foil log-probability difference. Text-only and None condition represent our two baselines.

Model	Dataset	text-only Vs. vision		vision $\to$ question		vision $\to$ content(A)		vision $\to$ content(B)		vision $\to$ user_text		vision $\to$ assistant
Model	Dataset	Acc	Acc	Flip rate	$\Delta$ Acc	Flip rate	$\Delta$ Acc	Flip rate	$\Delta$ Acc	Flip rate	$\Delta$ Acc	Flip rate	$\Delta$ Acc
Qwen2.5-VL	Spatial	0.54	0.67	8.19%	$+0.06$	17.62%	$-5.0$	15.0%	$-2.62$	26.38%	$-11.75$	8.38%	$-0.37$
	Temporal	0.39	0.43	8.25%	$-1.50$	14.62%	$-6.5$	12.12%	$-4.5$	18.75%	$-5.13$	6.06%	$-0.69$
	Causal	0.62	0.81	4.69%	$-0.81$	14.62%	$-7.87$	10.88%	$-4.87$	21.62%	$-15.50$	4.38%	$+0.25$
LLaVA-OneVision	Spatial	0.55	0.68	8.06%	$-0.56$	24.62%	$-7.87$	16.50%	$-6.50$	23.75%	$-10.63$	4.5%	$-1.38$
	Temporal	0.38	0.44	9.25%	$-4.87$	18.88%	$-0.87$	10.81%	$-0.44$	23.81%	$-10.56$	5.62%	$-1.00$

Table 3: Final-layer results for the text-only baseline and region-based Attention Knockout. We report accuracy for text-only and, for each knockout condition, Flip Rate and

\Delta

Accuracy relative to the unmasked none condition. Flip Rate is the proportion of changed predictions (

C\to F

F\to C

), while

\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{condition}}-\mathrm{Acc}_{\mathrm{none}}

is the accuracy change in percentage points; negative values indicate that the intervention harms performance.

Figure 3: Layer-wise effect of token-level attention knockout on (causal) candidate log-probability.

Figure 4: Layer-wise effect of token-level attention knockout on (spatial) candidate log-probability.

Figure 5: Layer-wise effect of token-level attention knockout on (temporal) candidate log-probability.

Temporal marker	$N$	Text-only baseline		Vision (None)
Temporal marker	$N$	Qwen	LLaVA	Qwen	LLaVA
After	443	0.42	0.36	0.56	0.65
Before	115	0.82	0.75	0.70	0.40
Beginning / Start	42	0.45	0.88	0.60	0.86
End	106	0.59	0.35	0.71	0.62
When	195	0.42	0.27	0.41	0.28

Table 4: Last-layer accuracy by temporal marker.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Overview of the Attention Knockout approach with inverted Text-Video prompt used for the text-to-vision masking direction.

Figure 7: Region-based attention knockout results. Rows indicate the visual prompt region prevented from attending to textual tokens (i.e., Text $\rightarrow$ Region ). Colors show the layer-wise caption-foil log-probability difference.

Figure 8: Region-based attention knockout results for English translations of MAIA subdatasets. Rows indicate the textual prompt region prevented from attending to visual tokens (i.e., Vision $\rightarrow$ Region ). Colors show the layer-wise caption-foil log-probability difference.

Figure 9: Layer-wise effect of token-level attention knockout on (causal) candidate log-probability. English data

Figure 10: Layer-wise effect of token-level attention knockout on (spatial) candidate log-probability. English data.

Figure 11: Layer-wise effect of token-level attention knockout on (temporal) candidate log-probability. English data.

Dataset	Option type	Sentence
Spatial	Caption	Alla fine della scena, la bambina sta camminando su un tappeto .
Spatial	Foil	Alla fine della scena, la bambina sta camminando per strada .
Temporal	Caption	Subito dopo aver tagliato la pizza, la ragazza la porge alla persona che ha di fronte.
Temporal	Foil	Prima di tagliare la pizza, la ragazza la porge alla persona che ha di fronte.
Causal	Caption	La ragazza è infastidita a causa di uno scarafaggio .
Causal	Foil	La ragazza è infastidita a causa di un forte rumore .

Appendix

Table 5: Original Italian examples of caption-foil pairs from the three MAIA subsets.

Figure 12: $\Delta$ log-probability (caption $-$ foil) across layers and token regions on causal items, split by model outcome: correct predictions ( $C>F$ , left) and incorrect predictions ( $F>C$ , right).

Figure 13: $\Delta$ log-probability (caption $-$ foil) across layers and token regions on spatial items, split by model outcome: correct predictions ( $C>F$ , left) and incorrect predictions ( $F>C$ , right).

Figure 14: $\Delta$ log-probability (caption $-$ foil) across layers and token regions on temporal items, split by model outcome: correct predictions ( $C>F$ , left) and incorrect predictions ( $F>C$ , right).

Explore similar work

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Apr 16, 2026Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras +1CoT Faithfulness Vision-Language Models
University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science
Where To Look? : Causal Tracing of Vision Encoders in VLM
Aug 11, 2026Naren Kumar S, Tirth Bhatt, Mayank SinghLarge Vision-Language Models VLM Interpretability
LINGO Research Group, Indian Institute of Technology Gandhinagar, India
Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
Oct 14, 2026Josh Barua, Kyle Mahowald, William RudmanVision-Language Models Modality Gap in VLMs
The University of Texas at Austin

Apr 16, 2026J/K move · Enter open · S save

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott

University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science

CoT Faithfulness Vision-Language Models VLM Interpretability VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear. We analyze reasoning dynamics in 18 VLMs covering instruction-tuned and reasoning-trained models from two different model families. We track confidence over Chain-of-Thought (CoT), measure the corrective effect of reasoning, and evaluate the contribution of intermediate reasoning steps. We find that models are prone to answer inertia, in which early commitments to a prediction are reinforced, rather than revised during reasoning steps. While reasoning-trained models show stronger corrective behavior, their gains depend on modality conditions, from text-dominant to vision-only settings. Using controlled interventions with misleading textual cues, we show that models are consistently influenced by these cues even when visual evidence is sufficient, and assess whether this influence is recoverable from CoT. Although this influence can appear in the CoT, its detectability varies across models and depends on what is being monitored. Reasoning-trained models are more likely to explicitly refer to the cues, but their longer and fluent CoTs can still appear visually grounded while actually following textual cues, obscuring modality reliance. In contrast, instruction-tuned models refer to the cues less explicitly, but their shorter traces reveal inconsistencies with the visual input. Taken together, these findings indicate that CoT provides only a partial view of how different modalities drive VLM decisions, with important implications for the transparency and safety of multimodal systems.

Abstract

Explore similar work

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

Where To Look? : Causal Tracing of Vision Encoders in VLM

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers