Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null

Genpei Zhang

cs.CVSep 1, 2026

Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null

Authors: Genpei Zhang

Organizations: University of Wisconsin–Madison

Abstract

Across three vision-language model architectures (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3-8B), we report a universal negative finding for mid-layer interpretability. On POPE -- the benchmark common to all three -- the mid layers encode the ground-truth answer in 68-91% of errors, yet this signal is not causally active for the final prediction: residual-stream patching yields 0% non-trivial flip at the layer level on all three architectures, and on two of three at the per-head level (Qwen: 0/12,600 patched forwards). The lone exception, InternVL3 layer-20 head-2, is a non-vocab, self-attending head whose effect is localized to that specific head (p < 1e-4). Despite the null, the errors separate operationally into three failure modes -- Perception Failure, Encoded-but-Disconnected, Prior-Override -- learnable above 60% on all three architectures, and the architecture's prior direction predicts which of two interventions elicits a category-specific response. We report these mitigation effects under oracle labels as evidence the categories are mechanistically real, not as a deployable method.

Figures & tables

		Phenomenology		Patching null		Diagnostic
Architecture	Family	EAS (POPE)	Peak band	Layer level	Per-head	CV acc	Target flip
LLaVA-1.5-7B	CLIP + LLaMA-2	74.5%	L12–L13 ( $\tfrac{1}{3}$ )	$0/50$	$1/32$ (noise)	$82.5\pm 3.8$	EBD $85.9$ %
Qwen2.5-VL-7B	SigLIP + Qwen2	91.0%	L10–L17 ( $\tfrac{1}{2}$ )	$0/44$	$0/12{,}600$	$68.0\pm 3.9$	PO $19.6$ %
InternVL3-8B	InternViT + Qwen2	68.5%	L18–L21 ( $\tfrac{2}{3}$ )	$1/50$	L20 H2: 26%	$79.7\pm 3.2$	PO $21.3$ %

Table 1: Three-architecture summary of all main-paper results. EAS (POPE) : mid-layer top-1 hit rate on POPE errors. Peak band : layers of strongest signal (relative depth). Patching null : per-pair non-trivial flip rate at the layer and per-head levels; InternVL3 L20 H2 is the only tuple rejecting the null. CV acc : 5-fold accuracy of the 24-feature classifier on PF/EBD/PO (logistic regression). Target flip : flip rate of the targeted class under oracle labels. Details in the appendices.

Figure 1: Per-category mid-layer top-1 margin distributions across the three architectures (LLaVA L13, Qwen L12, InternVL3 L20). The three failure categories separate by margin shape—broad-tailed (PF), low (EBD), tight near zero (PO)—which is the signal the 24-feature classifier exploits.

Figure 2: Bimodal POPE failure-mode separation across 5 model $\times$ benchmark cells: pred=No errors are $95$ – $100\%$ EAS-positive (bottom), pred=Yes are $<5\%$ (top). The split is fixed by prediction direction; the non-trivial content is that the two halves also differ in mid-layer signal.

Figure 3: Per-head patching flip-rate heatmaps at the last-text position across the three architectures (shared vmax $30\%$ ). The only cell exceeding the layer-level null upper CI is InternVL3 L20 H2 (circled, panel c); LLaVA and Qwen are essentially flat at zero.

Figure 4: InternVL3 layer 20 head 2 mechanism triplet (the per-head exception of Figure 3 ). (a) 5-variant sanity sweep (baseline vs. random, zero, different head, different layer). (b) Vocab projection: yes/no tokens absent from the head’s top-5. (c) Attention:

96\%

of mass on its own position. Causal but not vocab-encoding, and self-attending.

Arch.	Strategy	flip(PO)	flip(EBD)	flip(PF)
LLaVA	Re-Promote	$0.0$	$\mathbf{85.9}$	$85.7$
Qwen	Debias	$\mathbf{19.6}$	$0.0$	$0.0$
InternVL3	Debias	$\mathbf{21.3}$	$3.6$	$0.0$

Table 2: Per-category flip rates (%) under each architecture’s matched intervention; bold marks the targeted class. Counts

k/n

: LLaVA

0/51

122/142

6/7

; Qwen

31/158

0/24

0/18

; InternVL3

19/89

2/56

0/55

. Results are conditional on the operational label, not end-to-end; LLaVA’s PF (

6/7

) shows Re-Promote is not EBD-specific (see text). Debias and Re-Promote rows use separate runs.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Value	Source
LLaVA-1.5-7B SHA	b234b804	HuggingFace
Qwen2.5-VL-7B SHA	pinned at camera-ready †	HuggingFace
InternVL3-8B SHA	pinned at camera-ready †	HuggingFace
LLaVA precision	fp16	sdpa attn
Qwen precision	fp16	sdpa attn
InternVL3 precision	bfloat16	eager attn

Appendix

Table 3: Model and dataset configuration. “Prior direction” refers to the direction the model defaults to on POPE-style binary prompts, used in the Prior-Override category definition (Method). † The exact commit SHAs for Qwen2.5-VL-7B and InternVL3-8B were not frozen at submission and will be pinned in the camera-ready; all reported runs use a single fixed local checkout per model. InternVL3 runs in bfloat16 with eager attention (fp16 produced NaN logits on this architecture); this is a precision/stability choice and does not affect any qualitative conclusion—the layer-level patching null holds identically across all three models regardless of precision.

Architecture	LM layers	Scan window	Peak band (depth)	Phase transition
LLaVA-1.5-7B	$32$	L12–L24	L12–L13 ( $\sim\!\tfrac{1}{3}$ )	L18 $\to$ L19 ( $57.7\%\!\to\!98.5\%$ )
Qwen2.5-VL-7B	$28$	L10–L21	L10–L17 ( $\sim\!\tfrac{1}{2}$ )	gradual (no single-layer jump)
InternVL3-8B	$28$	L6–L26	L18–L21 ( $\sim\!\tfrac{2}{3}$ )	gradual (no single-layer jump)

Appendix

Table 4: Scan window versus peak band. The scan window (wider) defines the EAS condition; the peak band (narrower) carries the strongest signal and fixes the relative depth, computed against the language-model layer count. This separation resolves the apparent discrepancy between the wide InternVL3 window (L6–L26) and its

\sim\!\tfrac{2}{3}

peak depth (L18–L21). LLaVA alone shows a sharp single-layer phase transition (

+40.8

points in top-5 hit rate at L19, measured on correct samples); the others rise gradually. Top-5 phase-transition values are available for LLaVA only (Appendix H ).

Variant	Description	Flip rate	Fisher $p$ vs. baseline
Baseline	clean L20 H2 patch	$32\%$ ( $16/50$ )	—
Random	norm-matched gaussian	$14\%$ ( $7/50$ )	$0.056$
Zero	zero vector	$10\%$ ( $5/50$ )	$0.013$
Diff. head	L20 H3 (same layer)	$0\%$ ( $0/50$ )	$<10^{-4}$
Diff. layer	L19 H2 (same head idx)	$6\%$ ( $3/50$ )	$0.0017$

Appendix

Table 5: L20 H2 five-variant sanity sweep on an independent

50

-pair strict sample. The flip rate collapses under same-layer different-head substitution (

0/50

) and different-layer substitution, establishing

(\text{layer},\text{head})

locality; the borderline random comparison (

p=0.056

) indicates magnitude-sensitivity, which is why we describe the head conservatively (causal, non-vocab, self-attending) rather than assigning it a specific computational role.

Group	Features
Per-layer logits ( $\times 6$ layers)	top-1 / top-2 / top-3 logit value
Per-layer margin ( $\times 6$ layers)	top-1 minus top-2 logit
Final-layer logits	top-1 / top-2 logit value
Final-layer margin	top-1 minus top-2 logit

Appendix

Table 6: Feature groups for the

24

-dimensional classifier. All features come from the same forward pass that produces the prediction. The label-defining indicators (EAS-hit, prior-direction) and raw token identities are deliberately excluded (see text).

Architecture	PF	EBD	PO
LLaVA-1.5-7B	$0$ ( $0\%$ )	$148$ ( $74\%$ )	$52$ ( $26\%$ )
Qwen2.5-VL-7B	$24$ ( $12\%$ )	$18$ ( $9\%$ )	$158$ ( $79\%$ )
InternVL3-8B	$100$ ( $50\%$ )	$16$ ( $8\%$ )	$84$ ( $42\%$ )

Appendix

Table 7: Per-class counts and proportions of POPE errors (

n=200

per architecture), under the EAS scan window of Appendix B . Proportions are window-dependent; the mitigation runs of Per-Category Intervention use each script’s own window and therefore report slightly different counts, which we note where relevant.

Classifier	LLaVA-1.5-7B	Qwen2.5-VL-7B	InternVL3-8B
LR (primary)	$\mathbf{82.5\pm 3.8}$	$\mathbf{68.0\pm 3.9}$	$\mathbf{79.7\pm 3.2}$
GBM ( $n=100$ )	$84.2\pm 2.2$	$70.3\pm 2.7$	$76.3\pm 4.1$
Majority-class	$74.0$	$79.0$	$50.0$

Appendix

Table 8:

5

-fold stratified CV accuracy (%, mean

\pm

std across folds) on the PF/EBD/PO labels—the taxonomy used throughout the main paper, including routing. Primary classifier (LR) bolded. The majority-class baseline is the largest-class proportion (Table 7 ); on the prior-dominated Qwen this baseline (

79\%

) is itself high, so we report the prediction-direction control separately below rather than treating raw accuracy as the headline.

Input	LLaVA	Qwen	InternVL3
$\{$ EAS-hit, prior-dir $\}$ oracle (2 bits)	$100$	$100$	$100$
$24$ continuous features (deployed)	$82.5$	$68.0$	$79.7$
prediction-direction bit only	$62.5$	$94.0$	$93.0$
final-margin only (1 continuous)	$48.5$	$72.3$	$52.3$

Appendix

Table 9: Trivial-baseline control. The label-defining bits trivially give

100\%

; the deployed continuous features fall well short, confirming the classifier is not reconstructing its own label. On prior-dominated architectures a single direction bit is already strong, so the continuous features add the most where the label is not collinear with prediction direction (LLaVA,

+20

points).

Architecture	Classifier	Real	Random	Gap
LLaVA-1.5-7B	LR	$69.0$	$35.0$	$34.0$
	GBDT $n100$	$76.2$	$40.5$	$35.7$
	RF $n300$	$75.8$	$47.5$	$28.3$
Qwen2.5-VL-7B	LR	$59.3$	$38.3$	$21.0$
	GBDT $n100$	$66.0$	$32.3$	$33.7$
	RF $n300$	$66.7$	$35.7$	$31.0$

Appendix

Table 10: Random-feature ablation under the balanced margin-strength partition (used only here, not as the main-paper taxonomy):

5

-fold stratified CV accuracy (%) with deployed lens-derived features versus i.i.d. gaussian noise of identical shape, same splits/seed/hyperparameters. Real beats random by

21

–

43

points on every cell. The above-chance random RF on LLaVA/InternVL3 (

47

–

52\%

) reflects forest memorization of residual class structure; the random LR collapses to

32

–

38\%

Config	flip(PO)	flip(EBD)	flip(PF)
L12 $\alpha=0.5$	0.0%	54.9%	42.9%
L12 $\alpha=1.0$	0.0%	85.9%	85.7%
L12 $\alpha=1.5$	0.0%	96.5%	100%
L12 $\alpha=2.0$	0.0%	100%	100%
L13 $\alpha=1.0$	0.0%	90.9%	85.7%

Appendix

Table 11: Re-Promote sweep on LLaVA (

n=200

POPE errors). PO flip is

0\%

at every config; the EBD flip rate saturates near

100\%

above

\alpha\geq 1.5

but at the cost of also flipping the small PF population. Selected operating point bolded.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 268 papers

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models
Jul 6, 2026Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov +2VLM Evaluation Vision-Language Models
1MBZUAI · 2The University of Melbourne
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language Grounding Visual Evidence Sensitivity in VLMs
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Where To Look? : Causal Tracing of Vision Encoders in VLM
Aug 11, 2026Naren Kumar S, Tirth Bhatt, Mayank SinghLarge Vision-Language Models VLM Interpretability
LINGO Research Group, Indian Institute of Technology Gandhinagar, India

Jul 6, 2026J/K move · Enter open · S save

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov, Timothy Baldwin+1

1MBZUAI · 2The University of Melbourne

VLM Evaluation Vision-Language Models

Open paper AskPDF arXiv

Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.

Abstract

VLM Interpretability

Explore similar work

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Where To Look? : Causal Tracing of Vision Encoders in VLM