Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.
Figures & tables
Figure 1: Causal intervention on the VLM residual stream. By identifying the OCR direction via PCA on activation deltas (left) and projecting it out at the bottleneck (center), OCR capability is effectively suppressed while spatial reasoning and counting tasks are functionally separated and preserved (right).
Figure 2: Example original-inpainted pairs from EgoTextVQA. Left: original images with naturally-occurring text. Right: inpainted versions with text regions removed. Activation differences between pairs capture the “OCR signal.”
Model
Intervention
Count
Emb
RWQ
Qwen3-VL-4B
baseline
91.4
81.5
75.0
L17-19_pc5
95.7
79.5
76.0
Qwen3-VL-8B
baseline
90.2
82.0
72.0
L17-19_pc5
93.1
79.0
58.0
Qwen3-VL-2B
baseline
90.3
78.0
66.0
L12_pc5
95.9
72.0
60.0
Table 1: Cross-model retention results (%). Count=CountBench, Emb=EmbSpatial, RWQ=RealWorldQA. Qwen3-VL-4B achieves favorable trade-off; smaller/single-stage models show trade-offs or degradation. Qwen3-VL-8B replicates the 4B pattern at L17.
Intervention
Count
Emb
RWQ
Note
baseline
91.4
81.5
75.0
—
Single layer:
pca_L16_pc5
94.1
81.0
71.0
Count ↑ , RWQ ↓
pca_L17_pc1
90.3
80.5
73.0
Minimal
pca_L17_pc3
94.1
78.5
73.0
Trade-off
pca_L17_pc5
96.2
78.5
74.0
Good
Table 2: Complete Qwen3-VL-4B retention results (%). L17-19_pc5 achieves the best trade-off: Count +4.3pp, RWQ +1pp, with mild EmbSpatial degradation ( − 2pp).
EgoTextVQA
OCRBench
InfoVQA
Model
Drop
Layer
Drop
Layer
Drop
Layer
Qwen3-VL-4B
− 69.6
L18 (50%)
− 30.8
L30 (83%)
− 22.0
L4 (11%)
Qwen3-VL-2B
− 62.4
L26 (93%)
− 20.0
L26 (93%)
− 72.0
L26 (93%)
Phi-4
− 44.3
L8 (25%)
− 4.0
L1 (3%)
− 18.0
L10 (31%)
InternVL
− 75.3
L2 (6%)
− 69.0
L2 (6%)
− 29.0
L17 (49%)
Table 3: Cross-dataset generalization. Maximum accuracy drop (pp) when applying EgoTextVQA-learned PCA directions. Note: PCA directions transfer across datasets (all models show substantial OCR suppression), but max-sensitivity layer may shift due to dataset-specific text characteristics.
Figure 3: Small multiples view: unnormalized (absolute layer) comparison across all models. Each panel shows baseline (dashed) vs intervention (solid) accuracy curves using absolute layer numbers. Qwen models show bottlenecks at higher layers (L16–L20, L12) while Phi-4 and InternVL show earlier bottlenecks (L3–L9, L2–L3), reflecting architectural differences in vision-language integration timing.
Figure 4: Cross-dataset generalization of OCR suppression. Each panel shows one model’s accuracy delta across layers for all three datasets. PCA directions learned on EgoTextVQA (green) transfer effectively to OCRBench (blue) and InfoVQA (red), demonstrating that learned OCR representations generalize beyond the training distribution.
Figure 5: OCR bottleneck location by architecture and dataset. Each panel shows mean accuracy delta across three network depth stages. EgoTextVQA and OCRBench show consistent patterns: InternVL3.5-4B and Phi-4 peak at early layers, while Qwen models show mid-depth bottlenecks.
Model
Integration
Bottleneck
Qwen3-VL-4B
DeepStack
L16–L20
Qwen3-VL-2B
DeepStack
L8–10, L25–26
Phi-4
Single-stage
L3–L9
InternVL3.5-4B
Single-stage
L2–L3
Table 4: Cross-architecture OCR bottleneck analysis. DeepStack models (Qwen) show mid-depth bottlenecks; single-stage projection models (Phi-4, InternVL) show early bottlenecks.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
PCA Var.
OCR Drop
LaMa (default)
72.9%
− 69.6pp
SDXL Inpaint
71.2%
− 67.8pp
Simple blur
68.4%
− 64.2pp
Random boxes
12.3%
− 0.1pp
Appendix
Table 5: Inpainting artifact controls. Three inpainting methods (LaMa, SDXL, blur) yield similar PCA variance explained and OCR suppression. Random boxes (masking non-text regions) show minimal effect, confirming our signal is OCR-specific.
Experiment
Layers
Comp.
OCR Drop
Count Base
Count Δ
Outcome
pca_L16-20_pc3
L16–20
3
76.3pp
91.4%
+6.9pp
Ideal
pca_L17_pc5
L17
5
53.6pp
91.4%
+5.3pp
Good
pca_L16_pc5
L16
5
13.4pp
91.4%
+4.8pp
Good
pca_L17-19_pc5
L17–19
5
56.2pp
91.4%
+4.3pp
Best
pca_L20_pc5
L20
5
29.9pp
91.4%
+3.9pp
Good
pca_L17-19_pc3
L17–19
3
58.2pp
91.4%
+2.9pp
Good
Appendix
Table 6: Complete PCA intervention results sorted by Count Δ . Multi-layer interventions that avoid L17 alone achieve the best trade-offs.
Rank
Layer
Head
Selectivity Ratio
Power
1
16
4
1.41
0.55e-3
2
18
15
1.40
0.85e-3
3
8
8
1.40
0.26e-3
4
12
1
1.34
0.33e-3
5
17
13
1.27
0.68e-3
6
16
12
1.25
0.48e-3
Appendix
Table 7: Top 20 OCR-selective attention heads. Selectivity ratio = OCR attention / background attention. Heads concentrate in L12, L16–L20.
Intervention
OCR Drop
Count
Emb
RWQ
baseline
—
78.9
69.2
64.0
Bottleneck:
pca_L2_pc1
68.0pp
69.2
25.0
42.0
pca_L2_pc5
54.6pp
29.2
4.0
6.0
pca_L3_pc5
23.2pp
77.8
47.0
49.0
pca_L2-3_pc1
76.8pp
69.2
15.0
40.0
Appendix
Table 8: InternVL retention results (%). Bottleneck interventions (L2–L3) cause OCR drop and retention collapse.