From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists

Yuanxiang Huangfu; Hanmeng Zhong; Linqing Chen; Jeffrey Tiong Jee Hui

cs.CVSep 18, 2026

From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists

Authors: Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen, Jeffrey Tiong Jee Hui

Organizations: PatSnap Co., LTD. Suzhou, China

Abstract

Does a general vision--language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677-0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17-20 of the top 20 heads per task with all-head rank correlations of 0.874-0.942, while redistributing their functional and causal strengths.

Figures & tables

Figure 1: Token-level visual attention during Qwen2-VL-2B ( Wang et al., 2024 ) OCR for generated tokens (a) “Checklist” and (b) “yellow.” Columns compare the top text-OCR head, the retrieval head L22H05, and a head outside both rankings. Blue boxes mark target evidence, and all heatmaps use one absolute scale.

Figure 2: Evidence-grounded discovery and causal validation. Visible characters, cell text, and formula glyphs define evidence patches

E(z)

; only whole BPE tokens in exact free-generation match blocks are eligible. A head scores a hit when its global attention argmax falls inside the token evidence. Page-disjoint discoveries are then tested by held-out intervention against matched random sets.

Figure 3: Structural correspondence between independently identified OCR and retrieval/copy heads in Qwen2-VL-2B. Each point represents the same head’s retrieval and task-specific OCR rank percentiles. Orange points belong to both top-20 sets, blue rings to exactly one, and gray points to neither. Shaded bands mark the top-20 ranges. $\rho$ denotes all-head Spearman correlation; ranks below the 50th percentile are compressed for readability.

Task	#Pairs	Target source	Matched heads	Non-target source
Text	184	4.476	0.658	0.004
Table	105	2.425	0.812	-0.003
Formula	128	2.498	0.168	-0.003

Table 1: Counterfactual value patching in Qwen2-VL-2B. Values are donor-directed next-token logit shifts. Target uses OCR–retrieval overlap heads at the target evidence; Random uses matched random heads; Non-target uses the same overlap heads at unrelated visual positions.

Figure 4: Qwen2-VL OCR specialization preserves 18/20 top heads in each task. Bar height shows identity retention;

\rho

is the all-head Spearman correlation, and mass is the pre

\rightarrow

post top-20 OCR-score share.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Prompt
Text	Perform OCR on this image. Output only the visible text in natural reading order, verbatim. Preserve case, punctuation, symbols, spacing, and line breaks. No commentary, labels, Markdown, or code fences.
Table	Transcribe the table into compact HTML. Return only one <table>...</table> element with rows in reading order. Use <tr> and <td> or <th>; preserve every cell’s text exactly and use rowspan or colspan only when visually required. Use no other attributes, prose, Markdown, or code fence.
Formula	Read only the mathematical expression shown. Respond with its exact LaTeX transcription between $$ delimiters. Keep fractions, roots, superscripts, subscripts, accents, relations, brackets, and punctuation. Add no explanation.

Appendix

Table 2: Complete deployment prompts for untuned Qwen2-VL, selected by prompt calibration.

Task	Prompt
Text	Perform OCR on this image. Output only the visible text in natural reading order, verbatim. Preserve case, punctuation, symbols, spacing, and line breaks. No commentary, labels, Markdown, or code fences.
Table	Perform table OCR and output only canonical HTML beginning with <table> and ending with </table>. Preserve row and column order and all visible cell text. Use only table, tr, td, and th tags plus necessary rowspan/colspan attributes. Do not add commentary or a code block.
Formula	Transcribe the displayed mathematical formula exactly into LaTeX. Return one LaTeX string only, enclosed in $...$ . Preserve all symbols, operators, scripts, delimiters, and visible terminal punctuation. Do not explain and do not use a code fence.

Appendix

Table 3: Deployment prompts for untuned Qwen3-VL. The text prompt is identical to that of Qwen2-VL (Table 2 ).

Model	OCR task	Retrieval heads	Top-20 overlap	Spearman	OCR in $R$
Qwen2-VL	Text	31	13/20	0.846	92.3%
	Table	31	17/20	0.886	93.8%
	Formula	31	15/20	0.677	85.7%
Qwen3-VL	Text	30	15/20	0.835	74.1%
	Table	30	15/20	0.859	84.6%
	Formula	30	13/20	0.806	83.3%

Appendix

Table 4: Retrieval/copy reuse in untuned generalist VLMs. “OCR in

R

” is threshold-set containment; all top-20 overlaps exceed architecture-specific random expectation with hypergeometric

p<10^{-11}

Figure 5: Head-wise OCR scores for text, tables, and formulas in three OCR specialists. High-scoring heads are sparse and architecture dependent, with task-dependent variation in their locations and strengths.

Task	TF $\rho$	TF top-4	FG top-20
Text	0.9931	4/4	20/20
Table content	0.9951	4/4	20/20
Formula	0.9944	4/4	19/20

Appendix

Table 5: GLM-OCR cross-fold discovery stability. TF is the teacher-forced diagnostic and FG is free-generation replay.

Model	Text only	Table only	Formula only	Text–Table only	Table–Formula only	All three
GLM-OCR	4	2	7	5	2	11
MinerU2.5	3	1	2	1	2	16
PaddleOCR-VL-1.6	3	0	3	3	3	14

Appendix

Table 6: Exact Venn partitions of the task-specific top-20 OCR-head sets. Pairwise-only columns contain heads shared by exactly the named two tasks, excluding the third.

		Top-20 ablation		Random ablation
Model	Base	Score	$\Delta$	Score	$\Delta$
Text (NED)
GLM-OCR	0.033	0.337	$+0.303$	0.086	$+0.053$
MinerU2.5	0.027	0.070	$+0.043$	0.026	$-0.001$
PaddleOCR-VL-1.6	0.021	0.621	$+0.600$	0.023	$+0.002$
Table (TEDS)

Appendix

Table 7: Held-out causal validation in specialized OCR models. Top-20 and Random denote the targeted and matched-random ablation conditions, respectively. Score is the performance after intervention, and

\Delta

is the signed change from the intact baseline. Random results are averaged over five head sets exactly matched to the target set’s per-layer head counts for all three tasks.

Figure 6: Causal effects of OCR-head ablation across three specialized OCR models. (a) Task-specific damage from ablating the discovered top-20 heads and layer-matched random heads. Text, table, and formula use $\Delta$ NED, TEDS drop, and CDM drop, respectively. (b) Cross-task transfer of the task-specific top-20 OCR heads. Columns indicate the discovery task and rows the evaluation task.

Set	#H	$\Delta$ Acc.	Random	Excess	$p$
Text–Retrieval	12	0.56	0.16	0.39	.143
Table–Retrieval	15	0.22	0.21	0.01	.667
Formula–Retrieval	6	0.22	0.11	0.11	.381

Appendix

Table 8: POPE specificity control. Values are accuracy drops in percentage points, pooled over Random, Popular, and Adversarial splits. Random denotes the mean of 20 layer-matched controls; Excess is the targeted drop minus the random mean.

Figure 7: Token-level examples in MinerU2.5. Rows show text, table, and formula recognition. (a) Input crops with target evidence outlined in blue. (b) Visual attention of L16H03 at the selected generation step. Evidence and top-5 percentages are normalized by total attention to visual patches; argmax denotes the most-attended visual patch. (c) Corresponding output excerpts before and after task-specific top-20 ablation, with single-head effects reported separately. Input crops show local evidence and need not contain the full output excerpt.

Backbone	Task	Common $N$	Qualified B $\to$ S	Top-20 overlap	Spearman
Qwen2-VL	Text	319	17 $\to$ 20	18/20	0.9164
	Table	290	16 $\to$ 20	18/20	0.9423
	Formula	312	8 $\to$ 10	18/20	0.8736
Qwen3-VL	Text	319	27 $\to$ 33	17/20	0.9349
	Table	299	26 $\to$ 34	20/20	0.9372
	Formula	317	18 $\to$ 15	19/20	0.8946

Appendix

Table 9: Matched specialization replicates across Qwen generations. Discovery is computed independently in each configuration on common valid samples.

Model/task	Eval.	$k$	Base	B heads	S heads	Rnd mean
Qwen3-VL Text	Base	5	0.012	0.022 ( $+0.010$ )	0.019 ( $+0.006$ )	0.018 ( $+0.005$ )
		10	0.012	0.028 ( $+0.015$ )	0.018 ( $+0.005$ )	0.019 ( $+0.006$ )
		20	0.012	0.409 ( $+0.397$ )	0.303 ( $+0.290$ )	0.020 ( $+0.007$ )
Qwen3-VL Text	Specialized	5	0.019	0.022 ( $+0.002$ )	0.024 ( $+0.005$ )	0.018 ( $-0.001$ )
		10	0.019	0.022 ( $+0.003$ )	0.022 ( $+0.002$ )	0.019 ( $0.000$ )
		20	0.019	0.334 ( $+0.315$ )	0.439 ( $+0.420$ )	0.020 ( $+0.001$ )

Appendix

Table 10: Held-out cross-ranked causal effects. Cells report performance after ablation, with signed change from base in parentheses. Text uses NED (

\downarrow

), tables use TEDS (

\uparrow

), and formulas use CDM (

\uparrow

). B/S heads are ranked in the base/specialized configuration. Rnd mean averages ten layer-matched random control sets (five matched to each ranking in head count and per-layer allocation).

Task	Common $N$	Qualified G $\to$ S	Top-20 overlap	Spearman	Weighted layer G $\to$ S
Text	319	17 $\to$ 20	18/20	0.9164	18.695 $\to$ 18.681
Table	290	16 $\to$ 20	18/20	0.9423	18.754 $\to$ 18.697
Formula	312	8 $\to$ 10	18/20	0.8736	17.227 $\to$ 17.566

Appendix

Table 11: Matched OCR-head discovery before (G) and after specialization in Qwen2-VL. Top-head overlap, all-head rank correlation, and score-weighted layer position show little change in head organization.

Figure 8: Worked examples of $A\leftarrow B$ source-identity replacement. Image boxes mark changed glyphs, not attention intensity; links associate donor and recipient target regions. Panel (d) traces image and prefix processing to Q/K/V: A’s attention is fixed (green), non-target Values are retained (blue), and B supplies target Values only (orange). The overlap sets contain 13, 17, and 15 heads for text, tables, and formulas. Logits shift toward the donor without changing the next-token argmax.

Explore similar work

Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
Feb 26, 2026Jonathan Steinberg, Oren GalVision-Language Models VLM Interpretability
Swarms & AI Lab (SAIL), University of Haifa
Using OCR Heads to Verbalize Image Semantics
Sep 16, 2026Sheridan Feucht, Benno Krojer, Sarah Wang +3VLM Interpretability Mechanistic Interpretability
Northeastern University · Independent
Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
May 26, 2026Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1VLM Evaluation Document Understanding
Inria, Paris, France

Feb 26, 2026J/K move · Enter open · S save

Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models

Jonathan Steinberg, Oren Gal

Swarms & AI Lab (SAIL), University of Haifa

Vision-Language Models VLM Interpretability

Open paper AskPDF arXiv

Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.

Abstract

Explore similar work

Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models

Using OCR Heads to Verbalize Image Semantics

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions