Geometric Risk Control for Vision-Language Model OCR

Weile Gong; Zijian Lu; Mingcai Chen; Yiping Zuo; Xin He; Weibei Fan

cs.CVMar 20, 2026

Geometric Risk Control for Vision-Language Model OCR

Authors: Weile Gong, Zijian Lu, Mingcai Chen, Yiping Zuo, Xin He, Weibei Fan

Organizations: Nanjing University of Posts and Telecommunications, Nanjing, China

Abstract

Vision-language models (VLMs) enable flexible generative optical character recognition (OCR), while their open-ended decoders can expose wrong but fluent text with weak visual support. In audit-sensitive records, such an output can be more costly than abstention. Frozen or externally served VLMs therefore require an external decision layer that can determine whether a transcription has sufficient visual evidence for release. We introduce the Geometric Risk Controller (GRC), a model-agnostic controller that treats controlled geometric transformations as repeatable black-box probes, screens structurally implausible continuations, and releases the unique candidate supported by coherent cross-view evidence. The protocol provides empirical selective exposure control with explicit coverage and query cost under a reproducible fixed decision rule. Experiments across frozen VLMs and standard scene-text benchmarks consistently reduce mean, upper-tail, and catastrophic error among released outputs while retaining high coverage.

Figures & tables

Figure 1: Overview of GRC. Protocol $\pi$ applies $K$ transforms $I_{k}=T^{k}(I)$ . The frozen model $G_{\theta,\delta}$ with parameters $\theta$ , common prompt $p$ , and fixed decoder $\delta$ returns $y_{k}$ . Canonicalization yields $\tilde{y}_{k}$ . Validity flag $v_{k}$ rejects strings beyond the geometric length bound $L_{\mathrm{geom}}$ . The valid set is $\Omega(I)=\{k\mid v_{k}=1\}$ with $n(I)=|\Omega(I)|$ . Valid strings must have unique mode $s^{*}(I)$ . Exact support $q(I)$ and mean normalized edit-distance dispersion $\Delta(I)$ then enter Eq. 3 . The controller exposes $s^{*}(I)$ iff $n(I)\geq K_{\min}$ , $q(I)\geq\tau(m)$ , and $\Delta(I)\leq\kappa$ ; otherwise return $\bot$ . Here $K_{\min}$ is the minimum valid-view count, $\tau(m)$ the consensus threshold chosen by strictness $m$ , and $\kappa$ the dispersion cap. Only $m$ varies along the operating curve.

Figure 2: Evidence patterns for

K=5,m=3

. Fragmented support abstains. A majority may still abstain when dispersion is high. Tightly clustered evidence can pass at moderate support.

Figure 3: Representative fixed-protocol outcomes at

m=3

. GRC abstains on over-generation, unsupported substitution, and view-sensitive guesses, while retaining supported cases across different consensus levels. Evidence tags report valid views, consensus

q

, and

L_{\mathrm{geom}}

, or dispersion

\Delta

for the instability case.

		IIIT5K				ICDAR13
Model	Method	Cov.	CER	P99	MD@2	Cov.	CER	P99	MD@2
LLaVA-Phi3	Baseline	100.0	110.5	3087.8	33.7	100.0	147.4	3572.5	35.6
LLaVA-Phi3	GRC	89.5	8.4	100.0	0.3	89.0	9.3	100.0	3.7
Gemma3	Baseline	100.0	7.4	100.0	3.7	100.0	20.1	500.0	16.4
Gemma3	GRC	94.9	3.7	66.7	0.0	93.0	3.7	100.0	1.8
GLM-OCR	Baseline	100.0	6.3	100.0	3.7	100.0	3.3	88.3	1.8

Table 1: Main results for

K=5,m=3

. Coverage and CER are percentages. MD@2 is per mille. Baseline risk uses the full set. GRC risk uses only exposed outputs.

		IIIT5K			ICDAR13
Model	Method	Cov.	CER	MD@2	Cov.	CER	MD@2
LLaVA-Phi3	Conf.-Thr.	90.0	30.9	19.6	88.2	82.7	31.4
LLaVA-Phi3	GRC	89.5	8.4	0.3	89.0	9.3	3.7
Gemma3	Conf.-Thr.	95.8	7.6	2.2	95.5	22.5	15.1
Gemma3	GRC	94.9	3.7	0.0	93.0	3.7	1.8
GLM-OCR	Conf.-Thr.	94.8	4.4	2.2	95.9	3.2	5.8

Table 2: Approximately coverage-matched confidence comparison. Coverage and CER are percentages. MD@2 is per mille.

Figure 4: Coverage and risk trajectories under the shared protocol. Each point is a predeclared strictness

m

. Filled markers denote

m=3

. Increasing strictness consistently trades coverage for lower covered-output CER.

		IIIT5K		ICDAR13
Model	Setting	Cov.	MD@2	Cov.	MD@2
LLaVA-Phi3	Baseline	100.0	33.7	100.0	35.6
	$m=1$	91.3	0.3	92.1	3.7
	$m=5$	67.7	0.0	68.3	0.9
Gemma3	Baseline	100.0	3.7	100.0	16.4
	$m=1$	95.8	0.0	95.1	1.8

Table 3: Endpoint operating points under the shared

K=5

protocol. Coverage is percent and MD@2 is per mille.

Figure 5: Component analysis on LLaVA-Phi3 at

K=5,m=3

. Labels give coverage and bars give MD@2 (per mille). Consensus-only and structural-only controls both help. Their combination yields the lowest catastrophic exposure.

		IIIT5K		ICDAR13
$K$	Time	Cov.	MD@2	Cov.	MD@2
Baseline	1.0x	100.0	33.7	100.0	35.6
3	2.7x	85.4	0.0	86.3	1.8
5	4.5x	89.5	0.3	88.9	3.7
7	6.3x	89.5	0.3	89.1	3.7

Table 4: LLaVA-Phi3 query-budget check at

m=3

. Time is relative to one pass. Coverage is percent and MD@2 is per mille.

Explore similar work

May 26, 2026

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually unsupported text, suggesting reliance on language priors. Comparing open-weight VLMs with traditional OCR baselines on low-resource Ancient Greek critical editions, we show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions where traditional engines produce local recognition noise. To analyze visual evidence during decoding, we introduce controlled image perturbations and token-level grounding measures based on conditional versus image-free decoding distributions. Under character-level perturbations, VLMs diverge sharply from the perturbed ground truth while traditional OCR remains comparatively faithful; however, token-level analysis shows that prior reliance is model-specific: in an OCR-specialist model, fluent lexical errors are produced with little reliance on the image, whereas general-purpose VLMs remain conditioned on the visual input even when wrong. Decode-time interventions fail to reliably restore grounding, while post-OCR language-model correction improves several systems only by repairing text after generation. Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents and a broader set of models, showing that fluent output is not necessarily visually grounded and motivating interpretability-driven evaluation beyond aggregate accuracy.

Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1

Inria, Paris, France

VLM Evaluation Document Understanding Historical Document Analysis VLM Hallucination Vision-Language Grounding Optical Character Recognition

Ask PDF

Jun 24, 2026

How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their robustness under controlled visual degradation remains insufficiently understood. This gap is critical for OCR reasoning, where visual corruption can induce OCR errors and structural distortions, thereby introducing uncertainty into the reasoning task. To systematically study this problem, we introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations. It contains 812 samples across two complementary subsets: OCR1.0, covering documents, scene text, receipts, handwriting, and mathematical content, and OCR2.0, focusing on charts, geometry diagrams, and tables. To enable efficient yet informative evaluation, we conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability. We evaluate robustness using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and a composite Corruption Robustness Index (CRI), and benchmark 18 models spanning proprietary systems, open-source VLMs, and OCR+LLM pipelines. Our results show that higher clean accuracy does not necessarily imply stronger robustness, and that models can suffer pronounced degradation in the worst case on OCR tasks that are sensitive to structure, and charts and tables are substantially more fragile than document-like inputs under perturbation.

Yuxing Cheng, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

VLM Evaluation VLM Robustness Image Corruption Robustness Visual Reasoning Corruption Robustness

Ask PDF

Feb 26, 2026

Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models

Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.

Jonathan Steinberg, Oren Gal

Swarms & AI Lab (SAIL), University of Haifa

Vision-Language Models VLM Interpretability

Ask PDF