Geometric Risk Control for Vision-Language Model OCR

Weile Gong; Zijian Lu; Mingcai Chen; Yiping Zuo; Xin He; Weibei Fan

cs.CVMar 20, 2026

Geometric Risk Control for Vision-Language Model OCR

Authors: Weile Gong, Zijian Lu, Mingcai Chen, Yiping Zuo, Xin He, Weibei Fan

Organizations: Nanjing University of Posts and Telecommunications, Nanjing, China

Abstract

Vision-language models (VLMs) enable flexible generative optical character recognition (OCR), while their open-ended decoders can expose wrong but fluent text with weak visual support. In audit-sensitive records, such an output can be more costly than abstention. Frozen or externally served VLMs therefore require an external decision layer that can determine whether a transcription has sufficient visual evidence for release. We introduce the Geometric Risk Controller (GRC), a model-agnostic controller that treats controlled geometric transformations as repeatable black-box probes, screens structurally implausible continuations, and releases the unique candidate supported by coherent cross-view evidence. The protocol provides empirical selective exposure control with explicit coverage and query cost under a reproducible fixed decision rule. Experiments across frozen VLMs and standard scene-text benchmarks consistently reduce mean, upper-tail, and catastrophic error among released outputs while retaining high coverage.

Figures & tables

Figure 1: Overview of GRC. Protocol $\pi$ applies $K$ transforms $I_{k}=T^{k}(I)$ . The frozen model $G_{\theta,\delta}$ with parameters $\theta$ , common prompt $p$ , and fixed decoder $\delta$ returns $y_{k}$ . Canonicalization yields $\tilde{y}_{k}$ . Validity flag $v_{k}$ rejects strings beyond the geometric length bound $L_{\mathrm{geom}}$ . The valid set is $\Omega(I)=\{k\mid v_{k}=1\}$ with $n(I)=|\Omega(I)|$ . Valid strings must have unique mode $s^{*}(I)$ . Exact support $q(I)$ and mean normalized edit-distance dispersion $\Delta(I)$ then enter Eq. 3 . The controller exposes $s^{*}(I)$ iff $n(I)\geq K_{\min}$ , $q(I)\geq\tau(m)$ , and $\Delta(I)\leq\kappa$ ; otherwise return $\bot$ . Here $K_{\min}$ is the minimum valid-view count, $\tau(m)$ the consensus threshold chosen by strictness $m$ , and $\kappa$ the dispersion cap. Only $m$ varies along the operating curve.

Figure 2: Evidence patterns for

K=5,m=3

. Fragmented support abstains. A majority may still abstain when dispersion is high. Tightly clustered evidence can pass at moderate support.

Figure 3: Representative fixed-protocol outcomes at

m=3

. GRC abstains on over-generation, unsupported substitution, and view-sensitive guesses, while retaining supported cases across different consensus levels. Evidence tags report valid views, consensus

q

, and

L_{\mathrm{geom}}

, or dispersion

\Delta

for the instability case.

		IIIT5K				ICDAR13
Model	Method	Cov.	CER	P99	MD@2	Cov.	CER	P99	MD@2
LLaVA-Phi3	Baseline	100.0	110.5	3087.8	33.7	100.0	147.4	3572.5	35.6
LLaVA-Phi3	GRC	89.5	8.4	100.0	0.3	89.0	9.3	100.0	3.7
Gemma3	Baseline	100.0	7.4	100.0	3.7	100.0	20.1	500.0	16.4
Gemma3	GRC	94.9	3.7	66.7	0.0	93.0	3.7	100.0	1.8
GLM-OCR	Baseline	100.0	6.3	100.0	3.7	100.0	3.3	88.3	1.8

Table 1: Main results for

K=5,m=3

. Coverage and CER are percentages. MD@2 is per mille. Baseline risk uses the full set. GRC risk uses only exposed outputs.

		IIIT5K			ICDAR13
Model	Method	Cov.	CER	MD@2	Cov.	CER	MD@2
LLaVA-Phi3	Conf.-Thr.	90.0	30.9	19.6	88.2	82.7	31.4
LLaVA-Phi3	GRC	89.5	8.4	0.3	89.0	9.3	3.7
Gemma3	Conf.-Thr.	95.8	7.6	2.2	95.5	22.5	15.1
Gemma3	GRC	94.9	3.7	0.0	93.0	3.7	1.8
GLM-OCR	Conf.-Thr.	94.8	4.4	2.2	95.9	3.2	5.8

Table 2: Approximately coverage-matched confidence comparison. Coverage and CER are percentages. MD@2 is per mille.

Figure 4: Coverage and risk trajectories under the shared protocol. Each point is a predeclared strictness

m

. Filled markers denote

m=3

. Increasing strictness consistently trades coverage for lower covered-output CER.

		IIIT5K		ICDAR13
Model	Setting	Cov.	MD@2	Cov.	MD@2
LLaVA-Phi3	Baseline	100.0	33.7	100.0	35.6
	$m=1$	91.3	0.3	92.1	3.7
	$m=5$	67.7	0.0	68.3	0.9
Gemma3	Baseline	100.0	3.7	100.0	16.4
	$m=1$	95.8	0.0	95.1	1.8

Table 3: Endpoint operating points under the shared

K=5

protocol. Coverage is percent and MD@2 is per mille.

Figure 5: Component analysis on LLaVA-Phi3 at

K=5,m=3

. Labels give coverage and bars give MD@2 (per mille). Consensus-only and structural-only controls both help. Their combination yields the lowest catastrophic exposure.

		IIIT5K		ICDAR13
$K$	Time	Cov.	MD@2	Cov.	MD@2
Baseline	1.0x	100.0	33.7	100.0	35.6
3	2.7x	85.4	0.0	86.3	1.8
5	4.5x	89.5	0.3	88.9	3.7
7	6.3x	89.5	0.3	89.1	3.7

Table 4: LLaVA-Phi3 query-budget check at

m=3

. Time is relative to one pass. Coverage is percent and MD@2 is per mille.

Explore similar work

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
May 26, 2026Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1VLM Evaluation Document Understanding
Inria, Paris, France
How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations
Jun 24, 2026Yuxing Cheng, Yuan Wu, Yi ChangVLM Evaluation VLM Robustness
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
Feb 26, 2026Jonathan Steinberg, Oren GalVision-Language Models VLM Interpretability
Swarms & AI Lab (SAIL), University of Haifa

May 26, 2026J/K move · Enter open · S save

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot, Thibault Clérice

Inria, Paris, France

VLM Evaluation Document Understanding Historical Document Analysis VLM Hallucination Vision-Language Grounding Optical Character Recognition

Open paper AskPDF arXiv

Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually unsupported text, suggesting reliance on language priors. Comparing open-weight VLMs with traditional OCR baselines on low-resource Ancient Greek critical editions, we show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions where traditional engines produce local recognition noise. To analyze visual evidence during decoding, we introduce controlled image perturbations and token-level grounding measures based on conditional versus image-free decoding distributions. Under character-level perturbations, VLMs diverge sharply from the perturbed ground truth while traditional OCR remains comparatively faithful; however, token-level analysis shows that prior reliance is model-specific: in an OCR-specialist model, fluent lexical errors are produced with little reliance on the image, whereas general-purpose VLMs remain conditioned on the visual input even when wrong. Decode-time interventions fail to reliably restore grounding, while post-OCR language-model correction improves several systems only by repairing text after generation. Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents and a broader set of models, showing that fluent output is not necessarily visually grounded and motivating interpretability-driven evaluation beyond aggregate accuracy.

Abstract

Explore similar work

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models