VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR

Yani Guan; Dengpan Dong; Shuang Luo; Zi Wei; Joah Han; Dan Hannah; Yumin Zhang; Qichao Hu

cs.CVAug 23, 2026

VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR

Authors: Yani Guan, Dengpan Dong, Shuang Luo, Zi Wei, Joah Han, Dan Hannah, Yumin Zhang, Qichao Hu, +1 more

Organizations: SES AI Corporation

Abstract

Optical Chemical Structure Recognition (OCSR) converts 2D molecular depictions in the published literature into SMILES, and is increasingly important for constructing large-scale chemical training datasets. Automation at that scale requires identifying unreliable predictions in the absence of ground truth. Three families of label-free signals were compared on $263$ ACS journal depictions with verified ground truth: model confidence, re-rendering similarity, and agreement among recognizers. Pixel-space re-rendering performed little better than chance (AUROC $0.547$ , $95\%$ CI $[0.465,0.629]$ ), and an oracle-tuned threshold on it reduced correct labels per image from $0.745$ to $0.205$ . Agreement among four architecturally distinct recognizers instead reached an AUROC of $0.916$ ( $[0.880,0.952]$ ). The two-of-four rule accepted $81.7\%$ of images at $88.8\%$ precision, the three-of-four rule $52.1\%$ at $98.5\%$ . The same pattern held on CLEF-IP, UOB, and USPTO. This distinction is obscured on synthetic benchmarks, where re-rendered predictions naturally resemble their inputs. A substance filter removed $2{,}193$ false agreements on wildcards and R-group fragments, after which the three-of-four rule rejected all $68$ generic depictions. VERDICT was then applied to PMC Open Access, producing $6{,}146$ structure labels for $4{,}833$ molecules; chemist adjudication of $400$ released labels in two independent samples yielded precisions of $0.995$ for the three-of-four tier and $0.958$ for the two-of-four tier. VERDICT therefore enables validated labels for multimodal molecular databases linking structure images, machine-readable representations, and source-publication information. In SES AI's Molecular Universe platform, VERDICT further serves as an image-based interface for searching and retrieving molecular records.

Figures & tables

Figure 1: VERDICT groups four recognizer outputs by molecular identity and assigns each image to an agreement tier.

	rendered		real documents
recognizer	clean	degraded	ACS	CLEF-IP	UOB	USPTO
DECIMER	$.958$	$.550$	$.296$	$.732$	$.866$	$.586$
MolScribe	$.716$	$.272$	$.583$	$.726$	$.824$	$.766$
MolNexTR	$.802$	$.340$	$.520$	$.732$	$.828$	$.796$
VLM ⋆	$.958$	$.956$	$.459$	$.610$	$.830$	$.732$

Table 1: The four recognizers perform differently across rendered and real-document datasets.

min. heavy atoms	$\geq 1$	$\geq 2$	$\geq 3$	$\geq 4$	$\geq 5$	$\geq 6$	$\geq 7$	$\geq 8$
labels	$6{,}881$	$6{,}745$	$6{,}701$	$6{,}660$	$6{,}545$	$6{,}320$	$6{,}178$	$5{,}992$
gold slice	$3{,}215$	$3{,}213$	$3{,}212$	$3{,}207$	$3{,}166$	$3{,}097$	$3{,}020$	$2{,}917$

Table 2: Sensitivity to the heavy-atom threshold in the substance check, applied to the

6{,}881

strict-InChIKey labels. The deployed value is

\geq 6

Figure 2: Agreement separates correct from incorrect predictions, while pixel-space re-rendering does not. Results are shown for the $263$ ACS depictions with concrete ground truth.

signal	mean, correct	mean, wrong	range	AUROC	$p$
round-trip $s_{\mathrm{rt}}$	$0.0277$	$0.0247$	$0.0005$ – $0.0982$	$0.547$ $[.465,.629]$	$0.25$
MolScribe confidence	$0.876$	$0.810$	$0.001$ – $0.927$	$0.666$ $[.598,.734]$	$2\times 10^{-5}$
render-then-recognize	—	—	$\{0,1\}$	$0.566$ – $0.808$	—
agreement count $a$	—	—	$1$ – $4$	$0.916$ $[.880,.952]$	$<\!10^{-20}$

Table 3: Signal strength follows the space of evaluation. The two signals computed on molecular identity exceed the one computed on pixels, and the one signal drawing on multiple models exceeds those drawing on one. AUROC intervals are

95\%

DeLong intervals and

p

-values are two-sided Mann–Whitney tests ( DeLong et al. 1988 ) . The render-then-recognize row reports the range across the three engines of Table 4 .

signal	self-consistent	AUROC ( $95\%$ )	precision @ coverage
DECIMER	$236/256=0.922$	$0.566$ $[.534,.597]$	$0.411$ @ $89.7\%$
MolScribe	$219/254=0.862$	$0.721$ $[.664,.779]$	$0.817$ @ $83.3\%$
MolNexTR	$178/254=0.701$	$0.808$ $[.758,.856]$	$0.831$ @ $67.7\%$
agreement $a\!\geq\!2$	—	$0.916$ $[.880,.952]$	$0.888$ @ $81.7\%$

Table 4: Per-engine render-then-recognize self-consistency. The engine that most often reproduces its own output is the least informative one, so self-consistency measures stability rather than correctness.

recognizer	exact match	$95\%$ Wilson CI	answer rate
DECIMER	$0.376$	$[.320,.436]$	$97.3\%$
MolNexTR	$0.616$	$[.556,.673]$	$79.5\%$
VLM ⋆	$0.643$	$[.583,.698]$	—
MolScribe	$0.707$	$[.650,.759]$	$94.7\%$
four-engine quorum	$0.745$	$[.689,.794]$	$100\%$

Table 5: With abstention disabled, the four-engine quorum exceeds the best individual recognizer on ACS by less than four points. The dash for VLM ⋆ records that it emits a string for every image, so its answer rate is not defined against the same denominator as the parsing-limited engines.

set	$n$	quorum all	best engine	difference	$a\!\geq\!2$ cov./prec.	$a\!\geq\!3$ cov./prec.
ACS	$263$	$0.741$	$0.707$	$+3.4$	$0.810$ / $0.892$	$0.525$ / $0.978$
CLEF-IP	$444$	$0.923$	$0.917$	$+0.7$	$0.982$ / $0.933$	$0.854$ / $0.958$
UOB	$500$	$0.976$	$0.960$	$+1.6$	$0.994$ / $0.978$	$0.950$ / $0.992$
USPTO	$494$	$0.915$	$0.921$	$-0.6$	$0.923$ / $0.974$	$0.763$ / $0.989$

Table 6: Thresholds calibrated on ACS transfer to patent and hand-drawn depictions. Raw accuracy moves by at most

3.4

points, while the error rate among accepted predictions falls by a factor of

1.8

11.8

once abstention at

a\!\geq\!3

is allowed.

set	quorum	$a\!\geq\!2$ cov./prec.	$a\!\geq\!3$ cov./prec.	$a\!\geq\!4$ cov./prec.
USPTO	four engines	$0.923$ / $0.974$	$0.763$ / $0.989$	$0.431$ / $1.000$
USPTO	five engines	$0.968$ / $0.977$	$0.905$ / $0.989$	$0.755$ / $0.989$
CLEF-IP	four engines	$0.982$ / $0.933$	$0.854$ / $0.958$	$0.531$ / $0.979$
CLEF-IP	five engines	$0.998$ / $0.953$	$0.948$ / $0.962$	$0.818$ / $0.984$
ACS	four engines	$0.810$ / $0.892$	$0.525$ / $0.978$	$0.228$ / $0.983$
ACS	five engines	$0.837$ / $0.896$	$0.673$ / $0.955$	$0.422$ / $0.991$

Table 7: Adding a fifth recognizer extends the coverage–precision curve rather than shifting it. Read by count label the fifth engine appears to reduce ACS precision, because

a\!\geq\!3

of five requires only

60\%

agreement where

a\!\geq\!3

of four requires

75\%

. Read as a curve, the five-engine roster dominates at the high-precision end on ACS and CLEF-IP. UOB is omitted because archived five engine predictions are unavailable for that set.

gate	accepted	coverage	correct	precision	yield	certified $\geq$
VLM ⋆ alone	$263$	$100\%$	$169$	$0.643$ $[.583,.698]$	$0.643$	—
$a\!\geq\!1$ (emit all)	$263$	$100\%$	$196$	$0.745$ $[.689,.794]$	$0.745$	$0.677$
$a\!\geq\!2$ ( consensus )	$215$	$81.7\%$	$191$	$0.888$ $[.839,.924]$	$0.726$	$0.829$
$a\!\geq\!3$ ( gold )	$137$	$52.1\%$	$135$	$0.985$ $[.948,.996]$	$0.513$	$0.940$
$a=4$ (unanimous)	$59$	$22.4\%$	$58$	$0.983$ $[.910,.997]$	$0.221$	$0.893$
$a\!\geq\!2$ $+$ frontier backstop	$242$	$92.0\%$	$211$	$0.872$ $[.824,.908]$	$0.802$	$0.814$

Table 8: Two operating points at the deployed roster of four engines. Precision intervals are

95\%

Wilson intervals, yield is correct labels divided by all input images, and the certified column gives simultaneous distribution-free lower bounds. The final row previews the frontier-model backstop of § 4.1 .

Figure 3: The archived gpt-4o run resolves too few abstentions to be interpreted, while the later gpt-5.5 run resolves $20$ of $48$ .

backstop experiment	abstentions	correct rescues	rescue rate ( $95\%$ CI)
gpt-4o , archived run	$15$	$1$	$0.067$ $[.012,.298]$
gpt-5.5 , current run	$48$	$20$	$0.417$ $[.288,.557]$

Table 9: Rescue rates for the two backstop experiments. The runs use different image subsets and are not a head-to-head comparison of the two models.

system	accepted	coverage	precision	yield	est. API cost
gpt-5.5 alone	$202$	$76.8\%$	$0.812$ $[.752,.860]$	$0.624$	5.13$
quorum, emit all	$263$	$100\%$	$0.745$ $[.689,.794]$	$0.745$	—
quorum $a\!\geq\!2$	$215$	$81.7\%$	$0.888$ $[.839,.924]$	$0.726$	—
quorum $a\!\geq\!3$	$137$	$52.1\%$	$0.985$ $[.948,.996]$	$0.513$	—
quorum $a\!\geq\!2$ $+$ backstop	$242$	$92.0\%$	$0.872$ $[.824,.908]$	$0.802$	1.31$

Table 10: The quorum exceeds the standalone frontier model on coverage, precision, and yield, at a fraction of the cost. API costs assume

2.5

/

10

per million tokens and are approximate, since verified list pricing is unavailable.

system	$a=1$ ( $n=48$ )	$a=2$ ( $n=78$ )	$a=3$ ( $n=78$ )	$a=4$ ( $n=59$ )
quorum correct	$0.104$	$0.718$	$0.987$	$0.983$
gpt-5.5 correct	$0.396$	$0.500$	$0.731$	$0.831$
frontier $-$ quorum	$+29.2$ pt	$-21.8$ pt	$-25.6$ pt	$-15.3$ pt

Table 11: The two systems fail on different images. The frontier model leads only where no two recognizers agree. Refusals count as incorrect.

Figure 4: The pipeline converts PMC figures into $6{,}146$ released structure labels after consensus filtering and benchmark deduplication.

stage	gold ( $a\!\geq\!3$ )	consensus ( $a=2$ )	labels	distinct molecules	kept
strict InChIKey matching	$3{,}215$	$3{,}666$	$6{,}881$	—	$23.1\%$
$+$ canonical-SMILES fallback	$3{,}215$	$5{,}859$	$9{,}074$	—	$30.5\%$
substance-filtered	$3{,}097$	$3{,}223$	$6{,}320$	$4{,}944$	$21.2\%$
$+$ evaluation-set deduplication	$3{,}005$	$3{,}141$	$6{,}146$	$4{,}833$	$20.6\%$

Table 12: Substance filtering removes all

2{,}193

labels added by the broader matching rule and returns both matching rules to the same corpus.

output measure	DECIMER	MolScribe	MolNexTR	VLM ⋆
parseable output	$81.0\%$	$93.0\%$	$72.9\%$	$54.5\%$
valid identity key	$81.0\%$	$61.4\%$	$38.9\%$	$54.0\%$
substantive vote	$60.0\%$	$39.1\%$	$27.4\%$	$42.1\%$

Table 13: Parse rate inverts the ranking of engine contribution. The engine that parses most often supplies the fewest substantive votes.

Figure 5: Crops produced against the VLM reference count for each of $20$ probe figures. Points above the identity line are over-segmented and points below have lost structures. Estimated recall is $0.878$ , and most errors are extra crops rather than omissions.

tier	sample	correct	wrong	unscoreable	precision	$95\%$ CI
gold	first	$99$	$1$	$0$	$0.990$	$[.946,.998]$
	second	$99$	$0$	$1$	$1.000$	$[.963,1.00]$
	pooled	$\mathbf{198}$	$\mathbf{1}$	$\mathbf{1}$	$\mathbf{0.995}$	$\mathbf{[.972,.999]}$
consensus	first	$91$	$4$	$5$	$0.958$	$[.897,.984]$
	second	$92$	$4$	$4$	$0.958$	$[.898,.984]$
	pooled	$\mathbf{183}$	$\mathbf{8}$	$\mathbf{9}$	$\mathbf{0.958}$	$\mathbf{[.920,.979]}$

Table 14: Human review measures

99.5\%

precision for gold and

95.8\%

for consensus across two independent samples. Intervals are

95\%

Wilson intervals over scoreable labels.

Figure 6: Filtered structure labels increase distinct-molecule coverage and roughly double the population of most knowledge-base fields.

	original run	corrected run
input labels	$10{,}896$ (unfiltered)	$6{,}146$ (filtered)
records	$3{,}931$	$3{,}112$
distinct molecules	$1{,}698$	$2{,}552$
InChIKey	$52.4\%$	$100\%$
evidence snippet	$13.3\%$	$23.6\%$
text summary	$14.7\%$	$23.5\%$

Table 15: Filtering the input labels increases molecule and field coverage at unchanged extraction cost.

field	evaluated	matched criterion	result
numeric values	$235$	$231$ traceable	$98.3\%$
compound names	$393$	$320$ verbatim	$81.4\%$
evidence snippets	$653$	$325$ well supported	$49.8\%$
evidence snippets	$653$	$115$ with zero overlap	$17.6\%$

Table 16: Numeric fields are traceable and compound names mostly are, while half of the prose snippets lack strong verbatim support.

Explore similar work

May 7, 2026

MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition

Optical Chemical Structure Recognition (OCSR) aims to translate molecular diagrams in scientific literature into machine-readable formats, but current systems remain unreliable on real-world images due to substantial visual and chemical complexity. We introduce MOSAIC, a dual-dimensional difficulty framework with 37 fine-grained labels that jointly characterize visual interference and chemical semantic challenges in molecular diagrams. Based on this framework, we construct MolRecBench-Wild, a benchmark of 5,029 structures from 820 recent chemistry papers, covering the full difficulty spectrum observed in real publications. To enable faithful semantic evaluation beyond SMILES and MolFile, we propose CARBON, a representation language capable of expressing valence variations, icon-based groups, and other non-standard chemical semantics. We further adopt a dual-track evaluation protocol supporting both CARBON and SMILES outputs for broad model compatibility. Comprehensive experiments over 18 OCSR-capable models reveal severe performance degradation on MolRecBench-Wild, exposing a large gap between previous patent benchmarks and real-world academic scenarios.

Haote Yang, Hui Wang, Chen Zhu +14

Shanghai Artificial Intelligence Laboratory · King’s College London · East China University of Science and Technology +5

Optical Character Recognition

Ask PDF

Sep 27, 2026

ViCoR: Reliable Molecular Structure Extraction via Spatially Aligned Verification and Executable Revision

Reliable optical chemical structure recognition (OCSR) is essential for building high-quality chemical data from scientific literature, yet even small recognition errors can propagate into chemical databases and downstream models. In practice, recognized structures often require manual inspection and correction before use, making large-scale data curation costly and difficult to scale. We therefore study Selective Structure Recognition (SSR), a post-recognition setting that automatically produces reliable structured outputs while rejecting unresolved cases. Selection-only approaches can improve reliability by rejection, but cannot create additional correct outputs beyond those produced by the base recognizer. We propose ViCoR, a repair-before-rejection framework for iterative VerIfiCatiOn and Revision. Its key idea is to make observation-prediction correspondence explicit: coordinate-preserving rendering establishes spatial correspondence between the source image and predicted structure, while index anchoring maps localized visual discrepancies to executable graph edits without full-structure regeneration. A shared VLM is progressively trained from verification to revision. On two real-world OCSR benchmarks, ViCoR improves overall accuracy from 73.53% to 88.26% and from 61.83% to 84.32%, while achieving over 97% accepted accuracy at 85--89% coverage. The resulting molecular data further improve reaction-extraction F1 by 15.5 points and literature-sourced reaction prediction accuracy by 7.7 and 5.8 points, demonstrating the value of automated reliability control for scientific data curation and downstream chemical learning.

Yujian Yuan, Xin Cai, Yufan Chen +5

The Hong Kong University of Science and Technology · The Chinese University of Hong Kong · Institute of Computing Technology, Chinese Academy of Sciences.

Vision-Language Models Optical Character Recognition

Ask PDF

Apr 26, 2026

COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training

Optical chemical structure recognition (OCSR) translates molecular images into machine-readable representations like SMILES strings or molecular graphs, but remains challenging in real-world documents due to inexhaustible variations in chemical structures, shorthand conventions, and visual noise. Most existing deep-learning-based approaches rely on teacher forcing with token-level Maximum Likelihood Estimation (MLE). This training paradigm suffers from exposure bias, as models are trained under ground-truth prefixes but must condition on their own previous predictions during inference. Moreover, token-level MLE objectives hinder the optimization towards molecular-level evaluation criteria such as chemical validity and structural similarity. Here we introduce Minimum Risk Training (MRT) to OCSR and propose COMO (Closed-loop Optical Molecule recOgnition), a closed-loop framework that mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives, by iteratively sampling and evaluating the model's own predictions. Experiments on ten benchmarks including synthetic and real-world chemical diagrams from patent and scientific literature demonstrate that COMO substantially outperforms existing rule-based and learning-based methods with less training data. Ablation studies further show that MRT is architecture-agnostic, demonstrating its potential for broad application to end-to-end OCSR systems.

Zhuoqi Lyu, Qing Ke

Department of Data Science, College of Computing, City University of Hong Kong, Hong Kong, China

Optical Character Recognition

Ask PDF