VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR

Yani Guan; Dengpan Dong; Shuang Luo; Zi Wei; Joah Han; Dan Hannah; Yumin Zhang; Qichao Hu

cs.CVAug 23, 2026

VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR

Authors: Yani Guan, Dengpan Dong, Shuang Luo, Zi Wei, Joah Han, Dan Hannah, Yumin Zhang, Qichao Hu, +1 more

Organizations: SES AI Corporation

Abstract

Optical Chemical Structure Recognition (OCSR) converts 2D molecular depictions in the published literature into SMILES, and is increasingly important for constructing large-scale chemical training datasets. Automation at that scale requires identifying unreliable predictions in the absence of ground truth. Three families of label-free signals were compared on $263$ ACS journal depictions with verified ground truth: model confidence, re-rendering similarity, and agreement among recognizers. Pixel-space re-rendering performed little better than chance (AUROC $0.547$ , $95\%$ CI $[0.465,0.629]$ ), and an oracle-tuned threshold on it reduced correct labels per image from $0.745$ to $0.205$ . Agreement among four architecturally distinct recognizers instead reached an AUROC of $0.916$ ( $[0.880,0.952]$ ). The two-of-four rule accepted $81.7\%$ of images at $88.8\%$ precision, the three-of-four rule $52.1\%$ at $98.5\%$ . The same pattern held on CLEF-IP, UOB, and USPTO. This distinction is obscured on synthetic benchmarks, where re-rendered predictions naturally resemble their inputs. A substance filter removed $2{,}193$ false agreements on wildcards and R-group fragments, after which the three-of-four rule rejected all $68$ generic depictions. VERDICT was then applied to PMC Open Access, producing $6{,}146$ structure labels for $4{,}833$ molecules; chemist adjudication of $400$ released labels in two independent samples yielded precisions of $0.995$ for the three-of-four tier and $0.958$ for the two-of-four tier. VERDICT therefore enables validated labels for multimodal molecular databases linking structure images, machine-readable representations, and source-publication information. In SES AI's Molecular Universe platform, VERDICT further serves as an image-based interface for searching and retrieving molecular records.

Figures & tables

Figure 1: VERDICT groups four recognizer outputs by molecular identity and assigns each image to an agreement tier.

	rendered		real documents
recognizer	clean	degraded	ACS	CLEF-IP	UOB	USPTO
DECIMER	$.958$	$.550$	$.296$	$.732$	$.866$	$.586$
MolScribe	$.716$	$.272$	$.583$	$.726$	$.824$	$.766$
MolNexTR	$.802$	$.340$	$.520$	$.732$	$.828$	$.796$
VLM ⋆	$.958$	$.956$	$.459$	$.610$	$.830$	$.732$

Table 1: The four recognizers perform differently across rendered and real-document datasets.

min. heavy atoms	$\geq 1$	$\geq 2$	$\geq 3$	$\geq 4$	$\geq 5$	$\geq 6$	$\geq 7$	$\geq 8$
labels	$6{,}881$	$6{,}745$	$6{,}701$	$6{,}660$	$6{,}545$	$6{,}320$	$6{,}178$	$5{,}992$
gold slice	$3{,}215$	$3{,}213$	$3{,}212$	$3{,}207$	$3{,}166$	$3{,}097$	$3{,}020$	$2{,}917$

Table 2: Sensitivity to the heavy-atom threshold in the substance check, applied to the

6{,}881

strict-InChIKey labels. The deployed value is

\geq 6

Figure 2: Agreement separates correct from incorrect predictions, while pixel-space re-rendering does not. Results are shown for the $263$ ACS depictions with concrete ground truth.

signal	mean, correct	mean, wrong	range	AUROC	$p$
round-trip $s_{\mathrm{rt}}$	$0.0277$	$0.0247$	$0.0005$ – $0.0982$	$0.547$ $[.465,.629]$	$0.25$
MolScribe confidence	$0.876$	$0.810$	$0.001$ – $0.927$	$0.666$ $[.598,.734]$	$2\times 10^{-5}$
render-then-recognize	—	—	$\{0,1\}$	$0.566$ – $0.808$	—
agreement count $a$	—	—	$1$ – $4$	$0.916$ $[.880,.952]$	$<\!10^{-20}$

Table 3: Signal strength follows the space of evaluation. The two signals computed on molecular identity exceed the one computed on pixels, and the one signal drawing on multiple models exceeds those drawing on one. AUROC intervals are

95\%

DeLong intervals and

p

-values are two-sided Mann–Whitney tests ( DeLong et al. 1988 ) . The render-then-recognize row reports the range across the three engines of Table 4 .

signal	self-consistent	AUROC ( $95\%$ )	precision @ coverage
DECIMER	$236/256=0.922$	$0.566$ $[.534,.597]$	$0.411$ @ $89.7\%$
MolScribe	$219/254=0.862$	$0.721$ $[.664,.779]$	$0.817$ @ $83.3\%$
MolNexTR	$178/254=0.701$	$0.808$ $[.758,.856]$	$0.831$ @ $67.7\%$
agreement $a\!\geq\!2$	—	$0.916$ $[.880,.952]$	$0.888$ @ $81.7\%$

Table 4: Per-engine render-then-recognize self-consistency. The engine that most often reproduces its own output is the least informative one, so self-consistency measures stability rather than correctness.

recognizer	exact match	$95\%$ Wilson CI	answer rate
DECIMER	$0.376$	$[.320,.436]$	$97.3\%$
MolNexTR	$0.616$	$[.556,.673]$	$79.5\%$
VLM ⋆	$0.643$	$[.583,.698]$	—
MolScribe	$0.707$	$[.650,.759]$	$94.7\%$
four-engine quorum	$0.745$	$[.689,.794]$	$100\%$

Table 5: With abstention disabled, the four-engine quorum exceeds the best individual recognizer on ACS by less than four points. The dash for VLM ⋆ records that it emits a string for every image, so its answer rate is not defined against the same denominator as the parsing-limited engines.

set	$n$	quorum all	best engine	difference	$a\!\geq\!2$ cov./prec.	$a\!\geq\!3$ cov./prec.
ACS	$263$	$0.741$	$0.707$	$+3.4$	$0.810$ / $0.892$	$0.525$ / $0.978$
CLEF-IP	$444$	$0.923$	$0.917$	$+0.7$	$0.982$ / $0.933$	$0.854$ / $0.958$
UOB	$500$	$0.976$	$0.960$	$+1.6$	$0.994$ / $0.978$	$0.950$ / $0.992$
USPTO	$494$	$0.915$	$0.921$	$-0.6$	$0.923$ / $0.974$	$0.763$ / $0.989$

Table 6: Thresholds calibrated on ACS transfer to patent and hand-drawn depictions. Raw accuracy moves by at most

3.4

points, while the error rate among accepted predictions falls by a factor of

1.8

11.8

once abstention at

a\!\geq\!3

is allowed.

set	quorum	$a\!\geq\!2$ cov./prec.	$a\!\geq\!3$ cov./prec.	$a\!\geq\!4$ cov./prec.
USPTO	four engines	$0.923$ / $0.974$	$0.763$ / $0.989$	$0.431$ / $1.000$
USPTO	five engines	$0.968$ / $0.977$	$0.905$ / $0.989$	$0.755$ / $0.989$
CLEF-IP	four engines	$0.982$ / $0.933$	$0.854$ / $0.958$	$0.531$ / $0.979$
CLEF-IP	five engines	$0.998$ / $0.953$	$0.948$ / $0.962$	$0.818$ / $0.984$
ACS	four engines	$0.810$ / $0.892$	$0.525$ / $0.978$	$0.228$ / $0.983$
ACS	five engines	$0.837$ / $0.896$	$0.673$ / $0.955$	$0.422$ / $0.991$

Table 7: Adding a fifth recognizer extends the coverage–precision curve rather than shifting it. Read by count label the fifth engine appears to reduce ACS precision, because

a\!\geq\!3

of five requires only

60\%

agreement where

a\!\geq\!3

of four requires

75\%

. Read as a curve, the five-engine roster dominates at the high-precision end on ACS and CLEF-IP. UOB is omitted because archived five engine predictions are unavailable for that set.

gate	accepted	coverage	correct	precision	yield	certified $\geq$
VLM ⋆ alone	$263$	$100\%$	$169$	$0.643$ $[.583,.698]$	$0.643$	—
$a\!\geq\!1$ (emit all)	$263$	$100\%$	$196$	$0.745$ $[.689,.794]$	$0.745$	$0.677$
$a\!\geq\!2$ ( consensus )	$215$	$81.7\%$	$191$	$0.888$ $[.839,.924]$	$0.726$	$0.829$
$a\!\geq\!3$ ( gold )	$137$	$52.1\%$	$135$	$0.985$ $[.948,.996]$	$0.513$	$0.940$
$a=4$ (unanimous)	$59$	$22.4\%$	$58$	$0.983$ $[.910,.997]$	$0.221$	$0.893$
$a\!\geq\!2$ $+$ frontier backstop	$242$	$92.0\%$	$211$	$0.872$ $[.824,.908]$	$0.802$	$0.814$

Table 8: Two operating points at the deployed roster of four engines. Precision intervals are

95\%

Wilson intervals, yield is correct labels divided by all input images, and the certified column gives simultaneous distribution-free lower bounds. The final row previews the frontier-model backstop of § 4.1 .

Figure 3: The archived gpt-4o run resolves too few abstentions to be interpreted, while the later gpt-5.5 run resolves $20$ of $48$ .

backstop experiment	abstentions	correct rescues	rescue rate ( $95\%$ CI)
gpt-4o , archived run	$15$	$1$	$0.067$ $[.012,.298]$
gpt-5.5 , current run	$48$	$20$	$0.417$ $[.288,.557]$

Table 9: Rescue rates for the two backstop experiments. The runs use different image subsets and are not a head-to-head comparison of the two models.

system	accepted	coverage	precision	yield	est. API cost
gpt-5.5 alone	$202$	$76.8\%$	$0.812$ $[.752,.860]$	$0.624$	5.13$
quorum, emit all	$263$	$100\%$	$0.745$ $[.689,.794]$	$0.745$	—
quorum $a\!\geq\!2$	$215$	$81.7\%$	$0.888$ $[.839,.924]$	$0.726$	—
quorum $a\!\geq\!3$	$137$	$52.1\%$	$0.985$ $[.948,.996]$	$0.513$	—
quorum $a\!\geq\!2$ $+$ backstop	$242$	$92.0\%$	$0.872$ $[.824,.908]$	$0.802$	1.31$

Table 10: The quorum exceeds the standalone frontier model on coverage, precision, and yield, at a fraction of the cost. API costs assume

2.5

/

10

per million tokens and are approximate, since verified list pricing is unavailable.

system	$a=1$ ( $n=48$ )	$a=2$ ( $n=78$ )	$a=3$ ( $n=78$ )	$a=4$ ( $n=59$ )
quorum correct	$0.104$	$0.718$	$0.987$	$0.983$
gpt-5.5 correct	$0.396$	$0.500$	$0.731$	$0.831$
frontier $-$ quorum	$+29.2$ pt	$-21.8$ pt	$-25.6$ pt	$-15.3$ pt

Table 11: The two systems fail on different images. The frontier model leads only where no two recognizers agree. Refusals count as incorrect.

Figure 4: The pipeline converts PMC figures into $6{,}146$ released structure labels after consensus filtering and benchmark deduplication.

stage	gold ( $a\!\geq\!3$ )	consensus ( $a=2$ )	labels	distinct molecules	kept
strict InChIKey matching	$3{,}215$	$3{,}666$	$6{,}881$	—	$23.1\%$
$+$ canonical-SMILES fallback	$3{,}215$	$5{,}859$	$9{,}074$	—	$30.5\%$
substance-filtered	$3{,}097$	$3{,}223$	$6{,}320$	$4{,}944$	$21.2\%$
$+$ evaluation-set deduplication	$3{,}005$	$3{,}141$	$6{,}146$	$4{,}833$	$20.6\%$

Table 12: Substance filtering removes all

2{,}193

labels added by the broader matching rule and returns both matching rules to the same corpus.

output measure	DECIMER	MolScribe	MolNexTR	VLM ⋆
parseable output	$81.0\%$	$93.0\%$	$72.9\%$	$54.5\%$
valid identity key	$81.0\%$	$61.4\%$	$38.9\%$	$54.0\%$
substantive vote	$60.0\%$	$39.1\%$	$27.4\%$	$42.1\%$

Table 13: Parse rate inverts the ranking of engine contribution. The engine that parses most often supplies the fewest substantive votes.

Figure 5: Crops produced against the VLM reference count for each of $20$ probe figures. Points above the identity line are over-segmented and points below have lost structures. Estimated recall is $0.878$ , and most errors are extra crops rather than omissions.

tier	sample	correct	wrong	unscoreable	precision	$95\%$ CI
gold	first	$99$	$1$	$0$	$0.990$	$[.946,.998]$
	second	$99$	$0$	$1$	$1.000$	$[.963,1.00]$
	pooled	$\mathbf{198}$	$\mathbf{1}$	$\mathbf{1}$	$\mathbf{0.995}$	$\mathbf{[.972,.999]}$
consensus	first	$91$	$4$	$5$	$0.958$	$[.897,.984]$
	second	$92$	$4$	$4$	$0.958$	$[.898,.984]$
	pooled	$\mathbf{183}$	$\mathbf{8}$	$\mathbf{9}$	$\mathbf{0.958}$	$\mathbf{[.920,.979]}$

Table 14: Human review measures

99.5\%

precision for gold and

95.8\%

for consensus across two independent samples. Intervals are

95\%

Wilson intervals over scoreable labels.

Figure 6: Filtered structure labels increase distinct-molecule coverage and roughly double the population of most knowledge-base fields.

	original run	corrected run
input labels	$10{,}896$ (unfiltered)	$6{,}146$ (filtered)
records	$3{,}931$	$3{,}112$
distinct molecules	$1{,}698$	$2{,}552$
InChIKey	$52.4\%$	$100\%$
evidence snippet	$13.3\%$	$23.6\%$
text summary	$14.7\%$	$23.5\%$

Table 15: Filtering the input labels increases molecule and field coverage at unchanged extraction cost.

field	evaluated	matched criterion	result
numeric values	$235$	$231$ traceable	$98.3\%$
compound names	$393$	$320$ verbatim	$81.4\%$
evidence snippets	$653$	$325$ well supported	$49.8\%$
evidence snippets	$653$	$115$ with zero overlap	$17.6\%$

Table 16: Numeric fields are traceable and compound names mostly are, while half of the prose snippets lack strong verbatim support.

Explore similar work

MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition
May 7, 2026Haote Yang, Hui Wang, Chen Zhu +14Optical Character Recognition
Shanghai Artificial Intelligence Laboratory · King’s College London · East China University of Science and Technology +5
ViCoR: Reliable Molecular Structure Extraction via Spatially Aligned Verification and Executable Revision
Sep 27, 2026Yujian Yuan, Xin Cai, Yufan Chen +5Vision-Language Models Optical Character Recognition
The Hong Kong University of Science and Technology · The Chinese University of Hong Kong · Institute of Computing Technology, Chinese Academy of Sciences.
COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training
Apr 26, 2026Zhuoqi Lyu, Qing KeOptical Character Recognition
Department of Data Science, College of Computing, City University of Hong Kong, Hong Kong, China

May 7, 2026J/K move · Enter open · S save

MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition

Haote Yang, Hui Wang, Chen Zhu, Jingchao Wang+13

Shanghai Artificial Intelligence Laboratory · King’s College London · East China University of Science and Technology+5

Optical Character Recognition

Open paper AskPDF arXiv

Optical Chemical Structure Recognition (OCSR) aims to translate molecular diagrams in scientific literature into machine-readable formats, but current systems remain unreliable on real-world images due to substantial visual and chemical complexity. We introduce MOSAIC, a dual-dimensional difficulty framework with 37 fine-grained labels that jointly characterize visual interference and chemical semantic challenges in molecular diagrams. Based on this framework, we construct MolRecBench-Wild, a benchmark of 5,029 structures from 820 recent chemistry papers, covering the full difficulty spectrum observed in real publications. To enable faithful semantic evaluation beyond SMILES and MolFile, we propose CARBON, a representation language capable of expressing valence variations, icon-based groups, and other non-standard chemical semantics. We further adopt a dual-track evaluation protocol supporting both CARBON and SMILES outputs for broad model compatibility. Comprehensive experiments over 18 OCSR-capable models reveal severe performance degradation on MolRecBench-Wild, exposing a large gap between previous patent benchmarks and real-world academic scenarios.

Abstract

Explore similar work

MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition

ViCoR: Reliable Molecular Structure Extraction via Spatially Aligned Verification and Executable Revision

COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training