MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures

Tim Strohmeyer; Lucas Morin; Gerhard Ingmar Meijer; Valéry Weber; Ahmed Nassar; Peter Staar

cs.CVMar 30, 2026

MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures

Authors: Tim Strohmeyer, Lucas Morin, Gerhard Ingmar Meijer, Valéry Weber, Ahmed Nassar, Peter Staar

Organizations: IBM Research · ETH Zurich

Abstract

Automatically extracting chemical structures from documents is essential for the large-scale analysis of the literature in chemistry. Automatic pipelines have been developed to recognize molecules represented either in figures or in text independently. However, methods for recognizing chemical structures from multimodal descriptions (Markush structures) lag behind in precision and cannot be used for automatic large-scale processing. In this work, we present MarkushGrapher-2, an end-to-end approach for the multimodal recognition of chemical structures in documents. First, our method employs a dedicated OCR model to extract text from chemical images. Second, the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. Finally, the resulting encodings are effectively fused through a two-stage training strategy and used to auto-regressively generate a representation of the Markush structure. To address the lack of training data, we introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures. In addition, we present IP5-M, a large manually-annotated benchmark of real-world Markush structures, designed to advance research on this challenging task. Extensive experiments show that our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition, while maintaining strong performance in molecule structure recognition. Code, models, and datasets are released publicly.

Figures & tables

Figure 1: Model Use Case : MarkushGrapher-2 parses Markush backbones and variable regions from document image crops via joint multimodal encoding of vision, text, and layout.

Figure 2: Model Architecture : MarkushGraher-2 employs two complementary encoding pipelines. In the first pipeline, the input image is processed by a vision encoder (blue) followed by an MLP projector (yellow). In the second pipeline, the image is passed through an OCR model to extract textual content and bounding boxes, which is then fed into a Vision–Text–Layout (VTL) encoder together with the original image. The output of the MLP projector (e1) is concatenated with the resulting VTL embedding (e2). The combined representation is passed to a text decoder to generate a sequential description of the Markush structure and its substituents in tabular form.

Figure 3: Two-Phase Training : In Phase 1 (Adaptation), the OCSR encoder is frozen while the MLP projector and text decoder are trained for SMILES prediction to align with pretrained OCSR features. In Phase 2 (Fusion), the adapted modules are initialized, the VTL encoder is introduced, and the full model is trained end-to-end for CXSMILES prediction.

Models	M2S (103)				USPTO-M (74)				IP5-M (1000)
	P	R	F1	A@IoU_0.5	P	R	F1	A@IoU_0.5	P	R	F1	A@IoU_0.5
PaddleOCR v5	8.9	6.8	7.7	0.0	2.3	1.1	1.2	0.0	2.2	1.7	1.9	0.6
EasyOCR	9.8	10.7	10.2	0.0	24.8	14.2	18.0	0.0	23.5	15.2	18.4	2.7
ChemicalOCR (Ours)	86.9	87.4	87.2	32.0	93.5	92.6	93.0	63.5	85.6	87.4	86.5	69.5

Table 1: OCR on Chemical Images : Comparison of our ChemicalOCR model with existing OCR models. The evaluation is conducted on real-world benchmarks (M2S, USPTO-M, and IP5-M). Precision

P

, Recall

R

, and F1 are measured at individual bounding-box level. Accuracy

A

is measured at the image level; an image is considered correct if all OCR cells have an IoU

>0.5

and their recognized characters match the ground truth.

Figure 4: OCR - Qualitative Comparison : Comparison of OCR predictions by three models PaddleOCR v5, EasyOCR, and ChemicalOCR (Ours) for an exemplary chemical structure from the benchmarks M2S, USPTO-M, and IP5-M. Red labels indicate incorrect OCR, green labels indicate correct OCR and blue indicates predicted bounding boxes.

Methods	M2S (103)				USPTO-M (74)	WildMol-M (10000)	IP5-M (1000)
	CXSMILES	Table		Markush	CXSMILES	CXSMILES	CXSMILES
	A	A	F1	A	A	A	A
Image only
MolParser-Base	39	–	–	–	30	38.1 $\ddagger$	47.7
MolScribe	21 $\dagger$	–	–	–	7 $\dagger$	28.1	22.3
Multimodal

Table 2: Markush Structure Recognition: Comparison of our MarkushGrapher-2 model with existing models on CXSMILES and Substituent Table prediction. Models are evaluated on real-world benchmarks (M2S, UPSTO-M, WildMol-M, IP5-M). Accuracy

A

measures the percentage of correctly predicted samples, while F1 quantifies the similarity between predictions and ground truth, ranging from 0 (least similar) to 100 (most similar).

\dagger

scores taken from [ 20 ] ,

\ddagger

scores taken from [ 9 ] .

Methods	WildMol (10000)	JPO (450)	UOB (5740)	USPTO (5719)
Image only
MolParser-Base	76.9	78.9	91.8	93.0
MolScribe	66.4	76.2	87.4	93.1
DECIMER 2.7	56.0	64.0	88.3	59.9
MolGrapher	45.5	67.5	94.9	91.5
Multi-modal

Table 3: Molecular Structure Recognition : Comparison of our MarkushGrapher-2 model with existing models on SMILES prediction. Models are evaluated on real-world benchmarks (WildMol, JPO, UOB, and USPTO). Accuracy measures the percentage of correctly predicted molecular structure. Scores for MolParser, MolScribe, DECIMER, and MolGrapher scores are taken from [ 20 , 9 ] .

Figure 5: Markush Structure Recognition - Qualitative Comparison : Comparison of Markush structure predictions by five models MarkushGrapher-2 (Ours), MarkushGrapher-1, MolParser, DeepSeek OCR, and GPT-5 for an exemplary Markush structure from the benchmarks M2S, USPTO-M, WildMol-M, and IP5-M. Red labels indicate incorrect predictions, green labels indicate correct predictions.

Methods	M2S		USPTO-M		IP5-M
	A	A ${}_{\text{InChIKey}}$	A	A ${}_{\text{InChIKey}}$	A	A ${}_{\text{InChIKey}}$
without OCR	4	39	3	51	15.4	51.3
with OCR	56	80	55	69	53.7	73.3

Table 4: Effect of ChemicalOCR on Overall Model Performance : Comparison of MarkushGrapher-2 performance with and without OCR predictions as input. The table shows the CXSMILES prediction accuracy

A

and structural backbone prediction accuracy

A_{\text{InChIKey}}

Figure 6: Effect of OCR Predictions : Comparison of MarkushGrapher-2 predictions with and without OCR input. The green circle indicates prediction of a frequency variation indicator (i.e., repeating Sg groups).

Methods	M2S		JPO
	A	A ${}_{\text{InChIKey}}$	A	A ${}_{\text{InChIKey}}$
Fusion only	44	53	53.0	53.0
Adaptation and Fusion	50	68	61.5	61.5

Table 5: Effect of Two-Phase Training : Comparison of MarkushGrapher-2 performance with and without two-phase training. ‘Fusion only’ denotes the model trained in a single phase, while ‘Adaptation and Fusion’ refers to the two-phase training setup. Scores for CXSMILES prediction accuracy A and structural backbone prediction accuracy A

{}_{\text{InChIKey}}

are reported after 2 epochs for both configurations.

Figure 7: CXSMILES - Schema : A CXSMILES consists of two sections separated by a $|$ character. The first section is a SMILES that describes the molecular backbone of the Markush structure. The second section are the Markush features; variable groups (yellow) and attachment points (orange) are defined in the the R-groups section, cycle connections (green) are defined in the positional variation indicator section, and repeating structural groups (purple) are defined in the frequency variation indicator section.

Figure 8: USPTO-MOL-M Training dataset : Two examples of a chemical-structure image that is provided by the USPTO (left), the CXSMILES that we obtained (bottom) by processing the corresponding MOL file that is provided by the USPTO, and a visualization of said CXSMILES (right). Example 1 shows a simple Markush structure. Typical features of Markush structures, attach point (AP) and frequency variation ”n” (marked in green in CXSMILES and visualization (right)), are indicated. Example 2 shows a typical Markush structure. Typical features of Markush structures, variable groups and positional variation (color coded in CXSMILES and visualization (right)), are indicated.

USPTO-MOL-M (54k)
Variable group	91
Attach point	8
m-section	42
Sg-section	55
Mean num. atoms	20
Mean num. OCR cells	13

Table 6: USPTO-MOL-M - Statistics : Percentage of different (Markush) features found in the training dataset.

Figure 9: IP5-M Benchmark - Composition : Distribution of the Markush structure images extracted from patent documents of the IP5 offices.

Methods	M2S		USPTO
	CXSMILES	Table	SMILES
Pipeline 1	8	0	89.1
Pipeline 2	39	21	84.4
Pipeline 1 & 2	56	22	89.8

Table 7: Architectural Branches Ablation : Comparison of Pipelines 1, 2, and 1 & 2. Pipeline 1: vision encoder + projector + decoder; Pipeline 2: OCR + VTL encoder + decoder.

Checkpoints	M2S	USPTO-M	IP5-M
ChemicalOCR - pretrained	26.2	18.9	27.6
ChemicalOCR - finetuned	32.0	63.5	69.5

Table 8: Comparison of ChemicalOCR training stages : Comparison of OCR Accuracy A at IoU

>

0.5 of ChemicalOCR at different training stages. “Pretrained” denotes training on synthetic data only, while “finetuned” indicates additional training on 7k real samples.

Models	M2S (103)				USPTO-M (74)				IP5-M (1000)
	P	R	F1	A	P	R	F1	A	P	R	F1	A
@IoU $>$ 0.0
PaddleOCR v5	61.9	47.3	53.6	0.0	57.0	47.2	51.7	1.4	43.3	34.3	38.3	5.5
EasyOCR	10.4	11.4	10.8	0.0	29.8	17.0	21.7	0.0	30.6	19.7	24.0	3.5
ChemicalOCR (Ours)	90.0	90.6	90.3	37.9	96.2	95.3	95.8	73.0	90.3	92.2	91.3	73.7
@IoU $>$ 0.3

Table 9: Comparison of OCR performance for different IoU thresholds : Comparison of our ChemicalOCR model with existing OCR models for different IoU thresholds. The evaluation is conducted on real-world benchmarks (M2S, USPTO-M, and IP5-M). Precision

P

, Recall

R

, and F1 are measured at individual bounding-box level. Accuracy

A

is measured at the image level; an image is considered correct if all OCR cells have an IoU greater than the threshold (0.0, 0.3, 0.5), and their recognized characters match the ground truth.

Models	M2S		JPO
	Valid	Correct	Valid	Correct
DeepSeek-OCR	15	0	72	32
GPT-5	30	3	74	19
MarkushGrapher-2	95	56	92	71

Table 10: Comparison of (CX)SMILES predictions: Comparison of the percentage of chemically-valid and correct (CX)SMILES generated by multi-purpose models (DeepSeek-OCR and GPT-5) versus our dedicated MarkushGrapher-2 model. “Valid” reports the percentage of chemically-valid (CX)SMILES, and “Correct” denotes the percentage of correct CXSMILES predictions (i.e., accuracy).

Methods	IP5-M
	InChi	Variable	AP	m	Sg	A
MarkushGrapher-2	73.3(70.8)	74.8	73.9	78.8	30.7	53.7(51.5)

Table 11: Comparison of Markush features performances : Breakdown of model performance for backbone structure (InChi), variable groups, attach points, positional and frequency variation indicators and total accuracy. Scores reported in parentheses consider stereochemistry. (6.4% of samples contain stereochemistry).

Benchmarks	Samples	Percentage of CXSMILES with at least one				Mean num. atoms	Mean num. OCR cells
		Variable group	Attach point	m-section	Sg-section
M2S	103	97	0	30	25	19	15
USPTO-M	74	91	0	74	42	20	8
WildMol-M	10000	59	31	15	12	14	-
IP5-M	1000	60	28	17	26	14	7

Table 12: Benchmarks - Statistics : Comparison of the number of samples, percentage of dataset samples containing at least one Markush feature (variable group, attach point, positional variation indicator (m-section), frequency variation indicator (Sg-section)), mean number of heavy atoms, and mean number of OCR cells.

Figure 12: Qualitative Evaluation : MarkushGrapher-2 predictions of benchmark samples. A correct and an incorrect prediction is displayed for the features stereochemistry, variable groups, attach points, positional variation indicators, abbreviations, and frequency variation indicators.

Document Image Understanding

Document image understanding turns page images into recognized text, layout, structured data, and answers grounded in visual evidence.

See all 148 papers

MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition
Jul 30, 2026Alex Andonian, Samuel G Rodriques, Andrew D White +1Vision-Language Models Optical Character Recognition
1Edison Scientific
MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition
May 7, 2026Haote Yang, Hui Wang, Chen Zhu +14Optical Character Recognition
Shanghai Artificial Intelligence Laboratory · King’s College London · East China University of Science and Technology +5
MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognition
Aug 4, 2026Haote Yang, Jiang Wu, Jingchao Wang +42Document Information Extraction Optical Character Recognition

Jul 30, 2026J/K move · Enter open · S save

MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition

Alex Andonian, Samuel G Rodriques, Andrew D White, Siddharth M Narayanan

1Edison Scientific

Vision-Language Models Optical Character Recognition

Open paper AskPDF arXiv

Chemical structures appear in patents and the scientific literature as images. For programmatic usage, such as indexing in databases or constructing machine learning model training sets, they must be transformed into line notations. The two common forms of this task are translating an image of a single molecule (optical chemical structure recognition - OCSR) and translating a Markush structure that represents a family of molecules. While prior work in the former case is quite mature, Markush structure parsing remains a challenging task. In this work, we treat both tasks as an image-to-text translation problem. We then propose OCSRGlyph, a state-of-the-art OCSR model, improving performance over prior methods by carefully considering stereochemistry. For the Markush task, we introduce MarkushGlyph, a vision-language model that reads the entire Markush structure as an image. This contrasts with prior systems, which often use multiple stages to separately process visual and text input content. Finally, we introduce a new metric for determining the accuracy of Markush structure translations, handling failure modes present in prior metrics.

Abstract

Document Image Understanding

Explore similar work

MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition

MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition

MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognition