Reaction diagram parsing (RxnDP) is critical for extracting chemical synthesis information from literature. Although recent Vision-Language Models (VLMs) have emerged as a promising paradigm to automate this complex visual reasoning task, their application is fundamentally bottlenecked by the inability to align visual chemical entities with pre-trained knowledge, alongside the inherent discrepancy between token-level training and reaction-level evaluation. To address these dual challenges, this work enhances VLM-based RxnDP from two complementary perspectives: prompting representation and learning paradigms. First, we propose Identifier as Visual Prompting (IdtVP), which leverages naturally occurring molecule identifiers (e.g., bold numerals like 1a) to activate the chemical knowledge acquired during VLM pre-training. IdtVP enables powerful zero-shot and out-of-distribution capabilities, outperforming existing prompting strategies. Second, to further optimize performance within fine-tuning paradigms, we introduce Re3-DAPO, a reinforcement learning algorithm that leverages verifiable rewards to directly optimize reaction-level metrics, thereby achieving consistent gains over standard supervised fine-tuning. Additionally, we release the ScannedRxn benchmark, comprising scanned historical reaction diagrams with real-world artifacts, to rigorously assess model robustness and out-of-distribution ability. Our contributions advance the accuracy and generalization of VLM-based reaction diagram parsing. We will release data, models, and code on GitHub.
Figures & tables
Figure 1: Overview of the Id en t ifier as V isual P rompting ( IdtVP ) strategy. Left: VLMs learn visual-text alignment from identifier-rich interleaved literature during pre-training. Middle: IdtVP inputs diagrams annotated with molecule identifiers to activate this pre-trained knowledge. Right: The model outputs structured reactions, directly using these identifiers as “handles” to concisely represent complex molecules.
Strategy
RxnScribe-test
RxnCaption-15k-test
H-F1
S-F1
H-F1
S-F1
Gemini 3 Pro + BROS
3.90
71.20
4.92
49.80
Gemini 3 Pro + BIVP
45.27
81.32
47.40
73.40
Gemini 3 Pro + IdtVP
71.70
88.11
58.30
78.23
Qwen2.5-VL-7B + IdtVP
16.69
39.00
4.88
19.16
Intern-S1 + IdtVP
44.71
60.84
22.69
36.45
Table 1: Comparison of different visual prompting strategies under zero-shot setting. H-F1 and S-F1 denote Hybrid-F1 and Soft-F1 scores, respectively (%).
Figure 2: Attention heatmaps of different prompting strategies. The top panels illustrate the input images, textual prompts, and corresponding output formats for each strategy. The bottom panels visualize the Text-to-Image attention heatmaps, computed by averaging the attention weights across all heads per layer, followed by average pooling across all layers. Notably, IdtVP yields far more precise and comprehensive visual grounding on molecules and text compared to BROS and BIVP.
Figure 3: Overview of the proposed framework. Phase 1 constructs IdtVP data effectively supporting both zero-shot inference and model training . Phases 2-3 introduce a generalized optimization paradigm (SFT followed by Re 3 -DAPO) that is transferable to other prompting variants (e.g., BIVP).
Model
Image-level
Reaction-level
Total
Errors
Rate (%)
Total
Errors
Rate (%)
BIVP
3,882
500
12.88
20,992
3,131
14.92
IdtVP
5,833
272
4.66
31,300
2,538
8.11
Table 2: Analysis of sequential inconsistency in SFT models. Image-level statistics exclude single-reaction samples.
Model
Strategy
RxnScribe-test
RxnCaption-15k-test
ScannedRxn
Hybrid Match
Soft Match
Hybrid Match
Soft Match
Hybrid Match
Soft Match
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Trained Models
RxnID+RL
IdtVP
76.3
73.7
75.0
87.2
84.6
85.9
65.5
63.4
64.4
75.8
73.2
74.5
60.2
52.8
56.3
82.5
70.8
76.2
RxnID
IdtVP
75.8
73.5
74.6
87.2
84.5
85.6
64.2
61.1
61.2
75.2
70.6
72.8
58.1
51.4
54.5
80.5
69.1
74.4
RxnCaption+RL
BIVP
76.0
75.8
75.9
86.8
87.1
86.9
65.5
62.9
64.1
75.3
72.0
73.6
54.2
38.6
45.1
72.2
50.2
59.2
Table 3: Model performance comparison on RxnScribe-test, RxnCaption-15k-test, and the new ScannedRxn datasets. Best scores are in bold , second best are underlined . +RL means post trained by Re 3 -DAPO , same as below.
Detector
VLM
RxnCaption-15k-test
Soft-F1
Hybrid-F1
Mid-Mapper
IdtVP+RL
74.5
64.4
Gemini
78.2
58.3
Gemini
IdtVP+RL
75.6
64.7
Gemini
78.6
60.2
Ground Truth
IdtVP+RL
73.6
63.1
Table 4: Performance comparison of different identifier detectors on the RxnCaption-15k-test set. We evaluate the impact of detector precision on the Soft-F1 and Hybrid-F1 scores across different VLM prompting strategies.
Error Source
RS
RC
SR
Met.
Drp ↓
Met.
Drp ↓
Met.
Drp ↓
Upper Bound
100.0
-
100.0
-
100.0
-
MolYOLO Err.
(Ideal RxnDP VLM)
99.7
0.3
95.3
4.7
87.1
12.9
Mid-Mapper Err.
(Ideal RxnDP VLM)
98.5
1.2
93.8
1.5
84.7
2.4
Table 5: Main results on test set (RS=RxnScribe, RC=RxnCaption-15k-test, SR=ScannedRxn). Met. and Drp.( ↓ ) denote the Metric value and the performance drop relative to the upper bound, respectively.
Figure 4: Reward Δ change on the val dataset. The Y-axis denotes the reward improvement relative to the initial step (Step 0).
Ratio
Strategy
RxnCaption-15k-test
Soft-F1
Hybrid-F1
0 : 1
BIVP+RL
72.5
64.5
IdtVP+RL
73.6
64.4
1 : 0
BIVP+RL
73.8
62.7
IdtVP+RL
74.2
62.8
1 : 1
BIVP+RL
73.6
64.1
Table 6: Quantitative results on RxnCaption-15k-test. Comparison across reward ratios.
Figure 5: Overview of the Cross-Modal Verification pipeline. System processes double-stream inputs: visual parsing of raw diagrams via RxnID (top) and textual extraction from the manuscript via Idt-TE (bottom), enabling downstream applications such as Precision Refinement (Case A) and Contextual Enrichment (Case B).
Table 7: Prompt used for identifier extraction.
Figure 6: Overview of the annotation pipeline for the ScannedRxn dataset.
Figure 7: Visualization of sample data from the ScannedRxn dataset.
Figure 8: Fine-grained performance comparison across different reaction diagram types (Single, Multi-line, Tree, and Cyclic). We evaluate the F1 scores (Hybrid and Soft Match) by performing Supervised Fine-Tuning (SFT) on Qwen2.5-VL and testing the zero-shot capabilities on Gemini 3.0 Pro. Both are evaluated under three representation strategies: BROS, BIVP, and IdtVP.
Figure 9: Typical error cases of RxnID. We categorize the common failure modes into three types: (A) Low-resolution diagrams, where degraded visual quality frequently leads to inaccurate OCR predictions. (B) The presence of unconventional graphical icons (e.g., lightbulbs or sun symbols, highlighted by red boxes). Since these are neither standard chemical structures nor text, the model struggles to interpret them correctly. (C) Highly complex Cyclic diagrams containing a dense concentration of reaction steps, which challenge the model’s spatial reasoning capabilities.
Figure 10: The overall architecture of Idt-TE pipeline. Specifically, the Idt-TE pipeline operates in three stages: Phase 1 digitizes and structures the raw manuscript into logical paragraphs. Phase 2 employs a cascaded, context-aware LLM framework to resolve cross-paragraph dependencies and extract reaction entities based on semantic anchors. Finally, Phase 3 applies rigorous rule-based curation to validate and canonicalize the extracted records, ultimately enabling precision refinement and contextual enrichment of the visual predictions.
Type
Semantic Meaning
Example Format
"title"
References a specific section title (e.g., Example, Preparation, Intermediate).
["Example 3", "title"]
"id"
References a specific compound identifier.
["Compound 8", "id"]
"iupac_name"
References a prior synthesis step via its IUPAC name.
["2-chloroaniline", "iupac_name"]
"method"
References a universally defined general procedure or method.
["General Procedure I", "method"]
"last_compound"
Implicitly inherits the synthesis method of the immediately preceding reaction.
["last_compound", "last_compound"]
Table 8: Taxonomy of external procedural dependencies evaluated in Step 2.
Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.
Parsing chemical reaction diagrams from scientific literature is challenging due to heterogeneous layouts, intertwined visual elements, and the difficulty of integrating recognition and reasoning. Existing vision-language models advance multimodal understanding but still fail on complex diagrams, struggling to maintain spatial coherence and to integrate multidimensional information during reasoning. To address these issues, we propose MACReD, a hierarchical multi-agent framework that coordinates specialized agents for molecular perception, arrow understanding, text extraction, and reaction reconstruction within a unified VLM-guided architecture. The planning and perception layers use flexible, fine-grained detection to handle visual complexity, while the reasoning layer uses a multigraph fusion mechanism to integrate heterogeneous cues and enforce chemically consistent global reasoning. Experiments on the RxnScribe benchmark show that MACReD achieves state-of-the-art performance, with F1 scores of 75.2% and 84.6% under hard and soft match criteria, outperforming the RxnScribe baseline, which obtains 69.1% and 80.0%, respectively. These results demonstrate the robustness of MACReD across diverse diagram layouts, including multi-step and tree-structured reactions.
Chuang Tang, Chenhao Lin, Yin Xu +5
University of Science and Technology of China Hefei, China
While Large Language Models (LLMs) have revolutionized scientific text processing, they exhibit a significant capability gap when interpreting chemical reaction diagrams. We identify two fundamental bottlenecks restricting current systems: a Visual Deficit, where generic vision encoders struggle to resolve the strict topological connectivity of dense molecular graphs, and a Semantic Disconnect, where standard linear strings, such as SMILES, fail to effectively activate the model's latent chemical reasoning. To bridge these gaps, we propose the Chemical Visual Activation (ChemVA) framework, which employs a Visual Anchor mechanism to ground functional groups via hybrid-granularity detection, followed by a semantic alignment approach that translates visual features into entity names to maximize knowledge activation in LLMs. We evaluate our approach on OCRD-Bench, a newly constructed dataset featuring dense visual-semantic contexts and comprehensive reaction coverage to evaluate the full spectrum from recognition to reasoning. Extensive experiments on OCRD-Bench demonstrate that ChemVA achieves 92.0% structural recognition accuracy. By bridging visual and semantic bottlenecks, our framework delivers a consistent performance gain of approximately 20 percentage points across 9 diverse LLMs, enabling open-weight models to rival proprietary SOTA systems in complex chemical reasoning tasks.
Mingyang Rao, Kehua Feng, Zhihui Zhu +4
College of Computer Science and Technology, Zhejiang University Hangzhou, Zhejiang, China · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University Hangzhou, Zhejiang, China · Department of Chemistry, Fudan University Shanghai, China
Screen Tables and Substrate Scopes are information-dense figures widely used in synthetic chemistry literature, yet automated extraction of structured reaction data from them remains challenging. We introduce RxnExBench, a multimodal benchmark for fine-grained reaction-level evaluation. Built from high-impact chemistry journals, it contains 3,565 expert-annotated reactions and 2,361 molecular structures. The dataset adopts a unified schema with 15 reaction roles and 26 core fields, supporting downstream AI-for-Chemistry tasks such as retrosynthetic planning, yield prediction, and reaction condition recommendation. RxnExBench further provides a reasoning-driven difficulty taxonomy, a modular annotation pipeline, and the Oracle-MolRec protocol, which disentangles molecular recognition from higher-level reasoning. Evaluation of 9 state-of-the-art VLMs shows that performance degrades substantially as schema complexity and reasoning demands increase. Under the Oracle-MolRec setting, results further reveal that accurate molecular recognition alone is insufficient, with multimodal integration and chemistry-aware reasoning remaining the primary bottlenecks. We release all data and code to facilitate future research in chemistry-aware multimodal understanding and AI-for-Chemistry dataset construction. The dataset and code will be released at https://github.com/opendatalab/RxnExBench.
Jiaxing Sun, Junyuan Gao, Ruikang Zhang +16
Shanghai Artificial Intelligence Laboratory · Peking University · South China Normal University +7