Reaction diagram parsing (RxnDP) is critical for extracting chemical synthesis information from literature. Although recent Vision-Language Models (VLMs) have emerged as a promising paradigm to automate this complex visual reasoning task, their application is fundamentally bottlenecked by the inability to align visual chemical entities with pre-trained knowledge, alongside the inherent discrepancy between token-level training and reaction-level evaluation. To address these dual challenges, this work enhances VLM-based RxnDP from two complementary perspectives: prompting representation and learning paradigms. First, we propose Identifier as Visual Prompting (IdtVP), which leverages naturally occurring molecule identifiers (e.g., bold numerals like 1a) to activate the chemical knowledge acquired during VLM pre-training. IdtVP enables powerful zero-shot and out-of-distribution capabilities, outperforming existing prompting strategies. Second, to further optimize performance within fine-tuning paradigms, we introduce Re3-DAPO, a reinforcement learning algorithm that leverages verifiable rewards to directly optimize reaction-level metrics, thereby achieving consistent gains over standard supervised fine-tuning. Additionally, we release the ScannedRxn benchmark, comprising scanned historical reaction diagrams with real-world artifacts, to rigorously assess model robustness and out-of-distribution ability. Our contributions advance the accuracy and generalization of VLM-based reaction diagram parsing. We will release data, models, and code on GitHub.
Figures & tables
Figure 1: Overview of the Id en t ifier as V isual P rompting ( IdtVP ) strategy. Left: VLMs learn visual-text alignment from identifier-rich interleaved literature during pre-training. Middle: IdtVP inputs diagrams annotated with molecule identifiers to activate this pre-trained knowledge. Right: The model outputs structured reactions, directly using these identifiers as “handles” to concisely represent complex molecules.
Strategy
RxnScribe-test
RxnCaption-15k-test
H-F1
S-F1
H-F1
S-F1
Gemini 3 Pro + BROS
3.90
71.20
4.92
49.80
Gemini 3 Pro + BIVP
45.27
81.32
47.40
73.40
Gemini 3 Pro + IdtVP
71.70
88.11
58.30
78.23
Qwen2.5-VL-7B + IdtVP
16.69
39.00
4.88
19.16
Intern-S1 + IdtVP
44.71
60.84
22.69
36.45
Table 1: Comparison of different visual prompting strategies under zero-shot setting. H-F1 and S-F1 denote Hybrid-F1 and Soft-F1 scores, respectively (%).
Figure 2: Attention heatmaps of different prompting strategies. The top panels illustrate the input images, textual prompts, and corresponding output formats for each strategy. The bottom panels visualize the Text-to-Image attention heatmaps, computed by averaging the attention weights across all heads per layer, followed by average pooling across all layers. Notably, IdtVP yields far more precise and comprehensive visual grounding on molecules and text compared to BROS and BIVP.
Figure 3: Overview of the proposed framework. Phase 1 constructs IdtVP data effectively supporting both zero-shot inference and model training . Phases 2-3 introduce a generalized optimization paradigm (SFT followed by Re 3 -DAPO) that is transferable to other prompting variants (e.g., BIVP).
Model
Image-level
Reaction-level
Total
Errors
Rate (%)
Total
Errors
Rate (%)
BIVP
3,882
500
12.88
20,992
3,131
14.92
IdtVP
5,833
272
4.66
31,300
2,538
8.11
Table 2: Analysis of sequential inconsistency in SFT models. Image-level statistics exclude single-reaction samples.
Model
Strategy
RxnScribe-test
RxnCaption-15k-test
ScannedRxn
Hybrid Match
Soft Match
Hybrid Match
Soft Match
Hybrid Match
Soft Match
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Trained Models
RxnID+RL
IdtVP
76.3
73.7
75.0
87.2
84.6
85.9
65.5
63.4
64.4
75.8
73.2
74.5
60.2
52.8
56.3
82.5
70.8
76.2
RxnID
IdtVP
75.8
73.5
74.6
87.2
84.5
85.6
64.2
61.1
61.2
75.2
70.6
72.8
58.1
51.4
54.5
80.5
69.1
74.4
RxnCaption+RL
BIVP
76.0
75.8
75.9
86.8
87.1
86.9
65.5
62.9
64.1
75.3
72.0
73.6
54.2
38.6
45.1
72.2
50.2
59.2
Table 3: Model performance comparison on RxnScribe-test, RxnCaption-15k-test, and the new ScannedRxn datasets. Best scores are in bold , second best are underlined . +RL means post trained by Re 3 -DAPO , same as below.
Detector
VLM
RxnCaption-15k-test
Soft-F1
Hybrid-F1
Mid-Mapper
IdtVP+RL
74.5
64.4
Gemini
78.2
58.3
Gemini
IdtVP+RL
75.6
64.7
Gemini
78.6
60.2
Ground Truth
IdtVP+RL
73.6
63.1
Table 4: Performance comparison of different identifier detectors on the RxnCaption-15k-test set. We evaluate the impact of detector precision on the Soft-F1 and Hybrid-F1 scores across different VLM prompting strategies.
Error Source
RS
RC
SR
Met.
Drp ↓
Met.
Drp ↓
Met.
Drp ↓
Upper Bound
100.0
-
100.0
-
100.0
-
MolYOLO Err.
(Ideal RxnDP VLM)
99.7
0.3
95.3
4.7
87.1
12.9
Mid-Mapper Err.
(Ideal RxnDP VLM)
98.5
1.2
93.8
1.5
84.7
2.4
Table 5: Main results on test set (RS=RxnScribe, RC=RxnCaption-15k-test, SR=ScannedRxn). Met. and Drp.( ↓ ) denote the Metric value and the performance drop relative to the upper bound, respectively.
Figure 4: Reward Δ change on the val dataset. The Y-axis denotes the reward improvement relative to the initial step (Step 0).
Ratio
Strategy
RxnCaption-15k-test
Soft-F1
Hybrid-F1
0 : 1
BIVP+RL
72.5
64.5
IdtVP+RL
73.6
64.4
1 : 0
BIVP+RL
73.8
62.7
IdtVP+RL
74.2
62.8
1 : 1
BIVP+RL
73.6
64.1
Table 6: Quantitative results on RxnCaption-15k-test. Comparison across reward ratios.
Figure 5: Overview of the Cross-Modal Verification pipeline. System processes double-stream inputs: visual parsing of raw diagrams via RxnID (top) and textual extraction from the manuscript via Idt-TE (bottom), enabling downstream applications such as Precision Refinement (Case A) and Contextual Enrichment (Case B).
Table 7: Prompt used for identifier extraction.
Figure 6: Overview of the annotation pipeline for the ScannedRxn dataset.
Figure 7: Visualization of sample data from the ScannedRxn dataset.
Figure 8: Fine-grained performance comparison across different reaction diagram types (Single, Multi-line, Tree, and Cyclic). We evaluate the F1 scores (Hybrid and Soft Match) by performing Supervised Fine-Tuning (SFT) on Qwen2.5-VL and testing the zero-shot capabilities on Gemini 3.0 Pro. Both are evaluated under three representation strategies: BROS, BIVP, and IdtVP.
Figure 9: Typical error cases of RxnID. We categorize the common failure modes into three types: (A) Low-resolution diagrams, where degraded visual quality frequently leads to inaccurate OCR predictions. (B) The presence of unconventional graphical icons (e.g., lightbulbs or sun symbols, highlighted by red boxes). Since these are neither standard chemical structures nor text, the model struggles to interpret them correctly. (C) Highly complex Cyclic diagrams containing a dense concentration of reaction steps, which challenge the model’s spatial reasoning capabilities.
Figure 10: The overall architecture of Idt-TE pipeline. Specifically, the Idt-TE pipeline operates in three stages: Phase 1 digitizes and structures the raw manuscript into logical paragraphs. Phase 2 employs a cascaded, context-aware LLM framework to resolve cross-paragraph dependencies and extract reaction entities based on semantic anchors. Finally, Phase 3 applies rigorous rule-based curation to validate and canonicalize the extracted records, ultimately enabling precision refinement and contextual enrichment of the visual predictions.
Type
Semantic Meaning
Example Format
"title"
References a specific section title (e.g., Example, Preparation, Intermediate).
["Example 3", "title"]
"id"
References a specific compound identifier.
["Compound 8", "id"]
"iupac_name"
References a prior synthesis step via its IUPAC name.
["2-chloroaniline", "iupac_name"]
"method"
References a universally defined general procedure or method.
["General Procedure I", "method"]
"last_compound"
Implicitly inherits the synthesis method of the immediately preceding reaction.
["last_compound", "last_compound"]
Table 8: Taxonomy of external procedural dependencies evaluated in Step 2.
Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.
College of Computer Science and Technology, Zhejiang University Hangzhou, Zhejiang, China · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University Hangzhou, Zhejiang, China · Department of Chemistry, Fudan University Shanghai, China