cs.AIOct 23, 2026
SaveRxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature
Organizations: Shanghai Artificial Intelligence Laboratory · Peking University · South China Normal University · Shanghai Jiao Tong University ·
Abstract
Screen Tables and Substrate Scopes are information-dense figures widely used in synthetic chemistry literature, yet automated extraction of structured reaction data from them remains challenging. We introduce RxnExBench, a multimodal benchmark for fine-grained reaction-level evaluation. Built from high-impact chemistry journals, it contains 3,565 expert-annotated reactions and 2,361 molecular structures. The dataset adopts a unified schema with 15 reaction roles and 26 core fields, supporting downstream AI-for-Chemistry tasks such as retrosynthetic planning, yield prediction, and reaction condition recommendation. RxnExBench further provides a reasoning-driven difficulty taxonomy, a modular annotation pipeline, and the Oracle-MolRec protocol, which disentangles molecular recognition from higher-level reasoning. Evaluation of 9 state-of-the-art VLMs shows that performance degrades substantially as schema complexity and reasoning demands increase. Under the Oracle-MolRec setting, results further reveal that accurate molecular recognition alone is insufficient, with multimodal integration and chemistry-aware reasoning remaining the primary bottlenecks. We release all data and code to facilitate future research in chemistry-aware multimodal understanding and AI-for-Chemistry dataset construction. The dataset and code will be released at https://github.com/opendatalab/RxnExBench.
Figures & tables
Figure 1: Screen Table and Substrate Scope are information-rich figure types commonly used in chemical papers to present chemical reactions. A typical Screen Table or Substrate Scope usually contains: a “reaction template” presented as an image (purple box), “instantiated” chemical reactions in a tabular or substrate array format (yellow box), and a footnote for important supplementary information that cannot be fully displayed in the figure (red box). The task of RxnExBench is to extract structured information for all chemical reactions from these figures according to the defined schema.
| Benchmark | Figure Types | StructRxnEx | R-group Resolved | CLS Figures | Difficulty Levels | # Schema Fields | #Rxn | Avg Tokens /Fig | Avg Rxn /Fig |
|---|---|---|---|---|---|---|---|---|---|
| ChemTable | Screen Table, Substrate Scope | – | – | – | – | – | – | ||
| MERMES-T24 | Substrate Scope | – | – | – | – | – | – | ||
| ChemEAGLE | Substrate Scope | Level 3 | 11 | 1120 | 1,179.91 | 9.21 | |||
| MerMaid | Screen Table | Level 1,2 | 15 | 1343 | 2,707.64 | 12.32 | |||
| RxnExBench | Screen Table, Substrate Scope | Level 1,2,3 | 26 | 3565 | 7,205.28 | 13.40 |
Table 1: Comparison of related benchmarks and our RxnExBench . StructRxnEx indicates that the benchmark’s task is structured chemical reaction information extraction; R-group Resolved means that for all instantiated chemical reactions, the R-groups in all molecular structure diagrams have been resolved; “ CLS figures" means that the benchmark contains the “complex-line-substrate" figures (see Figure 3 for details).
Figure 2: Data collection and annotation of RxnExBench
| Type | #Figures | #Min Rxn | #Max Rxn | #Avg Rxn | #Total Rxn |
| Screen Table | 176 | 3 | 29 | 11.53 | 2,029 |
| Substrate Scope | 90 | 3 | 44 | 17.07 | 1,536 |
| Level 1 | 60 | 4 | 19 | 9.85 | 591 |
| Level 2 | 90 | 3 | 29 | 12.12 | 1,091 |
| Level 3 | 116 | 3 | 44 | 16.23 | 1,883 |
| Total | 266 | 3 | 44 | 13.40 | 3,565 |
Table 2: Statistics of reaction figures and counts.
Figure 3: In Screen Tables , each row typically corresponds to one instantiated reaction; in Substrate Scopes , each substrate likewise. Figures where all rows/substrates meet this condition are called “simple-line-substrate” ( SLS ) figures, as shown in (a). However, some rows or substrates may contain multiple reactions; these are termed “complex-line-substrate” ( CLS ) figures, as shown in (b).
Figure 4: Three stages of structured chemical reaction extraction from Screen Tables and Substrate Scopes , describing the processes of human expert reading, VLM understanding, and manual data annotation.
| Models | All Levels (266 Figures) | Level 1 (60 Figures) | Level 2 (90 Figures) | Level 3 (116 Figures) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | |
| Claude Opus 4.6 | 2.52% | 1.08% | 0.94% | 5.60% | 4.24% | 4.24% | 2.40% | 1.20% | 0.74% | 1.62% | 0.00% | 0.00% |
| Gemini 3.1 Pro | 19.47% | 9.57% | 8.86% | 40.42% | 20.56% | 19.86% | 15.46% | 10.22% | 8.25% | 15.24% | 5.77% | 5.77% |
| GPT-5.5 | 23.16% | 9.16% | 6.81% | 33.50% | 12.79% | 12.63% | 28.13% | 13.93% | 6.41% | 16.91% | 5.18% | 5.18% |
| GPT-4o | 0.09% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.17% | 0.00% | 0.00% |
| Doubao Seed 2.0 Pro | 19.81% | 6.59% | 5.35% | 33.56% | 11.36% | 10.68% | 25.70% | 7.14% | 4.55% | 11.80% | 4.71% | 4.10% |
Table 3: F1 scores of VLMs on RxnExBench (precision/recall in Appendix G). Results are shown for the full dataset and three difficulty levels under three evaluation criteria. C refers to Criterion . Boldfaced numbers denote the best performance, while underlined numbers represent the runner-up.
Figure 5: Average F1 scores of all VLMs across RxnExBench difficulty levels and evaluation criteria.
Figure 6: Comparison of reaction counts in figure between VLM predictions and ground truth, grouped by simple-line-substrate (SLS) and complex-line-substrate (CLS) figures. While VLMs achieve accurate reaction counts on SLS figures, they exhibit substantial omission errors on CLS figures.
Figure 7: Error type distributions for three representative models.
| Models | Criterion 1 | Criterion 2 | Criterion 3 |
|---|---|---|---|
| Claude Opus 4.6 | 30.04%( ↑27.52% ) | 17.20% ( ↑16.12% ) | 14.78% ( ↑13.84% ) |
| Gemini 3.1 Pro | 36.70%( ↑17.23% ) | 18.25% ( ↑8.68% ) | 15.88% ( ↑7.02% ) |
| GPT-5.5 | 42.67% ( ↑19.51% ) | 16.83%( ↑7.67% ) | 15.29% ( ↑8.48% ) |
| GPT-4o | 25.25%( ↑25.16% ) | 5.32%( ↑5.32% ) | 4.57%( ↑4.57% ) |
| Doubao Seed 2.0 Pro | 36.73% ( ↑16.92% ) | 11.94%( ↑5.35% ) | 10.41%( ↑5.06% ) |
| GLM-5V Turbo | 31.95%( ↑26.87% ) | 11.99%( ↑10.07% ) | 11.11%( ↑9.31% ) |
Table 4: VLM performance (F1 scores) under Oracle-MolRec inference setting on the RxnExBench-MolIdt subset. ( Red numbers in parentheses ) show improvement over the standard inference setting.
Figure 8: Example of Screen Table.
Figure 9: Example of Substrate Scope.
| Reaction roles | Fine grained fields | |
|---|---|---|
| reactants (list) | SMILES (str) text (str) amount (str) conversion rate (str) | |
| products (list) | SMILES (str) text (str) yield ratio (str) isolated yield (str) enantiomeric excess (str) regioisomeric ratio (str) enantiomeric ratio (str) alkene configuration (str) diastereomeric ratio (str) | |
| stage (list) | reagents (list) | SMILES (str) text (str) amount (str) |
| catalysts (list) | SMILES (str) text (str) amount (str) | |
Table 5: Structured chemical reaction schema of RxnExbench.
| Journal Name |
|---|
| Angewandte Chemie International Edition |
| Chemical Science |
| Journal of the American Chemical Society |
| ACS Catalysis |
| Green Chemistry |
| ACS Central Science |
Table 6: Journal List
| Figure Type | Screen Table | Substrate Scope |
|---|---|---|
| Reactions | 11.53 | 17.07 |
| Annotated Fields | 141.81 | 248.17 |
| Molecular structures | 36.17 | 43.67 |
| Atoms | 441.65 | 584.00 |
| Chemical bonds | 456.76 | 605.00 |
Table 7: Average Per-image Annotation Statistics
Figure 10: Screenshot for Molecular Structure Localization
Figure 11: Screenshot for Reaction Template Annotation
Figure 12: Screenshot for Pre-annotation of Instantiation References
Figure 13: Screenshot for Instantiation Execution
Figure 14: Screenshot for Molecular Structure (SMILES) Annotation
| Subset | Criterion 1 | Criterion 2 | Criterion 3 |
|---|---|---|---|
| Level 1 | 98.98% | 96.79% | 94.92% |
| Level 2 | 93.49% | 86.25% | 81.76% |
| Level 3 | 87.68% | 81.09% | 77.43% |
| All | 91.33% | 85.27% | 81.66% |
Table 8: Human expert performance (F1, %) on RxnExBench, broken down by difficulty level and evaluation criterion. Numbers are averaged over three independent experts. The overall human F1 of 91.33% under Criterion 1 serves as an empirical performance ceiling for this benchmark.
Figure 15: Example of Screen Table . The annotation results for this Screen Table are shown in Figures 27 and 28 .
Figure 16: Example of Substrate Scope . The annotation results for this Substrate Scope are shown in Figures 29 and 30 .
| Criterion | Downstream Task Category | Reaction Roles | Evaluation Description |
|---|---|---|---|
| 1 | Retrosynthesis Analysis [ 6 ] ; Main Product Prediction [ 10 ] ; Reaction Classification [ 28 ] | reactant, product, reagent | Focuses on the correctness of the primary reaction roles. (SMILES or text field indicating the identity of the substance) |
| 2 | Yield Prediction [ 1 ] ; Selectivity prediction [ 13 ] | reactant, product, reagent, catalyst, solvent | Focuses on the correctness of reaction roles and fine-grained fields. (SMILES or text indicating the identity of the substance, and fields such as amount and yield ratio that describe quantitative reaction information.) |
| 3 | Reaction Condition Recommendation; Reaction Kinetics Modeling; HTE Screening [ 19 , 25 ] | reactant, product, reagent, catalyst, ligand, solvent, pressure, reaction_time, reaction_temperature, pH | Focuses on the correctness of reaction roles and fine-grained fields.(SMILES or text indicating the identity of the substance, and fields such as amount and yield ratio that describe quantitative reaction information.) |
Table 9: Criterion details of RxnExBench.
Figure 17: Example of the input image of Oracle-MolRec-BBOX-IDX inference . The ground-truth bounding box of each molecule is drawn on the original image as visual prompts and a unique index number is labeled at the top-left corner. The corresponding prompt for this image is shown in Figure 23 .
| Models | All Levels (266 Figures) | Level 1 (60 Figures) | Level 2 (90 Figures) | Level 3 (116 Figures) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | |
| Claude Opus 4.6 | 2.50% | 1.07% | 0.93% | 5.58% | 4.23% | 4.23% | 2.38% | 1.19% | 0.73% | 1.59% | 0.00% | 0.00% |
| Gemini 3.1 Pro | 18.99% | 9.34% | 8.64% | 39.26% | 19.97% | 19.29% | 15.12% | 9.99% | 8.07% | 14.87% | 5.63% | 5.63% |
| GPT-5.5 | 22.97% | 9.09% | 6.76% | 33.67% | 12.86% | 12.69% | 28.14% | 13.93% | 6.42% | 16.62% | 5.10% | 5.10% |
| GPT-4o | 0.08% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.16% | 0.00% | 0.00% |
| Doubao Seed 2.0 Pro | 19.30% | 6.42% | 5.22% | 33.50% | 11.34% | 10.66% | 25.39% | 7.06% | 4.49% | 11.31% | 4.51% | 3.93% |
Table 10: Recall scores of VLMs on RxnExBench. Results are shown for the full dataset and three difficulty levels under three evaluation criteria. C refers to Criterion . Boldfaced numbers denote the best performance, while underlined numbers represent the runner-up.
| Models | All Levels (266 Figures) | Level 1 (60 Figures) | Level 2 (90 Figures) | Level 3 (116 Figures) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | |
| Claude Opus 4.6 | 2.55% | 1.09% | 0.95% | 5.61% | 4.25% | 4.25% | 2.42% | 1.21% | 0.74% | 1.64% | 0.00% | 0.00% |
| Gemini 3.1 Pro | 19.96% | 9.82% | 9.08% | 41.65% | 21.18% | 20.47% | 15.82% | 10.45% | 8.44% | 15.63% | 5.92% | 5.92% |
| GPT-5.5 | 23.34% | 9.23% | 6.87% | 33.33% | 12.73% | 12.56% | 28.11% | 13.92% | 6.41% | 17.20% | 5.27% | 5.27% |
| GPT-4o | 0.09% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.17% | 0.00% | 0.00% |
| Doubao Seed 2.0 Pro | 20.34% | 6.77% | 5.50% | 33.62% | 11.38% | 10.70% | 26.01% | 7.23% | 4.60% | 12.33% | 4.92% | 4.28% |
Table 11: Precision scores of VLMs on RxnExBench. Results are shown for the full dataset and three difficulty levels under three evaluation criteria. C refers to Criterion . Boldfaced numbers denote the best performance, while underlined numbers represent the runner-up.
| Models | Standard Inference | Oracle-MolRec-IDT | Oracle-MolRec-BBOX-CORD | Oracle-MolRec-BBOX-IDX | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | |
| Claude Opus 4.6 | 2.50% | 1.07% | 0.93% | 29.63% | 16.96% | 14.58% | 28.83% | 14.77% | 12.31% | 29.98% | 15.13% | 12.95% |
| Gemini 3.1 Pro | 18.99% | 9.34% | 8.64% | 35.15% | 17.47% | 15.21% | 31.85% | 16.32% | 13.98% | 34.99% | 19.18% | 16.16% |
| GPT-5.5 | 22.97% | 9.09% | 6.76% | 41.78% | 16.48% | 14.97% | 45.04% | 16.12% | 14.26% | 45.75% | 14.97% | 12.67% |
| GPT-4o | 0.08% | 0.00% | 0.00% | 24.15% | 5.08% | 4.37% | 22.16% | 5.36% | 4.25% | 21.37% | 4.01% | 2.94% |
| Doubao Seed 2.0 Pro | 19.30% | 6.42% | 5.22% | 35.31% | 11.48% | 10.01% | 35.19% | 11.32% | 10.44% | 36.30% | 10.56% | 9.93% |
Table 12: The Recall scores of each VLM under the four inference settings: Standard Inference , Oracle-MolRec-IDT , Oracle-MolRec-BBOX-CORD , and Oracle-MolRec-BBOX-IDX , on the RxnExBench-MolIDT subset. C refers to Criterion . Boldfaced numbers denote the best performance, while underlined numbers represent the runner-up.
| Criterion | Recall | Precision | F1-score |
|---|---|---|---|
| Criterion1 | 3.99% | 4.68% | 4.31% |
| Criterion2 | 0.59% | 0.69% | 0.64% |
| Criterion3 | 0.59% | 0.69% | 0.64% |
Table 13: Results of Mermaid on RxnExBench
| Criterion | Recall | Precision | F1-score |
|---|---|---|---|
| Criterion1 | 0.31% | 0.43% | 0.36% |
| Criterion2 | 0.00% | 0.00% | 0.00% |
| Criterion3 | 0.00% | 0.00% | 0.00% |
Table 14: Results of ChemEAGLE on RxnExBench
| Models | Standard Inference | Oracle-MolRec-IDT | Oracle-MolRec-BBOX-CORD | Oracle-MolRec-BBOX-IDX | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | |
| Claude Opus 4.6 | 2.55% | 1.09% | 0.95% | 30.47% | 17.44% | 14.99% | 29.50% | 15.12% | 12.60% | 30.80% | 15.54% | 13.30% |
| Gemini 3.1 Pro | 19.96% | 9.82% | 9.08% | 38.39% | 19.09% | 16.62% | 35.98% | 18.44% | 15.79% | 38.83% | 21.29% | 17.94% |
| GPT-5.5 | 23.34% | 9.23% | 6.87% | 43.60% | 17.20% | 15.62% | 45.89% | 16.43% | 14.53% | 46.30% | 15.15% | 12.82% |
| GPT-4o | 0.09% | 0.00% | 0.00% | 26.46% | 5.57% | 4.79% | 24.37% | 5.90% | 4.67% | 23.41% | 4.40% | 3.22% |
| Doubao Seed 2.0 Pro | 20.34% | 6.77% | 5.50% | 38.27% | 12.44% | 10.85% | 37.88% | 12.18% | 11.24% | 38.36% | 11.16% | 10.49% |
Table 15: The Precision scores of each VLM under the four inference settings: Standard Inference , Oracle-MolRec-IDT , Oracle-MolRec-BBOX-CORD , and Oracle-MolRec-BBOX-IDX , on the RxnExBench-MolIDT subset. C refers to Criterion . Boldfaced numbers denote the best performance, while underlined numbers represent the runner-up.
| Models | Standard Inference | Oracle-MolRec-IDT | Oracle-MolRec-BBOX-CORD | Oracle-MolRec-BBOX-IDX | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | C1 | C2 | C3 | |
| Claude Opus 4.6 | 2.52% | 1.08% | 0.94% | 30.04% | 17.20% | 14.78% | 29.16% | 14.94% | 12.45% | 30.39% | 15.34% | 13.12% |
| Gemini 3.1 Pro | 19.47% | 9.57% | 8.86% | 36.70% | 18.25% | 15.88% | 33.79% | 17.32% | 14.83% | 36.81% | 20.18% | 17.00% |
| GPT-5.5 | 23.16% | 9.16% | 6.81% | 42.67% | 16.83% | 15.29% | 45.46% | 16.28% | 14.39% | 46.02% | 15.06% | 12.74% |
| GPT-4o | 0.09% | 0.00% | 0.00% | 25.25% | 5.32% | 4.57% | 23.21% | 5.62% | 4.45% | 22.34% | 4.19% | 3.07% |
| Doubao Seed 2.0 Pro | 19.81% | 6.59% | 5.35% | 36.73% | 11.94% | 10.41% | 36.48% | 11.74% | 10.83% | 37.30% | 10.85% | 10.20% |
Table 16: The F1 scores of each VLM under the four inference settings: Standard Inference , Oracle-MolRec-IDT , Oracle-MolRec-BBOX-CORD , and Oracle-MolRec-BBOX-IDX , on the RxnExBench-MolIDT subset. C refers to Criterion . Boldfaced numbers denote the best performance, while underlined numbers represent the runner-up.
Figure 18: Venn diagrams showing error type distribution for each model. The diagrams illustrate the distribution and overlap of three error types: SMILES Error, Field Missing, and Content Error. Percentages indicate the proportion of each error category among all errors.
Figure 19: Standard inference prompt of Screen Table
Figure 20: Standard inference prompt of Substrate Scope
Figure 21: Oracle-MolRec-IDT inference prompt of Screen Table
Figure 22: Oracle-MolRec-IDT inference prompt of Substrate Scope
Figure 23: Oracle-MolRec-BBOX-IDX inference prompt of Screen Table
Figure 24: Oracle-MolRec-BBOX-IDX inference prompt of Substrate Scope
Figure 25: Oracle-MolRec-BBOX-CORD inference prompt of Screen Table
Figure 26: Oracle-MolRec-BBOX-CORD inference prompt of Substrate Scope
Figure 27: Annotation result of Screen Table.
Figure 28: Annotation result of Screen Table.
Figure 29: Annotation result of Substrate Scope.
Figure 30: Annotation result of Substrate Scope.