ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding

Xuanle Zhao; Xinyuan Cai; Xiang Cheng; Xiuyi Chen; Bo Xu

cs.CLApr 8, 2026

ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding

Authors: Xuanle Zhao, Xinyuan Cai, Xiang Cheng, Xiuyi Chen, Bo Xu

Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

While Vision-Language Models (VLMs) have demonstrated significant potential in chemical visual understanding, current models are predominantly optimized for direct visual question-answering tasks. This paradigm often results in "black-box" systems that fail to utilize the inherent capability of Large Language Models (LLMs) to infer underlying reaction mechanisms. In this work, we introduce ChemVLR, a chemical VLM designed to prioritize reasoning within the perception process. Unlike conventional chemical VLMs, ChemVLR analyzes visual inputs in a fine-grained manner by explicitly identifying granular chemical descriptors, such as functional groups, prior to generating answers. This approach ensures the production of explicit and interpretable reasoning paths for complex visual chemical problems. To facilitate this methodology, we implement a cross-modality reverse-engineering strategy, combined with a rigorous filtering pipeline, to curate a large-scale reasoning-and-captioning dataset comprising 760k high-quality samples across molecular and reaction tasks. Furthermore, we adopt a three-stage training framework that systemically builds model perception and reasoning capacity. Experiments demonstrate that ChemVLR achieves state-of-the-art (SOTA) performance, surpassing both leading proprietary models and domain-specific open-source baselines. We also provide comprehensive ablation studies to validate our training strategy and data generation designs. Code and model weights will be available at https://github.com/xxlllz/ChemVLR.

Figures & tables

Figure 1: Overview of our proposed Cross-Modality Reverse-Engineering strategy for data generation. The pipeline begins with an initial dataset constructed from textual SMILES QA pairs, enriched with auxiliary information, including IUPAC names, functional groups, and expert demonstrations. Once Gemini-2.5-Flash generates the reasoning and caption data, they are subjected to a rigorous three-stage filtering and verification process to ensure accuracy and quality. Subsequently, the filtered data are rendered into images using RDKit and Indigo code.

Setting	Mol. Rec.	Rxn. Rec.	Rxn. Pred.
SMILES	78.0%	58.0%	55.0%
+ Demo	80.0%	64.0%	62.0%
+ IUPAC	92.0%	71.0%	71.0%
+ RDKit	95.0%	73.0%	76.8%

Table 1: Effectiveness comparison of various data generation settings. We generate 10k samples for each category utilizing our reverse-engineering strategy, except for the last row which utilizes the whole reasoning set. The percentages represent the retention rate of high-quality data following the three-stage filtering protocol.

Figure 2: The training framework of ChemVLR proceeds in three stages. First, we conduct chemical-domain continual pre-training using caption data. Second, during the SFT stage, we train on a mixture of reasoning and instruction data. For the RL stage, we retain only non-trivial yet solvable instances and optimize utilizing DAPO.

Caption Dataset	Samples	Average	SD
ChEBI-MM	26k	151.57	80.46
ChemMLLM	70k	95.92	52.46
Molecule (Ours)	200k	409.93	139.03
Reaction (Ours)	200k	441.06	76.85

Table 2: Token Length Statistics of Caption Datasets. The mean and standard deviation (SD) of token counts are computed via the Qwen3-VL tokenizer.

Models	Paradigm	MMChemOCR		img2smiles		ChemRxn-V ${}_{\text{R}}$	ChemRxn-V ${}_{\text{P}}$
Models	Paradigm	Avg Sim.	Tani @ 1.0	Avg Sim.	Tani @ 1.0	Tani @ 1.0	Avg Sim.	Tani @ 1.0
Proprietary Models
zz Gemini-3-Flash	-	77.6	61.2	74.5	63.8	4.4	76.8	51.7
GPT-5-mini	-	28.8	5.7	20.4	5.1	0.7	24.3	2.3
GPT-4o	-	36.8	3.4	29.0	0.1	0.1	30.4	1.4
Open-Source General-Domain Models

Table 3: Combined evaluation on molecular and reaction tasks. Avg Sim. and [email protected] denote Average Tanimoto Similarity and Tanimoto [email protected]. ChemRxn-V

{}_{\text{R}}

and ChemRxn-V

{}_{\text{P}}

correspond to the reaction recognition and prediction subsets. Bold indicates the best performance.

Figure 3: The reward curve during the RL training process. The overall rewards are the weighted average of format and accuracy rewards.

Figure 4: Showcase of the reasoning process and final answer on the molcular recognition and reaction prediction tasks. The ChemVLR-8B generate the correct answer compare with other baselines.

Ablation	img2smiles	Rxn-V ${}_{\textbf{R}}$	Rxn-V ${}_{\textbf{P}}$
SFT only	81.5	60.2	56.8
RL only	2.1	0.2	0.1
CPT + SFT	84.7	63.5	59.3
ChemVLR-8B	92.7	74.4	67.8

Table 4: Ablation study of different training stages. Experiments are conducted using Qwen3-VL-8B-Instruct, with performance evaluated via [email protected]. For brevity, we refer to ChemRxn-V as Rxn-V.

Ablation	img2smiles	Rxn-V ${}_{\textbf{R}}$	Rxn-V ${}_{\textbf{P}}$
Full SFT Data	84.7	63.5	62.6
w/o IUPAC	80.8	56.2	56.1
w/o Reasoning	76.4	53.8	53.3
w/o Instuction	68.4	46.5	46.9

Table 5: Ablation study on SFT data composition. The instruction data contains the molecular-to-IUPAC data.

Ablation	img2smiles	Rxn-V ${}_{\textbf{R}}$	Rxn-V ${}_{\textbf{P}}$
Tani Simarity	90.2	70.8	65.9
Exact Matching	90.8	69.9	66.4
Struct-ID (Ours)	92.7	74.4	67.8

Table 6: Ablation study on different reward functions for RL. We compare our utilized structural identity reward against dense Tanimoto similarity and sparse exact string matching rewards.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Reasoning Data	Samples	Average	SD
Mol. Recognition	95k	811.10	307.44
Rxn. Recognition	73k	937.00	233.54
Rxn. Prediction	192k	1026.40	320.98

Appendix

Table 7: Statistics of reasoning thinking lengths in our constructed reasoning datasets. Token counts (mean and standard deviation) are calculated using the Qwen3-VL tokenizer.

Panel of Figure 5 — Figure 5: Full response of InternVL3.5-38B on the Figure 4 molecular recognition tasks.

Panel of Figure 6 — Figure 6: Full response of ChemVLR-8B-SFT on the Figure 4 molecular recognition tasks.

Panel of Figure 7 — Figure 7: Full response of ChemVLR-8B on the Figure 4 molecular recognition tasks.

Panel of Figure 8 — Figure 8: Full response of InternVL3.5-38B on the Figure 4 reaction prediction tasks.

Panel of Figure 9 — Figure 9: Full response of ChemVLR-8B on the Figure 4 molecular recognition tasks.

Panel of Figure 10 — Figure 10: Full response of ChemVLR-8B-SFT on the Figure 4 reaction prediction tasks.

Panel of Figure 11 — Figure 11: Full response of ChemVLR-8B on the Figure 4 reaction prediction tasks.

Panel of Figure 12 — Figure 12: The prompt to generate the reasoning process of reaction prediction.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 722 papers

ChemVA: Advancing Large Language Models on Chemical Reaction Diagrams Understanding
May 17, 2026Mingyang Rao, Kehua Feng, Zhihui Zhu +4Vision-Language Models VLM Reasoning
College of Computer Science and Technology, Zhejiang University Hangzhou, Zhejiang, China · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University Hangzhou, Zhejiang, China · Department of Chemistry, Fudan University Shanghai, China
Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models
Jul 14, 2026Xingyu Dang, Haocheng Tang, Junmei Wang +1LLM Reasoning
Department of Computer Science, Princeton University, Princeton, NJ, 08540, United States · School of Pharmacy, University of Pittsburgh, Pittsburgh, PA, 15261, United States · Khoury College of Computer Science, Northeastern University, Boston, MA, 02115, United States +2
Localize, Then Reason: Visual Latent Structural Reasoning for Molecular Properties and Edits
Aug 13, 2026Xingqiao Lin, Junmei Wang, Haocheng TangLatent Visual Reasoning Structured Reasoning
Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, United States · Department of Pharmaceutical Sciences and Computational Chemical Genomics Screening Center, School of Pharmacy, University of Pittsburgh, Pittsburgh, PA, USA · Khoury College of Computer Science, Northeastern University, Boston, MA, USA

May 17, 2026J/K move · Enter open · S save

ChemVA: Advancing Large Language Models on Chemical Reaction Diagrams Understanding

Mingyang Rao, Kehua Feng, Zhihui Zhu, Jiangzhen Fu+3

College of Computer Science and Technology, Zhejiang University Hangzhou, Zhejiang, China · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University Hangzhou, Zhejiang, China · Department of Chemistry, Fudan University Shanghai, China

Vision-Language Models VLM Reasoning

Open paper AskPDF arXiv

While Large Language Models (LLMs) have revolutionized scientific text processing, they exhibit a significant capability gap when interpreting chemical reaction diagrams. We identify two fundamental bottlenecks restricting current systems: a Visual Deficit, where generic vision encoders struggle to resolve the strict topological connectivity of dense molecular graphs, and a Semantic Disconnect, where standard linear strings, such as SMILES, fail to effectively activate the model's latent chemical reasoning. To bridge these gaps, we propose the Chemical Visual Activation (ChemVA) framework, which employs a Visual Anchor mechanism to ground functional groups via hybrid-granularity detection, followed by a semantic alignment approach that translates visual features into entity names to maximize knowledge activation in LLMs. We evaluate our approach on OCRD-Bench, a newly constructed dataset featuring dense visual-semantic contexts and comprehensive reaction coverage to evaluate the full spectrum from recognition to reasoning. Extensive experiments on OCRD-Bench demonstrate that ChemVA achieves 92.0% structural recognition accuracy. By bridging visual and semantic bottlenecks, our framework delivers a consistent performance gain of approximately 20 percentage points across 9 diverse LLMs, enabling open-weight models to rival proprietary SOTA systems in complex chemical reasoning tasks.

Abstract

Vision-Language Models

Explore similar work

ChemVA: Advancing Large Language Models on Chemical Reaction Diagrams Understanding

Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models

Localize, Then Reason: Visual Latent Structural Reasoning for Molecular Properties and Edits