cs.CLApr 8, 2026

ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding

Authors: Xuanle Zhao, Xinyuan Cai, Xiang Cheng, Xiuyi Chen, Bo Xu

Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While Vision-Language Models (VLMs) have demonstrated significant potential in chemical visual understanding, current models are predominantly optimized for direct visual question-answering tasks. This paradigm often results in "black-box" systems that fail to utilize the inherent capability of Large Language Models (LLMs) to infer underlying reaction mechanisms. In this work, we introduce ChemVLR, a chemical VLM designed to prioritize reasoning within the perception process. Unlike conventional chemical VLMs, ChemVLR analyzes visual inputs in a fine-grained manner by explicitly identifying granular chemical descriptors, such as functional groups, prior to generating answers. This approach ensures the production of explicit and interpretable reasoning paths for complex visual chemical problems. To facilitate this methodology, we implement a cross-modality reverse-engineering strategy, combined with a rigorous filtering pipeline, to curate a large-scale reasoning-and-captioning dataset comprising 760k high-quality samples across molecular and reaction tasks. Furthermore, we adopt a three-stage training framework that systemically builds model perception and reasoning capacity. Experiments demonstrate that ChemVLR achieves state-of-the-art (SOTA) performance, surpassing both leading proprietary models and domain-specific open-source baselines. We also provide comprehensive ablation studies to validate our training strategy and data generation designs. Code and model weights will be available at https://github.com/xxlllz/ChemVLR.

Figures & tables

Figure 1: Overview of our proposed Cross-Modality Reverse-Engineering strategy for data generation. The pipeline begins with an initial dataset constructed from textual SMILES QA pairs, enriched with auxiliary information, including IUPAC names, functional groups, and expert demonstrations. Once Gemini-2.5-Flash generates the reasoning and caption data, they are subjected to a rigorous three-stage filtering and verification process to ensure accuracy and quality. Subsequently, the filtered data are rendered into images using RDKit and Indigo code.
SettingMol. Rec.Rxn. Rec.Rxn. Pred.
SMILES78.0%58.0%55.0%
+ Demo80.0%64.0%62.0%
+ IUPAC92.0%71.0%71.0%
+ RDKit95.0%73.0%76.8%
Table 1: Effectiveness comparison of various data generation settings. We generate 10k samples for each category utilizing our reverse-engineering strategy, except for the last row which utilizes the whole reasoning set. The percentages represent the retention rate of high-quality data following the three-stage filtering protocol.
Figure 2: The training framework of ChemVLR proceeds in three stages. First, we conduct chemical-domain continual pre-training using caption data. Second, during the SFT stage, we train on a mixture of reasoning and instruction data. For the RL stage, we retain only non-trivial yet solvable instances and optimize utilizing DAPO.
Caption DatasetSamplesAverageSD
ChEBI-MM26k151.5780.46
ChemMLLM70k95.9252.46
Molecule (Ours)200k409.93139.03
Reaction (Ours)200k441.0676.85
Table 2: Token Length Statistics of Caption Datasets. The mean and standard deviation (SD) of token counts are computed via the Qwen3-VL tokenizer.
ModelsParadigmMMChemOCRimg2smilesChemRxn-V R{}_{\text{R}}ChemRxn-V P{}_{\text{P}}
Avg Sim.Tani @ 1.0Avg Sim.Tani @ 1.0Tani @ 1.0Avg Sim.Tani @ 1.0
Proprietary Models
zz Gemini-3-Flash-77.661.274.563.84.476.851.7
GPT-5-mini-28.85.720.45.10.724.32.3
GPT-4o-36.83.429.00.10.130.41.4
Open-Source General-Domain Models
Table 3: Combined evaluation on molecular and reaction tasks. Avg Sim. and [email protected] denote Average Tanimoto Similarity and Tanimoto [email protected]. ChemRxn-V R{}_{\text{R}} and ChemRxn-V P{}_{\text{P}} correspond to the reaction recognition and prediction subsets. Bold indicates the best performance.
Figure 3: The reward curve during the RL training process. The overall rewards are the weighted average of format and accuracy rewards.
Figure 4: Showcase of the reasoning process and final answer on the molcular recognition and reaction prediction tasks. The ChemVLR-8B generate the correct answer compare with other baselines.
Ablationimg2smilesRxn-V R{}_{\textbf{R}}Rxn-V P{}_{\textbf{P}}
SFT only81.560.256.8
RL only2.10.20.1
CPT + SFT84.763.559.3
ChemVLR-8B92.774.467.8
Table 4: Ablation study of different training stages. Experiments are conducted using Qwen3-VL-8B-Instruct, with performance evaluated via [email protected]. For brevity, we refer to ChemRxn-V as Rxn-V.
Ablationimg2smilesRxn-V R{}_{\textbf{R}}Rxn-V P{}_{\textbf{P}}
Full SFT Data84.763.562.6
w/o IUPAC80.856.256.1
w/o Reasoning76.453.853.3
w/o Instuction68.446.546.9
Table 5: Ablation study on SFT data composition. The instruction data contains the molecular-to-IUPAC data.
Ablationimg2smilesRxn-V R{}_{\textbf{R}}Rxn-V P{}_{\textbf{P}}
Tani Simarity90.270.865.9
Exact Matching90.869.966.4
Struct-ID (Ours)92.774.467.8
Table 6: Ablation study on different reward functions for RL. We compare our utilized structural identity reward against dense Tanimoto similarity and sparse exact string matching rewards.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Reasoning DataSamplesAverageSD
Mol. Recognition95k811.10307.44
Rxn. Recognition73k937.00233.54
Rxn. Prediction192k1026.40320.98
Appendix
Table 7: Statistics of reasoning thinking lengths in our constructed reasoning datasets. Token counts (mean and standard deviation) are calculated using the Qwen3-VL tokenizer.
Figure 5: Full response of InternVL3.5-38B on the Figure 4 molecular recognition tasks.
Figure 6: Full response of ChemVLR-8B-SFT on the Figure 4 molecular recognition tasks.
Figure 7: Full response of ChemVLR-8B on the Figure 4 molecular recognition tasks.
Figure 8: Full response of InternVL3.5-38B on the Figure 4 reaction prediction tasks.
Figure 9: Full response of ChemVLR-8B on the Figure 4 molecular recognition tasks.
Figure 10: Full response of ChemVLR-8B-SFT on the Figure 4 reaction prediction tasks.
Figure 11: Full response of ChemVLR-8B on the Figure 4 reaction prediction tasks.
Figure 12: The prompt to generate the reasoning process of reaction prediction.
More in this topic

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 722 papers

Explore similar work

CardsList
May 17, 2026

ChemVA: Advancing Large Language Models on Chemical Reaction Diagrams Understanding

While Large Language Models (LLMs) have revolutionized scientific text processing, they exhibit a significant capability gap when interpreting chemical reaction diagrams. We identify two fundamental bottlenecks restricting current systems: a Visual Deficit, where generic vision encoders struggle to resolve the strict topological connectivity of dense molecular graphs, and a Semantic Disconnect, where standard linear strings, such as SMILES, fail to effectively activate the model's latent chemical reasoning. To bridge these gaps, we propose the Chemical Visual Activation (ChemVA) framework, which employs a Visual Anchor mechanism to ground functional groups via hybrid-granularity detection, followed by a semantic alignment approach that translates visual features into entity names to maximize knowledge activation in LLMs. We evaluate our approach on OCRD-Bench, a newly constructed dataset featuring dense visual-semantic contexts and comprehensive reaction coverage to evaluate the full spectrum from recognition to reasoning. Extensive experiments on OCRD-Bench demonstrate that ChemVA achieves 92.0% structural recognition accuracy. By bridging visual and semantic bottlenecks, our framework delivers a consistent performance gain of approximately 20 percentage points across 9 diverse LLMs, enabling open-weight models to rival proprietary SOTA systems in complex chemical reasoning tasks.

Mingyang Rao, Kehua Feng, Zhihui Zhu +4

College of Computer Science and Technology, Zhejiang University Hangzhou, Zhejiang, China · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University Hangzhou, Zhejiang, China · Department of Chemistry, Fudan University Shanghai, China

Vision-Language ModelsVLM Reasoning
AskPDF
Jul 14, 2026

Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models

Reaction mechanisms consist of the step-by-step sequences of elementary reactions that explain chemical transformations. Learning the mechanism logic is therefore essential for enhancing the fundamental chemical intelligence of large language models (LLMs). The stepwise deduction of reaction mechanism aligns naturally with the reasoning paradigms of reasoning LLMs. However, current chemical LLMs primarily emphasize coarse-grained name reactions for product prediction and retrosynthesis, often leading to physical inconsistencies and hallucinations. In contrast, specialized small-scale generative models for mechanism inference typically suffer from restricted generalization capacity across diverse chemical spaces. To overcome these limitations, we built a novel, large-scale reasoning dataset of reaction mechanisms. Furthermore, we established the FukuyamaBench, a difficult benchmark derived from Fukuyama's Advanced Organic Reaction Mechanism book, to rigorously evaluate model performance on hierarchical mechanism reasoning. Our fine-tuned Qwen3-30B-A3B achieves 8.3% exact pathway match on FukuyamaBench Set~A, surpassing the specialized FlowER model (5.1%), demonstrating that mechanism-aware training substantially enhances chemical reasoning in language models.

Xingyu Dang, Haocheng Tang, Junmei Wang +1

Department of Computer Science, Princeton University, Princeton, NJ, 08540, United States · School of Pharmacy, University of Pittsburgh, Pittsburgh, PA, 15261, United States · Khoury College of Computer Science, Northeastern University, Boston, MA, 02115, United States +2

LLM Reasoning
AskPDF
Aug 13, 2026

Localize, Then Reason: Visual Latent Structural Reasoning for Molecular Properties and Edits

Local chemical perception and property reasoning are both essential for understanding how molecular structure determines properties. Current LLM-based chemical reasoning methods either receive SMILES/molecular images together with descriptions of local motifs, or reason directly from molecular images. Neither approach enables the model to focus on chemically meaningful regions before reasoning. To address this gap, we propose Visual Latent Structural Reasoning (VLSR), an end-to-end framework that jointly learns localization and reasoning from molecular images. Central to our approach is a localize-then-reason strategy. VLSR first learns to locate chemically meaningful regions in a molecular image. It then reasons about their property effects in a compact latent workspace before producing the final answer. Under the same inference setup, this design achieves 9.6X higher throughput than a comparable textual-reasoning baseline.

Xingqiao Lin, Junmei Wang, Haocheng Tang

Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, United States · Department of Pharmaceutical Sciences and Computational Chemical Genomics Screening Center, School of Pharmacy, University of Pittsburgh, Pittsburgh, PA, USA · Khoury College of Computer Science, Northeastern University, Boston, MA, USA

Latent Visual ReasoningStructured ReasoningMolecular Property Prediction
AskPDF