cs.CLMar 26, 2026

MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidation

Authors: Taolin Han, Shuang Wu, Jinghang Wang, Yuhao Zhou, Renquan Lv, Bing Zhao, Wei Hu

Organizations: Alibaba Group, Hangzhou, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) hold considerable potential for advancing scientific discovery, yet systematic assessment of their dynamic reasoning in real-world research remains limited. Current scientific evaluation benchmarks predominantly rely on static, single-turn Question Answering (QA) formats, which are inadequate for measuring model performance in complex scientific tasks that require multi-step iteration and experimental interaction. To address this gap, we introduce MolQuest, a novel agent-based evaluation framework for molecular structure elucidation built upon authentic chemical experimental data. Unlike existing datasets, MolQuest formalizes molecular structure elucidation as a multi-turn interactive task, requiring models to proactively plan experimental steps, integrate heterogeneous spectral sources (e.g., NMR, MS), and iteratively refine structural hypotheses. This framework systematically evaluates LLMs' abductive reasoning and strategic decision-making abilities within a vast and complex chemical space. Empirical results reveal that contemporary frontier models exhibit significant limitations in authentic scientific scenarios: notably, even state-of-the-art (SOTA) models achieve an accuracy of only approximately 50%, while the performance of most other models remains below the 30% threshold. This work provides a reproducible and extensible framework for science-oriented LLM evaluation, our findings highlight the critical gap in current LLMs' strategic scientific reasoning, setting a clear direction for future research toward AI that can actively participate in the scientific process.

Figures & tables

Figure 1: Molecular structure elucidation as a Constraint Satisfaction Problem (CSP).
Figure 2: The characteristics of our benchmark.
Figure 3: Data processing pipeline
Figure 4: Comparison of Chemical Reasoning and Decision-Making Between Large Language Models and Human Chemists in Molquest
Tool NameDescription
Check_DataCheck available data types in the database for the current molecule.
Measure_MWMeasure molecular weight (simulating Mass Spectrometry).
Measure_FormulaMeasure molecular formula (simulating High-Resolution MS).
Calculate_DBECalculate the Degree of Unsaturation (DBE).
Get_1H_NMRAcquire 1 H NMR spectrum data (proton signals).
Get_13C_NMRAcquire 13 C NMR spectrum data (carbon signals).
Table 1: The Action Space: List of simulated experimental tools available to the agent in MolQuest.
ModelAccuracy (%)Validity Rate (%)Average Similarity (%)Calibration Error (%)Formula Conservation (%)
 AgentBaselineAgentBaselineAgentBaselineAgentBaselineAgentBaseline
claude-haiku-4.511.519.6288.6886.0441.3040.7043.1936.4129.1523.03
claude-opus-4.525.6628.4997.7497.1758.7162.0524.9215.4361.5856.31
claude-sonnet-4.518.1117.5596.6095.0950.0451.9840.4734.9449.0242.26
deepseek-v3.17.366.7984.3486.2337.3135.4251.9755.5823.7116.41
deepseek-v3.211.325.6685.2886.0441.5631.6547.6852.5029.8716.01
Table 2: Main results on MolQuest under the Agent and Baseline settings.
Figure 5: the core results of the ablation study (dynamic vs. static)
ModelAvg. RoundsAccuracy (%)Acc/1M Tokens
claude-haiku-4.54.3611.514.57
claude-opus-4.54.7225.669.18
claude-sonnet-4.55.0218.115.88
deepseek-v3.15.907.362.51
deepseek-v3.24.4211.322.37
deepseek-v3.2-thinking4.6116.604.78
Table 3: Analysis of interaction efficiency and cost-effectiveness across models.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Chemical space visualization of the benchmark molecular set based on molecular fingerprints and t-SNE dimensionality reduction.
Figure 7: Statistics of pairwise Tanimoto similarity distribution for the benchmark molecular set.
Figure 8: Visualization of the pairwise similarity matrix heatmap of the benchmark molecular set based on molecular fingerprints.
Figure 9: Screenshot of the human review interface showing the "Red Flag" system for flagged data points.
More in this topic

Abductive Reasoning in Language Models

See all 10 papers

Explore similar work

CardsList
May 21, 2026

SciCore-Mol: Augmenting Large Language Models with Pluggable Molecular Cognition Modules

Large Language Models (LLMs) are central to the one-for-all intelligent paradigm, but they face a fundamental challenge when dealing with heterogeneous scientific data such as molecules: the inherent gap between discrete linguistic symbols and topological molecular or continuous reaction data leads to significant information loss and semantic noise in text-based reasoning. We propose SciCore-Mol, a modular framework that bridges this gap through three deeply integrated pluggable cognitive modules: a topology-aware perception module, a latent diffusion-based molecular generation module, and a reaction-aware reasoning module. Each module is coupled to the LLM backbone through learned representation interfaces, enabling richer information exchange than is possible with text-only tool feedback. Our experiments on diverse chemical tasks demonstrate that SciCore-Mol achieves strong comprehensive performance across molecular understanding, generation, reaction prediction, and general chemistry knowledge, with an 8B-parameter open-source system that is competitive with and in several dimensions surpasses proprietary large models. This work provides a systematic blueprint for equipping LLMs with scientific expertise through decoupled, pluggable, and flexibly orchestrated modules, with direct implications for drug design, chemical synthesis, and broader scientific discovery.

Yuxuan Chen, Changwei Lv, Yunduo Xiao +5

School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Tsinghua University, Beijing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China

AI-Assisted Scientific ResearchChemical Reaction PredictionMolecular GenerationLarge Language Model-Guided Scientific DiscoveryMolecular Representation Learning
AskPDF
Sep 23, 2026

MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design

Real-world molecular design remains challenging for large language model (LLM)-based agents. It requires them to interpret design contexts, satisfy multiple constraints, identify infeasible specifications, and reason over multi-step tool outputs. Existing benchmarks do not capture this complexity, focusing instead on explicit and narrow constraints, only feasible problems, and single-path solutions. To address this gap, we propose MolDesignBench, a scenario-grounded benchmark that more closely reflects real-world molecular design for evaluating tool-augmented LLM agents. MolDesignBench comprises 2K generation and optimization instances that combine implicit requirements embedded in design narratives with explicit property and functional-group constraints, including infeasible cases, and require the effective use of 17 specialized chemistry tools. Experiments across diverse frontier LLMs reveal low success rates--with the best achieving only ∼43\sim43%--and frequent failures in implicit-constraint reasoning, infeasibility detection, and tool reasoning. The corresponding fine-grained failure-mode analysis identifies implicit constraint interpretation and infeasibility detection as the primary bottlenecks, establishing MolDesignBench as a rigorous testbed to guide future research on chemical agents. The benchmark, tool interface, and evaluation code are publicly available.

Yongjun Jeong, Hanbum Ko, Ye Rin Kim +6

Department of Artificial Intelligence, Korea University · LG AI Research · Department of Statistics, Korea University

Molecular OptimizationLLM Agent EvaluationAI Agent BenchmarksConstraint-Guided Reasoning in Language ModelsMolecular GenerationMolecule Design
AskPDF
May 19, 2026

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom.

Yiyang Gu, Junwei Yang, Junyu Luo +15

State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5

AI for ScienceLLM Evaluation
AskPDF