cs.CLNov 9, 2026

ChemInsight: A Comprehensive Benchmark for Evaluating and Diagnosing Chemistry Understanding and Reasoning in Large Language Models

Authors: Changyu Hu, Yiming Zou, Xin Tang, Yiru Pan, Zicheng Huang, Deng Zhao, Qing Cui, Zhenduo Zhang, 

Organizations: Ant Group · Peking University · University of Electronic Science and Technology of China · Westlake University · 

Abstract

We introduce ChemInsight, a benchmark designed to evaluate large language models (LLMs) on chemistry tasks that require multi-step reasoning and structured chemical understanding. We propose a two-dimensional scoring system (scope and depth) to assess both task coverage and reasoning completeness. ChemInsight consists of 203 challenging chemistry questions organized into 50 bundles, which support the two-dimensional scoring evaluation scheme. We evaluate ten representative LLMs using ChemInsight and observe consistently low performance across models, with scope scores below 30% and depth scores below 11%, substantially lower than human expert performance. We then proceed to ask (1) What factor limits the advancement of current LLMs in chemistry? (2) To what extent are statistically driven language-model training methods effective at modeling chemistry symbolic language such as SMILES? And (3) Can LLM performance in chemistry be improved through intensive training on large, domain-specific corpora?

Figures & tables

Explore similar work

CardsList
  1. From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

    Jun 2, 2026Hongyu Guo, Hao Li, He Cao +2LLM EvaluationChemical Reasoning in LLMs

  2. Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models

    Jul 14, 2026Xingyu Dang, Haocheng Tang, Junmei Wang +1LLM Reasoning