ChemInsight: A Comprehensive Benchmark for Evaluating and Diagnosing Chemistry Understanding and Reasoning in Large Language Models
Organizations: Ant Group · Peking University · University of Electronic Science and Technology of China · Westlake University ·
Abstract
We introduce ChemInsight, a benchmark designed to evaluate large language models (LLMs) on chemistry tasks that require multi-step reasoning and structured chemical understanding. We propose a two-dimensional scoring system (scope and depth) to assess both task coverage and reasoning completeness. ChemInsight consists of 203 challenging chemistry questions organized into 50 bundles, which support the two-dimensional scoring evaluation scheme. We evaluate ten representative LLMs using ChemInsight and observe consistently low performance across models, with scope scores below 30% and depth scores below 11%, substantially lower than human expert performance. We then proceed to ask (1) What factor limits the advancement of current LLMs in chemistry? (2) To what extent are statistically driven language-model training methods effective at modeling chemistry symbolic language such as SMILES? And (3) Can LLM performance in chemistry be improved through intensive training on large, domain-specific corpora?
Figures & tables
| Model | Inference Mode | Scope Score | Depth Score | A | B | C | D | E | F |
|---|---|---|---|---|---|---|---|---|---|
| Gemini 3 Pro | Thinking | 0.278 | 0.107 | 0.217 | 0.202 | 0.548 | 0.396 | 0.291 | 0.191 |
| GPT-5 | Thinking | 0.215 | 0.069 | 0.076 | 0.078 | 0.275 | 0.450 | 0.360 | 0.207 |
| Gemini 2.5 Pro | Thinking | 0.212 | 0.059 | 0.100 | 0.106 | 0.371 | 0.421 | 0.304 | 0.181 |
| DeepSeek-V3.2-Speciale | Thinking | 0.208 | 0.067 | 0.120 | 0.120 | 0.370 | 0.354 | 0.279 | 0.163 |
| DeepSeek-V3.2 | Thinking | 0.181 | 0.047 | 0.089 | 0.091 | 0.359 | 0.351 | 0.244 | 0.156 |
| Qwen3-235B-A22B-Thinking-2507 | Thinking | 0.151 | 0.034 | 0.058 | 0.063 | 0.117 | 0.325 | 0.262 | 0.127 |