ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation

Eric Inae; Tim Gunn; Chris Bond; Meng Jiang

cs.LGAug 21, 2026

ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation

Authors: Eric Inae, Tim Gunn, Chris Bond, Meng Jiang

Organizations: University of Notre Dame Notre Dame, Indiana, USA

Abstract

The rapid advancement of large language models (LLMs) has led to increasing interest in their application to scientific domains such as chemistry. However, existing chemistry benchmarks often provide only a narrow view of model capability, focusing on limited task sets while overlooking robustness to variations in problem formulation and chemical representation. As a result, reported performance may overestimate a model's true ability to reason consistently across realistic settings. To address this challenge, we introduce ChemDIRT (Diversified Instruction, Representation, and Task Benchmark), a comprehensive evaluation framework designed to assess the robustness of chemical reasoning in LLMs. ChemDIRT systematically measures model performance across variations in instructions and molecular representations while spanning eight categories of chemistry tasks. By evaluating both accuracy and consistency under these controlled perturbations, ChemDIRT provides a more reliable assessment of model reasoning capabilities than conventional single-format benchmarks. We benchmark a diverse set of open- and closed-source LLMs, revealing substantial prompt sensitivity, representation dependence, and uneven performance across task families.

Figures & tables

Fig. 1: ClinTox classification accuracy across four semantically equivalent prompt variants. Despite asking the same question, changes in phrasing produce substantial performance shifts across all three models.

Fig. 2: Distribution of ChemDIRT evaluation examples across the eight chemical-reasoning categories.

Category	Subtask	Source/Test	$n$	Category	Subtask	Source/Test	$n$	Category	Subtask	Source/Test	$n$
Description	Carbon count	Ours	100	Design	IUPAC $\rightarrow$ SMILES	Ours	100	Regression	ESOL	113	113
	Ring count	Ours	100		SMILES $\rightarrow$ IUPAC	Ours	100		Lipophilicity	420	420
	Functional-group presence	Ours	100		Image $\rightarrow$ SMILES	Ours	100		FreeSolv	65	65
	Bond-type presence	Ours	100		Image $\rightarrow$ IUPAC	Ours	100		O 2 permeability	179	179
Classification	BACE	152	152		Description-based design	100	100	Computation	Physical-unit calculation	500	392
	BBBP	204	204	Mechanistic	Forward prediction (SMILES)	100	100		Coefficient balancing	500	100

TABLE I: ChemDIRT task–representation settings and evaluation counts (

N=4{,}484

). “Ours” denotes examples constructed for ChemDIRT.

Model	Full Accuracy	Subset Accuracy
ChemLLM	0.955	0.950
ChemDFM	0.902	0.900
ChemR	0.827	0.830
General-Reasoner	0.334	0.340
Falcon	0.043	0.050
Galactica	0.541	0.540

TABLE II: Full-test and selected-subset HIV accuracies for reference models (

n=100

ClinTox Classification Prompt Variants
Predict the clinical trial toxicity outcome for the molecule with SMILES: {SMILES} .
Did the compound represented by {SMILES} fail clinical trials due to toxicity?
Classify the molecule {SMILES} with respect to clinical trial toxicity.
Given the SMILES {SMILES} , determine whether the compound exhibits clinical trial toxicity.
Does the structure {SMILES} indicate failure in clinical trials due to toxicity?
Evaluate the clinical trial toxicity of the molecule encoded as {SMILES} .

TABLE III: Ten verified ClinTox prompt candidates. The reported evaluation uses the first four variants from this set, with {SMILES} replaced at inference time.

Fig. 3: Variation across prompts for three tasks. Bars show the minimum and maximum. Points show the mean. Colors identify models across panels.

Fig. 4: Representation-dependent accuracy for selected model–task pairs. Blue circles show SMILES means, orange squares show IUPAC means, and connecting bars show the labeled difference (IUPAC minus SMILES).

Model	Image $\rightarrow$ SMILES ( $n=100$ )				Image $\rightarrow$ IUPAC ( $n=100$ )
Model	Exact Match	MACCS	Morgan	RDK	Exact Match	MACCS	Morgan	RDK
ChemM-R1	$0.195\pm 0.021$	$0.871\pm 0.037$	$0.679\pm 0.045$	$0.717\pm 0.055$	$0.233\pm 0.205$	$0.787\pm 0.066$	$0.515\pm 0.165$	$0.665\pm 0.086$
ChemVLM-8B	$0.200\pm 0.012$	$0.869\pm 0.004$	$0.634\pm 0.008$	$0.690\pm 0.008$	$0.006\pm 0.010$	$0.587\pm 0.013$	$0.231\pm 0.009$	$0.324\pm 0.010$
Gemma3	$0.000\pm 0.000$	$0.375\pm 0.016$	$0.124\pm 0.007$	$0.219\pm 0.015$	$0.000\pm 0.000$	$0.525\pm 0.011$	$0.192\pm 0.003$	$0.292\pm 0.012$
Claude-Sonnet-5	$0.633\pm 0.028$	$0.981\pm 0.005$	$0.902\pm 0.016$	$0.935\pm 0.013$	$0.687\pm 0.039$	$0.980\pm 0.002$	$0.903\pm 0.014$	$0.922\pm 0.011$
Gemini-3.5-Flash-Lite	$0.300\pm 0.031$	$0.915\pm 0.007$	$0.746\pm 0.015$	$0.775\pm 0.010$	$0.095\pm 0.024$	$0.830\pm 0.011$	$0.514\pm 0.017$	$0.553\pm 0.015$

TABLE V: Image-task results for vision-capable models (mean

\pm

SD over evaluated conditions).

Chemical Reasoning in LLMs

See all 13 papers

From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
Jun 2, 2026Hongyu Guo, Hao Li, He Cao +2LLM Evaluation Chemical Reasoning in LLMs
Peking University, Shenzhen Graduate School · International Digital Economy Academy (IDEA)
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Oct 9, 2025Ruiling Xu, Yifan ZhangLLM Evaluation Benchmark Design
University of Illinois Urbana-Champaign
Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models
Jul 14, 2026Xingyu Dang, Haocheng Tang, Junmei Wang +1LLM Reasoning
Department of Computer Science, Princeton University, Princeton, NJ, 08540, United States · School of Pharmacy, University of Pittsburgh, Pittsburgh, PA, 15261, United States · Khoury College of Computer Science, Northeastern University, Boston, MA, 02115, United States +2

Jun 2, 2026J/K move · Enter open · S save

From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

Hongyu Guo, Hao Li, He Cao, Gongbo Zhang+1

Peking University, Shenzhen Graduate School · International Digital Economy Academy (IDEA)

LLM Evaluation Chemical Reasoning in LLMs Chemical Reaction Prediction LLM Reasoning

Open paper AskPDF arXiv

Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.

Abstract

Chemical Reasoning in LLMs

Explore similar work

From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models