ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

Aotian Chen; Frazier N. Baker; Reza Averly; Shan Zhang; Tianao Chen; Daniel Adu-Ampratwum; Xia Ning; Yuan Xue

cs.AIOct 12, 2026

ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

Authors: Aotian Chen, Frazier N. Baker, Reza Averly, Shan Zhang, Tianao Chen, Daniel Adu-Ampratwum, Xia Ning, Yuan Xue

Organizations: The Ohio State University · Adelaide University

Abstract

Current chemistry benchmarks primarily evaluate outcome prediction or reaction pathway generation, but rarely test whether a model can explain why an observed chemical effect occurs. We introduce ChemExplain, a benchmark of 1,180 multiple-choice chemistry questions drawn from 383 peer-reviewed papers. Each question pairs a molecular or reaction image with an observed chemical effect and four candidate explanations, each associated with a different structural feature visible in the image. Answering requires the model to perceive the depicted structure, apply chemical knowledge, and identify which feature most directly explains the effect, a capability we term explanatory attribution. The correct feature-mechanism relation is derived from supporting evidence in the source paper rather than invented by the question-generation LLM. We construct the benchmark automatically with a multi-agent framework that applies progressive quality control at three stages and validate it with eight chemistry domain experts. The strongest evaluated model reaches 78.6% accuracy, leaving considerable room for improvement. On a 600-question expert-reviewed subset, reasoning-trace analysis classifies 71.4% of the strongest model's errors as attribution failures. Attribution is also the largest error category for the open-source and chemistry-specific models included in our analysis.

Figures & tables

Figure 1: Task-level comparison with representative chemistry benchmarks. ChemCoTBench and oMeBench evaluate reaction outcome prediction and pathway generation, respectively. ChemExplain instead evaluates whether a model can identify which visible structural feature most directly explains an observed chemical effect.

Figure 2: Overview of the multi-agent benchmark construction framework. Four specialist agents perform acquisition, visual processing, text grounding, and question generation, with blue arrows indicating the main data flow. Bottom left: a traceability tree shows how one source paper branches into candidate structures and how invalid candidates are filtered out across stages. Bottom right: a dedicated quality control agent applies three progressive checks between stages: QC 1 validates 2D chemical structures, QC 2 verifies entity match and explanatory evidence, and QC 3 screens generated questions on four quality dimensions.

Benchmark	Size	Modality	Level	Curation	Source	Attr.
GPQA ( Rein et al., 2024 )	448	Text	Grad	Manual	Expert	✗
ChemLLMBench ( Guo et al., 2023 )	800	Text	Mixed	Semi	Expert	✗
ChemBench ( Mirza et al., 2025 )	2,788	Text	UG–Grad	Semi	Expert	✗
ChemCoTBench ( Li et al., 2025a )	1,495	Text	Mixed	Semi	Expert	✗
oMeBench ( Xu et al., 2025 )	2,871	Text	Mixed	Semi	Expert	✗
SUPERChem ( Zhao et al., 2025 )	500	Multimodal	Mixed	Manual	Expert	✗

Table 1: ChemExplain compared with existing chemistry benchmarks. UG = undergraduate, Grad = graduate. Source : origin of the correct answer. Attr. : whether the task requires explanatory attribution.

Figure 3: Construction yield and attrition from

\sim

5,000 source papers to 1,180 accepted questions. Green denotes retained samples, red denotes filtered samples annotated with rejection cause and loss percentage, and stage labels indicate the responsible agent.

Dimension	$\bar{H}$	AC 2			MAE $\downarrow$
Dimension	$\bar{H}$	H–H	H–L	SE	MAE $\downarrow$
Distractor Quality	4.50	0.909	0.948	0.009	0.497
Meaningfulness	4.42	0.898	0.936	0.009	0.505
Answer Unambiguity	4.43	0.899	0.929	0.011	0.407
Struct. Necessity	4.75	0.946	0.861	0.022	0.785
Avg. (Score)	4.53	0.913	0.919	0.013	0.549

Table 2: LLM-as-a-Judge reliability (

n{=}600

stratified sample from accepted questions, 8 human experts).

\bar{H}

: human expert mean; AC 2 : Gwet’s AC 2 ; H–H: human inter-rater; H–L: human–LLM; SE: analytic standard error of the H–L estimate; MAE: mean absolute error.

Model	Size	Avg.	Reasoning Type			Entry Type
Model	Size	Avg.	Mech.	Design	Cond.	Mol.	Rxn.
Random baseline	–	25.0	25.0	25.0	25.0	25.0	25.0
Proprietary VLMs
GPT-5.4	–	78.6	75.3	84.2	68.6	81.1	77.6
GPT-5	–	77.4	72.9	83.6	78.4	80.2	76.2
GPT-4o	–	68.1	65.6	72.5	60.8	68.6	67.9

Table 3: Performance on ChemExplain (majority-vote accuracy, %, 3 trials). Mech. = Mechanistic, Cond. = Condition, Mol. = molecule, Rxn. = reaction. Best per column in bold . All results are on the full 1,180-question benchmark. Human expert performance is reported separately in Appendix Tab. 6 .

Model	With image	Text-only	$\Delta$
GPT-5	77.0	23.5	$-$ 53.5
Claude Sonnet 4.6	74.0	20.5	$-$ 53.5
Gemini-3.1-Pro	78.5	24.5	$-$ 54.0

Table 4: Text-only control on the 600-question expert subset, removing only the structure image.

Figure 4: Accuracy and error type breakdown for three representative models on the 600-question evaluation subset. Dashed line: best human expert accuracy.

Configuration	Structures	Ent. Match	Expl.
Full pipeline (ours)	489	76%	43
w/o image-first (SMILES mode)	154	—	2
w/o VLM identifier	489	50%	34
w/ MolCoref identifier	489	35%	31
w/o context expansion	489	76%	26

Table 5: Construction pipeline ablation on 100 randomly sampled source papers. Each row below the full pipeline removes or replaces one component, and the columns report pipeline outputs. Structures: individual molecular or reaction images retained after segmentation. Ent. Match: fraction of retrieved text referring to the target molecule or reaction. Expl.: explanatory sentences.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Human Experts	Avg.	Mech.	Design	Cond.	Mol.	Rxn.
Average	73.3	71.0	75.9	72.8	72.2	74.4
Best	80.0	77.4	82.8	80.4	70.0	90.0
Worst	68.3	67.7	69.0	68.6	66.7	70.0

Appendix

Table 6: Human expert accuracy (%) on the 600-question stratified subset, reported for eight chemistry domain experts. Mech.: Mechanistic; Cond.: Condition; Mol.: molecule; Rxn.: reaction.

Metric	Value
Mean expert accuracy	73.3%
Labeled answer chosen by $\geq$ 5 of 8 experts	94.0%
Labeled answer is the most-voted option	99.7%
Mean vote lead over the next option	4.45
Median vote lead over the next option	4

Appendix

Table 7: Agreement of the eight experts’ blind answers with the labeled answer on the 600-question subset. Tab. 8 gives the full distribution of expert votes.

Votes	Questions	GPT-5.4 errors	Attr. errors
8/8	50 (8.3%)	5	4
7/8	100 (16.7%)	15	12
6/8	210 (35.0%)	45	36
5/8	204 (34.0%)	60	38
$\leq$ 4/8	36 (6.0%)	15	10
$\geq$ 5/8	564 (94.0%)	125	90

Appendix

Table 8: Distribution of blind expert votes for the labeled answer on the 600-question subset, with GPT-5.4 errors and the subset of these errors classified as attribution failures (Section 5.3 ). Votes: number of the eight experts selecting the labeled answer. Attr.: attribution.

Error Type	Count	Expert A Correct
Attribution Error	100	60
Knowledge Gap	30	30
Perception Error	10	10
Total	140	100

Appendix

Table 9: Expert A’s accuracy on the 140 questions GPT-5.4 answers incorrectly, grouped by error type (Section 5.3 ).

Subset	Expert A	GPT-5.4	$\Delta$
Within-domain	94.1	70.6	$\uparrow$ 23.5
Out-of-domain	42.3	84.6	$\downarrow$ 42.3
Overall	71.7	76.7	$\downarrow$ 5.0

Appendix

Table 10: Accuracy (%) of Expert A and GPT-5.4 on within-domain and out-of-domain subsets (

n

=340 and

n

=260 respectively), defined by Expert A’s self-annotations.

\Delta

: Expert A accuracy minus GPT-5.4 accuracy.

Model	Error type	N	C0	C1: Structure	C2: Knowledge	C3: Both
GPT-5.4	Perception	10	20.0 (2/10)	80.0 (8/10)	20.0 (2/10)	90.0 (9/10)
	Knowledge	30	16.7 (5/30)	23.3 (7/30)	70.0 (21/30)	76.7 (23/30)
	Attribution	100	14.0 (14/100)	17.0 (17/100)	16.0 (16/100)	19.0 (19/100)
Qwen3-VL-8B	Perception	60	15.0 (9/60)	70.0 (42/60)	16.7 (10/60)	76.7 (46/60)
	Knowledge	90	13.3 (12/90)	17.8 (16/90)	65.6 (59/90)	71.1 (64/90)
	Attribution	150	12.0 (18/150)	14.7 (22/150)	14.0 (21/150)	16.0 (24/150)

Appendix

Table 11: Correction rates (%) of originally incorrect questions under each condition, with counts in parentheses. C0 reruns the original prompt, C1 adds structural descriptions, C2 adds chemical knowledge, and C3 adds both. Bold marks a significant difference from C0 (two-sided exact McNemar test,

p<0.05

Model	Error type	C1 vs. C0	C2 vs. C0	C3 vs. C0
GPT-5.4	Perception	$b=6,c=0,\;p=.031$	$b=1,c=1,\;p=1.000$	$b=7,c=0,\;p=.016$
	Knowledge	$b=4,c=2,\;p=.688$	$b=18,c=2,\;p<.001$	$b=20,c=2,\;p<.001$
	Attribution	$b=8,c=5,\;p=.581$	$b=7,c=5,\;p=.774$	$b=10,c=5,\;p=.302$
Qwen3-VL-8B	Perception	$b=35,c=2,\;p<.001$	$b=5,c=4,\;p=1.000$	$b=39,c=2,\;p<.001$
	Knowledge	$b=8,c=4,\;p=.388$	$b=49,c=2,\;p<.001$	$b=54,c=2,\;p<.001$
	Attribution	$b=9,c=5,\;p=.424$	$b=8,c=5,\;p=.581$	$b=11,c=5,\;p=.210$

Appendix

Table 12: Two-sided exact McNemar tests comparing each condition with C0.

b

: questions correct under the condition but incorrect under C0.

c

: questions correct under C0 but incorrect under the condition.

Model	C0	C1	C2	C3
GPT-5.4	96.0	97.0	96.0	97.0
Qwen3-VL-8B	92.0	93.0	92.0	93.0

Appendix

Table 13: Accuracy (%) on 100 control questions that both models originally answer correctly. No oracle condition differs significantly from C0 (two-sided exact McNemar tests, all

p>0.05

Dimension	Benchmark	Subset	$\Delta$	$p$
Distractor Quality	4.447	4.450	$+$ 0.003	0.720
Meaningfulness	4.447	4.470	$+$ 0.023	0.400
Answer Unambiguity	4.480	4.460	$-$ 0.020	0.440
Struct. Necessity	4.505	4.630	$+$ 0.125	0.021

Appendix

Table 14: Judge scores on the 1–5 Likert scale for the full benchmark and the 600-question expert subset.

\Delta

: Subset minus Benchmark.

p

: two-sided Mann–Whitney U test.

Judge (H–L AC 2 )	Distr.	Mean.	Unamb.	Struct.	Avg.
GPT-5.2 (orig.)	0.948	0.936	0.929	0.861	0.919
Claude-Opus-4-7	0.933	0.939	0.926	0.952	0.938
Gemini-3.1-Pro	0.952	0.940	0.932	0.973	0.949

Appendix

Table 15: Human–LLM agreement (Gwet’s AC 2 ) on the 600-question expert subset for three judges from different model families. Distr.: Distractor Quality; Mean.: Meaningfulness; Unamb.: Answer Unambiguity; Struct.: Structure Necessity.

Config (Claude-Sonnet-4-6)	Struct.	Ent. Match	Expl.
Full pipeline	489	78%	41
w/o image-first (SMILES)	154	—	3
w/o VLM identifier	489	45%	31
w/ MolCoref identifier	489	36%	27
w/o context expansion	489	79%	24

Appendix

Table 16: Ablation on the same 100-paper subset as Tab. 5 , repeated with Claude-Sonnet-4-6. Struct.: individual molecular or reaction images retained after classification and segmentation. Ent. Match: fraction of retrieved text correctly referring to the target molecule or reaction. Expl.: explanatory sentences that qualify for question generation.

Dimension	$\bar{H}$	$\bar{L}$	Cause (%)
Structure Necessity	2.70	2.69	76.7
Answer Unambiguity	3.89	3.93	20.0
Distractor Quality	3.90	3.85	3.3
Meaningfulness	4.23	4.23	0.0
Overall	3.68	3.68	—

Appendix

Table 17: Quality scores for 60 rejected questions (

n{=}60

from 84 rejected candidates, 8 human experts).

\bar{H}

: mean human expert score (1–5);

\bar{L}

: mean LLM judge score. Cause (%): dimension with the lowest LLM score, indicating the primary reason for rejection.

Model	Accuracy	Paper-level 95% CI	One question per paper
GPT-5.4	78.6	[75.7, 81.4]	78.4 [76.0, 80.9]
GPT-5	77.4	[74.4, 80.2]	77.2 [74.9, 79.6]
GPT-4o	68.1	[65.0, 71.2]	67.8 [65.3, 70.2]
Claude Sonnet 4.6	78.2	[75.4, 80.9]	78.1 [75.7, 80.7]
Gemini-3.1-Pro	78.0	[75.1, 80.7]	77.9 [75.5, 80.4]
Qwen2.5-VL 7B	42.6	[39.0, 46.2]	42.5 [39.9, 45.2]

Appendix

Table 18: Accuracy (%) on the full benchmark with paper-level uncertainty. Paper-level 95% CI: percentile interval from 10,000 bootstrap resamples of source papers. One question per paper: mean accuracy and 95% range over 1,000 resamples that keep one randomly chosen question from each paper.

Model	Baseline	EGA	$\Delta$
Qwen2.5-VL (7B)	42.6	46.8	$+$ 4.2
Qwen3-VL (8B)	52.5	55.6	$+$ 3.1
ChemVLM (8B)	34.0	39.3	$+$ 5.3

Appendix

Table 19: Accuracy (%) with Evidence-Grounded Attribution on the full benchmark under the protocol of Tab. 3 .

\Delta

denotes the absolute percentage-point improvement.

Figure 5: Composition of the 1,180-question benchmark sourced from 383 papers, distributed across three reasoning categories: Mechanistic (55.9%), Design (39.7%), and Condition (4.3%).

Figure 6: A complete annotated ChemExplain question: a molecular structure image, an observed effect (green, identical across all options), and four candidate explanations each identifying a different structural feature (blue) and mechanism (orange). The correct answer (C, checkmark) is grounded in the source paper evidence (dashed box). Distractors reference other structural features genuinely present in the molecule.

Figure 7: SMILES extraction and round-trip validation in the SMILES-first pipeline (left), with two representative failure modes (right): (1) MolScribe fails to extract a valid SMILES from a structurally complex molecule; (2) extraction succeeds but the regenerated structure differs from the original, failing the similarity check.

Figure 8: Representative case where GPT-5.4 commits an attribution error while Expert A, answering without the source paper, correctly identifies the responsible structural feature within his domain.

Figure 9: Representative case where GPT-5.4 answers correctly while Expert A fails due to a domain knowledge gap on a question outside his primary expertise.

Figure 10: Representative attribution error from GPT-5.4 on ChemExplain. The model identifies the chemically relevant interaction and reasons plausibly about the options, yet selects the wrong feature as the primary explanation of the observed effect.

Figure 11: Perception error example. The model misidentifies or overlooks a structural feature visible in the image, leading to an incorrect answer.

Figure 12: Knowledge gap example. The model correctly perceives the structural features but lacks the chemical knowledge to connect them to the observed effect.

Figure 13: Structure necessity rejection (76.7% of rejections). The correct answer can be selected without analyzing the depicted structure.

Figure 14: Answer unambiguity rejection (20.0% of rejections). More than one option is scientifically defensible.

Figure 15: Prompt template for Stage 1 explanatory decomposition.

Figure 16: Prompt template for Stage 2 question and distractor generation.

Figure 17: Prompt template for LLM-as-a-Judge quality screening on four dimensions.

Figure 18: Human evaluation platform: blind answer phase, in which experts answer the question without access to the source paper text.

Figure 19: Human evaluation platform: evidence review and scoring phase, in which experts review the source paper evidence and score the four quality dimensions on a 1–5 Likert scale.

Figure 24: Condition question on a reaction.

Scientific Reasoning

Scientific reasoning studies how AI models use evidence, domain knowledge, and methods to explain phenomena, test hypotheses, and solve research problems.

See all 45 papers

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark
Oct 14, 2026Hanyao Guo, Mo Cui, Zihan Tan +21Chemical Reasoning in LLMs Scientific Reasoning in Language Models
Chemical Chain-of-Thought Functions as a Hallucination-Prone Molecular Scratchpad
Jul 23, 2026Jiatong Li, Yuxuan Ren, Weida Wang +2CoT Faithfulness Chemical Reasoning in LLMs
Blue Whale Lab, National University of Singapore · Fudan University · Hong Kong Polytechnic University
From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
Jun 2, 2026Hongyu Guo, Hao Li, He Cao +2LLM Evaluation Chemical Reasoning in LLMs
Peking University, Shenzhen Graduate School · International Digital Economy Academy (IDEA)

Oct 14, 2026J/K move · Enter open · S save

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

Hanyao Guo, Mo Cui, Zihan Tan, Leixin Sun+20

Chemical Reasoning in LLMs Scientific Reasoning in Language Models Reasoning Evaluation Multimodal Model Evaluation

Open paper AskPDF arXiv

Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still score answer endpoints alone. Such scoring conflates memorized facts with stepwise derivation, treats visual and textual inputs as interchangeable, and confounds long-horizon planning with stepwise execution. These blind spots prevent any diagnosis of where the reasoning chain breaks in a model. We present ChemArgus, a competition-level chemistry benchmark drawn from chemistry competition material, textbooks, and research papers, whose scoring replaces the answer endpoint with three measurement properties. Fine-grained scoring grades every subquestion point by point with partial credit, so a score lands at the level of the step rather than the outcome. Multimodal capability is priced on every problem containing a real image, paired with a controlled text description, so visual understanding is measured against text on identical tasks. Diagnostic attribution reads paired metrics that share one scoring unit and differ only in context construction, so a gap between two scores isolates a single cause, namely integration, propagation, or local reasoning. GPT-5.6-Sol scores 23.2% on unassisted whole-problem solving against 28.8% and 30.4% under the two sequential metrics, a gap measuring the integration burden. Through rubric partial credit and these cause-isolating score gaps, ChemArgus shows not only whether a model answers chemistry correctly but where its reasoning chain fails, so that every score it reports doubles as a diagnosis, a readout of the chain rather than a verdict, aiming to guide language models step by step on their path toward chemistry expertise, one corrected step at a time, each step priced by the rubric, point by point.

Abstract

Scientific Reasoning

Explore similar work

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

Chemical Chain-of-Thought Functions as a Hallucination-Prone Molecular Scratchpad

From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models