Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still score answer endpoints alone. Such scoring conflates memorized facts with stepwise derivation, treats visual and textual inputs as interchangeable, and confounds long-horizon planning with stepwise execution. These blind spots prevent any diagnosis of where the reasoning chain breaks in a model. We present ChemArgus, a competition-level chemistry benchmark drawn from chemistry competition material, textbooks, and research papers, whose scoring replaces the answer endpoint with three measurement properties. Fine-grained scoring grades every subquestion point by point with partial credit, so a score lands at the level of the step rather than the outcome. Multimodal capability is priced on every problem containing a real image, paired with a controlled text description, so visual understanding is measured against text on identical tasks. Diagnostic attribution reads paired metrics that share one scoring unit and differ only in context construction, so a gap between two scores isolates a single cause, namely integration, propagation, or local reasoning. GPT-5.6-Sol scores 23.2% on unassisted whole-problem solving against 28.8% and 30.4% under the two sequential metrics, a gap measuring the integration burden. Through rubric partial credit and these cause-isolating score gaps, ChemArgus shows not only whether a model answers chemistry correctly but where its reasoning chain fails, so that every score it reports doubles as a diagnosis, a readout of the chain rather than a verdict, aiming to guide language models step by step on their path toward chemistry expertise, one corrected step at a time, each step priced by the rubric, point by point.
Figures & tables
Benchmark
Image input
Image-text contrast
Rubric partial credit
Subquestion chains
Protocol separation
SciBench [ 18 ] [ICML24]
✓
✗
✗
✗
✗
SciCode [ 17 ] [NeurIPS24]
✗
✗
✗
✓
✗
GPQA [ 8 ] [arXiv23]
✗
✗
✗
✗
✗
DiscoveryWorld [ 65 ] [NeurIPS24]
✗
✗
✗
✗
✗
SciEval [ 27 ] [AAAI24]
✗
✗
✗
✗
✗
ChemBench [ 3 ] [arXiv24]
✗
✗
✗
✗
✗
Table 1: Comparison of ChemArgus against prior chemistry and adjacent scientific benchmarks on five properties. “image input” reports images as real multimodal blocks, and “image-text contrast” reports every figure paired with controlled text. “rubric point partial credit” reports partial credit per rubric point, and “subquestion chains” report problems split into shared-context subquestions. Finally, “protocol separation” reports one scoring unit under several protocols. ChemArgus is the only benchmark providing all five, green checkmarks for presence and red crosses for absence.
Model
Input
full problem
sequential carry
oracle scaffolded
GPT-5.6-Sol
multimodal
23.2
28.8
30.4
Gemini 3.8 Flash
multimodal
48.5
58.5
60.3
Claude Opus 5
multimodal
53.8
59.4
64.9
Qwen3.8-Max
multimodal
29.3
31.2
34.0
Kimi K3
multimodal
27.5
30.6
31.8
Grok 4.6
multimodal
33.8
41.8
45.5
Table 2: Results on the expanded model roster, normalized rubric score in %, rounded to one decimal. Bold marks the best score per metric. Multimodal runs receive real image blocks, text only runs receive curated text descriptions with zero image blocks, and text + reader runs pair a text model with the Qwen3-VL image reader, which converts each image to text.
Figure 3: Every reported run under the three metrics, grouped by input mode, normalized rubric score in %.
Figure 4: Score trajectory across the three metrics for every reported run, one panel per input mode, with scores as normalized rubric percentages.
Model
full problem
sequential carry
oracle scaffolded
mm
text
Δ
mm
text
Δ
mm
text
Δ
GPT-5.6-Sol
23.2
29.8
+6.5
28.8
35.6
+6.8
30.4
36.5
+6.1
Gemini 3.8 Flash
48.5
46.5
-2.0
58.5
59.0
+0.4
60.3
59.7
-0.6
Claude Opus 5
53.8
61.5
+7.7
59.4
61.7
+2.3
64.9
61.7
-3.2
Qwen3.8-Max
29.3
36.0
+6.7
31.2
39.0
+7.8
34.0
41.9
+7.9
Kimi K3
27.5
34.5
+7.0
30.6
36.0
+5.4
31.8
37.5
+5.7
Table 3: Modality contrast for the seven models reported under both inputs, normalized rubric score in %, where mm reports the multimodal run with real image blocks and text reports the text only run with curated text descriptions. Δ reports text only minus multimodal, in percentage points, positive values favor text only. Bold marks the best input score per metric.
Figure 5: Modality contrast for six of the seven models reported under both inputs, one panel per model, with Δ as text only minus multimodal in percentage points, positive values favoring text only.
Model
full problem
sequential carry
oracle scaffolded
text
reader
Δ
text
reader
Δ
text
reader
Δ
DeepSeek V4 Pro
25.0
36.7
+11.7
25.0
36.8
+11.8
27.6
37.0
+9.4
GLM-5.2
23.5
52.0
+28.5
24.9
52.2
+27.3
25.0
52.4
+27.3
GPT-5.6-Sol
29.8
31.1
+1.3
35.6
33.6
-2.1
36.5
34.0
-2.6
Gemini 3.8 Flash
46.5
33.8
-12.7
59.0
45.1
-13.9
59.7
49.7
-9.9
Claude Opus 5
61.5
46.2
-15.4
61.7
53.9
-7.8
61.7
54.7
-7.0
Table 4: Reader effect per model. Text + reader runs pair a text model with the Qwen3-VL image reader, which converts each image to text, and Δ marks text + reader minus text only, in percentage points, with positive values favoring the reader pairing.
Figure 6: Reader effect per model in three side-by-side metric panels. Deltas are text + reader minus text only in percentage points; purple marks reader gains and teal marks reader losses.
Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.
Hongyu Guo, Hao Li, He Cao +2
Peking University, Shenzhen Graduate School · International Digital Economy Academy (IDEA)
Current chemistry benchmarks primarily evaluate outcome prediction or reaction pathway generation, but rarely test whether a model can explain why an observed chemical effect occurs. We introduce ChemExplain, a benchmark of 1,180 multiple-choice chemistry questions drawn from 383 peer-reviewed papers. Each question pairs a molecular or reaction image with an observed chemical effect and four candidate explanations, each associated with a different structural feature visible in the image. Answering requires the model to perceive the depicted structure, apply chemical knowledge, and identify which feature most directly explains the effect, a capability we term explanatory attribution. The correct feature-mechanism relation is derived from supporting evidence in the source paper rather than invented by the question-generation LLM. We construct the benchmark automatically with a multi-agent framework that applies progressive quality control at three stages and validate it with eight chemistry domain experts. The strongest evaluated model reaches 78.6% accuracy, leaving considerable room for improvement. On a 600-question expert-reviewed subset, reasoning-trace analysis classifies 71.4% of the strongest model's errors as attribution failures. Attribution is also the largest error category for the open-source and chemistry-specific models included in our analysis.
The rapid advancement of large language models (LLMs) has led to increasing interest in their application to scientific domains such as chemistry. However, existing chemistry benchmarks often provide only a narrow view of model capability, focusing on limited task sets while overlooking robustness to variations in problem formulation and chemical representation. As a result, reported performance may overestimate a model's true ability to reason consistently across realistic settings. To address this challenge, we introduce ChemDIRT (Diversified Instruction, Representation, and Task Benchmark), a comprehensive evaluation framework designed to assess the robustness of chemical reasoning in LLMs. ChemDIRT systematically measures model performance across variations in instructions and molecular representations while spanning eight categories of chemistry tasks. By evaluating both accuracy and consistency under these controlled perturbations, ChemDIRT provides a more reliable assessment of model reasoning capabilities than conventional single-format benchmarks. We benchmark a diverse set of open- and closed-source LLMs, revealing substantial prompt sensitivity, representation dependence, and uneven performance across task families.