Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still score answer endpoints alone. Such scoring conflates memorized facts with stepwise derivation, treats visual and textual inputs as interchangeable, and confounds long-horizon planning with stepwise execution. These blind spots prevent any diagnosis of where the reasoning chain breaks in a model. We present ChemArgus, a competition-level chemistry benchmark drawn from chemistry competition material, textbooks, and research papers, whose scoring replaces the answer endpoint with three measurement properties. Fine-grained scoring grades every subquestion point by point with partial credit, so a score lands at the level of the step rather than the outcome. Multimodal capability is priced on every problem containing a real image, paired with a controlled text description, so visual understanding is measured against text on identical tasks. Diagnostic attribution reads paired metrics that share one scoring unit and differ only in context construction, so a gap between two scores isolates a single cause, namely integration, propagation, or local reasoning. GPT-5.6-Sol scores 23.2% on unassisted whole-problem solving against 28.8% and 30.4% under the two sequential metrics, a gap measuring the integration burden. Through rubric partial credit and these cause-isolating score gaps, ChemArgus shows not only whether a model answers chemistry correctly but where its reasoning chain fails, so that every score it reports doubles as a diagnosis, a readout of the chain rather than a verdict, aiming to guide language models step by step on their path toward chemistry expertise, one corrected step at a time, each step priced by the rubric, point by point.
Figures & tables
Benchmark
Image input
Image-text contrast
Rubric partial credit
Subquestion chains
Protocol separation
SciBench [ 18 ] [ICML24]
✓
✗
✗
✗
✗
SciCode [ 17 ] [NeurIPS24]
✗
✗
✗
✓
✗
GPQA [ 8 ] [arXiv23]
✗
✗
✗
✗
✗
DiscoveryWorld [ 65 ] [NeurIPS24]
✗
✗
✗
✗
✗
SciEval [ 27 ] [AAAI24]
✗
✗
✗
✗
✗
ChemBench [ 3 ] [arXiv24]
✗
✗
✗
✗
✗
Table 1: Comparison of ChemArgus against prior chemistry and adjacent scientific benchmarks on five properties. “image input” reports images as real multimodal blocks, and “image-text contrast” reports every figure paired with controlled text. “rubric point partial credit” reports partial credit per rubric point, and “subquestion chains” report problems split into shared-context subquestions. Finally, “protocol separation” reports one scoring unit under several protocols. ChemArgus is the only benchmark providing all five, green checkmarks for presence and red crosses for absence.
Model
Input
full problem
sequential carry
oracle scaffolded
GPT-5.6-Sol
multimodal
23.2
28.8
30.4
Gemini 3.8 Flash
multimodal
48.5
58.5
60.3
Claude Opus 5
multimodal
53.8
59.4
64.9
Qwen3.8-Max
multimodal
29.3
31.2
34.0
Kimi K3
multimodal
27.5
30.6
31.8
Grok 4.6
multimodal
33.8
41.8
45.5
Table 2: Results on the expanded model roster, normalized rubric score in %, rounded to one decimal. Bold marks the best score per metric. Multimodal runs receive real image blocks, text only runs receive curated text descriptions with zero image blocks, and text + reader runs pair a text model with the Qwen3-VL image reader, which converts each image to text.
Figure 3: Every reported run under the three metrics, grouped by input mode, normalized rubric score in %.
Figure 4: Score trajectory across the three metrics for every reported run, one panel per input mode, with scores as normalized rubric percentages.
Model
full problem
sequential carry
oracle scaffolded
mm
text
Δ
mm
text
Δ
mm
text
Δ
GPT-5.6-Sol
23.2
29.8
+6.5
28.8
35.6
+6.8
30.4
36.5
+6.1
Gemini 3.8 Flash
48.5
46.5
-2.0
58.5
59.0
+0.4
60.3
59.7
-0.6
Claude Opus 5
53.8
61.5
+7.7
59.4
61.7
+2.3
64.9
61.7
-3.2
Qwen3.8-Max
29.3
36.0
+6.7
31.2
39.0
+7.8
34.0
41.9
+7.9
Kimi K3
27.5
34.5
+7.0
30.6
36.0
+5.4
31.8
37.5
+5.7
Table 3: Modality contrast for the seven models reported under both inputs, normalized rubric score in %, where mm reports the multimodal run with real image blocks and text reports the text only run with curated text descriptions. Δ reports text only minus multimodal, in percentage points, positive values favor text only. Bold marks the best input score per metric.
Figure 5: Modality contrast for six of the seven models reported under both inputs, one panel per model, with Δ as text only minus multimodal in percentage points, positive values favoring text only.
Model
full problem
sequential carry
oracle scaffolded
text
reader
Δ
text
reader
Δ
text
reader
Δ
DeepSeek V4 Pro
25.0
36.7
+11.7
25.0
36.8
+11.8
27.6
37.0
+9.4
GLM-5.2
23.5
52.0
+28.5
24.9
52.2
+27.3
25.0
52.4
+27.3
GPT-5.6-Sol
29.8
31.1
+1.3
35.6
33.6
-2.1
36.5
34.0
-2.6
Gemini 3.8 Flash
46.5
33.8
-12.7
59.0
45.1
-13.9
59.7
49.7
-9.9
Claude Opus 5
61.5
46.2
-15.4
61.7
53.9
-7.8
61.7
54.7
-7.0
Table 4: Reader effect per model. Text + reader runs pair a text model with the Qwen3-VL image reader, which converts each image to text, and Δ marks text + reader minus text only, in percentage points, with positive values favoring the reader pairing.
Figure 6: Reader effect per model in three side-by-side metric panels. Deltas are text + reader minus text only in percentage points; purple marks reader gains and teal marks reader losses.