SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

Peiyao Xiao; Xiaogang Li; Xinyi Gao; Chengliang Xu; Ben Wang; Zichao Chen; Zeyu Wang; Lin Qu

cs.AIFeb 26, 2026

SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

Authors: Peiyao Xiao, Xiaogang Li, Xinyi Gao, Chengliang Xu, Ben Wang, Zichao Chen, Zeyu Wang, Lin Qu,

Organizations: Alibaba Group Beijing, China

Abstract

As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchmarks due to data contamination, insufficient complexity, and prohibitive human labor costs. Here we present SPM-Bench, an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM). We propose a fully automated data synthesis pipeline that ensures both high authority and low-cost. By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025. Through a hybrid cloud-local architecture where VLMs return only spatial coordinates "llbox" for local high-fidelity cropping, our pipeline achieves extreme token savings while maintaining high dataset purity. To accurately and objectively evaluate the performance of the LLMs, we introduce the Strict Imperfection Penalty F1 (SIP-F1) score. This metric not only establishes a rigorous capability hierarchy but also, for the first time, quantifies model "personalities" (Conservative, Aggressive, Gambler, or Wise). By correlating these results with model-reported confidence and perceived difficulty, we expose the true reasoning boundaries of current AI in complex physical scenarios. These insights establish SPM-Bench as a generalizable paradigm for automated scientific data synthesis.

Figures & tables

Figure 1. Distribution and important statistics of SPM-Bench (2,703 questions across 4 modalities: TEM, STM, SEM, AFM).

Figure 2. Statistical analysis of question complexity. Upper: stacked bar chart of question frequency. Lower: cognitive level distribution.

Figure 3. Overview of the data synthesis pipeline in SPM-Bench.

Figure 4. Parametric sensitivity of the SIP-F1 metric across varying

\lambda

and

\Gamma

values.

Strategy	Intermediate		Metric Scores
Strategy	$TP$	$FP$	EM	Std F1	SPC	SIP-F1
Perfect	2	0	1.00	1.0000	1.00	1.0000
Conservative	1	0	0.00	0.6667	0.50	0.4000
Aggressive	2	1	0.00	0.8000	0.00	0.2400
Gambling	2	2	0.00	0.6667	0.00	0.1500
All wrong	0	2	0.00	0.0000	0.00	0.0000

Table 1. Anti-Gaming Strategy Analysis (Ground truth: {A, B}).

Dimension	EM	Std F1	SPC	SIP-F1
Perfect match	1.0	1.0	1.0	1.0
50% omission	0.0	0.67	0.50	0.40
1 wrong selection	0.0	0.80+	0.0	$<$ 0.25
“Select-all”	0.0	High	0.0	$\sim$ 0.15
Discrimination	Low	Medium	Medium	High
Anti-gaming	High	Low	Medium	High

Table 2. Theoretical Comparison of Evaluation Metrics across Eight Dimensions.

Figure 5. Comparative analysis of model performance across eight scientific fields. Upper: radar chart showing field-specific competence envelopes for a 15-model subset (the complete 18-model results are reported in Table 3 ). Lower: heatmap providing exact EM scores per field, sorted by mean performance.

Figure 6. Analysis of accuracy under EM, SPC, and SIP-F1 evaluation metrics across a 15-model subset, sorted by SIP-F1 in descending order. The complete 18-model results are reported in Table 3 .

Figure 7. Behavioral profile analysis across a 15-model subset. Each subplot displays SIP-F1, confidence, perceived difficulty, and average tokens.

Model	EM	SPC	SIP-F1	$\Delta$	Personality
claude-opus-4-7	0.793	0.912	0.936	0.024	Wise
claude-sonnet-4-6	0.734	0.876	0.908	0.032	Wise
qwen3.5-plus	0.832	0.865	0.881	0.016	Wise
claude-opus-4-6	0.696	0.831	0.863	0.032	Wise
kimi-k2.5	0.790	0.840	0.855	0.015	Wise
qwen3.6-plus	0.658	0.819	0.790	0.030	Aggressive

Table 3. Thorough Performance of All 18 Models on SPM-Bench, sorted by SIP-F1. Figures 5 , 6 , and 7 present a 15-model subset for visual clarity.

Benchmark	Time	Mod.	Num.	Area	Type
SPM-Bench	Feb-26	MM	2.7k	Phys. Mat.	MCQ, VQA
MicroVQA++	Nov-25	MM	26k	Bio.	MCQ, VQA
AstroMMBench	Oct-25	MM	621	Astro.	MCQ
AMO-Bench	Oct-25	TXT	50	Math.	OE, Calc.
MatSciBench	Oct-25	MM	1.3k	Mat. Sci.	OE, Calc.
SEEPHYS	Oct-25	MM	2k	Phys.	OE, Reas.

Table 4. Taxonomy and Comparison of Emerging Scientific Reasoning Benchmarks.

Model	SIP-F1	ECE	Personality	Well-Calibrated?
qwen3.5-plus	0.881	0.082	Wise	Yes
gpt-5.2-1211-global	0.814	0.095	Wise	Yes
claude-opus-4.5	0.745	0.067	Conservative	Yes
gemini-3-flash-preview	0.710	0.183	Aggressive	No
qwen3-vl-plus	0.656	0.142	Aggressive	Marginal
gemini-2.5-flash-image-preview	0.646	0.201	Gambling	No

Table 5. Comparison of SIP-F1 and Expected Calibration Error (ECE) across six models, showing orthogonal evaluation dimensions.

Dimension	Claude Opus 4.7	GPT-5.5	Gemini 3.1 Pro	Qwen 3.6 Max
Scientific Accuracy	4.20	3.98	4.86	4.90
Question Clarity	3.93	3.64	3.92	4.35
Option Quality	4.18	3.80	4.66	4.77
Answer Correctness	4.12	4.03	4.72	4.81
Overall Quality	4.05	3.84	4.18	4.74
Average	4.10	3.86	4.47	4.71

Table 6. Benchmark Quality Assessment by Four Independent Model Evaluators (200 stratified questions, 1–5 scale).

Figure 8. Cross-model quality assessment of SPM-Bench questions.

Field Category	Questions	Share	Sub-fields
Condensed Matter Physics	967	35.8%	45+
Surface Science	596	22.0%	30+
Nanotechnology	121	4.5%	15+
Electron Microscopy	82	3.0%	10+
Magnetism & Spintronics	34	1.3%	8+
Materials Science	22	0.8%	12+

Table 7. Field Coverage of SPM-Bench

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$\|S_{c}\|$	$\{TP,FP\}$	$\Gamma$	$\lambda=0.3$	0.4	0.5	0.6	0.7	0.8
1	$\{1,1\}$	1	0.200	0.267	0.333	0.400	0.467	0.533
		2	0.150	0.200	0.250	0.300	0.350	0.400
		4	0.100	0.133	0.167	0.200	0.233	0.267
		6	0.075	0.100	0.125	0.150	0.175	0.200
		8	0.060	0.080	0.100	0.120	0.140	0.160
		10	0.050	0.067	0.083	0.100	0.117	0.133

Appendix

Table 8. Quantitative sensitivity analysis of the SIP-F1 metric across varied problem complexities (

|S_{c}|

), detection outcomes (

\{TP,FP\}

), and parameter configurations (

\Gamma,\lambda

). Highlighted cells distinguish performance tiers: blue points to scores approximately at 0.25 (Third Tier), while pink points to scores approximately at 0.4 (Second Tier).

Figure 9. Field-specific competence envelopes across model families. Unlike Figure 1 which organizes questions by microscopy type (TEM/STM/SEM/AFM), this chart categorizes by scientific field because field reflects distinct cognitive reasoning challenges (e.g., visual pattern recognition vs. quantum-mechanical inference), whereas microscopy type is a technical dimension that cuts across cognitive tasks.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3AI for Science LLM Evaluation
Southeast University · East China Normal University · Hong Kong University of Science and Technology
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
May 19, 2026Yiyang Gu, Junwei Yang, Junyu Luo +15AI for Science LLM Evaluation
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5
Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning
Apr 30, 2026Junpeng Ding, Zichen Tang, Haihong E +17AI for Science VLM Reasoning
Beijing University of Posts and Telecommunications

Jul 13, 2026J/K move · Enter open · S save

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Chuhan Shi, Xiaoquan Ren, Sicheng Song, Haobo Li+2

Southeast University · East China Normal University · Hong Kong University of Science and Technology

AI for Science LLM Evaluation Scientific Reasoning

Open paper AskPDF arXiv

Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.

Abstract

LLM Evaluation

Explore similar work

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning