CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
Authors: Jin Gene Wong, Anjney Midha, Joseph Mathew Tennyson, Wei-Lin Chiang, Ion Stoica, Zhi-Xun Shen
Organizations: Physics Department and Stanford Laboratory for Intelligent Matter, Stanford University, Stanford, CA 94305 · Applied Physics Department, Stanford University, Stanford, CA 94305 · Electrical Engineering & Computer Sciences, University of California, Berkeley, Berkeley, CA 94720 · Applied Physics Department, Physics Department, and Stanford Laboratory for Intelligent Matter, Stanford University, Stanford, CA 94305
Large Language Models (LLMs) are presently well assessed by expert-level mathematics and physics benchmarks which focus on theoretical, symbolic reasoning. However, we lack experimental benchmarks that evaluate data analysis in frontier research problems, a capability that future domain-specific foundation models must have. To this end, we present CME-SpectrumBench, an expert-designed benchmark consisting of 190 questions that measures how reliably frontier LLMs analyze research-level spectral scientific data in condensed matter experiments (CME). Using simulated CME spectral data, we prompt models to identify structures from 2D intensity arrays in the face of noise and limited instrumental resolution, a test of dense context reasoning. Models are scored by how accurately their numerical answers approach the ground truth. Under direct query, scores generally pale in comparison to handwritten code due to clear failure modes: models can find simple features but struggle to trace patterns or compute derived quantities, and performance typically degrades with increased data resolution and noise/convolution levels. LLM performance becomes comparable to handwritten code by allowing tool use (access to a Python interpreter) with/without ReAct prompting, in effect mapping questions onto agentic code generation problems. Our findings show that frontier models can serve as research assistants by interfacing with data through code, but still face difficulties processing information directly from large datasets - a key barrier future foundation models must overcome. The code and data are publicly available at https://huggingface.co/datasets/jingene/CMESpectrumBench.
Figures & tables
Figure 1: Sample spectral function A(k,ω) from the bottom of a band, from which we may draw three tiers of questions (Section 2.3) . Increasing noise levels are shown: (a) 0% , (b) 10% , (c) 40% .
Figure 2: (a) More recently released, closed–weight (proprietary) models generally perform better. Models with asterisks (*) are open–weight. Note expanded axes scaling, which top out at 1. Min. / max. refer to spectra with minimum and maximum noise or convolution. (b) Advanced prompting improved scores, particularly for questions with minimal noise and convolution (min.) , but gains were marginal when experimental complications were severe (max.) . A solid black line denotes identical scores under direct query and advanced prompting. The legend applies to both subfigures.
Figure 3: Advanced prompting improved scores slightly for Tier I and Tier III questions, and vastly for Tier II. However, questions with maximal noise or convolution (red) were still poorly tackled.
Figure 4: (a) Higher resolution (more pixels/token count) typically worsened scores. Three examples are shown, representing the three tiers of questions. A vertical dashed line at 1002 pixels indicates the resolution at which the standard evaluation is carried out in Fig. 2 . Shaded areas indicate error in the mean score (standard error) σx=σx/n , for score x , sample standard deviation σx , and sample size n . (b) Maintaining the same color scheme , exponential decay of score with resolution (beyond 1002 pixels) is shown across categories, highlighting the differences between tiers of questions.
Figure 5: Unexpected improvement and subsequent expected decline in accuracy of answers with increasing noise; ground truth in red. Noise levels are set at (a) 0% , (b) 10% , and (c) 40% of maximum dispersion intensity, for a quadratic dispersion (question category B2 ).
Model display name
Model API name
Context window (tokens)
Claude Sonnet 3.7
claude-3-7-sonnet-20250219
200k
Claude Sonnet 4.5
claude-4-5-sonnet-20250219
200k
Claude Opus 4.5
claude-opus-4-5
200k
Claude Opus 4.6
claude-opus-4-6
1 million
Claude Opus 4.7
claude-opus-4-7
1 million
DeepSeek–R1
deepseek-reasoner
128k
Table 6
Figure 6: Question domain A (Fermi level extraction). Along with F2 , this is the easiest domain, judging from scores in Figs. 13 and 14 . Spectra here ( A1 ) are featureless in momentum k (horizontal axis), and take on a Fermi–Dirac distribution in energy ω (vertical axis), Intensity∝e(ω−μ)/kBT+11 where μ corresponds to the Fermi level, kB is the Boltzmann constant, and T temperature. The Fermi level is the midpoint of spectral intensity and is visually obvious in the figure; the point of A1 is to retrieve this energy as a single number, a task not far removed from NIAH retrieval, although the addition of noise complicates this somewhat. Note that spectra in question domains B–D (Figs. 7 – 9 ) have an upper cut–off at the Fermi level as little information of interest is contained above the Fermi level; this also separates the problems tested by later questions from the task of Fermi level extraction itself. Question domain E (Fig. 10 ) is arrayed along two momentum axes, cut at the Fermi level in energy ω .
Figure 7: Question domain B (dispersion tracing). These cover a linear dispersion ( B1 ), curved/quadratic dispersion ( B2 ), linear dispersion with superstructure ( B3 ), a quadratic dispersion with a band bottom ( B4 ), a Dirac cone ( B5 ), and superconducting gap(s) ( B6 ). For B1 – B4 , models are prompted to trace the dispersion itself (white line) as function of momentum k or energy ω ; these are Tier II tasks. B4_bbE , B5 , and B6 respectively ask for the band bottom energy (bottom of parabola), Dirac cone energy (crossing point of two dispersions), and superconducting gap energy (given by half the distance between the two horizontal white lines); these are Tier I questions. Lastly, B1_vF , B2_vF , and B3_vF ask for Fermi velocity vF , which are the gradients ∂ω/∂k of the dispersion at the top of the spectra; these are Tier III.
Figure 8: Question domain C (line width tracing). Models are prompted to reproduce half–width half–maximum (HWHM) as a function of energy ω , visually shown as half the distance between the red lines. These cover cases where HWHM is constant ( C1 , impurity scattering), increases linearly away from the Fermi energy/top of spectrum ( C2 , marginal Fermi liquid), and increases quadratically ( C3 , Fermi liquid). C4 and C5 are variants of C2 and C3 with the presence of a single phonon at some phonon energy (horizontal white line). The presence of the phonon is signaled by a shift in the (red) dispersion away from the original linear (white) dispersion, an effect termed mass renormalization by physicists, and a broadening of line width below the phonon energy.
Figure 9: Question domain D (phonon energy determination). Horizontal white lines indicate phonon energies for a single phonon at five different coupling strengths λ ( D1 ), as well as two ( D2 ) and three phonons ( D3 ); models are prompted to state these energies.
Figure 10: Question domain E (doping determination). Spectra shown are Fermi surface maps in two momentum axes, kx and ky , over a single Brillouin zone (BZ). Fermi surfaces enclose areas whose size is linearly dependent on the doping level of a material. To keep matters internally consistent, we normalize each BZ to the same extent ( ±1/A˚ in both directions), and define Doping=1−2⋅Area(BZ)Area, such that a Fermi surface that takes up the whole BZ has a doping of −1 (completely electron–doped), and none of the BZ, +1 (completely hole–doped). Suffice it to say that conventions and BZ sizes vary in real life, but we wish to keep things simple here.
Figure 11: Question domain F (RIXS energy determination), including determination of feature energy ( F1 ), single phonon energy ( F2 ), phonon overtone energy ( F3 ), and second phonon overtone energy ( F4 ).
Figure 12: Question domain G (RIXS dispersion tracing), including tracing single plasmon dispersion ( G1 ), likewise done while differentiating between two ( G2 ) and three ( G3 ) plasmon dispersions; single magnon dispersion ( G4 ); tracing one of three orbiton dispersions ( G5–G7 ).
Figure 13: Scores from direct prompting , excluding handwritten code. Within each category (e.g. A1 ), the amount of noise (ARPES) or convolution (RIXS) increases as the plot descends. The colorplot ranges from 0 (deep purple) to 1 (bright yellow).
Figure 14: Scores from direct and advanced prompting . Within each question domain (e.g. A1 ), the amount of noise (ARPES) or convolution (RIXS) increases as the plot descends. The colorplot ranges from 0 (deep purple) to 1 (bright yellow).
Spectroscopic structure elucidation is central to molecular analysis, but recent Large Language Model (LLM)-based methods mostly formulate it as direct spectrum-to-SMILES generation. Although this paradigm can leverage paired spectral data, it does not explicitly model the analytical workflow used by spectroscopists, such as diagnostic peak interpretation, fragment reasoning, formula constraints, and chemical consistency checking. In this paper, we introduce S3C-LLM, a skill-guided and code-grounded agentic LLM for spectrum-to-structure elucidation. Rather than directly predicting a molecule, S3C-LLM retrieves modality-specific spectroscopy skills, executes analysis code to instantiate these skills on the input spectra, and integrates the resulting peak-level evidence and formula constraints before generating SMILES. Specifically, we contribute a self-evolving spectroscopy skill library, a thinking-augmented skill-code trajectory construction pipeline, and a two-stage training strategy that teaches Qwen3-4B through supervised fine-tuning (SFT) followed by our proposed step-level reinforcement learning (RL). Experiments on diverse benchmarks show that S3C-LLM consistently outperforms current general LLMs and spectrum-specific models across spectra, while using less than 1/10th of SpectraLLM's training corpus.
Xuanle Zhao, Xinyuan Cai, Xiang Cheng +1
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.
Chuhan Shi, Xiaoquan Ren, Sicheng Song +3
Southeast University · East China Normal University · Hong Kong University of Science and Technology
As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchmarks due to data contamination, insufficient complexity, and prohibitive human labor costs. Here we present SPM-Bench, an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM). We propose a fully automated data synthesis pipeline that ensures both high authority and low-cost. By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025. Through a hybrid cloud-local architecture where VLMs return only spatial coordinates "llbox" for local high-fidelity cropping, our pipeline achieves extreme token savings while maintaining high dataset purity. To accurately and objectively evaluate the performance of the LLMs, we introduce the Strict Imperfection Penalty F1 (SIP-F1) score. This metric not only establishes a rigorous capability hierarchy but also, for the first time, quantifies model "personalities" (Conservative, Aggressive, Gambler, or Wise). By correlating these results with model-reported confidence and perceived difficulty, we expose the true reasoning boundaries of current AI in complex physical scenarios. These insights establish SPM-Bench as a generalizable paradigm for automated scientific data synthesis.