cs.AINov 18, 2026

Benchmarking LLM abilities on postdoc-level natural science problems

Authors: Alina Avanesyan, Alexey Boldyrev, Liqun Deng, Denis Derkach, Mary Godunova, Mikhail Hushchyn, Dmitry Ilvovsky, Irina Knyazeva, 

Organizations: HSE University, Moscow, Russia · Independent researcher · N. P. Bechtereva Institute of the Human Brain RAS, St. Petersburg, Russia · AI Foundation and Algorithm Lab · 

Abstract

Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions

    Aug 31, 2026Guangxiang Zhao, Qilong Shi, Xusen Xiao +13LLM EvaluationScientific Reasoning

  2. SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

    Oct 23, 2026Tianyun Zhong, Wei Wang, Jialin Li +17Scientific Agent BenchmarksLLM Agent Evaluation

  3. Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

    Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3AI for ScienceLLM Evaluation