Authors: Kyle Waters, Lucas Nuzzi, Tadhg Looram, Alessandro Tomasiello, Ariel Ghislain Kemogne Kamdoum, Bikun Li, Damien Sileo, Egor Kretov, Francesco Fournier-Facio, Georgios Soloupis, Haile Kassahun, Hew Wolff, Jiaqi Cai, Lianghui Li, Marc Roth, Mohinder Naiya, Naixu Guo, Qicheng Tang, Richard Wheeler, Samuele Sala, Serguei Popov, Steven Dillmann, Yuqi Li
Organizations: PortexAI · University of Milano-Bicocca · University of Calgary · University of Chicago · Inria · Fraunhofer Institute for Individualized Medical Technology IMTE · University of Cambridge · Independent · McGill University · Massachusetts Institute of Technology · École Polytechnique Fédérale de Lausanne · Queen Mary University of London · Dot Ingredients · National University of Singapore · Georgia Institute of Technology · University of Edinburgh · Murdoch University · University of Porto · Stanford University · Stony Brook University
AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written benchmarks have proven effective at measuring AI reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted multimodal Terminus-2 agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains.
Table 3: Judge-vote agreement patterns by model across LLM-judge criteria combinations.
Model
Label
Runs
Pass rate
Avg. total / agent steps
claude-opus-4.6
installs_package
13
23%
10.9 / 9.9
claude-opus-4.6
no_substantive_tool_use
57
21%
5.3 / 4.3
gemini-3.1-pro
installs_package
24
17%
10.5 / 9.5
gemini-3.1-pro
calls_preinstalled_tools
2
0%
9.5 / 8.5
gemini-3.1-pro
no_substantive_tool_use
44
20%
6.1 / 5.1
gpt-5.4
installs_package
1
0%
6.0 / 5.0
Table 4: Observed solver-behavior clusters and associated Pass@1 outcomes.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Score
Result
Episodes
Trajectory summary
claude-opus-4.6
100%
PASS
10
Attempted Python package routes, then successfully installed python3-rdkit via apt; used RDKit-based parsing to compute molecular formula and hydrogen count; wrote exact final answer ( 350 ) to /app/answer.txt .
gpt-5.4
0%
FAIL
5
Tooling checks found no preinstalled chemistry libraries; install attempts via Python package paths failed; pivoted to a custom SMILES/valence parser and produced 399 , which failed exact-match grading against the reference answer 350 .
Appendix
Table 5: Case-study trajectory comparison for task 0sL35u .