Sci$Π$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Youran Sun; Xingyu Ren; Chugang Yi; Kejia Zhang; Di Wu; Yingkang Cao; Allen Zhou; Jianping Ye

cs.AIOct 15, 2026

Sci $Π$ vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Authors: Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu, Yingkang Cao, Allen Zhou, Jianping Ye,

Organizations: University of Maryland, College Park · The Chinese University of Hong Kong · Gradient Network · Stanford University ·

Abstract

Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct Sci $Π$ vot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while Sci $Π$ vot supports systematic evaluation of scientific judgment for LLM agents.

Figures & tables

Figure 1: How a case is built and used. The top row shows an original LLM plan (A), a senior researcher’s intervention, and the project’s subsequent progress. We use the completed history to reconstruct the human-corrected plan (B) and the evidence supporting it. Evaluators receive the pre-intervention project record and both plans, then choose which to pursue. Choosing B counts as correct.

Figure 2: Two human corrections from frontier research projects. In (A), a small language model scores zero on three LiveCodeBench groups ( Jain et al., 2024 ) , and the original LLM plan is to retest with a larger model. The senior researcher retains the small model, and a subsequent audit finds that the scorer had skipped answer extraction. In (B), a cloth fit reaches the search limits while four parameters stay fixed at zero, a restriction the agent had chosen in every earlier run. The senior researcher asks to fit all six parameters. Models receive the project record before the intervention and the two plans, without correctness labels or the explanations shown here. Appendix B gives the source evidence and model responses.

Figure 3: Accuracy by (a) scientific field (Table 3 ) and (b) judgment category (Table 3 ). Models run with the project record at medium reasoning effort. Section 4.1 describes the human baseline. Categories in (b) can overlap.

Benchmark	Scientific research task	Basis for evaluation	Fields
PaperBench	Replicate published ML experiments	Replication rubrics from human authors	ML
RE-Bench	Improve a defined ML R&D objective	Task scores normalized against human reference solutions	ML
MLE-bench	Build ML solutions for Kaggle competitions	Competition metrics and human leaderboards	ML
ScienceAgentBench	Write programs for scientific data workflows	Expert-validated tasks; execution checks and rubrics	Multiple
SDE	Reason about scientific scenarios; pursue discovery hypotheses	Answers checked by human experts; computational oracles	Multiple
Corral	Conduct scientific research; inspect tool traces	Task scores and human expert diagnoses	Multiple

Table 3: Representative scientific research benchmarks and how they establish good answers. Sci

\Pi

vot tests scientific planning and decision-making using experimental records and senior researchers’ corrections from real frontier scientific research. Some benchmarks evaluate the listed activities in separate tasks. Multiple means coverage across broad scientific fields. AI/ML subfields count as one field.

Model	Project (%)	Options only (%)	Best observed project (%)	Setting
Astra	59.5	63.7	63.5	xhigh
Opus	38.9	58.9	46.8	max
Grok	33.1	54.0	38.1	high
Sol	35.1	46.8	37.7	medium, after review 2
Terra	29.2	55.8	52.0	medium, after review 1
Human baseline	73.4	—	—	agent assistant available

Table 4: Model accuracy on the same 248 cases, averaged over both option orders. Project and Options only use medium effort. Best observed project is the highest score among seven conditions. These conditions are the five initial effort settings and the first and second review at medium effort. Complete 95% intervals are in Appendix A . Appendix D describes the human baseline. The options-only condition measures model context sensitivity.

Figure 4: Accuracy difference with the project record, relative to the options-only condition. Comparisons pair the same cases at medium effort. Bars show 95% case-bootstrap intervals.

Figure 5: More deliberation can restore a rejected plan after a correct choice. (a) Accuracy at each requested effort setting. (b) The initial answer at medium effort and its two recorded continuations. (c) Repairs and new errors across five models and both orders. Each review contains 2,480 response transitions on 248 cases. Panels (a) and (b) use a 25–67% accuracy range. The dashed line marks 50%. Effort settings are not matched token budgets. Repeated responses are not independent cases.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Project (%)	Options only (%)	Paired difference (pp)
Astra	59.5 [53.8, 65.1]	63.7 [58.5, 69.0]	-4.2 [-10.9, +2.4]
Opus	38.9 [33.5, 44.4]	58.9 [54.0, 63.7]	-20.0 [-26.8, -13.3]
Grok	33.1 [28.0, 38.1]	54.0 [48.6, 59.5]	-21.0 [-27.8, -14.3]
Sol	35.1 [29.8, 40.3]	46.8 [41.3, 52.0]	-11.7 [-18.5, -4.8]
Terra	29.2 [24.6, 34.1]	55.8 [50.6, 60.9]	-26.6 [-32.9, -20.0]

Appendix

Table 5: Medium-effort accuracies and paired Project minus Options only differences on 248 cases. Brackets give 95% percentile case-bootstrap confidence intervals after averaging the two option orders.

Model	Project 95% CI	Difference 95% CI
Astra	[52.6, 67.0]	[-11.8, +3.6]
Opus	[33.4, 45.2]	[-25.0, -14.5]
Grok	[28.3, 39.2]	[-25.6, -15.7]
Sol	[29.5, 42.6]	[-17.8, -4.5]
Terra	[24.2, 36.8]	[-32.6, -19.0]

Appendix

Table 6: Project-cluster 95% intervals for project accuracy (%) and the Project minus Options only difference (percentage points). Both estimates retain equal case weights.

Project identifier	Scientific question	Area	Cases
cpd-lattice	Can screened low-rank function proposals accelerate high-dimensional approximation and PDE solving?	Applied mathematics & scientific computing	4
fex-dim-lift-skeleton	When can solution structures found for low-dimensional PDEs be reused in higher dimensions?	Applied mathematics & scientific computing	12
fex-foundation-model	When does a pretrained expression-search prior help solve a new PDE?	Applied mathematics & scientific computing	12
fex-search-complexity	When can a search algorithm efficiently discover a PDE solution that has a compact expression?	Applied mathematics & scientific computing	4
fex-synchro-prior	Can checked spectral priors resolve frequency-search bottlenecks in symbolic PDE solvers?	Applied mathematics & scientific computing	1
ocl-calibration	How can calibration ensure that optimizer-selected decisions satisfy learned constraints?	Applied mathematics & scientific computing	8

Appendix

Table 7: Benchmark projects (1/2). The questions describe scientific objectives rather than established findings. Area labels describe the current sample. We do not count secondary methodological links separately.

Project identifier	Scientific question	Area	Cases
cd-projection-ladder	Does a failed reasoning trace disrupt later answers through problem-specific errors or general context interference?	AI & machine learning	1
cot-mon-stego-canary	How much does reading reasoning text improve misconduct detection beyond observing final actions?	AI & machine learning	4
elf-ecf	Can flow matching improve sentence-level language generation with fewer sampling steps?	AI & machine learning	12
elf-fk-steer	Can intermediate text previews guide generation toward factual and otherwise constrained outputs?	AI & machine learning	3
hybrid-scaling-decomp	Do model-selection criteria systematically favor particular neural scaling laws when the data cannot distinguish them?	AI & machine learning	4
looped-lora	Can small trainable adapters turn a frozen pretrained Transformer into an effective recurrent model?	AI & machine learning	24

Appendix

Table 8: Benchmark projects (2/2). The questions describe scientific objectives rather than established findings. Area labels describe the current sample. We do not count secondary methodological links separately.

Model	Low	Medium	High	xhigh	Max
Astra	57.7 / 64.5	59.5 / 63.7	62.1 / 64.9	63.5 / 64.5	63.1 / 62.7
Opus	40.5 / 59.5	38.9 / 58.9	43.3 / 59.1	46.0 / 58.9	46.8 / 57.9
Grok	32.1 / 53.8	33.1 / 54.0	38.1 / 55.8	33.3 / 56.5	33.1 / 53.4
Sol	36.1 / 54.4	35.1 / 46.8	35.9 / 54.2	28.0 / 53.8	29.2 / 55.8
Terra	31.7 / 56.5	29.2 / 55.8	30.4 / 55.4	30.2 / 55.6	34.9 / 57.3

Appendix

Table 9: Accuracy (%) at every requested effort setting. Each cell reports Project / Options only. Unparsed completed responses count as incorrect.

Model	$\Delta$ (pp)	Paired case 95% CI	Project-cluster 95% CI
Astra	+1.2	[-0.8, +3.2]	[-0.9, +2.9]
Opus	-5.4	[-8.7, -2.2]	[-9.6, -1.9]
Grok	-0.4	[-4.0, +3.2]	[-3.0, +3.0]
Sol	+5.6	[+0.2, +11.1]	[+1.2, +10.0]
Terra	-9.3	[-15.3, -3.0]	[-13.4, -4.3]
Five-model mean	-1.7	[-3.5, +0.2]	[-2.7, -0.3]

Appendix

Table 10: Accuracy change after the second review, relative to the first, in percentage points. We average both option orders within each of 248 cases before computing paired differences. Intervals use 100,000 paired percentile bootstrap samples. The project bootstrap resamples 26 whole projects with replacement and retains case weights. The mean averages the fixed set of five evaluated models within each case.

Table 14

	Language model		Cloth simulator
Model	Project	Options	Project	Options
Astra	2/2	1/2	0/2	2/2
Opus	1/2	2/2	0/2	0/2
Grok	2/2	2/2	1/2	0/2
Sol	2/2	1/2	0/2	0/2
Terra	0/2	1/2	0/2	1/2

Appendix

Table 13: Reference-plan choices across the two option orders at medium effort. “Options” denotes the options-only condition. Each model answers the same case twice, with the option order reversed. The ten choices per condition are not ten independent cases.

Case	Decision shorthand	Project	Options only
206	Check stiffness and travel limits	10/10	7/10
3	Start the resolution matrix; drop the baseline gate	8/10	3/10
21	Pilot the API pipeline at 20 items per cell	3/10	6/10
13	Train one seed for the full 8,000 iterations	4/10	6/10

Appendix

Table 14: Reference-plan choices at medium effort for the four examples in Appendix C . We selected these examples for their decision structure before inspecting model scores. Each count is out of ten responses from five fixed models and two option orders, not ten independent cases. Case numbers are zero-based indices in the benchmark.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment
May 30, 2026Qiuyu Tian, Haojie Yin, Yingce Xia +2LLM Agent Evaluation AI Agent Benchmarks
1Southeast University, Nanjing, China · 3Duke Kunshan University, Kunshan, China · 2Beijing Zhongguancun Academy, Beijing, China
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory

Apr 20, 2026J/K move · Enter open · S save

AI scientists produce results without reasoning scientifically

Martiño Ríos-García, Nawaf Alampara, Chandan Gupta, Indrajeet Mandal+4

Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India+3

AI Agent Reliability Scientific Reasoning LLM Agent Evaluation LLM Reliability Autonomous Scientific Discovery Scientific Reasoning in Language Models+2

Open paper AskPDF arXiv

Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously, yet whether their reasoning adheres to the epistemic norms that make scientific inquiry self-correcting is poorly understood. Here, we evaluate LLM-based scientific agents across eight domains, spanning workflow execution to hypothesis-driven inquiry, through more than 25,000 agent runs and two complementary lenses: (i) a systematic performance analysis that decomposes the contributions of the base model and the agent scaffold, and (ii) a behavioral analysis of the epistemological structure of agent reasoning. We observe that the base model is the primary determinant of both performance and behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Across all configurations, evidence is ignored in 68% of traces, refutation-driven belief revision occurs in 26%, and convergent multi-test evidence is rare. The same reasoning pattern appears whether the agent executes a computational workflow or conducts hypothesis-driven inquiry. They persist even when agents receive near-complete successful reasoning trajectories as context, and the resulting unreliability compounds across repeated trials in epistemically demanding domains. Thus, current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot repair them. Until reasoning itself becomes a training target, the scientific knowledge produced by such agents cannot be justified by the process that generated it.

Abstract

LLM Agent Evaluation

Explore similar work

AI scientists produce results without reasoning scientifically

ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration