Organizations: University of Chicago · University of Illinois Chicago · Harvard University · Boston University · Arizona State University · Yale University · University of Southern California · The University of Texas at Dallas · University of California, Los Angeles · Texas A&M University · Brown University · Department of Mechanical and Industrial Engineering, University of Toronto · Northeastern University · Departments of Neurology and Medicine, Stanford University · University of Illinois Urbana-Champaign · Purdue University · The Ohio State University · MatrAIx
Scientific agents collaborate with researchers whose expertise and preferences differ, but benchmarks usually hold the collaborator fixed. We introduce PersonaScienceBench: 200 tasks from five benchmarks across four scientific domains. Simulated researchers provide context and feedback according to persona profiles, while task objectives and grading remain fixed. Agents encounter researcher preferences only through conversation. Across eighteen modelconfigurations, task performance, fact recovery, and self-repetition reveal different strengths. Detailed GPT 6 Astra analyses show distinct agent responses across researcher profiles; interaction traces illustrate targeted clar ification and workflow revision. These behaviors can differ even when final scores match. PersonaScienceBench therefore evaluates how agents collaborate with diverse researchers alongside whether they complete the task.
Figures & tables
Figure 1: PersonaScienceBench overview. a, Separate public inputs from researcher-held context. b, Interact with a persona-conditioned researcher. c, Evaluate scientific outcomes, collaboration, and adaptation.
Figure 2: Performance by interaction length. Cumulative final scores from episodes ending by t , divided by the 200 task episodes in each setting. Curves describe completion lengths, not within-run learning.
Persona
Background and specialty
Primary language
Interaction preferences
001
Business and management; curriculum design
Hindi
Brief, direct communication.
003
Engineering; electrical work
Russian
Detailed, concrete explanations; speed and novel approaches.
046
Finance and economics; machine learning
Spanish
Formal, indirect communication.
155
Agriculture; agronomy
Mandarin
Detailed guidance; accuracy, familiar methods, and teamwork.
Table 1: Backgrounds and interaction preferences of the four researcher personas.
Table 3: Model-level scientific collaboration scorecard. Means with 95% confidence intervals in brackets; scores are normalized to 0–100. Actor and simulator share a model, and GPT-6 Astra with low reasoning effort judges all models. Estimation procedures differ across rows (Appendix B ), so cross-model comparisons are descriptive.
Persona
Life
Phys./ Eng.
Math./ CS
Soc./ Hum.
Avg.
Off
27.61 9.41–46.54
66.73 58.70–74.25
50.67 43.72–57.35
14.97 8.33–18.33
39.99 34.35–45.34
155
28.12 9.85–47.09
67.25 59.68–74.42
52.21 45.38–58.65
17.80 12.10–24.89
41.34 35.70–46.92
001
27.24 9.07–46.10
69.71 62.31–76.54
54.15 47.30–60.71
16.86 9.98–26.52
41.99 36.29–47.84
003
28.23 7.29–49.68
76.91 64.40–87.98
57.29 54.67–59.50
19.60 13.47–27.30
45.51 38.90–51.81
046
27.02 8.50–47.30
66.34 55.83–76.16
50.76 43.61–58.19
18.35 13.00–24.17
40.62 34.54–46.69
Table 4: Performance by scientific domain. Mean normalized score (0–100), with pointwise 95% confidence intervals below. Rows identify simulator personas evaluated on the same tasks. Domains: life sciences; physical sciences and engineering; mathematics and computer science; social sciences and humanities. Avg. weights domains equally.
Figure 3: Semantic separation and example trajectories. A, Excess cosine distance: simulator below, actor above the excluded diagonal. Stars: Holm-adjusted p<0.05 across 20 comparisons. B, BixBench (left) and SciConvBench (right); simulator (top), actor (bottom). Hollow/filled circles mark starts/ends; dots mark intermediate messages; overlapping endpoints indicate one message. UMAP coordinates are local to each subplot; curves only connect observations.
Figure 5: Interaction metrics by persona. A, Actor self-repetition with pointwise 95% study-cluster confidence intervals. B, Complete-fact recovery in failed SciConvBench disambiguation episodes.
Figure 6: Workflow revision with researcher feedback. Highlights added; […] marks omissions. Event identifiers are preserved.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Criterion
Required evidence
Scientific identity
Native task and study identifiers, objective, fixed constraints, deliverables, and a record of intentional changes.
Useful interaction
A legitimate context, decision, or review role for the researcher; no invented facts or quota of extra interactions.
Information separation
Allowlisted actor/researcher exports and runtime access checks that exclude hidden answers, tests, and solution artifacts.
Grounding and persona
Fact provenance, explicit unknowns and advice, versioned persona rendering, and declared knowledge/disclosure policies.
Runtime readiness
Resolved input roles and assets, hashes and mount mappings, actual resources, permitted tools, and required output formats.
Table 5: Acceptance rubric for benchmark conversion. Each criterion requires inspectable evidence; unresolved checks remain explicit rather than being inferred from successful serialization.
Figure 7: Incremental clarification with explanatory replies. Highlights added; […] marks omissions. Event identifiers are preserved.
Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.
Yinhao Tang, Youqing Fang, Yanan Sun +6
University of Science and Technology of China · Shanghai AI Laboratory
Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific Agents, an open-source corpus of 503 profession-specific AGENTS.md profiles, with Gemini 3.8 Flash via OpenRouter in the Pi agent harness. We compare matched profiles with four controls: a minimal baseline ("You are a helpful assistant"), the profile's opening role sentence, a generic scientific rigor guide, and a profile from an unrelated domain. Across nine text-based science benchmarks (4,531 sampled questions, 100 matched profiles), 4,488 items completed all five conditions after API-error retries, scored with automated, rule-based grading. The average profile-baseline accuracy difference is -0.6 percentage points (95% bootstrap interval [-1.5, +0.2] across fixed tasks), and no benchmark shows a statistically clear improvement. Matched profiles produced 1.5-2.3 times as many output tokens and cost 2.2-4.5 times more per successful call. On 60 tool-using BioMysteryBench bioinformatics problems (three runs each for baseline and profile), mean solve rates were 46.7% with the profile and 56.7% at baseline, a difference of -10.0 percentage points (95% interval [-16.7, -3.3]) driven by more frequent token- and time-limit stops under the profile. Longer prompts had one unexpected operational advantage: on SuperGPQA, frequent provider API drops left the short baseline with a correct first-pass answer on only 54.0% of items, against 71.6% with the profile. Generic and mismatched prompts were about as reliable, so this gain comes from prompt length or formatting rather than domain expertise. For the tested model and tasks, loading full profession profiles by default does not improve accuracy and costs considerably more; whether selective retrieval of profile sections or open-ended scientific tasks would change this remains to be tested.
Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.
Tianyun Zhong, Wei Wang, Jialin Li +17
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4