cs.AIOct 23, 2026
SavePersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers
Organizations: University of Chicago · University of Illinois Chicago · Harvard University · Boston University ·
Abstract
Scientific agents collaborate with researchers whose expertise and preferences differ, but benchmarks usually hold the collaborator fixed. We introduce PersonaScienceBench: 200 tasks from five benchmarks across four scientific domains. Simulated researchers provide context and feedback according to persona profiles, while task objectives and grading remain fixed. Agents encounter researcher preferences only through conversation. Across eighteen modelconfigurations, task performance, fact recovery, and self-repetition reveal different strengths. Detailed GPT 6 Astra analyses show distinct agent responses across researcher profiles; interaction traces illustrate targeted clar ification and workflow revision. These behaviors can differ even when final scores match. PersonaScienceBench therefore evaluates how agents collaborate with diverse researchers alongside whether they complete the task.
Figures & tables
Figure 1: PersonaScienceBench overview. a, Separate public inputs from researcher-held context. b, Interact with a persona-conditioned researcher. c, Evaluate scientific outcomes, collaboration, and adaptation.
Figure 2: Performance by interaction length. Cumulative final scores from episodes ending by , divided by the 200 task episodes in each setting. Curves describe completion lengths, not within-run learning.
| Persona | Background and specialty | Primary language | Interaction preferences |
|---|---|---|---|
| 001 | Business and management; curriculum design | Hindi | Brief, direct communication. |
| 003 | Engineering; electrical work | Russian | Detailed, concrete explanations; speed and novel approaches. |
| 046 | Finance and economics; machine learning | Spanish | Formal, indirect communication. |
| 155 | Agriculture; agronomy | Mandarin | Detailed guidance; accuracy, familiar methods, and teamwork. |
Table 1: Backgrounds and interaction preferences of the four researcher personas.
Table 3: Model-level scientific collaboration scorecard. Means with 95% confidence intervals in brackets; scores are normalized to 0–100. Actor and simulator share a model, and GPT-6 Astra with low reasoning effort judges all models. Estimation procedures differ across rows (Appendix B ), so cross-model comparisons are descriptive.
| Persona | Life | Phys./ Eng. | Math./ CS | Soc./ Hum. | Avg. |
|---|---|---|---|---|---|
| Off | 27.61 9.41–46.54 | 66.73 58.70–74.25 | 50.67 43.72–57.35 | 14.97 8.33–18.33 | 39.99 34.35–45.34 |
| 155 | 28.12 9.85–47.09 | 67.25 59.68–74.42 | 52.21 45.38–58.65 | 17.80 12.10–24.89 | 41.34 35.70–46.92 |
| 001 | 27.24 9.07–46.10 | 69.71 62.31–76.54 | 54.15 47.30–60.71 | 16.86 9.98–26.52 | 41.99 36.29–47.84 |
| 003 | 28.23 7.29–49.68 | 76.91 64.40–87.98 | 57.29 54.67–59.50 | 19.60 13.47–27.30 | 45.51 38.90–51.81 |
| 046 | 27.02 8.50–47.30 | 66.34 55.83–76.16 | 50.76 43.61–58.19 | 18.35 13.00–24.17 | 40.62 34.54–46.69 |
Table 4: Performance by scientific domain. Mean normalized score (0–100), with pointwise 95% confidence intervals below. Rows identify simulator personas evaluated on the same tasks. Domains: life sciences; physical sciences and engineering; mathematics and computer science; social sciences and humanities. Avg. weights domains equally.
Figure 3: Semantic separation and example trajectories. A, Excess cosine distance: simulator below, actor above the excluded diagonal. Stars: Holm-adjusted across 20 comparisons. B, BixBench (left) and SciConvBench (right); simulator (top), actor (bottom). Hollow/filled circles mark starts/ends; dots mark intermediate messages; overlapping endpoints indicate one message. UMAP coordinates are local to each subplot; curves only connect observations.
Figure 4: Behavioral differences (row minus column). A/B, Simulator/actor tokens (thousands/episode). C, SciConvBench clarification recall (percentage points). D, Actor tool calls/episode. Stars: A/B, Holm-adjusted ; C/D, absolute differences exceeding 2 percentage points/0.2 calls.
Figure 5: Interaction metrics by persona. A, Actor self-repetition with pointwise 95% study-cluster confidence intervals. B, Complete-fact recovery in failed SciConvBench disambiguation episodes.
Figure 6: Workflow revision with researcher feedback. Highlights added; […] marks omissions. Event identifiers are preserved.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Criterion | Required evidence |
|---|---|
| Scientific identity | Native task and study identifiers, objective, fixed constraints, deliverables, and a record of intentional changes. |
| Useful interaction | A legitimate context, decision, or review role for the researcher; no invented facts or quota of extra interactions. |
| Information separation | Allowlisted actor/researcher exports and runtime access checks that exclude hidden answers, tests, and solution artifacts. |
| Grounding and persona | Fact provenance, explicit unknowns and advice, versioned persona rendering, and declared knowledge/disclosure policies. |
| Runtime readiness | Resolved input roles and assets, hashes and mount mappings, actual resources, permitted tools, and required output formats. |
| Evaluation compatibility | Native submission mapping, evaluator version, declared design/execution scope, scoring changes, and error behavior. |
Table 5: Acceptance rubric for benchmark conversion. Each criterion requires inspectable evidence; unresolved checks remain explicit rather than being inferred from successful serialization.
Figure 7: Incremental clarification with explanatory replies. Highlights added; […] marks omissions. Event identifiers are preserved.