SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents
Organizations: Lawrence Livermore National Laboratory, Livermore, CA 94550, USA
Abstract
We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.
Figures & tables
| Tool | Purpose | Limit |
|---|---|---|
| request_measurement | New noisy data from the hidden sample for an agent-designed experiment | 4 per task |
| run_simulation | Run the simulator (or a candidate simulator) for chosen parameters, design and solver settings | compute budget |
| fit_parameters | Least-squares fit; returns estimates, linearized uncertainty, misfit per sensor | compute budget |
| analyze | Residual runs test; early/late refit; amplitude-scaling comparison | compute budget |
| get_readings , get_residuals | Fetch raw points of data or misfit | free |
| submit | Parameters with 95% intervals, hidden-target prediction, conclusions | once |
| Family | What is wrong | Right behaviour |
|---|---|---|
| T1 | Data cannot determine all parameters | Honestly wide intervals, or an experiment that separates them |
| T2 | Missing physics ( , joint, radiation, side loss) | Detect inadequacy; intervals that reflect it |
| T3 | Default solver under-resolved | Converge the solver before attributing misfit to physics |
| T10 | As T2, physical-property wording, extrapolated target | Calibrated physical properties despite a rejected simulator |
| T5 | Under-resolved solver and missing physics | Fix numerics, then still handle the physics |
| T6 | Candidate simulators offered; one, or none, is right | Pick the right candidate; recognize when none fits |
| Rank | Model (default setting) | Score at default | Coverage at default | Best tested score (setting) |
|---|---|---|---|---|
| 1 | Claude Fable 5.2 (adaptive) | 0.020 | 99% | same as default |
| 2 | GPT-6 Astra (provider default) | 0.037 | 94% | 0.018 (max) |
| 3= | Claude Sonnet 5.5 (high) | 0.75 | 56% | 0.097 (max) |
| 3= | Claude Opus 5.5 (medium) | 0.77 | 62% | 0.028 (xhigh) |
| 5= | Claude Haiku 5.5 (medium) | 1.35 | 49% | 0.12 (max) |
| 5= | Claude Sonnet 4.5 (no thinking) | 1.39 | 39% | 1.33 (thinking) |
| Model | Reasoning setting | Runs/task | Interval score vs default [95% CI] | Coverage | Submitted | Tokens/episode | USD/episode |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | default | 3 | – | 0.94 | 1.00 | 3.3k | 1.04 |
| max | 1 | 2.1 better [1.5–3.0 ] | 1.00 | 1.00 | 19.2k | 3.10 | |
| Claude Fable 5.2 | low ‡ | 1 | 12 worse [8.0–18 ] | 0.80 | 1.00 | 2.8k | 0.24 |
| medium ‡ | 1 | 5.5 worse [3.9–8.0 ] | 0.85 | 1.00 | 4.1k | 0.35 | |
| default | 3 | – | 0.99 | 1.00 | 20.8k | 1.56 | |
| max ‡ | 1 | 1.4 worse [1.1–1.8 ] | 0.98 | 1.00 | 98.3k | 5.55 |
| Agent control | Scientific setting | Evaluation | |||||
| Interactive | Designs expts | Physics sim. | Flawed sim. | Calibrated intervals | Process graded | Judge-free | |
| ScienceAgentBench [ 3 ] | – | – | – | – | – | – | |
| CORE-Bench [ 4 ] | ✓ | – | – | – | – a | – | ✓ |
| MLE-bench [ 5 ] | ✓ | – | – | – | – | ✓ b | |
| RE-Bench [ 6 ] | ✓ | – | – | – | – | ✓ | |
| DiscoveryWorld [ 8 ] | ✓ | ✓ | – | – | ✓ | ||
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Access | Identifier (revision) | Runs (UTC) | Episodes |
|---|---|---|---|---|
| GPT-6 Astra | Amazon Bedrock | us.openai.gpt-6-astra | 2026-10-21 to 2026-10-29 | 345 |
| Claude Fable 5.2 | Amazon Bedrock | us.anthropic.claude-fable-5-2 | 2026-11-10 to 2026-11-11 | 477 |
| Claude Opus 5.5 | Amazon Bedrock | us.anthropic.claude-opus-5-5 | 2026-10-21 to 2026-11-03 | 1,746 |
| Claude Sonnet 5.5 | Amazon Bedrock | us.anthropic.claude-sonnet-5-5 | 2026-10-21 to 2026-10-29 | 1,315 |
| Claude Haiku 5.5 | Amazon Bedrock | us.anthropic.claude-haiku-5-5 | 2026-10-21 to 2026-10-30 | 1,449 |
| Claude Sonnet 4.5 | Amazon Bedrock | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | 2026-10-21 to 2026-10-31 | 345 |
| Model | T1 | T2 | T3 | T10 | T5 | T6 | T6 | Ctrl |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 0.97 | 0.84 | 1.00 | 0.97 | 0.99 | 1.00 | 0.96 | 0.99 |
| Claude Fable 5.2 | 0.98 | 0.98 | 1.00 | 1.00 | 1.00 | 1.00 | 0.93 | 1.00 |
| Claude Opus 5.5 | 1.00 | 0.58 | 1.00 | 0.66 | 0.72 | 1.00 | 0.30 | 1.00 |
| Claude Sonnet 5.5 | 1.00 | 0.62 | 1.00 | 0.56 | 0.62 | 1.00 | 0.15 | 1.00 |
| Claude Haiku 5.5 | 0.95 | 0.53 | 1.00 | 0.48 | 0.54 | 1.00 | 0.22 | 0.99 |
| Claude Sonnet 4.5 | 0.83 | 0.35 | 0.37 | 0.41 | 0.46 | 0.96 | 0.26 | 0.95 |
| Model | T1 | T2 | T3 | T10 | T5 | T6 | T6 | Ctrl |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 0.006 | 0.033 | 0.001 | 0.027 | 0.020 | 0.001 | 0.061 | 0.001 |
| Claude Fable 5.2 | 0.006 | 0.016 | 0.002 | 0.024 | 0.019 | 0.002 | 0.029 | 0.001 |
| Claude Opus 5.5 | 0.022 | 1.532 | 0.012 | 1.086 | 0.247 | 0.007 | 2.088 | 0.010 |
| Claude Sonnet 5.5 | 0.027 | 0.279 | 0.013 | 2.177 | 0.473 | 0.015 | 2.083 | 0.008 |
| Claude Haiku 5.5 | 0.032 | 1.593 | 0.009 | 2.487 | 2.068 | 0.006 | 2.517 | 0.006 |
| Claude Sonnet 4.5 | 0.027 | 1.683 | 0.110 | 2.116 | 1.082 | 0.002 | 2.435 | 0.002 |
| Comparison (model vs next in the ranking) | Tasks | Interval score [95% CI] |
|---|---|---|
| Claude Fable 5.2 vs GPT-6 Astra | 38 | 1.9 better [1.3–2.8 ] |
| GPT-6 Astra vs Claude Sonnet 5.5 | 38 | 20 better [13–32 ] |
| Claude Sonnet 5.5 vs Claude Opus 5.5 | 38 | no detectable change [0.76–1.40 ] |
| Claude Opus 5.5 vs Claude Haiku 5.5 | 38 | 1.8 better [1.4–2.2 ] |
| Claude Haiku 5.5 vs Claude Sonnet 4.5 | 38 | no detectable change [0.90–1.18 ] |
| Claude Sonnet 4.5 vs Qwen3.8-27B | 38 | no detectable change [0.88–1.22 ] |
| Model | Episodes | Submitted | Seed s.d. | Cost (USD) |
|---|---|---|---|---|
| GPT-6 Astra | 279 | 279 | 0.21 | 267.26 |
| Claude Fable 5.2 | 279 | 279 | 0.12 | 388.02 |
| Claude Opus 5.5 | 279 | 279 | 0.16 | 36.77 |
| Claude Sonnet 5.5 | 279 | 279 | 0.17 | 16.65 |
| Claude Haiku 5.5 | 279 | 279 | 0.09 | 1.98 |
| Claude Sonnet 4.5 | 279 | 279 | 0.08 | 83.94 |
| Model | Cov. | Est. err. | Half-width | Names effect | Low power | One dataset | Final misfit | Expts |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 0.94 | 0.001 | 0.013 | 0.66 | 0.70 | 0.96 | 1.4 | 4.0 |
| Claude Fable 5.2 | 0.99 | 0.001 | 0.010 | 0.89 | 0.50 | 0.94 | 1.5 | 4.0 |
| Claude Opus 5.5 | 0.62 | 0.070 | 0.035 | 0.35 | 0.08 | 0.39 | 21.2 | 2.1 |
| Claude Sonnet 5.5 | 0.56 | 0.068 | 0.022 | 0.25 | 0.42 | 0.56 | 16.3 | 2.6 |
| Claude Haiku 5.5 | 0.49 | 0.074 | 0.024 | 0.14 | 0.07 | 0.19 | 25.8 | 2.6 |
| Claude Sonnet 4.5 | 0.39 | 0.069 | 0.014 | 0.03 | 0.17 | 0.04 | 27.6 | 3.4 |
| Model | Hint | Coverage | Change, points [95% CI] | IS | IS vs no hint [95% CI] | Control IS |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | A | 0.62 | 1.05 | 0.010 | ||
| B | 0.67 | 0.52 | 1.4 better [1.2–1.8 ] | 0.008 | ||
| H | 0.63 | 1.07 | no detectable change [0.96–1.29 ] | 0.008 | ||
| S † | 0.66 | 0.45 | 2.5 better [1.6–3.8 ] | 0.008 | ||
| Claude Sonnet 5.5 | A | 0.56 | 0.96 | 0.008 | ||
| B | 0.55 | 1.28 | no detectable change [0.71–1.17 ] | 0.007 |
| Model | Hint | Glued joint | Radiation | Edge loss | |
|---|---|---|---|---|---|
| GPT-6 Astra | A | 0.99 (0.001) | 0.93 (0.001) | 1.00 (0.000) | 0.85 (0.003) |
| Claude Fable 5.2 | A | 1.00 (0.001) | 0.99 (0.001) | 1.00 (0.001) | 0.97 (0.002) |
| Claude Opus 5.5 | A | 0.77 (0.131) | 0.70 (0.046) | 0.73 (0.027) | 0.35 (0.083) |
| Claude Opus 5.5 | S | 0.80 (0.094) | 0.63 (0.016) | 0.98 (0.003) | 0.32 (0.083) |
| Claude Sonnet 5.5 | A | 0.70 (0.131) | 0.52 (0.042) | 0.81 (0.019) | 0.28 (0.082) |
| Claude Sonnet 5.5 | S | 0.72 (0.136) | 0.43 (0.017) | 0.98 (0.003) | 0.22 (0.083) |
| Model | Tasks | Interval score with S: times better [95% CI] | Coverage change | Low-power experiment | Answer from one dataset | Typical estimate error |
|---|---|---|---|---|---|---|
| (points) | (no hint S) | (no hint S) | (no hint S) | |||
| Claude Opus 5.5 | 34 | 2.2 [1.5, 3.3] | +1 | 6% 93% | 39% 89% | 13% 6% |
| Claude Sonnet 5.5 | 34 | 1.7 [1.2, 2.4] | +0 | 41% 99% | 53% 85% | 13% 7% |
| Claude Haiku 5.5 | 34 | 1.7 [1.3, 2.2] | +4 | 6% 82% | 20% 49% | 14% 10% |
| Model | Reasoning setting | Names effect | One dataset | Low power | Final misfit |
|---|---|---|---|---|---|
| GPT-6 Astra | default | 0.66 | 0.96 | 0.70 | 1.4 |
| max | 0.71 | 0.97 | 0.71 | 1.5 | |
| Claude Fable 5.2 | low ‡ | 0.45 | 0.61 | 0.47 | 10.6 |
| medium ‡ | 0.61 | 0.79 | 0.53 | 5.5 | |
| default | 0.89 | 0.94 | 0.50 | 1.5 | |
| max ‡ | 0.92 | 1.00 | 0.29 | 2.7 |
| Check | Exact reference | Tolerance |
| Thick plate, constant heating | Semi-infinite front-face rise, | 0.5% |
| Steady state, one layer | Series resistance, | 0.01% |
| Steady state, two layers and a joint | Series resistance; jump at the joint | 0.01% |
| Flash heating, insulated rear | Rear-face curve [ 10 ] | 0.2% of final rise |
| Half-rise time | 1% | |
| Temperature-dependent conductivity | Steady state by the Kirchhoff transform | 0.01% |
| Model | Setting | Episodes | Keyword match | Both judges |
|---|---|---|---|---|
| GPT-6 Astra | default | 114 | 58% | 66% |
| max | 38 | 68% | 71% | |
| Claude Fable 5.2 | default | 114 | 98% | 89% |
| low | 38 | 71% | 45% | |
| max | 38 | 97% | 92% | |
| medium | 38 | 87% | 61% |
| Suite | Check (scripted policies) | Values |
| v0 | param_interval_score: oracle heuristic naive | 0.009 / 0.018 / 1.54 |
| target_interval_score: oracle heuristic naive | 1 / 6 / 206 | |
| adequacy_correct: oracle heuristic naive | 1 / 1 / 0.745 | |
| attribution_correct: oracle heuristic naive | 1 / 1 / 0.553 | |
| T1 stop_correct: oracle heuristic naive | 1 / 0.944 / 0.278 | |
| T1 experiment_selection: heuristic random | 1 / 0.11 |