We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.
Figures & tables
Figure 1: The task, shown on a development task (no held-out answer is disclosed); red and blue mark the hot front and cooler rear, and magenta what is hidden from the agent. (a) A plate of unknown material is heated on its front face and loses heat to the air at its rear. The agent sees one experiment, may design four more (heater power and shape, duration, thermometer positions), and may run and fit a simulator of a one-piece plate at any resolution. It reports the conductivity k (how easily heat flows through the material), the heat capacity ρc (how much heat it stores) and the heat-loss coefficient h (how fast it loses heat to the air), each with a 95% range, scored against the exact truth. Hidden from the agent, this sample has a glued joint that the simulator lacks. (b) The simulator’s best fit looks near-perfect, yet its conductivity is 12% low; the misfit (bottom) is patterned, up to about 25 times the noise: the clue that physics is missing. (c) The evaluation pipeline. (d) The hidden flaw in each task family (described in Table 2 ); “basic version” in Figure 3 is missing physics with nothing else wrong.
Figure 2: How often the AI’s stated 95% ranges contained the true property value, on held-out tasks (3 properties per run; 3 runs per task at default, 1–3 at the highest setting; unsubmitted runs count as misses). An honest AI scores about 95%; far below means confidently wrong, and above 95% is cautious rather than better. When the simulator is missing real physics, only GPT-6 Astra and Claude Fable 5.2 are reliable at default settings; the Claude 5.5 models recover only at their highest reasoning setting, and older and open-weight models not at all.
Tool
Purpose
Limit
request_measurement
New noisy data from the hidden sample for an agent-designed experiment
4 per task
run_simulation
Run the simulator (or a candidate simulator) for chosen parameters, design and solver settings
compute budget
fit_parameters
Least-squares fit; returns estimates, linearized uncertainty, misfit per sensor
Parameters with 95% intervals, hidden-target prediction, conclusions
once
Table 1: Agent tools and limits.
Family
What is wrong
Right behaviour
T1
Data cannot determine all parameters
Honestly wide intervals, or an experiment that separates them
T2
Missing physics ( k(T) , joint, radiation, side loss)
Detect inadequacy; intervals that reflect it
T3
Default solver under-resolved
Converge the solver before attributing misfit to physics
T10
As T2, physical-property wording, extrapolated target
Calibrated physical properties despite a rejected simulator
T5
Under-resolved solver and missing physics
Fix numerics, then still handle the physics
T6
Candidate simulators offered; one, or none, is right
Pick the right candidate; recognize when none fits
Table 2: Task families. Physics-wrong tasks (T2, T10, the physics traps of T5, and the T6 side-loss tasks) are those in which the simulator lacks real physics and no correct alternative is offered.
Figure 3: Coverage by task family at default settings (held-out trap tasks; 3 runs per task). Left: the physics is right, but the data are weak, the solver is too coarse, or the right simulator must be picked from a menu; current models handle these. Right (shaded): the simulator is missing real physics, alone, with the task asking for true material properties, with a coarse solver, or in a menu with no right candidate (3 tasks); only Astra and Fable 5.2 stay calibrated.
Rank
Model (default setting)
Score at default
Coverage at default
Best tested score (setting)
1
Claude Fable 5.2 (adaptive)
0.020
99%
same as default
2
GPT-6 Astra (provider default)
0.037
94%
0.018 (max)
3=
Claude Sonnet 5.5 (high)
0.75
56%
0.097 (max)
3=
Claude Opus 5.5 (medium)
0.77
62%
0.028 (xhigh)
5=
Claude Haiku 5.5 (medium)
1.35
49%
0.12 (max)
5=
Claude Sonnet 4.5 (no thinking)
1.39
39%
1.33 (thinking)
Table 3: Main results. Score: geometric-mean interval score on the 38 physics-wrong tasks (lower is better); coverage counts unsubmitted runs as misses; “=” marks models that paired tests cannot separate. Default settings differ by provider (in brackets); every setting tested, with tokens and cost, is in Table 4 .
Figure 4: (a) A development task whose conductivity rises with temperature, while the simulator assumes it constant: each point is the conductivity the simulator infers from one experiment, against how hard that experiment heated the sample. Heated hard, the simulator reports the conductivity at a higher temperature than the room-temperature value the task asks for. Opus ran only such experiments and reported a compromise that misses; Astra added gentle experiments and reports the right value. Average temperature rise: how much an experiment heated the sample above room temperature; because conductivity here rises with temperature, a hard-heating experiment measures a different value from the one asked for. (b) On physics-wrong tasks, every model except Astra and Fable 5.2 has errors larger than its stated ranges; theirs are about 70–100 times smaller, with ranges no wider.
Figure 5: What agents did on physics-wrong tasks at default settings (share of episodes). Left: naming the missing effect (both LLM judges agree), which the Claude 5.5 models do far less often than Astra at default. Right (shaded): the two behaviours that correct the measurement, a low-power experiment and an answer based on a single dataset, which Astra and Fable 5.2 do far more often than the others.
Figure 6: What a one-paragraph hint changes on physics-wrong tasks (3 runs per task). B: general scientific practice; H: “your ranges assume the simulator is right, so widen them”; S: Astra’s strategy stated outright. (a) Coverage; an honest model scores about 95%, as Astra does without a hint. (b) Typical distance of the best guess from the truth (bar) and how far the stated range reaches from the guess (mark); a bar past its mark means the ranges typically miss. H changes nothing; S halves the error, but the ranges shrink with it, so coverage stays far below Astra’s. Paired intervals: Appendix Table 12 . † Written after inspecting Astra on this split; replicated on fresh tasks (Appendix Table 14 ).
Model
Reasoning setting
Runs/task
Interval score vs default [95% CI]
Coverage
Submitted
Tokens/episode
USD/episode
GPT-6 Astra
default
3
–
0.94
1.00
3.3k
1.04
max
1
2.1 × better [1.5–3.0 × ]
1.00
1.00
19.2k
3.10
Claude Fable 5.2
low ‡
1
12 × worse [8.0–18 × ]
0.80
1.00
2.8k
0.24
medium ‡
1
5.5 × worse [3.9–8.0 × ]
0.85
1.00
4.1k
0.35
default
3
–
0.99
1.00
20.8k
1.56
max ‡
1
1.4 × worse [1.1–1.8 × ]
0.98
1.00
98.3k
5.55
Table 4: Reasoning settings on the 38 physics-wrong tasks of the 66-task set. Interval score: how many times better or worse than the same model’s three-run default, paired over tasks, with 95% intervals (bold: the pre-registered primary test); since lower scores are better, “10 × better” means a score one-tenth as large. Submitted is over all 66 tasks; tokens are median output tokens per episode. ‡ Added after the primary results. Behaviour by setting: Appendix Table 15 .
Figure 7: Physics-wrong tasks. (a) The Claude 5.5 models climb steeply as they reason more; Sonnet 4.5, Gemma and Muse do not, and Qwen3.8 reasons at length yet stays low. Astra is high at default with few tokens; Fable 5.2 at default matches Astra at max with similar tokens, and falls short at Astra’s default token count. (b) Fable’s default costs half as much as Astra at max ; Haiku 5.5 at max reaches 0.90 for under ten cents. Opus at max includes its budget-limited episodes rerun at twice the budget (0.93 without; § 7 ).
Agent control
Scientific setting
Evaluation
Interactive
Designs expts
Physics sim.
Flawed sim.
Calibrated intervals
Process graded
Judge-free
ScienceAgentBench [ 3 ]
–
–
–
–
–
–
∼
CORE-Bench [ 4 ]
✓
–
–
–
– a
–
✓
MLE-bench [ 5 ]
✓
∼
–
–
–
–
✓ b
RE-Bench [ 6 ]
✓
∼
–
–
–
–
✓
DiscoveryWorld [ 8 ]
✓
✓
∼
–
–
✓
∼
Table 5: Closest benchmarks. ✓: yes; ∼ : partly; –: no; ?: not established from the paper. “Flawed sim.”: the agent’s simulator is deliberately wrong or under-resolved. “Calibrated intervals”: agent-reported uncertainty is scored. “Judge-free”: no language-model judge in scoring.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Access
Identifier (revision)
Runs (UTC)
Episodes
GPT-6 Astra
Amazon Bedrock
us.openai.gpt-6-astra
2026-10-21 to 2026-10-29
345
Claude Fable 5.2
Amazon Bedrock
us.anthropic.claude-fable-5-2
2026-11-10 to 2026-11-11
477
Claude Opus 5.5
Amazon Bedrock
us.anthropic.claude-opus-5-5
2026-10-21 to 2026-11-03
1,746
Claude Sonnet 5.5
Amazon Bedrock
us.anthropic.claude-sonnet-5-5
2026-10-21 to 2026-10-29
1,315
Claude Haiku 5.5
Amazon Bedrock
us.anthropic.claude-haiku-5-5
2026-10-21 to 2026-10-30
1,449
Claude Sonnet 4.5
Amazon Bedrock
us.anthropic.claude-sonnet-4-5-20250929-v1:0
2026-10-21 to 2026-10-31
345
Appendix
Table 6: Models evaluated. API models were called through Amazon Bedrock (us-east-1); open-weight models were served with SGLang v0.5.21 on one AWS g7e.24xlarge instance (four NVIDIA RTX PRO 6000 GPUs). Episodes include all conditions and effort settings.
Model
T1
T2
T3
T10
T5
T6 menu
T6 none
Ctrl
GPT-6 Astra
0.97
0.84
1.00
0.97
0.99
1.00
0.96
0.99
Claude Fable 5.2
0.98
0.98
1.00
1.00
1.00
1.00
0.93
1.00
Claude Opus 5.5
1.00
0.58
1.00
0.66
0.72
1.00
0.30
1.00
Claude Sonnet 5.5
1.00
0.62
1.00
0.56
0.62
1.00
0.15
1.00
Claude Haiku 5.5
0.95
0.53
1.00
0.48
0.54
1.00
0.22
0.99
Claude Sonnet 4.5
0.83
0.35
0.37
0.41
0.46
0.96
0.26
0.95
Appendix
Table 7: Coverage on trap tasks by family, and on controls (held-out split, 3 seeds). Interval scores are in Appendix Table 8 .
Model
T1
T2
T3
T10
T5
T6 menu
T6 none
Ctrl
GPT-6 Astra
0.006
0.033
0.001
0.027
0.020
0.001
0.061
0.001
Claude Fable 5.2
0.006
0.016
0.002
0.024
0.019
0.002
0.029
0.001
Claude Opus 5.5
0.022
1.532
0.012
1.086
0.247
0.007
2.088
0.010
Claude Sonnet 5.5
0.027
0.279
0.013
2.177
0.473
0.015
2.083
0.008
Claude Haiku 5.5
0.032
1.593
0.009
2.487
2.068
0.006
2.517
0.006
Claude Sonnet 4.5
0.027
1.683
0.110
2.116
1.082
0.002
2.435
0.002
Appendix
Table 8: Median parameter interval score on trap tasks by family and on controls (lower is better).
Comparison (model vs next in the ranking)
Tasks
Interval score [95% CI]
Claude Fable 5.2 vs GPT-6 Astra
38
1.9 × better [1.3–2.8 × ]
GPT-6 Astra vs Claude Sonnet 5.5
38
20 × better [13–32 × ]
Claude Sonnet 5.5 vs Claude Opus 5.5
38
no detectable change [0.76–1.40 × ]
Claude Opus 5.5 vs Claude Haiku 5.5
38
1.8 × better [1.4–2.2 × ]
Claude Haiku 5.5 vs Claude Sonnet 4.5
38
no detectable change [0.90–1.18 × ]
Claude Sonnet 4.5 vs Qwen3.8-27B
38
no detectable change [0.88–1.22 × ]
Appendix
Table 9: Paired comparisons on physics-wrong tasks: each model against the next in the ranking, as how many times better its interval score is, with 95% bootstrap intervals over tasks.
Model
Episodes
Submitted
Seed s.d.
Cost (USD)
GPT-6 Astra
279
279
0.21
267.26
Claude Fable 5.2
279
279
0.12
388.02
Claude Opus 5.5
279
279
0.16
36.77
Claude Sonnet 5.5
279
279
0.17
16.65
Claude Haiku 5.5
279
279
0.09
1.98
Claude Sonnet 4.5
279
279
0.08
83.94
Appendix
Table 10: Completion and cost on the held-out split (condition A). Seed s.d.: mean over tasks of the standard deviation of log10 interval score across seeds, physics-wrong tasks.
Model
Cov.
Est. err.
Half-width
Names effect
Low power
One dataset
Final misfit
Expts
GPT-6 Astra
0.94
0.001
0.013
0.66
0.70
0.96
1.4
4.0
Claude Fable 5.2
0.99
0.001
0.010
0.89
0.50
0.94
1.5
4.0
Claude Opus 5.5
0.62
0.070
0.035
0.35
0.08
0.39
21.2
2.1
Claude Sonnet 5.5
0.56
0.068
0.022
0.25
0.42
0.56
16.3
2.6
Claude Haiku 5.5
0.49
0.074
0.024
0.14
0.07
0.19
25.8
2.6
Claude Sonnet 4.5
0.39
0.069
0.014
0.03
0.17
0.04
27.6
3.4
Appendix
Table 11: Behaviour on physics-wrong tasks (condition A). Est. err. and half-width are medians in log10 ; “low power”: an experiment at ≤0.5× the initial power; final misfit: rms/noise of the fit the answer came from.
Model
Hint
Coverage
Change, points [95% CI]
IS
IS vs no hint [95% CI]
Control IS
Claude Opus 5.5
A
0.62
1.05
0.010
B
0.67
+5[+1,+8]
0.52
1.4 × better [1.2–1.8 × ]
0.008
H
0.63
+0[−2,+3]
1.07
no detectable change [0.96–1.29 × ]
0.008
S †
0.66
+4[−2,+10]
0.45
2.5 × better [1.6–3.8 × ]
0.008
Claude Sonnet 5.5
A
0.56
0.96
0.008
B
0.55
−2[−6,+3]
1.28
no detectable change [0.71–1.17 × ]
0.007
Appendix
Table 12: Hint ablation on physics-wrong tasks, paired over tasks against no hint, with 95% bootstrap intervals. Coverage change in percentage points; IS: median interval score. † Written after inspecting test-split traces; replicated on fresh tasks (Table 14 ).
Model
Hint
Glued joint
Radiation
k(T)
Edge loss
GPT-6 Astra
A
0.99 (0.001)
0.93 (0.001)
1.00 (0.000)
0.85 (0.003)
Claude Fable 5.2
A
1.00 (0.001)
0.99 (0.001)
1.00 (0.001)
0.97 (0.002)
Claude Opus 5.5
A
0.77 (0.131)
0.70 (0.046)
0.73 (0.027)
0.35 (0.083)
Claude Opus 5.5
S
0.80 (0.094)
0.63 (0.016)
0.98 (0.003)
0.32 (0.083)
Claude Sonnet 5.5
A
0.70 (0.131)
0.52 (0.042)
0.81 (0.019)
0.28 (0.082)
Claude Sonnet 5.5
S
0.72 (0.136)
0.43 (0.017)
0.98 (0.003)
0.22 (0.083)
Appendix
Table 13: Coverage (median estimate error in log10 ) by hidden mechanism on physics-wrong tasks, without and with hint S.
Model
Tasks
Interval score with S: times better [95% CI]
Coverage change
Low-power experiment
Answer from one dataset
Typical estimate error
(points)
(no hint → S)
(no hint → S)
(no hint → S)
Claude Opus 5.5
34
2.2 × [1.5, 3.3]
+1
6% → 93%
39% → 89%
13% → 6%
Claude Sonnet 5.5
34
1.7 × [1.2, 2.4]
+0
41% → 99%
53% → 85%
13% → 7%
Claude Haiku 5.5
34
1.7 × [1.3, 2.2]
+4
6% → 82%
20% → 49%
14% → 10%
Appendix
Table 14: Pre-registered re-test of hint S on new tasks nobody had seen (34 physics-wrong tasks of a fresh split; 3 runs per task; compared task by task without and with S). Interval score: how many times better with S (above 1 is better; the pre-registered test required the whole interval above 1, which holds for all three models). Coverage change in percentage points. The last three columns show the behaviour change: models follow the stated strategy, and their typical estimate error roughly halves, yet coverage barely moves.
Model
Reasoning setting
Names effect
One dataset
Low power
Final misfit
GPT-6 Astra
default
0.66
0.96
0.70
1.4
max
0.71
0.97
0.71
1.5
Claude Fable 5.2
low ‡
0.45
0.61
0.47
10.6
medium ‡
0.61
0.79
0.53
5.5
default
0.89
0.94
0.50
1.5
max ‡
0.92
1.00
0.29
2.7
Appendix
Table 15: Behaviour on physics-wrong tasks at each reasoning setting (shares of episodes; final misfit: median rms/noise of the fit the answer came from), as in Appendix Table 11 .
Check
Exact reference
Tolerance
Thick plate, constant heating
Semi-infinite front-face rise, (2q/e)t/π
0.5%
Steady state, one layer
Series resistance, q(L/k+1/h)
0.01%
Steady state, two layers and a joint
Series resistance; jump at the joint =qRc
0.01%
Flash heating, insulated rear
Rear-face curve [ 10 ]
0.2% of final rise
Half-rise time 0.1388L2/α
1%
Temperature-dependent conductivity
Steady state by the Kirchhoff transform
0.01%
Appendix
Table 16: Verification of the simulators against exact results (automated tests run with every code change; tests/test_solver_verification.py , tests/test_axisym.py ). Tolerances are relative unless a unit is given.
Model
Setting
Episodes
Keyword match
Both judges
GPT-6 Astra
default
114
58%
66%
max
38
68%
71%
Claude Fable 5.2
default
114
98%
89%
low
38
71%
45%
max
38
97%
92%
medium
38
87%
61%
Appendix
Table 17: Share of physics-wrong episodes whose conclusions name the hidden effect, by the earlier keyword match and by the consensus of the two language-model judges, for every model and reasoning setting.
Suite
Check (scripted policies)
Values
v0
param_interval_score: oracle < heuristic < naive
0.009 / 0.018 / 1.54
target_interval_score: oracle < heuristic < naive
1 / 6 / 206
adequacy_correct: oracle ≥ heuristic > naive
1 / 1 / 0.745
attribution_correct: oracle ≥ heuristic > naive
1 / 1 / 0.553
T1 stop_correct: oracle ≥ heuristic > naive
1 / 0.944 / 0.278
T1 experiment_selection: heuristic > random
1 / 0.11
Appendix
Table 18: Scripted-baseline gate on the development split (all checks passed). Values list the policies in the order named in each check; IS: interval score.
Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct SciΠvot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while SciΠvot supports systematic evaluation of scientific judgment for LLM agents.
Youran Sun, Xingyu Ren, Chugang Yi +15
University of Maryland, College Park · The Chinese University of Hong Kong · Gradient Network +3
Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference solutions, or simulators with a known generative structure. Real scientific requests arrive differently. They are underspecified, they carry attachments, and they lack ground truth. We report K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs. Three blinded language-model judges scored every run against an eight-dimension rubric. On a rubric whose 8-anchor is defined as work a domain scientist would accept with minor edits, no model clears the line under all three judges. gpt-5.6-sol has the highest pooled mean, 8.04, but its 95% interval [7.80, 8.23] spans the threshold, and two of the three judges rank claude-opus-5 first instead. We therefore report the ordering of systems as the reproducible quantity, the absolute level as an attribute of the instrument, and the top of the table as unresolved. Across all 39,934 scored judgments -- the eight dimension scores plus a holistic overall for each assessment, excluding not-applicable cells -- 47.6% fall below the 8-point threshold. Difficulty is not uniform across the rubric: scientific accuracy averages 6.22 against 7.33 for communication, on identical denominators and in the same direction within every one of the nine models. The single leading failure tag is overclaiming, on 31.4% of assessments. We argue that the informative quantity for scientific agents is not a leaderboard position but the joint distribution of what was delivered, what was claimed, and what artifacts were produced.
Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface. SciGame contains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.