Unsatz: Evaluating Scientific Discovery Without a Given Ansatz
Organizations: Department of Computer Science, University of California, Los Angeles
Abstract
Before a scientist fits any parameter, they must decide what the model contains: which quantities exist, which are hidden, how they interact, and how they are measured. Physicists call such a trial form an ansatz. Benchmarks for automated equation discovery mostly supply it, by fixing the variables, a library of candidate terms, or a law to be modified. We introduce Unsatz, a benchmark in which an LLM agent must construct the ansatz itself. Each task is a simulated system, generated from a private seed as a program in a small modeling language: ordinary, delay, or partial differential equations, discrete maps, static and causal models, or stochastic processes, possibly with hidden states, distorted sensors, and uninformative measurement channels. The agent buys experiments within a budget, writes candidate programs in the same language, obtains their coefficients from the benchmark's fitter, and is scored only on how its final program predicts experiments it never saw: conditions outside the range it could explore, interventions, and longer runs. Across 80 systems in 20 domains, nine language models solve between 21% and 60%. On 44 of these systems, classical equation-discovery methods almost never succeed, although the same data usually suffice once the true equations are given with unknown coefficients; the difficulty lies in finding the structure, not in estimating it. Because mechanisms recur across systems, we also give agents the program they discovered for a related system; it helped only when the new system shared the mechanism and the first investigation had succeeded. Stronger agents make fewer unscientific errors; beyond these, agents find which variables a system has but not how they act, and prefer revisions that lower the training error to revisions that predict better.
Figures & tables
| Benchmark | Data | Hidden systems | Variables | Scored on |
|---|---|---|---|---|
| SRBench | Fixed | Known equations, real datasets | Given | Symbolic match, test fit |
| LLM-SRBench | Fixed | Rewritten and synthetic equations | Given | LLM-judged match, out-of-domain error |
| BoxingGym | Chosen, 10 | 10 probabilistic models | Given | Prediction error, explanation |
| DiscoveryWorld | Actions, 1,000 | Variants of 24 task templates | Given | Task completion, LLM-graded report |
| NewtonBench | Chosen, 200 | Modified canonical laws | Given | LLM-judged match, data error |
| PhysGym | Chosen, 100 | PHYBench physics problems | Given or masked | Symbolic match |
| Domain | Class | What must be inferred | Domain | Class | What must be inferred |
|---|---|---|---|---|---|
| Deterministic dynamics | Static and causal systems | ||||
| Mechanics † | ODE | Forces, driving, and couplings | Causal † | Causal | Dependencies under intervention |
| Kinetics † | ODE | Reactions with hidden species | Scaling † | Static | Relations via intermediate quantities |
| Discrete † | Map | Nonlinear update rules | Metrology † | Static | A latent response, two sensors |
| Ecology † | ODE | Two hidden resource stores | Spectroscopy † | Static | Peaks under four settings |
| Network † , S1 † , S2 | ODE | Directed couplings of nodes | Stochastic systems | ||
| Solved by kind of system | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Solved | 95% CI | Dyn. (36) | Spatial (12) | Static (16) | Stoch. (16) | Median | Invalid | Truncated |
| GPT 6 Astra | 48/80 | 49–70% | 18 | 12 | 9 | 9 | 0.045 | 2 | 0 |
| Claude Fable 5.1 | 38/80 | 37–58% | 10 | 12 | 9 | 7 | 0.106 | 3 | 1 |
| Muse Spark 1.3 | 24/80 | 21–41% | 4 | 12 | 7 | 1 | 0.202 | 5 | 7 |
| Grok 4.6 | 23/80 | 20–39% | 5 | 9 | 6 | 3 | 0.301 | 3 | 3 |
| Qwen 3.8 Max | 23/80 | 20–39% | 8 | 10 | 3 | 2 | 0.276 | 9 | 40 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Configuration |
|---|---|
| Programs | At most 16 fitted parameters, including latent initial values; at most 400 expression nodes (1,200 for Network-S2) |
| Generators | 1–4 terms per equation and 1–2 factors per term; power exponents ; up to 2 latent states; 0–2 auxiliary channels; signal-to-noise ratio log-uniform in in mechanics, kinetics, discrete, causal, and scaling |
| Fitter | Trust-region reflective least squares, 6 starts of at most 60 residual evaluations each (finite-difference Jacobian columns are not counted); starting coefficients have random sign and log-uniform magnitude in , latent initial values are uniform in , and an agent-supplied start replaces one of the six; stochastic processes use Nelder–Mead, 6 starts of 60 evaluations |
| Held-out tests | : for ODE and map worlds, for delay and spatial worlds, for jump, agent, and lattice worlds; causal and scaling 10 and 6; metrology and spectroscopy 16 or 24 and 8; SDE 4 conditions; stochastic scores use 512 simulated paths per condition |
| Stochastic scores | Excess CRPS for predicted and true , divided by the true standard deviation (at least 0.1) and maximized over recorded times; excess Brier score |
| Calibration | True-coefficient fits on designs of 8, 12, 16, 24, 32, 44, or 60 experiments on three noise seeds; five further seeds at the full budget |
| Setting | Channels | Samples/run | |||||
|---|---|---|---|---|---|---|---|
| Mechanics | 2–4 | 1.46–16.9 | 41 | 337 | 89 | 179 | 187 |
| Kinetics | 2–3 | 0.655–12.2 | 41 | 120 | 178 | 322 | 102 |
| Discrete | 3–5 | 30 | 31 | 121 | 246 | 203 | 181 |
| Causal | 5–6 | — | Point/sample | 160 | 2,135 | 171 | 1,181 |
| Scaling | 4–7 | — | Point/sample | 215 | 149 | 109 | 88 |
| Ecology | 2–3 | 8 | 25 | 243 | 166 | 192 | 144 |
| Setting | Ordinary | Extended |
|---|---|---|
| Frequency | ||
| Field | ||
| Temperature | ||
| Resolution |
| Setting | Observe | Initial | Kick | Clamp | Input | Probe | Assay | Gold |
|---|---|---|---|---|---|---|---|---|
| Mechanics | 1–2 | 2–5 | 4–6 ∗ | 8–11 ∗ | 9 ∗ | 6–14 | 15–40 | — |
| Kinetics | 1–2 | 2–4 | 4–6 ∗ | 7–11 ∗ | 4–10 ∗ | 16 ∗ | 15–40 | — |
| Discrete | 1–2 | 2–4 | 3–5 ∗ | 5–11 ∗ | 6–9 ∗ | 9–10 ∗ | 15–40 | — |
| Causal | 1 | — | — | 3–8 | — | — | — | — |
| Scaling | 1 | — | — | — | — | 4–10 | — | — |
| Ecology | 1–2 | 2–4 | 4–6 | 6–9 | 4–10 | 10–14 | 15–40 | — |
| Syntax | Meaning |
|---|---|
| x+y , x-y , x*y , -x | Addition, subtraction, multiplication, and negation |
| x/y | Protected division as defined above |
| pow(x,r) | |
| sat(x,k,a) | |
| exp(x) | |
| log1p(x) |
| Spelling | Meaning or expansion |
|---|---|
| x**r or xˆr | pow(x,r) |
| sqrt(x) | pow(x,1/2) |
| log(x) | log1p(x) |
| cos(x) | |
| sat(x,k) | sat(x,k,1) |
| abs(x) |
| Related world | Control world | |||||
|---|---|---|---|---|---|---|
| Model | Studies | First world | Fresh | Given program | Fresh | Given program |
| GPT 6 Astra | 4 | 1 | 2 | 2 | 1 | 0 |
| Claude Fable 5.1 | 4 | 2 | 0 | 2 | 0 | 0 |
| DeepSeek v4.1 Flash | 4 | 0 | 0 | 0 | 0 | 0 |
| Qwen 3.8 Max | 4 | 1 | 1 | 1 | 0 | 0 |
| Muse Spark 1.3 | 4 | 1 | 0 | 1 | 0 | 0 |
| Model | Provider | Context | Output | Effort | Cache |
|---|---|---|---|---|---|
| GPT 6 Astra | OpenAI | 1,050,000 | 128,000 | max | — |
| Claude Fable 5.1 | Anthropic | 1,000,000 | 128,000 | max | 1h |
| GLM 5.3 | Z.AI | 1,048,576 | 131,072 | max | — |
| Grok 4.6 | xAI | 500,000 | 450,000 | max | — |
| Muse Spark 1.3 | Meta | 1,048,576 | 943,718 | max | — |
| Qwen 3.8 Max | Alibaba | 1,000,000 | 131,072 | xhigh | — |
| Label | OpenRouter identifier |
|---|---|
| GPT 6 Astra | openai/gpt-6-astra |
| Claude Fable 5.1 | anthropic/claude-fable-5.1 |
| GLM 5.3 | z-ai/glm-5.3 |
| Grok 4.6 | x-ai/grok-4.6 |
| Qwen 3.8 Max | qwen/qwen3.8-max-0902 |
| Muse Spark 1.3 | meta/muse-spark-1.3 |
| Setting | World | GPT | Claude | GLM | Grok | Muse | Qwen | Gemini | Kimi | DeepSeek |
|---|---|---|---|---|---|---|---|---|---|---|
| Mechanics | 1 | 0.458 | 0.389 | 0.259 | 0.597 | 0.357 | 0.469 | 0.495 | 0.844 | 0.369 |
| 2 | 0.005 | 0.005 | 0.056 | 0.073 | 0.005 | 0.005 | 0.210 | 0.230 | I | |
| 3 | 0.293 | 0.208 | 0.130 | 0.512 | 0.432 | 0.210 | 0.542 | 0.633 | 0.730 | |
| 4 | 3.063 | 3.037 | 3.704 | D | 1.106 | 3.806 | 3.942 | 3.082 | 3.948 | |
| Kinetics | 1 | D | 0.429 | D | 0.659 | 0.854 | 0.400 | 0.956 | 0.515 | 0.355 |
| 2 | 0.300 | D | 1.405 | D | 109.923 | 0.273 | 2.491 | 0.365 | D |
| Model | 1: CRPS | 1: Brier | 2: CRPS | 2: Brier | 3: CRPS | 3: Brier | 4: CRPS | 4: Brier |
|---|---|---|---|---|---|---|---|---|
| GPT 6 Astra | 0.1122 | 0.0004 | 0.0327 | 0.0316 | 0.0090 | 0.0014 | 0.0144 | 0.0015 |
| Claude Fable 5.1 | 0.0857 | 0.0006 | 0.0563 | 0.0011 | 0.0236 | 0.0037 | 0.0282 | 0.0006 |
| GLM 5.3 | — | — | — | — | — | — | 0.0294 | 0.0017 |
| Grok 4.6 | — | — | 0.3768 | 0.0009 | 0.0377 | 0.0030 | 0.0116 | 0.0011 |
| Muse Spark 1.3 | 0.0710 | 0.0007 | 0.0832 | 0.0003 | 0.0087 | 0.0018 | 0.0426 | 0.0010 |
| Qwen 3.8 Max | 0.0548 | 0.0009 | — | — | 0.0088 | 0.0020 | 0.0338 | 0.0014 |
| Setting | World 1 | World 2 | World 3 | World 4 |
|---|---|---|---|---|
| Mechanics | 0.0087 | 0.0023 | 0.0471 | 0.0115 |
| Kinetics | 0.0738 | 0.0169 | 0.0751 | 0.0129 |
| Discrete | 0.0103 | 0.0575 | 0.0159 | 0.0069 |
| Causal | 0.0651 | 0.0372 | 0.0294 | 0.0565 |
| Scaling | 0.0474 | 0.0570 | 0.0144 | 0.0223 |
| Ecology | 0.0569 | 0.0196 | 0.0178 | 0.0125 |
| Method | Dynamical | NRMSE | Static | NRMSE | All deterministic | NRMSE |
|---|---|---|---|---|---|---|
| GPT 6 Astra | 11/24 | 0.133 | 6/12 | 0.084 | 20/40 | 0.099 |
| Claude Fable 5.1 | 6/24 | 0.217 | 6/12 | 0.113 | 15/40 | 0.168 |
| GLM 5.3 | 4/24 | 0.299 | 3/12 | 0.260 | 10/40 | 0.278 |
| Grok 4.6 | 2/24 | 0.504 | 3/12 | 0.256 | 8/40 | 0.361 |
| Muse Spark 1.3 | 2/24 | 0.481 | 4/12 | 0.180 | 9/40 | 0.282 |
| Qwen 3.8 Max | 4/24 | 0.497 | 3/12 | 0.272 | 7/40 | 0.358 |
| World | CRPS excess | Brier excess | Pass |
|---|---|---|---|
| 1 | 0.0170 | 0.0005 | Yes |
| 2 | 0.0571 | 0.0012 | No |
| 3 | 0.0084 | 0.0020 | Yes |
| 4 | 0.0228 | 0.0011 | Yes |
| Model | Pass | Finite error | Simulation | Missing channel | No submission |
|---|---|---|---|---|---|
| GPT 6 Astra | 22 | 21 | 1 | 0 | 0 |
| Claude Fable 5.1 | 17 | 25 | 1 | 1 | 0 |
| GLM 5.3 | 11 | 26 | 1 | 2 | 4 |
| Grok 4.6 | 10 | 31 | 2 | 0 | 1 |
| Muse Spark 1.3 | 10 | 33 | 1 | 0 | 0 |
| Qwen 3.8 Max | 9 | 29 | 1 | 2 | 3 |
| Model | Measurements | Fit cost | Predicted intensity range | Hidden error |
|---|---|---|---|---|
| GPT 6 Astra | 157 | 0.00398 | [0.0354, 0.2037] | 0.0121 |
| Claude Fable 5.1 | 162 | 0.00323 | [0.0346, 0.1941] | 0.0157 |
| Kimi K3 | 192 | 2.13896 | [0.0340, 0.0340] | Missing readouts |
| DeepSeek v4.1 Flash | 222 | 4.81083 | [0.0000, 0.0000] | Missing readouts |
| Domain | GPT | Claude | Muse | Grok | Qwen | GLM | DeepSeek | Gemini | Kimi | All |
|---|---|---|---|---|---|---|---|---|---|---|
| Dynamics | ||||||||||
| Mechanics | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 6 |
| Kinetics | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| Discrete | 2 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 5 |
| Ecology | 1 | 1 | 1 | 0 | 2 | 2 | 0 | 0 | 0 | 7 |
| Network | 2 | 2 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 5 |
| Model | Budget used (%) | Truncated | No submission | Final worse | Solve lost | Cost ($) |
|---|---|---|---|---|---|---|
| GPT 6 Astra | 93 | 0 | 0 | 10 | 1 | 959 |
| Claude Fable 5.1 | 84 | 1 | 0 | 34 | 2 | 1687 |
| Muse Spark 1.3 | 88 | 7 | 4 | 27 | 1 | 618 |
| Grok 4.6 | 77 | 3 | 2 | 23 | 0 | 397 |
| Qwen 3.8 Max | 66 | 40 | 7 | 34 | 0 | 498 |
| GLM 5.3 | 83 | 20 | 11 | 25 | 1 | 446 |
Scientific Discovery
Scientific discovery uses computational methods to find hypotheses, laws, mechanisms, and materials from data, literature, and experiments.
See all 90 papers