cs.AIOct 26, 2026
SaveSlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot Undo
Organizations: College of Informatics, Huazhong Agricultural University
Abstract
AI agents are increasingly proposed as autonomous experimenters, yet the benchmarks that test them mostly offer cheap experiments that answer at once. Experiments in physical facilities are different: they take months, occupy scarce equipment and cannot be undone. We present SlowLab, a benchmark in which an agent runs a one-year campaign in four simulated greenhouse compartments on a site whose crop, facility and prices are hidden. The agent decides when to plant which climate policy, whether to stop a crop and which measurements to read while crops grow, and at the end recommends one policy, which a private evaluator scores in weather the agent never saw. In a preregistered study on 48 unseen sites, three low-cost language-model agents ended near a standard grower setting and clearly below a Bayesian-optimisation baseline that starts from that setting. They ran the facility as two synchronous batches, so measurements taken during the season never changed what they planted, and these measurements improved no method's result. Letting other methods re-read the evidence a campaign had collected did not yield significantly better recommendations, which points to where the models start and which experiments they try. Code is released under the MIT license.
Figures & tables
Figure 1: SlowLab in one picture. Left: a real campaign year (test site 0, first repeat, Full feedback of the first model alphabetically; selection fixed before plotting). Bars are crops in the four compartments C1–C4 (grey: cleaning), ticks are channel reads; each crop’s margin is known only when it closes. Right: across 48 unseen sites, the final policies of three language models scored well below a prior-informed local Bayesian-optimisation baseline (site-level paired differences, BCa 95% intervals; shaded: the EUR/m 2 smallest effect of interest).
| Delayed | Irrevers- | Parallel | Mid-run | Hidden | Unseen-con- | Non-LLM | Pre- | |
| outcomes | ible | units | signals | instance | dition score | baseline | registered | |
| BoxingGym ( Gandhi et al., 2025 ) | – | – | – | – | ✓ | – | ||
| SciGym ( Duan et al., 2025 ) | – | – | – | – | ✓ | ✓ | – | – |
| CARE ( Liu et al., 2026 ) | – | – | – | – | – | ✓ | ||
| LLMs for BO ( Gupta et al., 2025 ) | – | – | – | – | ✓ | – | ||
| Vending-Bench ( Backlund and Petersson, 2025 ) | ✓ | ✓ | – | – |
Table 1: SlowLab against the closest benchmarks and environments. Columns: outcomes arrive only after a long delay during which the agent keeps acting; starting an experiment commits a resource that cannot be released without loss; a small fixed set of units must be scheduled in parallel; partial measurements are readable while an experiment runs; each instance has hidden parameters to be learned by experimenting; the final output is scored under conditions the agent never observed; a non-language-model baseline is evaluated alongside language-model agents; the evaluation was preregistered. ✓: yes; : partly; –: no; n/a: no language-model agents; ?: not reported.
Figure 2: One campaign through the agent’s eyes; Section 3.3 describes steps 1–6. Everything inside the dashed executor is hidden from the agent. Shields: every message is hash-audited, and messages to the model also pass the information firewall. The language-model harness is shown in Appendix Figure 6 . Icons: Freepik, flaticon.com.
| Method | Full | Endpoint | Pre-experiment |
|---|---|---|---|
| Prior-informed local BO (baseline) | 33.62 | 34.34 | – |
| GLM-5.3 Flash | 29.62 | 30.38 | 28.35 |
| MiMo V2.6 Flash | 28.90 | 29.24 | 26.86 |
| DeepSeek V4.1 Flash | 28.02 | 28.31 | 25.36 |
| Fixed reference (no experiments) | 29.34 | 29.34 | 29.34 |
Table 2: Mean score of the recommended policy over the 48 test sites (EUR/m 2 , three evaluation years each). The fixed reference runs no experiments, so its score is the same in every column.
Figure 3: Language models end near the grower setting, well below the baseline. Left: site-level paired difference between each model’s final (Full) policy and the baseline’s; dots are sites, bars BCa 95% intervals, dotted lines the smallest effect of interest ( EUR/m 2 ); p-values Holm-adjusted. Middle: gain over the model’s own pre-experiment recommendation (BH-adjusted). Right: heat and light per crop of the recommended policies in the evaluation runs.
Figure 4: No method profits from in-season process feedback: site-level paired difference between the Full and Endpoint branches of each method (Holm-adjusted).
Figure 5: Reading the final evidence is not the bottleneck. Reader swap gain: each Reader’s recommendation from a history packet minus the recommendation of the method that collected the packet, by packet source (16 packets each, BCa 95% intervals) and pooled over sites (diamonds; BH-adjusted over the joint secondary family).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Test | Original | Corrected |
|---|---|---|
| E1 DeepSeek baseline | 3.09 [ 4.35, 1.80], 0.001 | 5.60 [ 6.91, 4.00], 0.001 |
| E1 GLM baseline | 1.43 [ 2.93, 0.03], 0.29 | 3.99 [ 5.34, 2.41], 0.001 |
| E1 MiMo baseline | 2.05 [ 3.57, 0.72], 0.044 | 4.71 [ 6.39, 3.23], 0.001 |
| E2 DeepSeek Full Endpoint | 0.30 [ 1.07, +0.29], 1.00 | 0.30 [ 1.07, +0.29], 0.81 |
| E2 GLM Full Endpoint | 0.71 [ 1.49, +0.03], 0.32 | 0.76 [ 1.61, +0.11], 0.39 |
| E2 MiMo Full Endpoint | 0.19 [ 1.74, +0.65], 1.00 | 0.34 [ 1.88, +0.49], 0.81 |
Table 3: Original and corrected results (EUR/m 2 ; mean [BCa 95% interval], : Holm for E1/E2, BH over the joint secondary family otherwise).
Figure 6: The language-model harness. Top: the stages of a campaign. Bottom: one call of the main loop. The prompt is assembled from the system message, a digest of the state, the six most recent turns and the model’s own notes (1); it passes the information firewall (2); the model returns one JSON object, and every message is hash-audited (3); the reply is parsed (4) and sent to the executor or the analysis tools, or returned as an error (5); the reply and its result join the history for the next call (6). Icons: Freepik, flaticon.com.
Figure 7: Four Full-feedback campaigns on test site 0 (selection fixed before plotting: first repeat of each model, first seed of the baseline). Bars are crops in the four compartments C1–C4 (grey: cleaning), red ticks are channel reads. The models plant their second wave only when the first closes; the baseline staggers two compartments at mid-season.
| Day | Action | Stated reason |
|---|---|---|
| 0 | advance to day 10 | Let the crops establish before reading climate and canopy data. |
| 10 | observe (2 channels) | Check climate tracking in compartment 0 so far. |
| 30–120 | observe, advance (9 steps) | Check canopy growth progress toward maturity. |
| 150 | advance to day 180 | Let the baseline crops finish their 180-day cycle so totals are released. |
| 180 | runs, observe (4 steps) | Get accrued resources and contribution margin for the two completed baseline crops. |
| 180 | start (C3, C4; 2 steps) | Now let me explore a more aggressive policy in compartments 2 and 3, and start planning. |
Table 4: The Full branch of the campaign in Figure 1 : day, action and the model’s stated reason (verbatim).
| Element | Specification | Basis |
|---|---|---|
| Facility | 4 independent compartments of 96 m 2 ; GreenLight fa502ed ; Cabauw weather | simulator; audited weather |
| Calendar | 365-day campaign; crop 180 d; cleaning 2 d; last start day 183; control step 300 s | benchmark design |
| Policy | day 18–26 ∘ C; night 15–22 ∘ C ( day); CO 2 400–1200 ppm; light 0–16 h; vent offset 2–6 ∘ C; RH limit 70–90% | design bounds |
| Actions | start, stop (crop discarded), advance, observe, recommend; analysis tools incl. process predictor | benchmark design |
| Observations | air temperature, RH, CO 2 , canopy LAI proxy; cumulative harvest, heat, light, CO 2 dosing; every 300 s | sensor noise v0 (assumed) |
| Feedback | Full: all records at any time; Endpoint: status while running, totals at close | experimental factor (E2) |
Table 5: The SlowLab environment (task contract v8, site distribution v1). “Assumed” marks declared assumptions without a calibration source; “literature” marks ranges constrained by published values.