Operon-S1: a fast decision controller for long-horizon biomedical agents
Organizations: Genoria AI
Abstract
Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own. Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Every decision is recorded with its consequence. Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. Task-level results in this study use the public checkpoint. The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.
Figures & tables
| Point | Judgement | Question (type) | Action | Round-1 label |
|---|---|---|---|---|
| S1 | analysis family | ten classes (choice) | scopes S6 | construction |
| S2 | skills | which skills may help, with a no-skill option (choice); relevance of the top three (yes/no) | brief: up to three skill names, never their text | construction |
| S3 | skill need | is a skill needed (yes/no) | recorded only | – |
| S4 | environment | which environment, when there is more than one (choice) | brief: environment path | – |
| S5 | files | is each input file relevant (yes/no) | brief: relevant files | – |
| S6 | conventions | convention per slot that the task does not state (choice) | brief: “the task appears to require …; verify” | – |
| Benchmark | Tasks | Metric | Network | Judge | Per-task limits |
|---|---|---|---|---|---|
| LAB-Bench SeqQA | 600 | accuracy | database allowlist | none | 1 CPU, 2 GiB, 1,200 s |
| LAB-Bench 2 seqqa2 | 400 | accuracy (official validators) | database allowlist | none | 1 CPU, 2 GiB, 1,200 s, last 60 s reserved |
| LAB-Bench 2 protocolqa2 | 125 | accuracy (official judge prompt) | offline | LLM (substitute model) | 2 CPU, 4 GiB, 2,400 s |
| BixBench-Verified-50 | 50 | open-answer accuracy | database allowlist | LLM for range answers (substitute model) | 2 CPU, 16 GiB, 3,600 s |
| SciAgentArena | 189 | mean correctness | offline | none | 4 CPU, 8 GiB, 3,600 s |
| BiomniBench-DA | 50 | rubric score, 0–100 | web via logged proxy | LLM rubric (substitute model) | 2 CPU, 16 GiB, 7,200 s |
| Version | Training data | Gate split | Δ v0 | Held-out states |
|---|---|---|---|---|
| Version 0 | public checkpoint, zero-shot | – | – | seqqa2 family 0.056; error class 0.174 |
| rsi-v2 | 2,500 rows of curriculum decisions | 6 questions | positive on all 6 (family 0.26 → 0.99) | – |
| rsi-v5 | synthetic protocol decisions, labels by construction | 4 questions | positive on all 4 (answer type 0.24 → 1.00) | – |
| rsi-v6 | 1,000 new synthetic rows and 200 replay rows per arm, mix set by controllers’ errors on the development halves | 250 rows per question, 4 questions | +0.452 to +0.868, lower bounds above 0 | seqqa2 family 1.000; error class 0.408 |
| Question | Generator | Label source | Train | Cal | Gate | OOD | Majority | Version 0 (gate) | Rule (gate) |
|---|---|---|---|---|---|---|---|---|---|
| Analysis family | G1 | construction | 845 | 115 | 428 | 110 | 0.92 | 0.26 | 0.79 |
| Answer type | G1 | construction | 845 | 115 | 428 | 110 | 0.49 | 0.40 | 0.48 |
| Skill choice | G2 | construction | 428 | 115 | 428 | 110 | 0.69 | 0.02 | 0.68 |
| Skill relevance | G2 | construction | 5,892 | 1,550 | 5,739 | 1,430 | 0.87 | 0.13 | 0.05 |
| Answer format | G3 | verifier | 440 | 52 | 212 | 62 | 0.50 | 0.44 | 0.66 |
| Error class | G5 | execution | 4,808 | 729 | 1,474 | 0 | 0.08 | 0.43 | 0.85 |
| Benchmark | Tasks | Δ score [95% CI] | Δ wall (%) | Δ steps (%) | Baseline σ (rounds used for σ) | Tasks changed between baseline rounds |
|---|---|---|---|---|---|---|
| SeqQA | 600 | +0.004 [−0.005, +0.013] | −2.3 [−8.2, +3.9] | −0.8 [−5.2, +4.0] | 0.001 (2) | 2.2% |
| seqqa2 | 400 | +0.004 [−0.009, +0.017] | −1.8 [−11.5, +8.1] | −4.5 [−9.1, +0.2] | 0.007 (3) | 3.0% |
| protocolqa2 | 125 | −0.000 [−0.061, +0.061] | −2.2 [−8.9, +4.8] | −3.6 [−8.4, +1.6] | 0.021 (5) | 13.1% |
| BixBench-Verified-50 | 50 | −0.030 [−0.092, +0.023] | +2.5 [−10.2, +15.9] | +9.2 [−1.0, +20.0] | 0.014 (2) | 10.0% |
| SciAgentArena | 189 | +0.014 [−0.027, +0.049] | −6.9 [−20.2, +5.9] | +0.5 [−10.0, +9.6] | 0.010 (5) | 8.5% |
| BiomniBench-DA | 50 | +0.770 [−2.270, +3.840] | +40.5 [+18.0, +66.4] | −1.3 [−8.5, +6.5] | 0.979 (3) | 0.0% |
| Arm | Benchmark | Tasks | Right in both | Gained | Lost | Wrong in both | P | P (Holm) |
|---|---|---|---|---|---|---|---|---|
| No-op | SeqQA | 600 | 584 | 7 | 5 | 4 | 0.774 | 1.000 |
| No-op | seqqa2 | 400 | 375 | 2 | 5 | 18 | 0.453 | 1.000 |
| No-op | protocolqa2 | 125 | 82 | 10 | 12 | 21 | 0.832 | 1.000 |
| No-op | BixBench-Verified-50 | 50 | 42 | 0 | 2 | 6 | 0.500 | 1.000 |
| No-op | SciAgentArena | 189 | 101 | 8 | 7 | 73 | 1.000 | 1.000 |
| Rule-based | SeqQA | 600 | 586 | 8 | 3 | 3 | 0.227 | 0.906 |
| Comparison | Tasks | Δ score [95% CI] | Δ wall (%) | Δ steps (%) |
|---|---|---|---|---|
| JEV + protocol profile − baseline | 125 | +0.011 [−0.023, +0.045] | −11.7 [−17.8, −5.0] | −4.2 [−9.2, +1.0] |
| General LLM + protocol profile − baseline | 125 | +0.035 [−0.002, +0.073] | −15.0 [−21.3, −8.4] | −9.7 [−14.1, −5.4] |
| JEV + protocol profile − JEV (default harness) | 125 | −0.021 [−0.077, +0.035] | −9.8 [−18.7, −0.2] | +0.9 [−5.8, +7.8] |
| General LLM + protocol profile − general LLM (default harness) | 125 | +0.035 [−0.024, +0.093] | −13.3 [−21.3, −4.6] | +1.4 [−5.5, +8.5] |
| LAYA version 0 + protocol profile − baseline | 125 | +0.060 [+0.014, +0.108] | – | −7.9 [−12.3, −3.3] |
| LAYA version 0 + protocol profile − baseline, held-out half | 62 | +0.055 [−0.011, +0.129] | – | −8.9 [−15.5, −1.3] |
| Question | n | Majority | Rule | LAYA v0 | JEV | General LLM | Headroom |
|---|---|---|---|---|---|---|---|
| Error class (P3) | 3,434 | 0.383 | 0.953 | 0.164 [0.132, 0.194] / 0.360 (836) | 0.864 [0.841, 0.885] / 0.036 (868) | 0.846 [0.820, 0.871] / 0.026 (891) | on 3 of 6 benchmarks |
| Analysis family (S1) | 4,554 | 0.878 | 0.593 | 0.087 [0.052, 0.126] / 0.399 (1,139) | 0.974 [0.961, 0.984] / 0.013 (1,139) | 0.963 [0.944, 0.980] / 0.040 (941) | yes |
| Answer type (S7) | 2,400 | 1.000 | 0.933 | 0.663 [0.625, 0.702] / 0.267 (600) | 1.000 [1.000, 1.000] / 0.039 (600) | 0.998 [0.994, 1.000] / 0.046 (472) | yes (one label value) |
| Answer format (D20) | 481 | 0.996 | 0.996 | 0.994 [0.985, 1.000] / 0.053 (477) | 0.680 [0.638, 0.722] / 0.132 (485) | 0.903 [0.875, 0.928] / 0.054 (473) | no |
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 275 papers