Structure Before Intelligence: Gating What Agentic Scientific Workflows Release
Authors: Ashish Acharya, Aman Al Amin, Hong Qin
Organizations: Department of Computer Science, Old Dominion University, Norfolk, Virginia, USA · Department of Computer Science, School of Data Science, Old Dominion University, Norfolk, Virginia, USA
An agentic scientific workflow can run to completion and still release wrong results: a silent numerical error, a stale input, or a defect in a shared upstream step can contaminate several outputs while a sibling branch stays correct. We treat trust in such results as a release decision. A harness around the workflow's directed acyclic graph (DAG) decides which outputs may be used: it enforces mandatory checks, contains failures to the affected branch, recomputes from tracked versions, adds validators, and releases a result only if a risk model estimates its probability of being invalid as low. Language-model agents work inside the harness, choosing corrective actions and reviewing releases, but cannot override it. On 60 held-out paired instances of two workflows with injected faults, we switched components on and off. Each supplied a different property. Containment and recovery raised the number of correct results. Validation supplied safety: fixed rules without validation released 37 invalid results, while validating every candidate released none. Risk estimation supplied efficiency, matching that safety with 31% fewer validators at somewhat lower yield, although better targeting than random validation was not shown. Lineage evidence helped on one workflow. The language model added no measurable benefit: whether invalid results escaped depended on the harness, not the model. This proof of concept points to a practical path: placing agents inside explicit release-control structure can keep released scientific results trustworthy, and it gives more capable agents a measurable baseline to improve on.
Figures & tables
Figure 1. The harness as a release gate: it decides which outputs may be used. A fault in shared ancestor S can reach both deliverables A and B . For each candidate the controller estimates pt(d) from check evidence and, optionally, lineage; it releases iff p<p∗ and otherwise validates, recomputes, or withholds. Only the manifest states what may be used, and later evidence is recorded as a retraction. The language-model agent acts inside the layer and cannot override the release rule.
Published table
SIR model
Evidence tier
Published values
Synthetic parameters
Shared node
source table
model specification
Deliverables
A report, B report
summary A, summary B
Checks (mand./opt.)
3 / 5
4 / 4
Alternatives
rebuild from verified source; decimal mean
load from registry; 3 tighter solvers
Table 1. Workflow contracts (both: 7 logical nodes, at most 3 attempts per node, 6 repairs, and cost 60).
Risk-Lineage : 0 invalid, 31% fewer validators than Validate-All
lower yield; targeting not shown
Lineage
yield, one workflow
Risk-Lineage above Risk-Local on the published table only
one mechanism
Language model
nothing shown
no agent arm beat its base policy
reviewer cannot reject
Table 3. What each harness component supplied (evaluation split; evidence in the paragraphs below).
Valid yield
Invalid/
False
Vali-
Policy
Pub. table
SIR
released
block
dators
Cost
Halt
26/60
22/60
0/48
0
0
13.00
Contain
32/60
32/60
0/64
0
0
15.50
Rules
42/60
41/60
37/120
0
0
11.07
Risk-Local
2/60
49/60
0/55
20
235
15.80
Risk-Lineage
51/60
49/60
0/106
0
248
19.65
Table 4. Deterministic policies, evaluation split. Valid yield is of 60 requested deliverables per workflow; invalid releases and false blocking (clean and benign deliverables not released, of 40) pool both workflows; validators are inserted validator runs (total); cost is mean contract cost per instance. Released versions include those retracted after late evidence ( Rules 7 (all invalid), Risk-Local 4, Risk-Lineage 6, Validate-All 7, Random-Matched 7; none otherwise), which never count toward valid yield. Every retracted valid version was built on a replacement, made by recovery before release, of the node that a post-release notice named; because a notice attaches to the logical node, not to a version, these retractions were unnecessary and cost valid yield. ∗ Post hoc (Section 7.6 ).
Figure 2. Valid deliverables (of 120) against mean cost per episode for each arm of Table 4 . Crosses mark arms that released invalid deliverables (count in parentheses); ∗ post hoc. Scatter plot of valid yield against cost for seven arms.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Deterministic policies
With agent (+Agent)
Family
Halt
Contain
Rules
Risk-Local
Risk-Lineage
Halt
Rules
Risk-Lineage L/M
Risk-Lineage G
Clean
10/10; 0
10/10; 0
10/10; 0
0/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
Benign
10/10; 0
10/10; 0
10/10; 0
0/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
Silent error
2/10; 0
5/10; 0
5/10; 5
0/10; 0
9/10; 0
2/10; 0
5/10; 5
9/10; 0
6/10; 0
Stale
3/10; 0
5/10; 0
10/10; 0
2/10; 0
10/10; 0
3/10; 0
10/10; 0
10/10; 0
10/10; 0
Shared ancestor
0/10; 0
0/10; 0
0/10; 10
0/10; 0
10/10; 0
0/10; 0
0/10; 10
10/10; 0
8/10; 0
Appendix
Table S1. Published-table workflow, evaluation split. Each cell: valid yield (of requested); invalid releases; bold cells released at least one invalid deliverable. Agent columns: Halt+Agent and Rules+Agent give the same cells for all three models, and Risk-Lineage+Agent the same for Llama and Mistral (L/M; G: Gemma; asserted by the generator).
Deterministic policies
With agent (+Agent)
Family
Halt
Contain
Rules
Risk-Local
Risk-Lineage
Halt
Rules
Risk-Lineage L/M
Risk-Lineage G
Clean
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
Benign
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
10/10; 0
Silent error
0/10; 0
5/10; 0
5/10; 5
7/10; 0
7/10; 0
0/10; 0
5/10; 5
7/10; 0
6/10; 0
Stale
0/10; 0
5/10; 0
10/10; 0
10/10; 0
10/10; 0
0/10; 0
10/10; 0
10/10; 0
10/10; 0
Shared ancestor
0/10; 0
0/10; 0
0/10; 10
10/10; 0
10/10; 0
0/10; 0
0/10; 10
10/10; 0
0/10; 0
Appendix
Table S2. SIR workflow, evaluation split. Each cell: valid yield (of requested); invalid releases; bold cells released at least one invalid deliverable. Agent columns: Halt+Agent and Rules+Agent give the same cells for all three models, and Risk-Lineage+Agent the same for Llama and Mistral (L/M; G: Gemma; asserted by the generator).
Arm set
Arms
Ep.
Infra.
Compl.
Budget
Stop
Error
Deterministic
5
300
0
260
0
40
0
Sensitivity
2
120
0
120
0
0
0
Agentic
9
540
0
403
17
120
0
Appendix
Table S3. Execution integrity of the evaluation runs (episode counts and final status, generated from the run records). Sensitivity: the two risk policies with their original weights (Section S6 ).
Arm
n (invalid)
Missing
Brier [95% CI]
Ref.
Skill
ECE [95% CI]
Fill / orig. Brier (orig. n , missing)
Risk-Local
327 (31)
7
0.0153 [0.0108, 0.0218]
0.086
0.82
0.078 [0.065, 0.092]
0.110 / 0.122 (296, 38)
Risk-Lineage
447 (42)
10
0.0036 [0.0015, 0.0076]
0.085
0.96
0.021 [0.015, 0.025]
0.050 / 0.083 (255, 202)
Risk-Local (orig.)
60 (16)
0
0.1956 [0.1491, 0.2498]
0.196
0.00
0.001
0.196 / 0.196 (60, 0)
Risk-Lineage (orig.)
60 (16)
0
0.1956 [0.1478, 0.2513]
0.196
0.00
0.005
0.196 / 0.196 (60, 0)
Risk-Lineage+Agent Gemma
401 (74)
10
0.0038 [0.0015, 0.0082]
0.150
0.97
0.022 [0.016, 0.027]
0.056 / 0.079 (268, 143)
Released subsets: Risk-Local n=55 , Risk-Lineage n=106 , Risk-Lineage+Agent Gemma n=84 ; 0 invalid in each.
Appendix
Table S4. Calibration on the evaluation split. Post hoc (primary): risk of the latest consultation of each labeled version before release or withholding, usually after validator evidence. Fill sensitivity: the original analysis’s risk where it exists, the post-hoc risk otherwise. Original: the original analysis (omits accepts; keeps the latest event’s risk). Missing: versions without any consultation (external stale inputs, missing by design). Intervals: scenario-clustered bootstrap, 2000 replicates. On released subsets no version is invalid, so the reference Brier is 0 and skill is undefined.
Table S6. Agentic arms, evaluation split (60 instances, 120 requested deliverables per arm). Calls and tokens are summed over Halt+Agent , Rules+Agent , and Risk-Lineage+Agent and counted from the call logs. Every oracle-invalid approval was under Rules+Agent . Δ is per instance (two deliverables), with a case-group bootstrap 95% interval.
Invalid/
Valid
Vali-
Arm
Validation
released
yield
dators
Cost
Rules
rule, never fired
37/120
83/120
0.00
11.07
Validate-All
always
0/119
112/120
5.98
25.17
Random-Defective †
random, low volume
14/113
99/120
2.32
17.32
Random-Matched
random, matched
3/111
101/120
4.18
21.76
Risk-Lineage
risk-targeted
0/106
100/120
4.13
19.65
Appendix
Table S7. Post-hoc validation ablation, evaluation split (60 instances, 120 requested deliverables per arm). All three arms reuse the rules and recovery ladder of Rules . Validators: mean inserted validator runs per instance; cost: mean contract cost per instance. Differences: ratio-of-sums (invalid/released, yield) or per-instance mean (cost), scenario bootstrap 95% interval.
Option
Action
Nodes
Description
Published table workflow
retry
retry
any
Run the node’s current operation again on the same inputs, with no changes.
repair
repair
any
Run the node’s current operation again with the previous attempt’s check results attached as diagnostics.
recompute
invalidate, recompute
any
Invalidate the current version and recompute the node from its current inputs, applying any input updates published during the run.
table_from_verified_transcription
select alternative
source_table
Rebuild the source table from the frozen verified transcription of the publication instead of the runtime table text.
means_decimal
select alternative
a_means
Recompute the per-padding means with exact decimal arithmetic instead of floating point.
Appendix
Table S8. Proposal menus, as fixed before the agentic evaluation run. “Any”: every node of the workflow.
Table S10. Action-benefit table (development records only). Entries are action, situation, mandatory-fail status, and issue kind. “Bound”: worst-case valid-before rate (fewer than five valid-before records).
Model
Arm
Valid
Inv./rel.
Calls
Tokens
Grok 4.3
Halt+Agent
48
0/48
0/48
63.0k/2.2k
Rules+Agent
83
37/120
16/128
184.8k/7.0k
Risk-Lineage+Agent
100
0/106
36/114
191.3k/7.8k
Gemini 3.6 Flash
Halt+Agent
48
0/48
0/48
55.0k/1.7k
Rules+Agent
83
37/120
16/128
160.8k/5.4k
Risk-Lineage+Agent
100
0/106
36/114
166.6k/6.0k
Appendix
Table S11. Post-hoc paid-API addendum (not pooled). Valid yield of 120; calls proposer/reviewer and tokens prompt/completion from the call logs, excluding Gemini’s reasoning tokens (69.1k in total, billed as output); all reviewer verdicts were approvals.
LLM-enabled AI workflows increasingly produce outputs through iterative generate-evaluate-revise loops. Each iteration can improve the candidate, but it also creates a release decision: when to stop and output the current result? This raises a statistical challenge because deployment-time evaluator scores are adaptively generated and repeatedly monitored, yet the likelihood models or exchangeability assumptions typically used for calibration are unavailable. We propose an always-valid release wrapper for existing generator-evaluator pipelines. The wrapper builds a hard-negative reference pool of high-scoring failures, calibrates deployment-time evaluator scores against this pool, and accumulates the resulting evidence with an e-process. This separates two roles: the reference pool turns black-box scores into conservative evidence, while the e-process provides validity under optional stopping. In theory, we show that a conservative reference pool yields finite-sample control of the probability of releasing on infeasible tasks, that is, tasks for which the given workflow is not capable of producing a reliable solution. We also characterize conditions under which the same conservative rule still achieves nontrivial release on feasible tasks. In an MBPP+ coding-agent case study, the wrapper reduces premature incorrect release relative to baseline stopping rules while still releasing on tasks for which the workflow repeatedly accumulates moderate supporting evidence.
Young Hyun Cho, Will Wei Sun
Department of Statistics, Purdue University. · Mitch Daniels School of Business, Purdue University.
AI-workflow governance cannot be reduced to checking the final answer: an apparently safe answer may rest on a fragile evidence path that ordinary evaluation cannot see, localize or govern. We call this hidden fragility a "structural iceberg": hallucinations and unsupported claims may form its visible tip, while consequential weakness remains submerged. Stochastic semantic evidence graphs (SSEGs) expose these icebergs by preserving workflow channels, propagating local uncertainty and identifying the hidden paths on which an apparently safe output depends. ALCE and RAGTruth show that visible failures at the tip -unsupported citations and hallucinated spans -rest on distinct submerged weaknesses and therefore require different interventions. Across retrieval, tool-use and controlled stress tests, SSEG localizes those weaknesses, supports targeted repair, produces no false automatic passes in 35,000 known-truth cases and reduces ToolSandbox review by 28.8% across 96 executions from two agent models. The same structural view carries into end-to-end governance: in a separately sealed 1,200-case FinGovBench study, adding SSEG to GPT-OSS-20B reduces unsafe releases from 452/660 to 8/660 while releasing all 540 safe cases and correctly distinguishing 592/600 matched workflow pairs. An unchanged-gate transfer to Qwen3-8B releases all 540 safe cases and none of 660 unsafe cases, whereas flat-UQ releases 520 unsafe cases. SSEG therefore moves governance below surface-level output checking, turning hidden evidence dependencies into auditable, path-specific decisions about intervention, revalidation and release.
Matthew F. Dixon, Bertrand Nortier, Miquel Noguer i Alonso
BeliefLens · AI Finance Institute (AIFI) · University of Leicester +1
Agentic AI systems are increasingly being integrated into scientific workflows, yet their behavior under realistic conditions remains insufficiently understood. We evaluate CMBAgent across two workflow paradigms and eighteen astrophysical tasks. In the One-Shot setting, access to domain-specific context yields an approximately ~6x performance improvement (0.85 vs. ~0 without context), with the primary failure mode being silent incorrect computation - syntactically valid code that produces plausible but inaccurate results. In the Deep Research setting, the system frequently exhibits silent failures across stress tests, producing physically inconsistent posteriors without self-diagnosis. Overall, performance is strong on well-specified tasks but degrades on problems designed to probe reasoning limits, often without visible error signals. These findings highlight that the most concerning failure mode in agentic scientific workflows is not overt failure, but confident generation of incorrect results. We release our evaluation framework to facilitate systematic reliability analysis of scientific AI agents.
Shivam Rawat, Lucie Flek
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.