Structure Before Intelligence: Gating What Agentic Scientific Workflows Release

Ashish Acharya; Aman Al Amin; Hong Qin

cs.AIOct 14, 2026

Structure Before Intelligence: Gating What Agentic Scientific Workflows Release

Authors: Ashish Acharya, Aman Al Amin, Hong Qin

Organizations: Department of Computer Science, Old Dominion University, Norfolk, Virginia, USA · Department of Computer Science, School of Data Science, Old Dominion University, Norfolk, Virginia, USA

Abstract

An agentic scientific workflow can run to completion and still release wrong results: a silent numerical error, a stale input, or a defect in a shared upstream step can contaminate several outputs while a sibling branch stays correct. We treat trust in such results as a release decision. A harness around the workflow's directed acyclic graph (DAG) decides which outputs may be used: it enforces mandatory checks, contains failures to the affected branch, recomputes from tracked versions, adds validators, and releases a result only if a risk model estimates its probability of being invalid as low. Language-model agents work inside the harness, choosing corrective actions and reviewing releases, but cannot override it. On 60 held-out paired instances of two workflows with injected faults, we switched components on and off. Each supplied a different property. Containment and recovery raised the number of correct results. Validation supplied safety: fixed rules without validation released 37 invalid results, while validating every candidate released none. Risk estimation supplied efficiency, matching that safety with 31% fewer validators at somewhat lower yield, although better targeting than random validation was not shown. Lineage evidence helped on one workflow. The language model added no measurable benefit: whether invalid results escaped depended on the harness, not the model. This proof of concept points to a practical path: placing agents inside explicit release-control structure can keep released scientific results trustworthy, and it gives more capable agents a measurable baseline to improve on.

Figures & tables

Panel of Figure 1 — Figure 1. The harness as a release gate: it decides which outputs may be used. A fault in shared ancestor $S$ can reach both deliverables $A$ and $B$ . For each candidate the controller estimates $p_{t}(d)$ from check evidence and, optionally, lineage; it releases iff $p<p^{\ast}$ and otherwise validates, recomputes, or withholds. Only the manifest states what may be used, and later evidence is recorded as a retraction. The language-model agent acts inside the layer and cannot override the release rule.

	Published table	SIR model
Evidence tier	Published values	Synthetic parameters
Shared node	source table	model specification
Deliverables	A report, B report	summary A, summary B
Checks (mand./opt.)	3 / 5	4 / 4
Alternatives	rebuild from verified source; decimal mean	load from registry; 3 tighter solvers

Table 1. Workflow contracts (both: 7 logical nodes, at most 3 attempts per node, 6 repairs, and cost 60).

Policy	Behavior	Comparison enabled
Halt	Static graph, all checks, global stop	Global intervention baseline
Contain	Static graph, all checks, branch containment	Value of containment
Rules	Dynamic, fixed-rule validation and recovery	Value of bounded adaptation
Risk-Local	Dynamic, risk-guided, local evidence	Value of estimated risk
Risk-Lineage	Dynamic, risk-guided, local + ancestral evidence	Value of lineage
Validate-All ∗	Rules , validating every candidate	Value of validation

Table 2. Policies. ∗ Post-hoc ablation arms (Section 7.6 ).

Component	Supplies	Key evidence	Caveat
Containment, recovery	valid yield	Contain above Halt ; Rules above Contain	the Rules gain came with invalid releases
Validation	safety	Rules : 37 invalid; Validate-All : 0	highest cost
Risk estimation	efficiency	Risk-Lineage : 0 invalid, 31% fewer validators than Validate-All	lower yield; targeting not shown
Lineage	yield, one workflow	Risk-Lineage above Risk-Local on the published table only	one mechanism
Language model	nothing shown	no agent arm beat its base policy	reviewer cannot reject

Table 3. What each harness component supplied (evaluation split; evidence in the paragraphs below).

	Valid yield		Invalid/	False	Vali-
Policy	Pub. table	SIR	released	block	dators	Cost
Halt	26/60	22/60	0/48	0	0	13.00
Contain	32/60	32/60	0/64	0	0	15.50
Rules	42/60	41/60	37/120	0	0	11.07
Risk-Local	2/60	49/60	0/55	20	235	15.80
Risk-Lineage	51/60	49/60	0/106	0	248	19.65

Table 4. Deterministic policies, evaluation split. Valid yield is of 60 requested deliverables per workflow; invalid releases and false blocking (clean and benign deliverables not released, of 40) pool both workflows; validators are inserted validator runs (total); cost is mean contract cost per instance. Released versions include those retracted after late evidence ( Rules 7 (all invalid), Risk-Local 4, Risk-Lineage 6, Validate-All 7, Random-Matched 7; none otherwise), which never count toward valid yield. Every retracted valid version was built on a replacement, made by recovery before release, of the node that a post-release notice named; because a notice attaches to the logical node, not to a version, these retractions were unnecessary and cost valid yield. ∗ Post hoc (Section 7.6 ).

Panel of Figure 2 — Figure 2. Valid deliverables (of 120) against mean cost per episode for each arm of Table 4 . Crosses mark arms that released invalid deliverables (count in parentheses); ∗ post hoc. Scatter plot of valid yield against cost for seven arms.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Deterministic policies					With agent (+Agent)
Family	Halt	Contain	Rules	Risk-Local	Risk-Lineage	Halt	Rules	Risk-Lineage L/M	Risk-Lineage G
Clean	10/10; 0	10/10; 0	10/10; 0	0/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0
Benign	10/10; 0	10/10; 0	10/10; 0	0/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0
Silent error	2/10; 0	5/10; 0	5/10; 5	0/10; 0	9/10; 0	2/10; 0	5/10; 5	9/10; 0	6/10; 0
Stale	3/10; 0	5/10; 0	10/10; 0	2/10; 0	10/10; 0	3/10; 0	10/10; 0	10/10; 0	10/10; 0
Shared ancestor	0/10; 0	0/10; 0	0/10; 10	0/10; 0	10/10; 0	0/10; 0	0/10; 10	10/10; 0	8/10; 0

Appendix

Table S1. Published-table workflow, evaluation split. Each cell: valid yield (of requested); invalid releases; bold cells released at least one invalid deliverable. Agent columns: Halt+Agent and Rules+Agent give the same cells for all three models, and Risk-Lineage+Agent the same for Llama and Mistral (L/M; G: Gemma; asserted by the generator).

	Deterministic policies					With agent (+Agent)
Family	Halt	Contain	Rules	Risk-Local	Risk-Lineage	Halt	Rules	Risk-Lineage L/M	Risk-Lineage G
Clean	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0
Benign	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0	10/10; 0
Silent error	0/10; 0	5/10; 0	5/10; 5	7/10; 0	7/10; 0	0/10; 0	5/10; 5	7/10; 0	6/10; 0
Stale	0/10; 0	5/10; 0	10/10; 0	10/10; 0	10/10; 0	0/10; 0	10/10; 0	10/10; 0	10/10; 0
Shared ancestor	0/10; 0	0/10; 0	0/10; 10	10/10; 0	10/10; 0	0/10; 0	0/10; 10	10/10; 0	0/10; 0

Appendix

Table S2. SIR workflow, evaluation split. Each cell: valid yield (of requested); invalid releases; bold cells released at least one invalid deliverable. Agent columns: Halt+Agent and Rules+Agent give the same cells for all three models, and Risk-Lineage+Agent the same for Llama and Mistral (L/M; G: Gemma; asserted by the generator).

Arm set	Arms	Ep.	Compl.	Budget	Stop
Deterministic	5	300	260	0	40
Sensitivity	2	120	120	0	0
Agentic	9	540	403	17	120

Appendix

Table S3. Execution integrity of the evaluation runs (episode counts and final status, generated from the run records). Sensitivity: the two risk policies with their original weights (Section S6 ).

Released subsets: Risk-Local $n=55$ , Risk-Lineage $n=106$ , Risk-Lineage+Agent Gemma $n=84$ ; 0 invalid in each.
Arm	$n$ (invalid)	Missing	Brier [95% CI]	Ref.	Skill	ECE [95% CI]	Fill / orig. Brier (orig. $n$ , missing)
Risk-Local	327 (31)	7	0.0153 [0.0108, 0.0218]	0.086	0.82	0.078 [0.065, 0.092]	0.110 / 0.122 (296, 38)
Risk-Lineage	447 (42)	10	0.0036 [0.0015, 0.0076]	0.085	0.96	0.021 [0.015, 0.025]	0.050 / 0.083 (255, 202)
Risk-Local (orig.)	60 (16)	0	0.1956 [0.1491, 0.2498]	0.196	0.00	0.001	0.196 / 0.196 (60, 0)
Risk-Lineage (orig.)	60 (16)	0	0.1956 [0.1478, 0.2513]	0.196	0.00	0.005	0.196 / 0.196 (60, 0)
Risk-Lineage+Agent Gemma	401 (74)	10	0.0038 [0.0015, 0.0082]	0.150	0.97	0.022 [0.016, 0.027]	0.056 / 0.079 (268, 143)

Appendix

Table S4. Calibration on the evaluation split. Post hoc (primary): risk of the latest consultation of each labeled version before release or withholding, usually after validator evidence. Fill sensitivity: the original analysis’s risk where it exists, the post-hoc risk otherwise. Original: the original analysis (omits accepts; keeps the latest event’s risk). Missing: versions without any consultation (external stale inputs, missing by design). Intervals: scenario-clustered bootstrap, 2000 replicates. On released subsets no version is invalid, so the reference Brier is 0 and skill is undefined.

Arm	Bin	$n$	Mean risk	Observed invalid
Risk-Local	[0.0, 0.1)	236	0.048	0.000
Risk-Local	[0.1, 0.2)	3	0.145	0.000
Risk-Local	[0.2, 0.3)	58	0.250	0.017
Risk-Local	[0.9, 1.0]	30	0.999	1.000
Risk-Lineage	[0.0, 0.1)	361	0.015	0.000
Risk-Lineage	[0.1, 0.2)	45	0.107	0.022

Appendix

Table S5. Post-hoc reliability bins, all labeled versions (non-empty bins only).

	Llama 3.3	Gemma 3	Mistral S. 3.2
Calls, proposer / reviewer	52 / 290	70 / 266	52 / 290
Tokens, prompt / completion	326.3k / 14.1k	344.4k / 14.7k	338.0k / 9.8k
Reviewer approvals	290/290	266/266	290/290
of which oracle-invalid	44	44	44
Valid yield, Risk-Lineage+Agent	100/120	84/120	100/120
Invalid/released, Rules+Agent	37/120	37/120	37/120

Appendix

Table S6. Agentic arms, evaluation split (60 instances, 120 requested deliverables per arm). Calls and tokens are summed over Halt+Agent , Rules+Agent , and Risk-Lineage+Agent and counted from the call logs. Every oracle-invalid approval was under Rules+Agent .

\Delta

is per instance (two deliverables), with a case-group bootstrap 95% interval.

		Invalid/	Valid	Vali-
Arm	Validation	released	yield	dators	Cost
Rules	rule, never fired	37/120	83/120	0.00	11.07
Validate-All	always	0/119	112/120	5.98	25.17
Random-Defective †	random, low volume	14/113	99/120	2.32	17.32
Random-Matched	random, matched	3/111	101/120	4.18	21.76
Risk-Lineage	risk-targeted	0/106	100/120	4.13	19.65

Appendix

Table S7. Post-hoc validation ablation, evaluation split (60 instances, 120 requested deliverables per arm). All three arms reuse the rules and recovery ladder of Rules . Validators: mean inserted validator runs per instance; cost: mean contract cost per instance. Differences: ratio-of-sums (invalid/released, yield) or per-instance mean (cost), scenario bootstrap 95% interval.

Option	Action	Nodes	Description
Published table workflow
retry	retry	any	Run the node’s current operation again on the same inputs, with no changes.
repair	repair	any	Run the node’s current operation again with the previous attempt’s check results attached as diagnostics.
recompute	invalidate, recompute	any	Invalidate the current version and recompute the node from its current inputs, applying any input updates published during the run.
table_from_verified_transcription	select alternative	source_table	Rebuild the source table from the frozen verified transcription of the publication instead of the runtime table text.
means_decimal	select alternative	a_means	Recompute the per-padding means with exact decimal arithmetic instead of floating point.

Appendix

Table S8. Proposal menus, as fixed before the agentic evaluation run. “Any”: every node of the workflow.

Feature	Set	Meaning	Constant
n_checks	L	checks recorded on the version
n_fail	L	failed checks
n_unknown	L	checks with unknown outcome	yes
n_optional	L	optional checks run
max_value	L	maximum absolute check value
has_value	L	a check value is available

Appendix

Table S9. Risk features. L: local ( Risk-Local , Risk-Lineage ); A: ancestral ( Risk-Lineage only).

Entry	Rate	Inv.	Val.	Source
validator, candidate, no fail	se $0.842$ , fa $0.000$	38	110	estimated
repair, blocked, fail, check	fix $0.089$	45	0	bound
select alt., blocked, fail, check	fix $0.875$	8	0	bound
recompute, blocked, no fail, version	fix $1.0$	36	0	bound
recompute, invalidated, no fail, late	fix $1.0$	15	2	bound

Appendix

Table S10. Action-benefit table (development records only). Entries are action, situation, mandatory-fail status, and issue kind. “Bound”: worst-case valid-before rate (fewer than five valid-before records).

Model	Arm	Valid	Inv./rel.	Calls	Tokens
Grok 4.3	Halt+Agent	48	0/48	0/48	63.0k/2.2k
	Rules+Agent	83	37/120	16/128	184.8k/7.0k
	Risk-Lineage+Agent	100	0/106	36/114	191.3k/7.8k
Gemini 3.6 Flash	Halt+Agent	48	0/48	0/48	55.0k/1.7k
	Rules+Agent	83	37/120	16/128	160.8k/5.4k
	Risk-Lineage+Agent	100	0/106	36/114	166.6k/6.0k

Appendix

Table S11. Post-hoc paid-API addendum (not pooled). Valid yield of 120; calls proposer/reviewer and tokens prompt/completion from the call logs, excluding Gemini’s reasoning tokens (69.1k in total, billed as output); all reviewer verdicts were approvals.

Agentic Workflows

Agentic workflows coordinate AI agents, tools, and humans across multi-step tasks, emphasizing reliable execution, efficiency, and oversight.

See all 218 papers

When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems
May 13, 2026Young Hyun Cho, Will Wei SunAnytime-Valid Inference Sequential Hypothesis Testing
Department of Statistics, Purdue University. · Mitch Daniels School of Business, Purdue University.
Finding Icebergs in Language-Model Workflow: Diagnosing Latent Structural Fragility with Stochastic Semantic Evidence Graphs
Sep 3, 2026Matthew F. Dixon, Bertrand Nortier, Miquel Noguer i AlonsoLLM Safety Evaluation AI Agent Auditing
BeliefLens · AI Finance Institute (AIFI) · University of Leicester +1
Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows
Apr 28, 2026Shivam Rawat, Lucie FlekAgent Failure Analysis AI for Science
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.

May 13, 2026J/K move · Enter open · S save

When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems

Young Hyun Cho, Will Wei Sun

Department of Statistics, Purdue University. · Mitch Daniels School of Business, Purdue University.

Anytime-Valid Inference Sequential Hypothesis Testing LLM Agent Reliability

Open paper AskPDF arXiv

LLM-enabled AI workflows increasingly produce outputs through iterative generate-evaluate-revise loops. Each iteration can improve the candidate, but it also creates a release decision: when to stop and output the current result? This raises a statistical challenge because deployment-time evaluator scores are adaptively generated and repeatedly monitored, yet the likelihood models or exchangeability assumptions typically used for calibration are unavailable. We propose an always-valid release wrapper for existing generator-evaluator pipelines. The wrapper builds a hard-negative reference pool of high-scoring failures, calibrates deployment-time evaluator scores against this pool, and accumulates the resulting evidence with an e-process. This separates two roles: the reference pool turns black-box scores into conservative evidence, while the e-process provides validity under optional stopping. In theory, we show that a conservative reference pool yields finite-sample control of the probability of releasing on infeasible tasks, that is, tasks for which the given workflow is not capable of producing a reliable solution. We also characterize conditions under which the same conservative rule still achieves nontrivial release on feasible tasks. In an MBPP+ coding-agent case study, the wrapper reduces premature incorrect release relative to baseline stopping rules while still releasing on tasks for which the workflow repeatedly accumulates moderate supporting evidence.

Abstract

Agentic Workflows

Explore similar work

When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems

Finding Icebergs in Language-Model Workflow: Diagnosing Latent Structural Fragility with Stochastic Semantic Evidence Graphs

Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows