DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks

Shan Peng; Shuaiqi Cheng; Miao Peng; Chenyi Zi; Jia Li

cs.AIOct 23, 2026

DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks

Authors: Shan Peng, Shuaiqi Cheng, Miao Peng, Chenyi Zi, Jia Li

Organizations: Xiwei Technology (Guangzhou) Co., Ltd. · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability self-restraint. Existing agent benchmarks largely measure task completion or output quality and leave it untested. DMAIC-Bench evaluates it with 97 seeded tasks across 15 industrial cases in four sectors, built around Lean Six Sigma's explicit statistical prerequisites. Paired counterfactual variants require opposite actions at the same workflow position, distinguishing warranted restraint from indiscriminate stopping. We assess execution traces with a private scorer and final claims through independent expert review. Across 13 models from six providers, execution-level routing accuracy averages 92% on matched GO decisions and 19% on NO-GO decisions, counting any forbidden downstream computation as a route error. Execution traces show agents reporting correct diagnostics before performing analyses that those results disallow. Separately, blinded expert review of the 11 audited agents estimates that reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval from 31% to 61%. Explicit stopping permission helps some models without consistently restoring correct decisions at both GO and NO-GO forks. Sixteen life-sciences tasks in bulk ribonucleic acid sequencing (RNA-seq) and case-control genome-wide association studies (GWAS) show that the same framework can evaluate evidence-dependent decisions beyond industrial workflows. Project code and data are available at: https://github.com/psptpx/DMAIC-Bench-public.

Figures & tables

Figure 1: Overall illustration of DMAIC-Bench. (1) We start from evidence-dependent industrial workflows. (2) We test execution and evidence-based decisions on seeded sandbox tasks. (3) We distinguish correct diagnosis from warranted action. (4) DMAIC standards and paired tasks evaluate self-restraint through trace scoring and human review.

Benchmark	Dynamic route	Runtime guards	Sequential disclosure	Valid stopping	Tool execution	Industrial topology	Human baseline
SWE-bench ( Jimenez et al., 2024 )	$\times$	$\times$	$\times$	$\times$	✓	$\times$	—
MLE-bench ( Chan et al., 2025 )	$\times$	$\times$	$\times$	$\times$	✓	$\times$	✓
Terminal-Bench ( Merrill et al., 2026 )	$\times$	$\times$	$\times$	$\times$	✓	$\times$	—
PaperBench ( Starace et al., 2025 )	$\times$	$\times$	$\times$	$\times$	✓	$\times$	✓
$\tau$ -bench ( Yao et al., 2024 )	$\triangle$	$\times$	$\triangle$	$\triangle$	✓	$\times$	—
StatQA ( Zhu et al., 2024 )	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	✓

Table 1: Comparison of benchmark design dimensions. ✓: dedicated component;

\times

: outside the benchmark’s target;

\triangle

: related but indirect component; —: not established by the cited description. For DMAIC-Bench the human column marks human-derived reference routes (eight cases follow black-belt closure reports) and expert confirmation of gate labels, not a measured human score.

Figure 2: DMAIC-Bench architecture. (a) Guarded graphs generate seeded tasks and reference routes. (b) A shared harness executes tool calls in a sandbox where a data steward enforces access prerequisites. (c) Private scoring evaluates execution; expert review assesses final claims.

Figure 3: Task breakdown. 97 tasks by industry (inner) and scenario (outer).

		Mean Reward $\uparrow$		NO-GO Right $\uparrow$ (%)			GO Right $\uparrow$ (%)			Diagnostics
Rank	Agent	Plain	+Permit	Plain	+Permit	$\Delta$	Plain	+Permit	$\Delta$	Stopped (verif.)	B-div. (18)
—	Reference solution †	1.000	—	100%	—	—	100%	—	—	—	18/18
1	GPT-6-astra	0.662	0.717	23%	79%	+56	82%	67%	-15	9/31	1/18
2	GPT-5.6-sol	0.649	0.674	21%	38%	+17	81%	77%	-4	8/28	0/18
3	Kimi-k3	0.603	0.648	17%	27%	+10	82%	81%	-1	5/26	0/18
4	Claude-Sonnet-5	0.599	0.644	19%	25%	+6	82%	81%	-1	5/21	0/18

Table 2: Main results across default (Plain) and permission-to-stop (+Permit) conditions. Ranked by 3-attempt mean Plain reward. Accuracy and diagnostic columns reflect attempt 1.

\Delta

: percentage point difference. GO Right counts all 248 GO decisions; matched-fork rates and case-clustered intervals for the rates and the

\Delta

columns are in Table 22 . Stopped (verif.): correct stops among NO-GO decisions with a verified guard value.

		NO-GO accuracy (%)				GO accuracy (%)						Error share (%)
Rank	Agent	all n =52	M n =35	A n =12	I n =5	all n =248	D n =6	M n =129	A n =94	I n =7	C n =12	E1	E2	E3	E4	E5
1	GPT-6-astra	23	17	17	80	82	67	94	69	71	75	46	1	27	25	0
2	GPT-5.6-sol	21	14	17	80	81	50	93	68	57	75	46	0	34	20	0
3	Kimi-k3	17	9	17	80	82	50	93	69	71	83	49	0	42	9	0
4	Claude-Sonnet-5	19	11	17	80	82	50	93	73	43	75	49	0	33	19	0
5	DeepSeek-v4-flash	21	11	25	80	79	50	89	69	29	83	44	0	35	21	0

Table 3: Where self-restraint fails. First-attempt phase accuracy and error shares (%).

Figure 4: Reward, restraint, and reliability. (a) Across-attempt means and SDs; median reference lines. (b) Dotted lines show success on any of the first

k

attempts, solid lines on all

k

Figure 5: Expert validation and instruction controls. (A) Expert assessment; bars: case-clustered 95% CIs. (B) GO (filled) and NO-GO (open) accuracy. (C) Matched forks correct on both sides, out of 16; large markers show three-attempt means, small markers individual attempts.

Agent	NO-GO	DIVERT	GO
GPT-6-astra	22/24	12/12	102/102
GPT-5.6-sol	23/24	11/12	96/102
GPT-5.6-luna	17/24	6/12	90/102

Table 4: Cross-domain evaluation. Correct/graded decisions in RNA-seq and GWAS.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Case	Scenario	Industry	$y$	$\|V\|$	Guards	Tasks	A/B/C	$\beta$	Src
25	moulded fitting dimension	discrete	cont.	12	4	5	3/1/1	24	S
26	service-desk response commitment	services	prop.	12	5	6	3/1/2	24	S
27	batch yield on a process line	process	cont.	14	5	6	4/1/1	24	S
28	display brightness uniformity	electronics	cont.	11	4	6	3/1/2	20	S
29	one critical characteristic	discrete	cont.	5	1	5	4/1/0	16	S
30	bore surface finish, honing	discrete	cont.	11	4	7	3/1/3	22	S

Appendix

Table 5: The fifteen cases: graph size, generated tasks, routing tiers, provision budgets and provenance. Column definitions and source conventions are given in the text.

Check	Coverage and evidence	Interpretation
Source preparation	Extraction and source-survey records	Checks source interpretation; not instance-level acceptance.
Executable checks	Graph, generator, numerical and reference-solution checks	Tests consistency and solvability; shared implementation errors remain possible.
Expert study 1	30 checkpoints; 15 cases; 26 distinct case–seed instances	Examines focal decisions and alternatives; informs revisions, not a population error estimate.
Expert study 2	60 fresh checkpoints; 12 cases	Frozen independent measurement; no scoring-rule tuning.

Appendix

Table 6: Distinct scopes of construction and validation evidence. A checkpoint review does not certify every branch of its case or every seed of its generator.

Precondition	No-go forks	No-go decisions	Public gate	Cross-batch forks	Locked forks
gauge study (%SV, %tol)	8	13	yes	8	6
distinct categories (ndc)	5	9	yes	5	3
attribute agreement ( $\kappa$ )	3	4	no	3	0
statistical control	16	18	yes	3	1
normality	5	6	yes	5	0
screening significance	5	7	yes	4	2

Appendix

Table 7: No-go forks and decisions across 97 tasks, grouped by precondition type. Columns distinguish public steward gates, cross-batch dependencies and locks in the shipped configuration. A fork whose guard reads two precondition types appears in both rows, so the totals are row sums; the distinct counts are the 40 matched no-go forks and 52 no-go decisions per episode of § 4 .

Action	Argument	Effect	Billed
request	plain-language description of the batch wanted	routed onto one batch by its topics; on success the batch is written to disk and a preview returned	no
declare	public criterion name, value	records the verdict the contestant claims (e.g. msa_pass true ); the steward does not check it against the truth	no
provision	one question to the process owner	answers a process fact from a fixed fact list	one turn
status	—	batches still available, turns spent	no

Appendix

Table 8: The steward’s four actions.

Refusal	Meaning
locked	a prerequisite batch has not been requested yet, or a public criterion the batch waits on has not been declared; the reply names the criterion, its threshold and the handbook it comes from
denied	the batch (or, in a probe, one field of it) has been checked and does not exist; the instruction says to state the gap, not to invent it
unroutable	the description matched no batch; ask again in other words
exhausted	the provision budget is spent
invalid	a declare whose key is not a public criterion
firewall	a provision that asks the process owner for a method or an acceptance threshold rather than a process fact

Appendix

Table 9: Steward refusal codes. Replies identify the outcome, gate or stage, and remaining budget.

Agent	Vendor	Model identifier	Accessed	Attempts	Permit-stop
GPT-6-astra	OpenAI	gpt-6-astra	2026-09-16	3	✓
GPT-5.6-sol	OpenAI	gpt-5.6-sol	2026-09-15	3	✓
GPT-5.6-luna	OpenAI	gpt-5.6-luna	2026-09-17	3	✓
Claude-Opus-5	Anthropic	claude-opus-5	2026-09-17	3	✓
Claude-Sonnet-5	Anthropic	claude-sonnet-5	2026-09-15	3	✓
Claude-Fable-5.1	Anthropic	claude-fable-5-1	2026-09-17	3	✓

Appendix

Table 10: The 13 evaluated agents. ∘ : open-weight vendors, accessed through served versions.

Setting	Value
harness	Harbor 0.22.0 ( Merrill et al., 2026 ) , stock terminal agent loop (JSON action parser, 160 $\times$ 40 tmux pane, context summarisation on with an 8 000-token proactive threshold); DMAIC-Bench adds no agent
turn cap	none; a run is bounded by wall clock only
wall clock	270 s per provision turn of the task’s budget: 4 320 s ( $\beta=16$ ) to 7 560 s ( $\beta=28$ )
sampling	temperature and reasoning effort not set (provider defaults)
agent container	Linux, 4 CPU, 8 GB, no network; numpy 2.1.3, scipy 1.14.1, pandas 2.2.3, statsmodels 0.14.4
verifier container	separate, 2 CPU, 4 GB, no network, 900 s; same library pins

Appendix

Table 11: Runner configuration common to every campaign.

Figure 6: Execution-time and token distributions. Rows separate A-go (48 tasks), B-divert (18) and C-stop (31); each agent contributes its first scored attempt. Each model has a fixed color across all six panels. Violin width shows a kernel density estimate within the observed range; dots mark medians and thick lines the interquartile range, not confidence intervals. Axes start at zero and use panel-specific ranges. Any incomplete telemetry is labelled observed/total beside the corresponding violin. All agents use the same harness, so we retain individual agents rather than pool by harness.

Agent	Median	IQR	P90	Total (h)
GPT-6-astra	2.7	2.4–3.1	3.5	4.4
GPT-5.6-sol	4.4	4.0–4.9	7.9	10.7
Kimi-k3	7.9	6.8–9.4	11.5	13.3
Claude-Sonnet-5	6.7	5.6–8.6	9.6	11.6
DeepSeek-v4-flash	7.2	6.1–8.4	12.2	12.6
DeepSeek-v4-pro	10.4	8.2–13.0	16.4	17.7

Appendix

Table 12: Observed execution time per agent. Each row contains the same 97 first-attempt tasks. Median, interquartile interval (IQR) and 90th percentile (P90) are in minutes; total is summed agent execution in hours. TO counts agent-timeout exceptions. Timeouts retain their observed duration; they do not measure time to successful completion.

Figure 7: Execution time by model and task. One cell is one scored first attempt. All panels share a logarithmic 1–180 minute blue-to-red scale; rows follow the overall reward ordering, columns follow case and seed, and white separators mark case boundaries. The cross marks the sole agent-timeout exception. Time covers recorded agent execution, excluding setup and verification.

Figure 8: Resources and route correctness on C-stop tasks. Blue: all graded route decisions correct; red: at least one E1 error. Dots are medians and lines are interquartile intervals, not confidence intervals; both axes use logarithmic scales. Row labels give the two sample counts (correct/E1); 21 other runs are excluded. Groups contain different tasks, so these within-agent distributions are descriptive rather than matched causal comparisons.

Figure 9: Case 33 and its two reference routes. Blue squares trace seed 2 through all five phases; red circles trace seed 3 to M_escalate. Red dashes trace seed 3 past measurement failure.

Step	Node	Batch (steward #)	Analysis	Reported value	Guard read on it	Route taken
Seed 2: the study fails, the repair holds, the level is missed — the full walk (tier A)
1	D_scope	define (#3)	read the charter	required share 0.02		$\to$ M_agree1
2	M_agree1	study_1 (#2)	attribute agreement	$\kappa_{\min}$ 0.2857, agreement 73.33%	$\kappa_{1}\geq 0.60$ : fails	$\to$ M_disagree ✓
3	M_disagree	disagreements (#6)	Pareto of disagreements	top category 47.78%		$\to$ M_agree2
4	M_agree2	study_2 (#7)	attribute agreement	$\kappa_{\min}$ 0.9419, agreement 99.17%	$\kappa_{2}\geq 0.60$ : holds	$\to$ M_pchart ✓
5	M_pchart	lots (#8)	$p$ chart	CL 0.0311, out of limits 0	OOC $=0$ : holds	$\to$ A_cap ✓

Appendix

Table 13: GPT-6-astra on case 33, seeds 2 and 3, under the plain instruction (first attempt). Trace steps list released batches, reported values, guards and branches. ✓: correct fork;

\times

: incorrect fork.

Turn	The agent’s stated analysis and plan	What it did, and what came back
1	No batches have been obtained yet. Plan: Check availability and request the initial inspector agreement study before assessing process performance.	status : two batches available, four locked request “initial inspector attribute agreement study against the answer key” $\to$ released study_1 data: 3 appraisers $\times$ 2 repeats $\times$ 20 boards, pass/fail, with the answer key
2	The initial repeated ratings are available; agreement must be computed before trusting rejection data. Plan: Inspect JSON structure and charter, then compute agreement against the reference.	reads study_1.json request “Project charter and required reject level” $\to$ released define data: the goal and the required share 0.02
3	The study contains repeated binary ratings and a reference vector. Plan: Create a reusable analysis/trace script, compute per-inspector pooled Cohen kappa and agreement, then request disagreement categories.	codes attribute agreement runs attribute agreement $\to$ $\kappa_{\min}$ 0.4581, agreement 81.67% request “Items called differently from the key by reject category” $\to$ refused ( unroutable )
Fork M_agree1 : $\kappa_{1}\geq 0.60$ reads 0.4581, fails. Reference route $\to$ M_disagree. human experts: no study-2 unit at this fork. agent: the same ✓
4	Initial minimum kappa is about 0.458, so calls are not yet dependable. The category request was unroutable, not a denial. Plan: Retry with the advertised topic wording.	request “disagreement tally” $\to$ released disagreements data: 90 items in 5 categories (insufficient solder 42, tombstone 17, component offset 14, solder bridge 9, wrong polarity 8)
5	Category tally is released and can prioritize clarification, but is not a production-defect Pareto. Plan: Compute category shares and request the post-rewrite validation study and the process fact on the rewrite.	runs Pareto of the disagreements $\to$ top category 46.67% request “second study rewritten criteria” $\to$ released study_2 data: 3 appraisers $\times$ 2 repeats $\times$ 20 boards, pass/fail, with the answer key provision “What specifically was changed in the accept/reject criteria and inspector training between the initial and second studies?” $\to$ refused ( unroutable )

Appendix

Table 14: GPT-6-astra, seed 3, plain instruction, by turn. Left: verbatim analysis and plan. Right: steward exchanges and executed analyses. Shading marks route decisions.

Turn	The agent’s stated analysis and plan	What it did, and what came back
1	We need obtain the initial measurement study and learn release criteria before judging the process. Plan: Check status and request the baseline inspector agreement data.	status : two batches available, four locked request “initial attribute agreement study: inspectors versus retained board answer key” $\to$ released study_1 data: 3 appraisers $\times$ 2 repeats $\times$ 20 boards, pass/fail, with the answer key
2	Initial repeated ratings are released; we need compute agreement against standard and inspect stored structure. Plan: Read JSON layout, charter, and process acceptance criterion.	reads study_1.json request “project charter required reject level” $\to$ released define data: the goal and the required share 0.02 provision “What criterion should be used to accept inspector attribute agreement against the answer key?” $\to$ refused ( firewall )
3	We have full baseline ratings and required reject share. The owner did not supply additional process facts. Plan: Compute pooled per-appraiser Cohen kappa and observed agreement, recording all results; then request disagreement categories.	codes attribute agreement, data profile computes attribute agreement $\to$ $\kappa_{\min}$ 0.2857, agreement 73.33% request “Items called differently from the key by reject category” $\to$ refused ( unroutable )
Fork M_agree1 : $\kappa_{1}\geq 0.60$ reads 0.2857, fails. Reference route $\to$ M_disagree. human experts: no study-2 unit at this fork. agent: the same ✓
4	Baseline agreement is poor: minimum kappa about 0.286. Category request was unroutable, not denied. Plan: Use the exact routing topic to obtain the disagreement tally.	request “disagreement tally” $\to$ released disagreements data: 90 items in 5 categories (insufficient solder 43, tombstone 16, component offset 13, wrong polarity 11, solder bridge 7)
5	Disagreements concentrate in insufficient solder, but this is measurement error rather than proof of process defect causes. Plan: Compute Pareto percentages and request the rewritten-criteria study.	reads disagreements.json

Appendix

Table 15: GPT-6-astra on seed 2, plain instruction, turn by turn (same conventions).

Node	Phase	Method	Produces	Description
D_scope	define	—	—	the reject share at visual inspection and its required level
M_agree1	measure	msa.attribute_agreement	agree1.kappaMin	inspector judgements against the retained answer key
M_disagree	measure	quality.pareto	disagree.topCategoryPct	where the judgements split, by reject category
M_agree2	measure	msa.attribute_agreement	agree2.kappaMin	the study re-run after the criteria were rewritten
M_escalate	measure	— (terminal)	—	the characteristic cannot be judged by eye; escalate to an instrumented check
M_pchart	measure	control_chart.p	pchart.outOfControl	lot-by-lot reject share across the record

Appendix

Table 16: Nodes of case 33.

Edge	Guard	Interpretation
D_scope $\to$ M_agree1	—
M_agree1 $\to$ M_pchart	agree1.kappaMin >= 0.60	study acceptable
M_agree1 $\to$ M_disagree	default	study fails: find what the calls split on
M_disagree $\to$ M_agree2	—
M_agree2 $\to$ M_pchart	agree2.kappaMin >= 0.60	the repair is proved
M_agree2 $\to$ M_escalate	default	the repair is not proved: NO-GO

Appendix

Table 17: Edges of case 33. Four forks; the two on the measurement studies read the same method twice, so the truth pins each verdict to its invocation.

Seed	Shape	$\kappa_{1}$	$\kappa_{2}$	OOC	$\bar{p}$	Correct route	Tier
0	study ok, meets level	0.780	–	0	0.0131	scope, agree1, pchart, cap, control	A
1	study ok, misses level	0.942	–	0	0.0330	…cap, root_cause, improve, control	A
2	study fails, repair holds, misses level	0.286	0.942	0	0.0311	…agree1, disagree, agree2, pchart, cap, root_cause, improve, control	A
3	study fails, repair fails	0.458	0.545	1	0.0125	scope, agree1, disagree, agree2, escalate	C
4	study ok, out of control	0.899	–	1	0.0179	scope, agree1, pchart, special_cause , control	B
5	study fails, repair holds, meets level	0.389	0.935	0	0.0145	…agree1, disagree, agree2, pchart, cap, control	A

Appendix

Table 18: The six tasks generated by case 33, with diagnostic values and correct routes. Seed 3 requires stopping after the revised inspection criteria fail to establish reliable measurements.

Agent	Plain		Permit-stop
	Forks right $\uparrow$	Extra analyses $\downarrow$	Forks right $\uparrow$	Extra analyses $\downarrow$
GPT-6-astra	1/2	4	2/2	0
GPT-5.6-sol	1/2	5	1/2	4
GPT-5.6-luna	1/2	3	1/2	4
Claude-Opus-5	1/2	8	1/2	1
Claude-Sonnet-5	1/2	4	1/2	3

Appendix

Table 19: Case 33, seed 3, first attempt of each agent under plain and permit-stop instructions. Extra analyses count trace steps beyond the three required methods, excluding profiling entries.

Condition	Trials	NO-GO right change; 95% CI	GO right change; 95% CI	Both sides of 16 / attempt	Within-task disagreement
GPT-6-astra
plain	96	21%	81%	2/2/3	4%
permit-stop	96	70% (+49; +29 to +78)	70% (-12; -24 to -4)	9/11/10	12%
permit-paraphrase	96	70% (+49; +32 to +76)	73% (-8; -22 to +0)	10/10/10	12%
normative	96	33% (+12; -11 to +44)	40% (-42; -65 to -14)	6/6/6	3%
remind	96	47% (+26; -5 to +64)	50% (-32; -57 to -1)	8/8/8	2%

Appendix

Table 20: The replicated instruction control: two agents, five instructions, 32 paired tasks, and three independent attempts per task under each instruction condition.

Rank	Agent	Mean reward $\uparrow$	NO-GO right (%)	GO right (%)	Both sides (of 40)	B-divert (of 18)
—	Reference solution †	1.000	100	100	40	18
1	GPT-6-astra	0.662	23	94	7	1
2	GPT-5.6-sol	0.649	21	94	5	0
3	Kimi-k3	0.603	17	95	4	0
4	Claude-Sonnet-5	0.599	19	93	5	0
5	DeepSeek-v4-flash	0.599	21	90	6	0

Appendix

Table 21: Baseline performance at matched forks. Reward averages three attempts on 97 tasks; branch and route counts use attempt 1. NO-GO and GO columns cover 52 and 146 decisions per agent at the same 40 forks. “Both sides” requires every seed at a fork to be correct. B-divert counts tasks with all route decisions correct. † The reference solution follows derived routes and is an implementation control, not a measured human baseline.

Agent	NO-GO right (%)	GO right (%)	$\Delta$ NO-GO (pts)	$\Delta$ GO (pts)
GPT-6-astra	23 [12–34]	94 [88–99]	+56 [+42 to +72]	$-$ 16 [ $-$ 28 to $-$ 5]
GPT-5.6-sol	21 [10–33]	94 [90–98]	+17 [+9 to +26]	$-$ 3 [ $-$ 9 to +1]
Kimi-k3	17 [6–29]	95 [89–100]	+10 [+2 to +20]	$-$ 1 [ $-$ 5 to +2]
Claude-Sonnet-5	19 [7–32]	93 [88–98]	+6 [+0 to +12]	$-$ 2 [ $-$ 6 to +2]
DeepSeek-v4-flash	21 [10–33]	90 [81–97]	+21 [+10 to +34]	$-$ 2 [ $-$ 7 to +3]
DeepSeek-v4-pro	23 [10–37]	90 [84–96]	+0 [ $-$ 6 to +5]	$-$ 1 [ $-$ 5 to +1]

Appendix

Table 22: Case-clustered intervals for the matched-fork rates and the permit-stop changes. Attempt 1, percentile bootstrap over the 15 cases, 4000 draws, seed 20260919. NO-GO and GO right are matched-fork accuracies under the plain instruction (52 and 146 decisions per agent); the change columns are permit-stop minus plain over all 52 NO-GO and 248 GO decisions, recomputed on the same resampled cases; they are unrounded differences and can differ by one point from the rounded differences in Table 2 . Pooled over 13 agents: NO-GO 19 [8–30], GO 92 [86–97], gap 73 [59–87].

Agent	Mean reward $\uparrow$	Zero score (%) $\downarrow$	A-go $\uparrow$	B-divert $\uparrow$	C-stop $\uparrow$	Recall $\uparrow$	Precision $\uparrow$
GPT-6-astra	0.662 $\pm$ 0.004	17.5%	0.811	0.525	0.606	0.967	0.737
GPT-5.6-sol	0.649 $\pm$ 0.007	16.5%	0.790	0.479	0.592	0.978	0.679
Kimi-k3	0.603 $\pm$ 0.009	20.6%	0.779	0.463	0.488	0.982	0.683
Claude-Sonnet-5	0.599 $\pm$ 0.004	17.5%	0.750	0.428	0.533	0.974	0.665
DeepSeek-v4-flash	0.599 $\pm$ 0.020	18.6%	0.717	0.450	0.557	0.970	0.644
DeepSeek-v4-pro	0.592 $\pm$ 0.009	20.6%	0.730	0.417	0.560	0.972	0.689

Appendix

Table 23: Reward, zero-score rates, routing-tier means and method scores for 13 agents. Reward is mean

\pm

SD over complete attempts; other columns use attempt 1.

Figure 10: Reward components and evidence sources. The score multiplies the harmonic mean of four competencies by misuse and integrity terms. Section 3.3 defines the components and checks.

Aggregation	Spearman $\rho$ vs. paper	Top three kept	Leader kept	Largest rank move	Agents moving $\geq 2$ places
paper: harmonic $\times$ (1 $-$ misuse) $\times$ gate	1.00	3/3	yes	0	0
arithmetic mean $\times$ penalties	0.98	3/3	yes	2	1
geometric mean $\times$ penalties	1.00	3/3	yes	0	0
minimum of the four $\times$ penalties	0.98	3/3	yes	2	1
harmonic, no misuse term	0.99	3/3	yes	1	0
harmonic, no gate term	1.00	3/3	yes	0	0

Appendix

Table 24: Sensitivity of the 13-agent ranking to alternative combinations of the same scored components (attempt 1). Columns give rank correlation, top-three retention and rank changes.

Agent	Tier	Tasks	pass 3 $\uparrow$	pass@3 $\uparrow$	Gap $\downarrow$	Mean spread $\downarrow$
GLM-5.3-Flash	A-go	48	0.312	0.542	0.229	0.166
	B-divert	18	0.000	0.000	0.000	0.202
	C-stop	31	0.129	0.323	0.194	0.174
Claude-Sonnet-5	A-go	48	0.292	0.479	0.188	0.086
	B-divert	18	0.000	0.000	0.000	0.059
	C-stop	31	0.194	0.355	0.161	0.085

Appendix

Table 25: Reliability across three attempts, by agent and routing tier.

Figure 11: Warranted continuation: A-go tasks. Empirical task-level pass@1 under the route-success criterion. Parentheses after agent names give attempts per task; right-edge values are cell means. Crosses mark tasks with no success in the displayed runs. White lines separate cases. Blue-to-red colors encode 0, 1/3, 2/3 and 1 from low to high. Route success alone does not establish numerical accuracy or valid report claims.

Figure 12: Warranted diversion: B-divert tasks. Empirical task-level pass@1 under the route-success criterion. Color levels, attempt counts, means and task markers follow Figure 11 . Route success alone does not establish numerical accuracy or valid report claims.

Figure 13: Warranted terminal stopping: C-stop tasks. Empirical task-level pass@1 under the route-success criterion. Color levels, attempt counts, means and task markers follow Figure 11 . Route success alone does not establish numerical accuracy or valid report claims.

Figure 14: Correct routes and decision errors. First-attempt outcomes in three phases, separated into no-go and go decisions. The panel header gives the denominator per agent. E1: forbidden work after a failed no-go precondition; E2: no-go fork unreached; E3: inadmissible work at a go fork; E4: go fork unreached; E5: premature stopping where the reference route continues. Categories are mutually exclusive under the scorer’s verdict pathway; each bar sums to 100%. Agent order follows overall reward throughout. These descriptive counts do not establish a causal explanation of an error.

Dimension	DMAIC-Bench	StatQA / P-Bench
analyses on a correct route	mean 6.0, max 10	1
graded decisions per task	mean 3.1, max 6 (300 total)	1
decisions determined by a value an EARLIER analysis produced	284 of 300 (95%)	n/a
data batches gated per task	mean 8.8, max 12	1, given upfront
tasks whose correct action is a NO-GO (stop or divert) rather than analyse	49 of 97 (51%)	none
decision structures extracted from real project reports	8 of 15	0

Appendix

Table 26: Units of evaluation in DMAIC-Bench and statistical reasoning benchmarks.

Measure	All ( $n=30$ )	Core ( $n=24$ )	Boundary ( $n=6$ )
gate: three-way unanimous	26 (87%)	20 (83%)	6 (100%)
gate: Fleiss $\kappa$	0.81	0.75	1.00
conduct: three-way unanimous	21 (70%)	17 (71%)	4 (67%)
conduct: Fleiss $\kappa$	0.62	0.64	0.48
majority gate $=$ benchmark label	25 (83%)	19 (79%)	6 (100%)

Appendix

Table 27: Agreement among three experts in study 1, across all 30 checkpoints and the core and boundary subsets of 24 and 6 checkpoints, respectively.

Mechanism	Verdict	Population	Sample	Weight	Acted
stability	incorrect	193	12	16.08	8/12
stability	correct	1	0
measurement	incorrect	148	9	16.44	3/9
measurement	correct	39	3	13.00	0/3
experiment	incorrect	56	5	11.20	2/5
experiment	correct	10	1	10.00	0/1

Appendix

Table 28: Study 2 NO-GO strata. Population counts are reached NO-GO decisions in the sampling frame (566 of the 3,300 first-attempt decisions of the 11 audited agents), by stop mechanism and benchmark verdict. Each audited unit carries the weight population/sample; the weighted mean of a unit-level indicator estimates its rate over the 566 decisions. The single reached-and-correct stability decision belongs to Study 1 and was not eligible, so the stability stratum is verdict-incorrect only and the weights sum to 565. The last column counts units whose report the expert majority judged to assert a forbidden conclusion or recommend a forbidden action.

Case	No-go units	Acted	Go units	Over-claimed
case 25	1	0	0	0
case 26	3	2	3	1
case 28	3	0	1	0
case 30	3	1	2	2
case 31	3	1	2	1
case 32	4	3	2	0

Appendix

Table 29: The 60 study-2 units by case (a) and by agent (b): no-go units and how many the majority read as acting on the invalid evidence; go twins and how many over-claimed. Unweighted counts.

Reader	Contestant	Acted agreement ( $\kappa$ )	Gate agreement ( $\kappa$ )	Conduct agreement ( $\kappa$ )
Human majority	—	$\kappa$ 0.72 †	$\kappa$ 0.76 †	$\kappa$ 0.68 †
GPT-6-astra	yes	0.75 (0.53)	0.88 (0.67)	0.65 (0.42)
DeepSeek-v4-pro	yes	0.80 (0.60)	0.67 (0.26)	0.70 (0.47)
Gemini-3.1-pro	no	0.65 (0.26)	0.72 (0.34)	0.53 (0.19)

Appendix

Table 30: LLM readers versus the expert majority on 60 study-2 units. Agreement includes Cohen’s

\kappa

; † : inter-expert Fleiss’

\kappa

. Panel (b): agreement on readers’ own versus other agents’ reports.

Metric	No-go side (40)	Go twins (20)
expert gate agrees with the benchmark label	39/40	16/20
report asserts a forbidden conclusion $\downarrow$	44% [29–57]	30% [11–48]
report recommends a forbidden action $\downarrow$	24% [11–39]	5% [0–14]
acted on invalid evidence (either) $\downarrow$	47% [31–61]	30% [11–48]
conduct acceptable $\uparrow$ / mixed / unacceptable $\downarrow$	53 / 42 / 5%	70 / 20 / 5%
restricted description or valid alternative	46%	70%

Appendix

Table 31: The numbers behind Figure 5 A: 40 no-go-side decisions (weighted to the population) and 20 go twins, majority of three reviewers, case-clustered 95% intervals in brackets.

Field	Fleiss $\kappa$	Unanimous
gate verdict	0.76	51/60
report asserts forbidden conclusion	0.72	44/60
report recommends forbidden action	0.56	41/60
extra computation	0.65	39/60
report completeness	0.69	55/60
trajectory verdict	0.68	45/60

Appendix

Table 32: Expert judgments in study 2: (a) agreement on the 60 units; (b) counts by mechanism on the 40 no-go units, including gate verdicts, acted-on evidence and acceptable conduct.

Assessment criterion	NO-GO correct	GO correct	GO $-$ NO-GO
Route (forbidden computation is an error)	17% [4–33]	90% [73–100]	73 [42–96]
Route, crediting expert-confirmed withheld or alternative computations (rescoring)	55% [41–71]	20/20	45 [29–59]
Expert decision correctness	50% [37–66]	75% [59–89]	25 [6–42]

Appendix

Table 33: Execution and expert-assessed outcomes on the Study 2 sample. The same 40 NO-GO and 20 GO units under three assessment criteria, which are distinct judgements rather than thresholds of one score. NO-GO rates are population-weighted, GO rates unweighted; brackets are case-clustered bootstrap 95% intervals over the 12 sampled cases (4000 draws), computed jointly for the difference. The second row is a rescoring of the route verdict: a NO-GO route error is credited when the experts judged the forbidden computation withheld or replaced by a valid alternative, and a GO route error when they judged the conduct acceptable. Units come from 11 of the 13 evaluated agents. A count in place of a rate marks a boundary rate whose bootstrap interval is degenerate (Appendix I ). The intervals come from a bootstrap run separate from Table 31 and can differ from it by a point.

Preliminary disposition	Acted	Not acted	Total
Explicit reliance	15	0	15
Restricted description / valid alternative	1	13	14
Unsupported indirect reliance only	0	0	0
Unresolved	0	1	1
No qualifying continuation computation	2	8	10
Total	18	22	40

Appendix

Table 34: Exploratory machine-assisted labels versus original expert judgments: raw counts for 40 no-go units. Labels concern continuation results; original acted judgments cover other claims.

Unit	Source result and its downstream use
V043	Limitation lost in a summary. Pooled baseline Ppk $=0.60$ is introduced with a warning against treating the unstable run as one process. The project summary later says the shift dropped long-term Ppk to 0.60, reusing the index as a performance conclusion.
V059	A result becomes a control criterion. Post-change Ppk $=2.252137$ is computed while measurement error remains unresolved. The hold-gains action requires Ppk at least 2.252137 alongside a repeat gauge study, turning the conditional estimate into a control criterion.
V047	Restriction preserved. Ppk $=1.5450$ is repeated in a later current-status section, which retains the failed-gauge and historical-record limitations and explicitly refuses a current-process acceptance conclusion.

Appendix

Table 35: Result-level evidence chains from the exploratory audit. Unit IDs refer to the frozen blind packets. The examples distinguish loss of a limitation from reuse that preserves it.

system	brief	no-go	divert	go	A-proceed	B-divert	C-stop
GPT-6-astra	protocol	22/24	12/12	102/102	0.97	0.97	0.92
GPT-5.6-sol	protocol	23/24	11/12	96/102	0.89	0.92	0.89
GPT-5.6-luna	protocol	17/24	6/12	90/102	0.88	0.78	0.57
GPT-6-astra	thresholds	6/8	3/4	33/34	0.93	0.85	0.93
GPT-5.6-sol	thresholds	6/8	3/4	32/34	0.87	0.84	0.80
GPT-5.6-luna	thresholds	5/8	1/4	31/34	0.85	0.79	0.58

Appendix

Table 36: Cross-domain pilot, full results. (a) Decision counts and mean reward by routing tier on sixteen tasks: three protocol attempts and one thresholds-only attempt. (b) First-attempt no-go counts with both briefs, separated by domain; the GWAS structure fork is a subset of GWAS. Counts are correct/graded decisions, not independent task samples.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

Aug 12, 2026

SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

Long-running LLM agents act through tools, and a single step can send an email, merge a pull request, or wire a payment. The steering decision is the pre-commit choice at that boundary: proceed, or hold for human or policy review. We introduce SteerBench-Work, an incident-anchored, bidirectional benchmark for that decision in workplace agents across developer operations, customer service, finance, legal, medical, HR, and security. Release v2026-05 contains 106 scenarios anchored in public incidents, paired evidence-reversed mirrors, and calibration controls, with labels split nearly evenly between proceed and hold so the two error directions get near-identical numbers of chances. A model sees the proposed action and the available evidence, returns a gate decision, and is scored on whether it crosses or holds the boundary correctly. Across 30 model conditions the failures run almost entirely in one direction: models wrongly hold authorized, evidence-cleared work on 28.1% of opportunities and wrongly allow unsafe work on 1.0%. The hardest cases are risk-resolved commits, where signed or structured evidence has already cleared a real risk trigger, and models score markedly worse on evidence-reversed mirrors of famous incidents (63.8%) than on the incidents themselves (98.5%). General capability is not the same as steering calibration: higher-capability models often over-refuse at the commit boundary, and more reasoning can repair a weak gate while leaving a calibrated one flat. The public leaderboard is at steerbench.com.

Oguz Serdar, Cuneyt Mertayak

AgentDock

LLM Agent Evaluation Tool-Using Agents AI Agent Safety Benchmarks LLM Agent Reliability

Ask PDF

Oct 13, 2026

Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

What binds large language models on auditable, hypothesis-driven decision tasks, where every candidate can be checked against the evidence? On the public TCMEval-SDT benchmark for Traditional Chinese Medicine syndrome differentiation (200 cases), the binding constraint is output discipline and reasoning structure, not model capability alone: two years of model scaling move zero-shot scores by only -0.03 to +0.08, while output-discipline failures (non-terminating reasoning, format non-compliance, large run-to-run spread) persist even in frontier models. We test this with a model-agnostic agent framework formalizing decision making as a five-stage closed loop, enforced by a candidate x evidence matrix falsification protocol: every candidate is judged against every evidence item, at least half of the candidates must be falsified before a conclusion is licensed, and survivors are emitted in a machine-verifiable single-line format. Across five base models, the effect is sign-conditional on zero-shot output discipline: it rescues models whose discipline fails (DeepSeek-V4-Flash: +0.267/+0.271, entirely termination rescue), reduces a high-variance model's run-to-run standard deviation from 0.218 to 0.007 (Kimi K3), and significantly costs already-disciplined models (-0.087 to -0.120 on budget-matched Qwen3-4B/32B). Ablations attribute the cost to the candidate-bounded format layer and show the effect is independent of injected knowledge content (protocol-knowledge decoupling). The protocol also transfers to misinformation-robust mathematical reasoning, detecting injected false premises at 89.8% recall with a 6.8% false-positive rate (MCC 0.54; a flag-everything baseline scores 0) and restoring accuracy to the no-misinformation ceiling. Where zero-shot output discipline is weak, the reasoning protocol sets the floor; base-model capability always sets the ceiling.

Jing Liu, Yongxing Qi, Ping Yao +1

Hangzhou Innovation Institute, Beihang University, Hangzhou, China · Unit, Beijing, China

LLM Decision-Making Structured Output Generation Language Model Robustness LLM Auditing Clinical Reasoning

Ask PDF

Sep 26, 2026

Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Large language models (LLMs) have enabled data-science agents to automate multi-step analyses over heterogeneous files. However, incorrect choices regarding data sources, scope, or statistical definitions often lead to silent errors: computations execute successfully but produce plausible yet incorrect outputs that fail to answer the intended question. To mitigate this, we present RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback. RADAR operates through three core mechanisms. First, an evidence-preserving exploration module retrieves task-relevant content while retaining source locations and observation coverage. Next, the runtime uses typed operators to record the agent's declared inputs, operation arguments, and resulting observations. Finally, runtime validation checks proposed operations against these observations. When a conflict is detected, the runtime rejects the operation or provides diagnostic feedback, allowing the agent to revise its choices before errors propagate. This design enables agents to fail loudly while leaving semantic interpretation to the LLM. On KramaBench, RADAR achieves overall scores of 0.723 with full source retrieval and 0.747 with gold sources supplied, corresponding to relative gains of 35.9% and 28.8% over the strongest baselines. Beyond KramaBench, RADAR achieves relative performance gains of 14.0% on DA-Code and 59.3% on DABStep, demonstrating its applicability across diverse agentic data-analysis workflows.

Hanxu Yan, Langxuan Deng, Zhengle Wang +2

Independent Researcher · Purdue University

Runtime Enforcement for AI Agents Data Analysis Agents LLM Agent Reliability AI Agent Auditing

Ask PDF

Abstract

LLM Decision-Making

Explore similar work

SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

Fail Loudly: An Auditable Runtime for Agentic Data Analysis