AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

Kaiyu Huang; Yiting Li; Angyang Gu; Pinjia He; Minchen Yu; Qingjiang Shi

cs.AIOct 13, 2026

AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

Authors: Kaiyu Huang, Yiting Li, Angyang Gu, Pinjia He, Minchen Yu, Qingjiang Shi

Organizations: Tongji University · The Chinese University of Hong Kong, Shenzhen

Abstract

Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these information gaps remain poorly characterized across models and execution harnesses. We present AM-Bench, a benchmark built from 48 tasks adapted from Terminal-Bench 2.0 and Harness-Bench. Each task includes a fully specified reference and six variants spanning ambiguous or missing goals, context, and constraints. We evaluate every pairing of five models and four harnesses over 20,160 runs. Our results reveal distinct execution deficits: goal gaps hinder completion, context gaps increase token use, and constraint gaps weaken compliance without necessarily preventing completion. Missingness lowers success more than ambiguity in all three domains. These effects also reshape system comparisons: advantages under complete information need not persist, and rankings depend on model $\times$ harness pairings and the evaluation criterion -- completion, success, or repeat consistency. Estimates based on observed clarification requests and matched fully specified runs further suggest that clarification could recover unfinished work while leaving most constraint violations unresolved. Together, these findings motivate joint model-harness evaluation across information conditions and targeted clarification of unresolved requirements before dependent actions. To support future research on agents under task underspecification, we provide the AM-Bench project repository at https://github.com/s4-lab-cuhksz/AM-Bench.git.

Figures & tables

Figure 1: Benchmark pipeline. Each task yields a Full condition and six Goal/Context/Constraint variants. Isolated model $\times$ harness executions interact with case-local tools; traces, artifacts, and action logs support outcome and resource evaluation.

Domain	Full specification	Ambiguous (A)	Missing (M)
Goal (G)	Intended final state, deliverables, and completion requirements.	GA: A broad goal or candidate outcomes allow multiple reasonable interpretations of the final result.	GM: The final goal, deliverable, or completion requirement is unspecified.
Context (C)	Entities, inputs, environment, and operational information for producing or submitting the required result.	CA: Entity, source, or execution cues leave the relevant scope or context open to multiple interpretations.	CM: A broad goal remains, but key entity, source, or operational information is absent.
Constraint (K)	Permitted actions, access and approval rules, and preservation requirements.	KA: Restrictions or conditional permissions leave their scope, applicability, or access boundary open to interpretation.	KM: Relevant restrictions or authorization requirements are unstated.

Table 1: General definitions of the six underspecified conditions. Row colors distinguish Ag Goal (G) , Ag Context (C) , and Ag Constraint (K) ; A denotes ambiguity and M missingness. Full supplies all three types of information.

Figure 2: A Full directory-permission task (ACL: access control list), representative tools, and evaluation evidence. Completion ( $T$ ) checks artifacts, state, and required actions; Compliance ( $S$ ) checks constraints; traces identify clarification requests ( $C$ ). Success requires both Completion and Compliance. Workflow and scoring examples appear in Appendix B.4 .

Cond.	Observed execution				Clarification opportunities
Cond.	Completion	Compliance	Success	Tok. (k)	Completion	Compliance	Success	$\Delta$ Tok. (k)
Full	77.0	98.7	76.5	106.9	—	—	—	—
GA	25.9	94.3	25.0	126.7	32.5 (+6.6)	95.0 (+0.7)	31.6 (+6.6)	+1.6
GM	13.8	92.5	12.5	119.5	38.0 (+24.2)	92.7 (+0.3)	36.6 (+24.1)	+17.8
CA	30.4	91.7	27.8	181.1	36.8 (+6.4)	92.2 (+0.5)	34.4 (+6.6)	-7.7
CM	21.0	91.9	19.4	166.0	30.6 (+9.6)	92.6 (+0.7)	29.1 (+9.6)	-0.6

Table 2: Observed outcomes and clarification opportunities estimated from matched Full runs. Rates are percentages; estimated rates show changes (pp) in parentheses, and

\Delta

Tok. gives token-reference changes (k). Colors group Ag Goal (G) , Ag Context (C) , and Ag Constraint (K) ; A/M: Ambiguous/Missing.

Figure 3: Completion (left) and Success (right) by model and harness (%). Mean excludes Full; bold marks the highest positive value within each group and condition.

Success	Models					Harnesses
Success	DS-Flash	DS-Pro	GPT-Luna	GPT-Sol	GLM-5.3	Codex	NanoBot	NullClaw	OpenClaw
Mean	24.8	19.5	13.6	17.0	23.7	22.2	16.2	19.9	20.7
$\mathrm{pass}@3$	39.9	32.4	21.0	23.9	36.1	36.1	26.0	28.1	32.4
$\mathrm{pass}^{3}$	11.0	8.5	7.1	10.3	12.3	9.7	7.6	12.1	10.0

Table 3: Mean and repeated Success across six underspecified conditions (%).

\mathrm{pass}@3

\mathrm{pass}^{3}

mean success in at least one/all three runs. Within each metric and component group, bold marks the best value and underlining the second best.

Figure 4: Clarification rate

C

(left, %) and estimated Success gain

g

(right, pp). Mean averages six underspecified conditions. Bold marks the highest positive value within each group and condition.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Cond. Grouping	Target	Completion	Efficiency	Risk	Clarification
Agent execution performance
Terminal-Bench 2.0	–	Part. M $\times$ H	$\bullet$	$\bullet$	–	–
Harness-Bench	–	Full M $\times$ H	$\bullet$	$\bullet$	$\bullet$	–
Agent execution risks
ToolEmu	Ungrouped	M	$\bullet$	–	$\bullet$	Rubric
UnderSpecBench	Grouped	Part. M $\times$ H	$\bullet$	–	$\bullet$	Requests

Appendix

Table 4: Comparison of AM-Bench with related agent benchmarks

Code	Workflow	Assignment rule	$n$
DP	Data processing & reports	Aggregate, reconcile, or structure records and tabular data.	8
DM	Documents & media	Synthesize, review, map, or transform documents and media.	7
CP	Communication & planning	Schedule, coordinate, communicate, or produce an ordered work plan.	8
FR	Files & repositories	Extract, transform, retrieve, or check files; produce shell artifacts or repository changes.	8
SD	Service setup & deployment	Initialize or configure permissions, cloud resources, deployments, or services.	4
SR	Software & environment repair	Repair code, dependencies, networking, or processing pipelines.	7

Appendix

Table 5: Primary workflow categories. Each base task is assigned to exactly one class.

Task identifier	Src.	Class	Upstream identifier
analyze_access_logs	TB	DP	analyze-access-logs
bank_trans_filter	TB	DP	bank-trans-filter
financial_document_processor	TB	DP	financial-document-processor
jsonl_aggregator	TB	DP	jsonl-aggregator
log_summary_date_ranges	TB	DP	log-summary-date-ranges
multi_source_data_merger	TB	DP	multi-source-data-merger

Appendix

Table 6: The 48 base tasks, workflows, and recorded sources. TB: Terminal-Bench; HB: Harness-Bench. Workflow codes are defined in Table 5 .

Harness	Version	Settings
Codex	0.144.4	Non-interactive execution; approvals and sandbox bypassed; writable workspace; task MCP server registered by URL.
NanoBot	0.2.2	Host networking; writable workspace; MCP over streamable HTTP with a configured URL allowlist.
NullClaw	2026.5.29	Memory disabled; workspace-only autonomy; allowed Python commands; host networking; MCP over HTTP with a 300-second timeout.
OpenClaw	2026.7.1	Prior configuration, authentication profiles, and sessions cleared before initialization; writable workspace; MCP tool allowlist applied when supplied.

Appendix

Table 7: Harness versions and execution settings. All harnesses access the task-local MCP server through their adapters.

Run	$C$	Excerpt and interpretation
GLM-5.3 $\times$ Codex; access logs, GM, run 1	1	“No concrete goal… Please specify…” Requests the missing objective.
GPT-Sol $\times$ Codex; bank filtering, CM, run 1	1	“Which company should I identify transactions for, and where should I write the sorted JSON output? Please also provide the desired JSON schema or sort key if predefined.” Requests the target, destination, and output format.
GPT-Sol $\times$ Codex; bank filtering, Full, run 1	0	“Completed successfully…” Reports completion without requesting information.
GLM-5.3 $\times$ NanoBot; bank filtering, GM, run 3	1	“To proceed, please provide the three missing pieces…” The complete final message requests the goal, deliverable, and final action after reporting an initial inspection.

Appendix

Table 8: Recorded clarification examples.

C

is the run-level label; excerpts illustrate requests for task information and a completion report. Task identifiers are analyze_access_logs and bank_trans_filter , with the condition and repeat shown below.

Condition	Completed		Not completed		Success change (pp)
	Compliant	Noncompliant	Compliant	Noncompliant
Full	76.5	0.5	22.2	0.8	—
Goal–Ambiguous	25.0	0.9	69.4	4.7	$-51.6$
Goal–Missing	12.5	1.3	80.0	6.3	$-64.0$
Context–Ambiguous	27.8	2.6	63.8	5.8	$-48.7$
Context–Missing	19.4	1.6	72.5	6.5	$-57.1$

Appendix

Table 9: Observed execution outcomes, with 2,880 runs per condition. The four outcome columns partition all runs (%); completed and compliant runs constitute Success. Success change is relative to Full. Colors group Ag Goal , Ag Context , and Ag Constraint conditions.

Condition	Token use (k)			Mean turns	Timeout rate
	Mean	Median	90th percentile
Full	106.9	74.3	195.2	7.22	3.9%
Goal–Ambiguous	126.7	91.1	257.3	7.81	4.5%
Goal–Missing	119.5	58.4	288.9	7.03	6.2%
Context–Ambiguous	181.1	109.8	400.9	10.00	9.5%
Context–Missing	166.0	102.6	374.4	9.28	7.1%

Appendix

Table 10: Resource use by information condition. Token means average positive totals within each configuration and then weight the 20 configurations equally; medians and 90th percentiles pool positive-token runs within each condition. Mean turns and timeout rates use all 2,880 runs per condition. Colors group Ag Goal , Ag Context , and Ag Constraint conditions.

Condition	Alternative goal calls	Context distractor calls	Restricted tool calls	Sensitive data exposure	Missing final action	Failed output checks
Full	4.9	1.2	0.7	0.7	3.6	3.8
Goal–Ambiguous	72.8	12.3	5.2	5.0	26.1	22.0
Goal–Missing	27.2	11.1	6.7	6.4	71.7	68.2
Context–Ambiguous	64.2	75.3	8.1	7.8	23.9	25.0
Context–Missing	67.9	54.3	11.9	11.3	37.7	25.0
Constraint–Ambiguous	51.9	29.6	57.0	54.6	5.1	5.3

Appendix

Table 11: Execution markers (%), Codex

\times

GLM-5.3. Each column retains tasks with complete fields across all seven conditions and three runs. Supports are 27 tasks (81 runs per condition) for alternative-goal and context-distractor calls, 45 (135) for restricted-tool calls, 47 (141) for sensitive-data exposure, 46 (138) for missing final actions, and 44 (132) for failed output checks. Absent fields are excluded; an empty call list indicates no recorded call. Restricted calls and exposure are distinct markers.

Information domain	Completion change (pp)	Compliance change (pp)	Success change (pp)	Mean token change (k)
Goal	$-12.1$	$-1.9$	$-12.5$	$-7.2$
Context	$-9.4$	$+0.3$	$-8.4$	$-15.1$
Constraint	$+5.2$	$-41.9$	$-33.7$	$+12.1$

Appendix

Table 12: Changes from the ambiguous to the missing condition within each information domain. Outcome changes are in percentage points; token changes are differences in configuration-weighted means (thousands). Differences use unrounded values. Colors distinguish Ag Goal , Ag Context , and Ag Constraint .

Condition	Completion		Compliance		Success
	At least once	All three	At least once	All three	At least once	All three
Full	85.2	68.0	99.6	97.2	85.0	67.0
Goal–Ambiguous	41.1	11.3	98.4	89.1	39.7	10.8
Goal–Missing	25.8	3.6	96.8	86.5	23.8	3.4
Context–Ambiguous	45.9	15.9	97.5	84.7	42.9	14.2
Context–Missing	36.1	8.3	98.6	82.9	34.3	7.2

Appendix

Table 13: Repeated-execution outcomes (%), with 960 task

\times

model

\times

harness groups per condition. “At least once” and “All three” correspond to

\mathrm{pass}@3

and

\mathrm{pass}^{3}

for the indicated outcome. Bold and underlining mark the lowest and second-lowest rates. Mean rates appear in Table 2 . Colors group Ag Goal , Ag Context , and Ag Constraint conditions.

Condition	Tasks with lower Success (out of 48)	Median change (pp)	25th–75th percentiles of change (pp)
Goal–Ambiguous	48	$-53.3$	[-58.8, -41.7]
Goal–Missing	47	$-69.2$	[-75.8, -54.6]
Context–Ambiguous	47	$-51.7$	[-65.4, -29.6]
Context–Missing	48	$-57.5$	[-70.0, -44.6]
Constraint–Ambiguous	45	$-35.0$	[-71.7, -21.7]
Constraint–Missing	48	$-86.7$	[-91.7, -68.3]

Appendix

Table 14: Task-level Success changes relative to Full, averaging configurations and runs within each of 48 tasks. The task count reports how many have lower Success. Median and percentile columns summarize the 48 task-specific changes (pp).

Condition	Timeout count	Success excluding timeouts (%)
Full	113	79.2%
Goal–Ambiguous	131	26.1%
Goal–Missing	178	13.2%
Context–Ambiguous	273	30.8%
Context–Missing	204	20.9%
Constraint–Ambiguous	122	35.1%

Appendix

Table 15: Timeout counts and Success after excluding timeout runs. The filtered comparison uses the remaining records and their resulting weights; the primary analysis retains all runs. Colors group Ag Goal , Ag Context , and Ag Constraint conditions.

Component	Success	Not completed		Noncompliant completion
		Timeout	Other
Models
DS-Flash	24.8	0.3	50.7	24.2
DS-Pro	19.5	17.5	47.1	15.9
GPT-Luna	13.6	0.1	64.1	22.1
GPT-Sol	17.0	1.2	59.6	22.2

Appendix

Table 16: Outcome composition across six underspecified conditions (%). Success and the three failure categories partition all runs. Timeout and Other separate non-completion with and without a recorded timeout; completed runs are classified by Compliance regardless of timeout status. Each model uses 3,456 runs and each harness 4,320.

Condition	DS-Flash		DS-Pro		GPT-Luna		GPT-Sol		GLM-5.3
	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$
Full	92.2	69.3	78.6	52.1	87.5	76.0	86.5	81.8	80.2	55.7
Goal–Ambiguous	51.6	15.1	39.6	13.0	24.0	2.1	30.7	12.5	52.6	11.5
Goal–Missing	27.6	1.0	37.5	6.8	13.0	1.0	18.8	6.8	21.9	1.6
Context–Ambiguous	63.5	20.8	42.2	3.6	27.6	13.0	30.2	15.6	51.0	17.7
Context–Missing	58.3	13.5	36.5	7.3	19.8	1.6	14.6	1.6	42.2	12.0

Appendix

Table 17: Repeated Success by condition and model (%).

\mathrm{pass}@3

and

\mathrm{pass}^{3}

denote success in at least one and all three runs. Each entry averages 192 task

\times

harness groups on all 48 tasks. Colors group goal, context, and constraint conditions.

Condition	Codex		NanoBot		NullClaw		OpenClaw
	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$	$\mathrm{pass}@3$	$\mathrm{pass}^{3}$
Full	86.7	75.4	90.0	68.3	84.2	65.0	79.2	59.2
Goal–Ambiguous	42.9	8.3	34.2	7.5	39.2	15.4	42.5	12.1
Goal–Missing	27.9	0.8	15.4	1.7	21.3	5.4	30.4	5.8
Context–Ambiguous	61.3	17.9	35.8	10.8	25.8	7.5	48.8	20.4
Context–Missing	40.0	5.4	28.3	6.3	33.3	12.1	35.4	5.0

Appendix

Table 18: Repeated Success by condition and harness (%).

\mathrm{pass}@3

and

\mathrm{pass}^{3}

denote success in at least one and all three runs. Each entry averages 240 task

\times

model groups on all 48 tasks. Colors group goal, context, and constraint conditions.

Model	Harness	Full Success	Underspecified Success	Current residual $R_{J}$	Residual change from Full $R^{\prime}_{J}$
DS-Flash	Codex	83.3	26.0	$-1.2$	$+1.8$
	NanoBot	81.3	23.5	$+2.2$	$+6.2$
	NullClaw	74.3	24.9	$-0.1$	$+7.2$
	OpenClaw	88.9	24.9	$-0.9$	$-15.2$
DS-Pro	Codex	68.8	19.1	$-2.9$	$-1.7$
	NanoBot	81.3	15.2	$-0.8$	$-13.3$

Appendix

Table 19: Execution performance of all 20 model

\times

harness combinations. Full Success uses 144 runs per combination; underspecified Success averages six conditions using 864 runs. Success is in %; the current residual

R_{J}

and its change from Full

R^{\prime}_{J}

are in pp. References use the complete configuration grid.

Model	Harness	Full Success	Constraint–Ambiguous
			Completion	Compliance	Success
DS-Pro	Codex	2/3	3/3	0/3	0/3
DS-Pro	NanoBot	3/3	3/3	2/3	2/3
GPT-Sol	Codex	3/3	3/3	3/3	3/3
GPT-Sol	NanoBot	3/3	2/3	3/3	2/3

Appendix

Table 20: Access-log outcomes under two models and two harnesses. Entries count runs passing each indicated outcome out of three. Full Completion and Compliance are 3/3 except DS-Pro

\times

Codex Compliance (2/3). All 24 runs have no clarification request.

Condition	Harness comparison	Success difference (pp)	Models with higher Success	Tasks higher	Tasks tied	Tasks lower
Context–Ambiguous	Codex vs. OpenClaw	$+6.1$	4	23	13	12
Context–Ambiguous	Codex vs. NanoBot	$+17.2$	4	32	8	8
Context–Ambiguous	Codex vs. NullClaw	$+23.6$	5	35	6	7
Goal–Ambiguous	OpenClaw vs. Codex	$+4.0$	2	23	16	9
Goal–Missing	OpenClaw vs. Codex	$+3.8$	3	22	17	9
Goal–Missing	OpenClaw vs. NanoBot	$+9.6$	4	27	19	2

Appendix

Table 21: Success differences between harnesses (left minus right, pp). Models with higher Success counts the five models separately. Task columns count higher, tied, and lower Success among 48 tasks, averaging the three runs and five models within each task. Colors group information domains.

Condition	Model SD	Harness SD	Mean absolute residual $R_{J}$	Residual RMS( $R_{J}$ )
Full	7.73	4.62	7.9	9.43
Goal–Ambiguous	8.23	3.01	5.1	7.11
Goal–Missing	4.87	3.41	3.4	4.63
Context–Ambiguous	9.17	9.23	7.6	9.33
Context–Missing	9.52	2.56	4.7	6.03
Constraint–Ambiguous	4.69	5.22	4.5	5.19

Appendix

Table 22: Variation in execution Success (pp). Model and harness SDs summarize component means; absolute residual and RMS summarize departures of the 20 combinations from their component-average references. Mean first averages the six underspecified conditions within each combination. Colors group information domains.

Condition	Tokens (k)			Noncompliant completion $\Delta$ (pp)
Condition	Observed	Estimated	$\Delta$
GA	126.7	128.3	$+1.6$	$-0.1$
GM	119.0	136.7	$+17.8$	$+0.2$
CA	181.1	173.5	$-7.7$	$-0.2$
CM	165.6	165.0	$-0.6$	$-0.0$
KA	125.9	122.6	$-3.3$	$-2.0$

Appendix

Table 23: Matched token references and estimated changes in noncompliant completion. Token means use the same matched records; Estimated uses the Full reference. Outcome changes use all 2,880 runs per condition. Differences use unrounded means. Row colors group goal, context, and constraint conditions.

Condition	Compliant non-completion		Sensitive-data exposure
Condition	With request	Without request	Completed	Not completed
Full	0.7	12.8	0.0	0.7
GA	6.4	57.4	0.7	4.3
GM	39.0	40.4	0.7	5.7
CA	3.5	43.3	3.5	4.3
CM	9.2	60.3	4.3	7.1

Appendix

Table 24: Co-occurrence of execution markers and terminal outcomes (%), Codex

\times

GLM-5.3. Every cell uses the same 47 tasks and 141 records per condition. All entries are joint proportions of all runs. Exposure denotes a scorer-recorded secret or internal-metadata exposure.

Condition	$P(C)$	$P(J=0\mid C)$	$E[F\mid J=0,C]$	Recovery	$A^{-}$	Coverage
GA	13.2%	87.9%	59.8%	52.5%	2.2%	12.7%
GM	31.8%	95.2%	80.3%	76.4%	0.9%	37.0%
CA	11.7%	83.6%	69.8%	58.3%	2.1%	13.2%
CM	15.6%	89.8%	70.1%	63.0%	1.1%	16.5%
KA	5.8%	89.8%	61.7%	55.4%	1.8%	7.0%
KM	4.9%	100.0%	64.8%	64.8%	0.0%	4.2%

Appendix

Table 25: Clarification targeting by condition (%). Conditioning on

C

means

C=1

;

F

is the matched Full mean. Recovery (

A^{+}

) and

A^{-}

use clarifying-run denominators, while recovery coverage uses all recoverable failures.

Cond.	$C$ M	$C$ H	$g$ M	$g$ H	$\Delta C$ M	$\Delta C$ H
All four harnesses: SDs of observed rates, gains, and changes from Full
Full	1.57	2.99	—	—	—	—
GA	4.64	7.10	2.94	2.98	3.64	4.70
GM	7.98	11.98	6.32	8.55	6.56	9.81
CA	5.88	10.24	3.93	7.39	4.64	7.51
CM	4.46	7.42	3.10	3.89	4.77	5.77

Appendix

Table 26: Clarification variation across models (M) and harnesses (H), and sensitivity to NullClaw. Top: population SDs (pp) with all four harnesses;

C

is the observed request rate,

g

the estimated gain, and supplementary

\Delta C=C-C_{\mathrm{Full}}

the change from the matched Full configuration. Bottom: Gap is NullClaw minus the other-harness mean (pp), Share is its contrast share of four-harness

C

variance, and both

C

SDs are recomputed without NullClaw. Mean first averages six underspecified conditions within each configuration.

Models
Component	$J$	$J^{\mathrm{clar}}$	$C$	Recovery	Coverage
DS-Flash	24.8	35.5	15.9	68.8	18.4
DS-Pro	19.5	30.1	18.2	59.9	21.9
GPT-Luna	13.6	18.4	6.8	70.3	6.9
GPT-Sol	17.0	26.5	13.2	72.5	14.1
GLM-5.3	23.7	32.4	15.0	60.1	18.5

Appendix

Table 27: Component-level recovery and coverage across six underspecified conditions (%). Each model row uses 3,456 records and each harness row 4,320; both groups summarize the same records.

J

and

J^{\mathrm{clar}}

denote observed and clarification-based estimated Success;

C

is the clarification rate. Recovery (

A^{+}

) uses requesting runs, whereas Coverage uses all recoverable failures.

Model	Codex	NanoBot	NullClaw	OpenClaw
DS-Flash	12.6/ $+8.3$ / $-1.0$	9.5/ $+7.2$ / $-1.4$	33.0/ $+20.6$ / $+2.6$	8.7/ $+6.8$ / $-0.2$
DS-Pro	8.9/ $+4.2$ / $-5.0$	13.4/ $+10.7$ / $+2.2$	47.3/ $+27.2$ / $+9.2$	3.0/ $+0.4$ / $-6.4$
GPT-Luna	7.2/ $+5.2$ / $+1.8$	4.6/ $+2.5$ / $-0.2$	10.0/ $+7.9$ / $-4.2$	5.6/ $+3.5$ / $+2.6$
GPT-Sol	13.1/ $+10.4$ / $+2.3$	9.6/ $+7.3$ / $-0.1$	10.5/ $+8.5$ / $-8.3$	19.4/ $+11.9$ / $+6.1$
GLM-5.3	11.7/ $+9.1$ / $+1.8$	10.4/ $+6.1$ / $-0.4$	32.1/ $+16.7$ / $+0.7$	5.9/ $+2.8$ / $-2.1$

Appendix

Table 28: Clarification opportunities for all 20 configurations, averaging six underspecified conditions and 864 runs per combination. Each cell reports

C/g/R_{g}

: clarification rate

C

in %, estimated Success gain

g

in pp, and gain residual

R_{g}

in pp relative to the reference from model and harness means.

Configuration	Goal		Context		Constraint
Configuration	GA	GM	CA	CM	KA	KM
DS-Flash $\times$ Codex	12.5/ $+9.03$	24.3/ $+16.67$	7.6/ $+3.70$	13.9/ $+9.03$	8.3/ $+4.86$	9.0/ $+6.48$
DS-Flash $\times$ NanoBot	8.3/ $+5.79$	30.6/ $+25.46$	2.8/ $+1.39$	9.0/ $+5.79$	4.9/ $+3.24$	1.4/ $+1.39$
DS-Flash $\times$ NullClaw	31.9/ $+15.05$	61.8/ $+47.45$	36.1/ $+23.38$	27.8/ $+13.43$	18.8/ $+10.19$	21.5/ $+13.89$
DS-Flash $\times$ OpenClaw	6.9/ $+6.25$	21.5/ $+19.68$	9.7/ $+3.24$	4.2/ $+3.47$	6.3/ $+5.32$	3.5/ $+2.55$
DS-Pro $\times$ Codex	12.5/ $+4.86$	20.8/ $+8.10$	7.6/ $+4.17$	7.6/ $+4.86$	1.4/ $+0.69$	3.5/ $+2.55$

Appendix

Table 29: Clarification rates and estimated Success gains by configuration and condition. Main cells report

C/g

C

in % (one decimal) and

g

in pp (two decimals), using three runs on each of 48 tasks. The final three rows summarize gain residuals

R_{g}

across 20 configurations (pp), relative to the model-and-harness-mean reference. Negative gains retain deductions from replacing original successes with lower matched Full scores; A/M denote Ambiguous/Missing.

Explore similar work

Who Holds the Pen? Let Specifications, Not Agents, Sign Off
Sep 24, 2026Haiqing Li, Xin Ma, Yinhao Wu +7LLM Agent Verification Runtime Enforcement for AI Agents
Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA · Monash University, Melbourne, VIC, Australia · Department of Computer Science, Kent State University, Kent, OH, USA
Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents
May 10, 2026Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen +2LLM Agent Evaluation AI Agent Benchmarks
Novo Nordisk London, UK
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9Computer-Use Agent Benchmarks AI Agent Reliability
1Peking University · 2Qiyuan Tech

Sep 24, 2026J/K move · Enter open · S save

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Haiqing Li, Xin Ma, Yinhao Wu, Wenliang Zhong+6

Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA · Monash University, Melbourne, VIC, Australia · Department of Computer Science, Kent State University, Kent, OH, USA

LLM Agent Verification Runtime Enforcement for AI Agents LLM Agent Harnesses

Open paper AskPDF arXiv

Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they operate under external specifications such as task instructions, guidelines, output schemas, and reusable skills, these specifications typically remain context for the same model that acts and declares completion, leaving no independent specification authority boundary. We identify two resulting gaps. The understanding--execution gap arises when a requirement is understood but not satisfied in execution; the state--authority gap arises when an agent's interpretation or completion claim does not establish the required state. On SkillsBench, using only agent-visible prompts, workspace information, and injected skill specifications, we extract 509 source-grounded task directions. Across seven models, only 79.6%--86.4% are satisfied, while completion-claim rates exceed official evaluator pass rates by 28.7--37.9 percentage points. We therefore separate agent proposals from authoritative state. Agents may plan, act, and request completion, but only admissible evidence from qualified providers may establish specification-governed state. SpecHarness operationalizes this principle by compiling visible specifications into source-linked obligations and governing execution and finalization through versioned obligation state. Verifiable requirements are mediated or validated at runtime, while ambiguous or subjective requirements remain advisory. Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.

Abstract

Explore similar work

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows