Organizations: University of Pennsylvania · Independent Researcher · Columbia University · Northeastern University · University of Chicago · University of Rochester · Shanghai Jiao Tong University · University of Washington · Lehigh University · Syracuse University
Human approval provides effective control only if rejected work stops and authorized work completes. We formalize refusal safety and bounded authorization completeness, and audit three execution boundaries: the model selects a call, the harness schedules and resumes it, and the workflow routes the decision to a protected effect. Paired decisions, exact traces, and native business-state witnesses support experiments with three models, five prompt arms, four runtime approval designs, two additional framework replays, and 100 public workflow sources. The paired model panel leaves seven of 300 approvals unfinished despite zero denied effects; parallel confirmation loses an approved branch in three tasks per model, restoration omits 336/672 CrewAI approvals, and four of 59 comparable n8n workflows execute after Reject. Comparing intent, generated calls, dispatch, and native effects identifies the first observable divergence and its repair target. This yields a practical audit contract that makes both refusal safety and authorization completeness independently verifiable, converting human approval from a procedural formality into an enforceable control boundary between intent and real-world effect.
Figures & tables
Model
Complete [95% CI]
Denied effect
GPT-6.1-Sol
96/100 [90.2,98.4]
0/100
DeepSeek V4.1 Flash
100/100 [96.3,100]
0/100
Claude Haiku 5.5
97/100 [91.5,99.0]
0/100
TABLE I: Paired Retail results: 100 cases per model. Completion has descriptive 95% Wilson intervals; each model’s zero-leak upper bound is 3.7%.
Fig. 2: Five-arm approval failure rate for the displayed panel. O: original; S: action-specific; S ′ : wording-only; C: system card; L: ledger. Each cell shows its own rate and denominator; heatmap intensity indicates omission severity.
O
S
S+ledger
Tasks completing 5/5
17
20
20
Tasks below 5/5
3
0
0
Total completions / 100
96
100
100
TABLE II: Repeated diagnostic: 20 selected tasks, five samples per arm. Here L adds a ledger to S; expansion L instead adds a ledger to O.
Fig. 3: Harness-level results across six test families. Consolidated run counts include repair controls; native anomalies and their mechanisms are shown by model.
Protocol
T1 breach
T2 W
T2 O
T3 O
GPT-6.1-Sol original
4
4
2
0
DeepSeek V4.1 Flash adapted
4
4
8
1
Claude Haiku 5.5 sensitivity
4
4
0
7
TABLE III: Banking transitions: 16 episodes per family including controls. W: old-target effect after withdrawal; O: new approved work omitted. Counts can overlap within an episode.
Stage
Sources
Attempted public source IDs
100
Terminal paired suffixes (33 nonterminal)
67
Approve positive controls (2 excluded)
65
Policy-comparable pairs (6 excluded)
59
Reject suppresses protected effect
55
Reject executes protected effect
4
TABLE IV: Public workflow funnel; exclusions are sequential.
Fig. 4: Approval triage: follow the first divergence from the recorded decision, then check native effects. Each diagnostic branch follows recorded calls, dispatch, and native effects.
Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy.
Language-model agents increasingly participate in operational workflows where approvals, binds, and payments create commitments. An approval valid when issued may no longer authorize execution after relevant facts, evidence, policy, or authority change. We call this the evaluation--execution gap. We specify a state-bound runtime assurance layer that separates probabilistic measurement from deterministic adjudication, binds an execution allowance to the decision-relevant basis of an approval, invalidates that allowance after relevant logged changes, and rechecks admissibility at dispatch. This distinguishes the continued validity of a prior approval from a current-state policy check. We describe eight failure modes, an assurance model, a reference architecture, and prospective interventions with retrospective remediation. In a scripted 42-case replay, the state-bound configuration identifies 15 later-invalidated approvals that a deliberately non-re-evaluating point-in-time comparator does not revisit; it holds 14 affected actions before dispatch and compensates one landed effect. These results illustrate behavior under the stipulated policies and event orders, not superiority over dispatch-time enforcement designs or production effectiveness. Under explicit interception, ingestion, dependency, and serialization assumptions, the model establishes dispatch-time validity within the registered execution surface.
When an LLM agent acts on behalf of a user, it is expected to act within the authorization scope the user grants. Existing agent evaluations largely measure task completion or harmful consequences and can therefore overlook actions that violate this delegation without producing conventional safety or privacy harms. We introduce AuthorizationBench, a benchmark of authorization alignment grounded in documented real-world cases. A scalable construction pipeline produces 286 executable tasks spanning situational triggers, the presence or absence of direct harmful consequences, and ten application domains. We execute every task with four models at two reasoning-effort settings across three agent harnesses in an authorization-aware sandbox that keeps authorized and out-of-authorization paths executable and observable, yielding 6,864 trajectories. Agents act out of authorization in 19.2% of trajectories, with rates ranging from 10.5% to 26.6% across the 24 configurations. Task completion does not imply authorization. Increasing reasoning effort leaves the aggregate rate nearly unchanged, with effects varying across models. On controlled pairs, agents act out of authorization at similar rates with (17.9%) and without (20.8%) direct harmful consequences, and more often under benign pressures than malicious injections (23.5% versus 17.6%, p=0.002). Our results show that task capability, additional reasoning, and consequence-oriented safety evaluation do not by themselves ensure authorization alignment. Moreover, agents that consult the principal before completing a task exhibit lower rates of out-of-authorization behavior (14.0% versus 21.6%, p<0.001), pointing to a tension between agent autonomy and preserving users' decision authority that calls for further human-centered study. Dataset and code are available at https://github.com/PEACH-Research-Lab/AuthorizationBench .
Zhiping Zhang, Franklin Mingzhe Li, Zihao Huang +1
Northeastern University · Carnegie Mellon University · UNC-Chapel Hill