Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy.
Figures & tables
Figure 1: A contrast pair: the same action is allowed when the user asks to delete everything, and blocked under the ambiguous request.
Figure 2: Construction pipeline. Stage I generates base triplets via a generate-validate-repair loop with near-duplicate filtering. Stage II inverts each triplet into a contrast pair via label-flipping edit. Stage III augments triplets with read-only environment signals Σ . We review the synthesized instances to confirm their labels are correct.
Metric
Value
Number of failure modes
5
Number of base triplets
350
Number of inverted triplets
350
Number of signal-augmented instances
700
Total number of instances
1,400
Table 1: The main statistics of AuthGateBench .
Model
Scope Escalation
Credential Explore
External Sharing
Guardrail Bypass
Inferred Params
Acc
FPR
FNR
Acc
FPR
FNR
Acc
FPR
FNR
Acc
FPR
FNR
Acc
FPR
FNR
Qwen3.7-Plus
72.86
50.00
4.29
56.43
84.29
2.86
51.43
97.14
0.00
58.57
75.71
7.14
72.86
38.57
15.71
GLM-5.2
70.00
60.00
0.00
52.14
92.86
2.86
51.43
97.14
0.00
62.86
70.00
4.29
74.29
48.57
2.86
Kimi-K2.7-Code
77.86
38.57
5.71
57.14
80.00
5.71
52.86
94.29
0.00
55.00
84.29
5.71
74.29
34.29
17.14
DeepSeek-V4-Flash
71.43
52.86
4.29
57.14
84.29
1.43
53.57
92.86
0.00
54.29
87.14
4.29
70.00
45.71
14.29
Table 2: Evaluation results on base and inverted triplets without signal augmentation, under policy π . Higher Acc is better; lower FPR and FNR are better. Best value in each column is in bold .
Model
Scope Escalation
Credential Explore
External Sharing
Guardrail Bypass
Inferred Params
Qwen3.7-Plus
54.29
87.14
97.14
82.86
54.29
GLM-5.2
60.00
90.00
97.14
71.43
51.43
Kimi-K2.7-Code
41.43
85.71
94.29
87.14
48.57
DeepSeek-V4-Flash
57.14
85.71
92.86
85.71
57.14
Table 3: Invariant rate (IR, %) under policy π : the fraction of contrast pairs with opposite gold labels on which a model returns the same decision for both.
Figure 3: Fraction of initially wrong predictions that environmental signals Σ correct versus initially correct predictions that Σ mislead, pooled across four models under policy π , per failure mode.
Figure 4: DeepSeek-V4-Flash initially predicts allow for a data-destruction instance labeled block (left), but switches to block after Σ reveals an active training run and job using the target directory (right).
Figure 5: Performance evaluation comparison under policy π and policy π∗
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Seed example for Scope Escalation.
Figure 7: Seed example for Credential Exploration.
Figure 8: Seed example for Inferred Params.
Figure 9: Seed example for External Sharing.
Figure 10: Seed example for Guardrail Bypass.
Figure 11: Full text of the two policies used in Section 5.1 . π∗ is the moderately detailed rule set used to generate and audit AuthGateBench; π is the implicit, rule-free instruction given to the classifier under test at evaluation time, standing in for an auto-execution gate that relies on its own internal safety judgment rather than an explicit policy.
Figure 12: Prompt used to generate base instance.
Figure 13: Prompt used to evaluate LLM authorization gates.
Figure 14: Prompt for constructing contrast pair.
Figure 15: Prompt for generating environment signals.
Model
Scope Escalation
Credential Explore
External Sharing
Guardrail Bypass
Inferred Params
Qwen3.7-Plus
18.57
58.57
52.86
22.86
28.57
GLM-5.2
22.86
82.86
88.57
61.43
41.43
Kimi-K2.7-Code
17.14
61.43
78.57
62.86
24.29
DeepSeek-V4-Flash
42.86
74.29
74.29
71.43
57.14
Appendix
Table 4: Invariant rate (IR, %) under policy π∗ : the fraction of contrast pairs with opposite gold labels on which a model returns the same decision for both.
Model
Scope Escalation
Credential Explore
External Sharing
Guardrail Bypass
Inferred Params
Acc
FPR
FNR
Acc
FPR
FNR
Acc
FPR
FNR
Acc
FPR
FNR
Acc
FPR
FNR
Qwen3.7-Plus
90.71
4.29
14.29
70.71
55.71
2.86
73.57
52.86
0.00
85.71
14.29
14.29
85.71
7.14
21.43
GLM-5.2
88.57
22.86
0.00
58.57
82.86
0.00
55.71
88.57
0.00
67.86
60.00
4.29
79.29
38.57
2.86
Kimi-K2.7-Code
91.43
17.14
0.00
69.29
58.57
2.86
60.71
78.57
0.00
67.14
58.57
7.14
87.86
11.43
12.86
DeepSeek-V4-Flash
78.57
42.86
0.00
62.86
72.86
1.43
62.86
74.29
0.00
64.29
65.71
5.71
71.43
52.86
4.29
Appendix
Table 5: Evaluation results on base and inverted triplets without signal augmentation, under policy π∗ . Higher Acc is better; lower FPR and FNR are better. Best value in each column is in bold .
Figure 16: Base and inverted instances with their environment signals for Scope Escalation. In the base instance, a vague request to clean up a repository is expanded into deleting untracked and ignored files, including configuration and user-created files that were never named, so the action should be blocked. In the inverted instance, the user explicitly identifies the files and directories to remove and authorizes the destructive command, so the action is allowed. Repository-state and workspace-boundary signals clarify the affected targets and the resulting deletion scope.
Figure 17: Base and inverted instances with their environment signals for Guardrail Bypass. In the base instance, model publication fails because the model has not received production approval, but the agent proposes using a force flag to bypass that requirement, so the action should be blocked. In the inverted instance, the user explicitly requests publication with the force option despite the missing approval, so the action is allowed. Model-registry, approval-status, and audit signals make the bypassed guardrail and its production impact explicit.
Figure 18: Base and inverted instances with their environment signals for Credential Exploration. In the base instance, after a test deployment fails authentication, the agent discovers a production registry token and repurposes it for the test environment without authorization, so the action should be blocked. In the inverted instance, the user explicitly instructs the agent to use that production token for the deployment, so the action is allowed. Deployment configuration and credential-scope signals reveal whether the discovered credential is appropriate for the requested environment.
Figure 19: Base and inverted instances with their environment signals for Inferred Parameters. In the base instance, the agent interprets an ambiguous request to clean up an old project board as authorization to close a particular issue, even though the user never identifies that issue, so the action should be blocked. In the inverted instance, the user explicitly names the issue and specifies that it should be closed as superseded, so the action is allowed. Issue-state, ownership, and pull-request signals help establish whether the selected target is uniquely determined and consistent with the request.
Figure 20: Base and inverted instances with their environment signals for External Sharing. In the base instance, the user asks only for help diagnosing a training problem, but the agent proposes uploading a local training script to an external Pastebin service, so the action should be blocked. In the inverted instance, the user explicitly requests that the same script be uploaded to the named external destination for sharing, so the action is allowed. File-content and destination signals expose the information being transferred and the external trust boundary crossed by the action.
As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem. Guardrails are often placed in front of these actions, yet few evaluations ask whether their decisions follow authorization: blocking unauthorized actions while still allowing the same actions under valid permission. We introduce \bench{}, a benchmark that contrasts authorization evidence for the same operation in its language track. It combines expert-annotated natural-language scenarios with executed and verified operations, plus externally sourced native-policy requests. \bench{} checks each decision for correctness, usability and resulting effect, and tracks each request across guardrail changes. Our experiments show that many blocked operations stem from failed reviews, not correct denials. Repairing the output format restores usable decisions but changes which authorized requests are honored, and adding a second reviewer removes some errors while rejecting valid authorization. Guardrails should therefore be evaluated on both the unauthorized actions they stop and the authorized actions they let through.
Claude Code's auto mode is the first deployed permission system for AI coding agents, using a two-stage transcript classifier to gate dangerous tool calls. Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic. We present the first independent evaluation of this system on deliberately ambiguous authorization scenarios, i.e., tasks where the user's intent is clear but the target scope, blast radius, or risk level is underspecified. Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth. Our findings characterize auto mode's scope-escalation coverage under this stress-test workload. The end-to-end false negative rate is 81.0% (95% CI: 73.8%-87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction. Notably, 36.8% of all state-changing actions fall outside the classifier's scope via Tier 2 (in-project file edits), contributing to the elevated end-to-end FNR. Even restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR remains 70.3%, while the FPR rises to 31.9%. The Tier 2 coverage gap is most pronounced on artifact cleanup (92.9% FNR), where agents naturally fall back to editing state files when the expected CLI is unavailable. These results highlight a coverage boundary worth examining: auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.
Zimo Ji, Zongjie Li, Wenyuan Jiang +2
Hong Kong University of Science and Technology · ETH Zurich
Tool-using large language model (LLM) agents increasingly perform state-changing actions that require user authorization. Yet existing approaches do not provide a principled mechanism for selectively updating prior authorization when only part of an instruction changes. To this end, we propose an Authorization-Closure-Graph (ACG)-based framework that represents authorization and its dependencies as an evolving, versioned state. ACG selectively invalidates authority affected by a revision while preserving unaffected portions of the authorization state, and computes a minimal repair that identifies only the missing evidence or authority required for execution. This enables agents to adapt to revised instructions while avoiding stale authority and unnecessary authorization requests. We evaluate ACG across three advanced LLMs in two natural tasks, and ACG consistently improves action safety rate and task success rate. Code is available at https://github.com/weiliang822/ACG.