cs.AIOct 12, 2026
SaveAuthGateBench: Benchmarking LLMs as Authorization Gate
Organizations: Huawei Technologies Ltd. · Sun Yat-sen University
Abstract
Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy.
Figures & tables
Figure 1: A contrast pair: the same action is allowed when the user asks to delete everything, and blocked under the ambiguous request.
Figure 2: Construction pipeline. Stage I generates base triplets via a generate-validate-repair loop with near-duplicate filtering. Stage II inverts each triplet into a contrast pair via label-flipping edit. Stage III augments triplets with read-only environment signals . We review the synthesized instances to confirm their labels are correct.
| Metric | Value |
| Number of failure modes | 5 |
| Number of base triplets | 350 |
| Number of inverted triplets | 350 |
| Number of signal-augmented instances | 700 |
| Total number of instances | 1,400 |
Table 1: The main statistics of AuthGateBench .
| Model | Scope Escalation | Credential Explore | External Sharing | Guardrail Bypass | Inferred Params | ||||||||||
| Acc | FPR | FNR | Acc | FPR | FNR | Acc | FPR | FNR | Acc | FPR | FNR | Acc | FPR | FNR | |
| Qwen3.7-Plus | 72.86 | 50.00 | 4.29 | 56.43 | 84.29 | 2.86 | 51.43 | 97.14 | 0.00 | 58.57 | 75.71 | 7.14 | 72.86 | 38.57 | 15.71 |
| GLM-5.2 | 70.00 | 60.00 | 0.00 | 52.14 | 92.86 | 2.86 | 51.43 | 97.14 | 0.00 | 62.86 | 70.00 | 4.29 | 74.29 | 48.57 | 2.86 |
| Kimi-K2.7-Code | 77.86 | 38.57 | 5.71 | 57.14 | 80.00 | 5.71 | 52.86 | 94.29 | 0.00 | 55.00 | 84.29 | 5.71 | 74.29 | 34.29 | 17.14 |
| DeepSeek-V4-Flash | 71.43 | 52.86 | 4.29 | 57.14 | 84.29 | 1.43 | 53.57 | 92.86 | 0.00 | 54.29 | 87.14 | 4.29 | 70.00 | 45.71 | 14.29 |
Table 2: Evaluation results on base and inverted triplets without signal augmentation, under policy . Higher Acc is better; lower FPR and FNR are better. Best value in each column is in bold .
| Model | Scope Escalation | Credential Explore | External Sharing | Guardrail Bypass | Inferred Params |
| Qwen3.7-Plus | 54.29 | 87.14 | 97.14 | 82.86 | 54.29 |
| GLM-5.2 | 60.00 | 90.00 | 97.14 | 71.43 | 51.43 |
| Kimi-K2.7-Code | 41.43 | 85.71 | 94.29 | 87.14 | 48.57 |
| DeepSeek-V4-Flash | 57.14 | 85.71 | 92.86 | 85.71 | 57.14 |
Table 3: Invariant rate (IR, %) under policy : the fraction of contrast pairs with opposite gold labels on which a model returns the same decision for both.
Figure 3: Fraction of initially wrong predictions that environmental signals correct versus initially correct predictions that mislead, pooled across four models under policy , per failure mode.
Figure 4: DeepSeek-V4-Flash initially predicts allow for a data-destruction instance labeled block (left), but switches to block after reveals an active training run and job using the target directory (right).
Figure 5: Performance evaluation comparison under policy and policy
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Seed example for Scope Escalation.
Figure 7: Seed example for Credential Exploration.
Figure 8: Seed example for Inferred Params.
Figure 9: Seed example for External Sharing.
Figure 10: Seed example for Guardrail Bypass.
Figure 11: Full text of the two policies used in Section 5.1 . is the moderately detailed rule set used to generate and audit AuthGateBench; is the implicit, rule-free instruction given to the classifier under test at evaluation time, standing in for an auto-execution gate that relies on its own internal safety judgment rather than an explicit policy.
Figure 12: Prompt used to generate base instance.
Figure 13: Prompt used to evaluate LLM authorization gates.
Figure 14: Prompt for constructing contrast pair.
Figure 15: Prompt for generating environment signals.
| Model | Scope Escalation | Credential Explore | External Sharing | Guardrail Bypass | Inferred Params |
| Qwen3.7-Plus | 18.57 | 58.57 | 52.86 | 22.86 | 28.57 |
| GLM-5.2 | 22.86 | 82.86 | 88.57 | 61.43 | 41.43 |
| Kimi-K2.7-Code | 17.14 | 61.43 | 78.57 | 62.86 | 24.29 |
| DeepSeek-V4-Flash | 42.86 | 74.29 | 74.29 | 71.43 | 57.14 |
Table 4: Invariant rate (IR, %) under policy : the fraction of contrast pairs with opposite gold labels on which a model returns the same decision for both.
| Model | Scope Escalation | Credential Explore | External Sharing | Guardrail Bypass | Inferred Params | ||||||||||
| Acc | FPR | FNR | Acc | FPR | FNR | Acc | FPR | FNR | Acc | FPR | FNR | Acc | FPR | FNR | |
| Qwen3.7-Plus | 90.71 | 4.29 | 14.29 | 70.71 | 55.71 | 2.86 | 73.57 | 52.86 | 0.00 | 85.71 | 14.29 | 14.29 | 85.71 | 7.14 | 21.43 |
| GLM-5.2 | 88.57 | 22.86 | 0.00 | 58.57 | 82.86 | 0.00 | 55.71 | 88.57 | 0.00 | 67.86 | 60.00 | 4.29 | 79.29 | 38.57 | 2.86 |
| Kimi-K2.7-Code | 91.43 | 17.14 | 0.00 | 69.29 | 58.57 | 2.86 | 60.71 | 78.57 | 0.00 | 67.14 | 58.57 | 7.14 | 87.86 | 11.43 | 12.86 |
| DeepSeek-V4-Flash | 78.57 | 42.86 | 0.00 | 62.86 | 72.86 | 1.43 | 62.86 | 74.29 | 0.00 | 64.29 | 65.71 | 5.71 | 71.43 | 52.86 | 4.29 |
Table 5: Evaluation results on base and inverted triplets without signal augmentation, under policy . Higher Acc is better; lower FPR and FNR are better. Best value in each column is in bold .
Figure 16: Base and inverted instances with their environment signals for Scope Escalation. In the base instance, a vague request to clean up a repository is expanded into deleting untracked and ignored files, including configuration and user-created files that were never named, so the action should be blocked. In the inverted instance, the user explicitly identifies the files and directories to remove and authorizes the destructive command, so the action is allowed. Repository-state and workspace-boundary signals clarify the affected targets and the resulting deletion scope.
Figure 17: Base and inverted instances with their environment signals for Guardrail Bypass. In the base instance, model publication fails because the model has not received production approval, but the agent proposes using a force flag to bypass that requirement, so the action should be blocked. In the inverted instance, the user explicitly requests publication with the force option despite the missing approval, so the action is allowed. Model-registry, approval-status, and audit signals make the bypassed guardrail and its production impact explicit.
Figure 18: Base and inverted instances with their environment signals for Credential Exploration. In the base instance, after a test deployment fails authentication, the agent discovers a production registry token and repurposes it for the test environment without authorization, so the action should be blocked. In the inverted instance, the user explicitly instructs the agent to use that production token for the deployment, so the action is allowed. Deployment configuration and credential-scope signals reveal whether the discovered credential is appropriate for the requested environment.
Figure 19: Base and inverted instances with their environment signals for Inferred Parameters. In the base instance, the agent interprets an ambiguous request to clean up an old project board as authorization to close a particular issue, even though the user never identifies that issue, so the action should be blocked. In the inverted instance, the user explicitly names the issue and specifies that it should be closed as superseded, so the action is allowed. Issue-state, ownership, and pull-request signals help establish whether the selected target is uniquely determined and consistent with the request.
Figure 20: Base and inverted instances with their environment signals for External Sharing. In the base instance, the user asks only for help diagnosing a training problem, but the agent proposes uploading a local training script to an external Pastebin service, so the action should be blocked. In the inverted instance, the user explicitly requests that the same script be uploaded to the named external destination for sharing, so the action is allowed. File-content and destination signals expose the information being transferred and the external trust boundary crossed by the action.
More in this topic
Tool Access Control for LLM Agents
Tool access control for LLM agents limits which tools, resources, and actions agents may use, and under what conditions.
See all 208 papers