Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment

Zhiping Zhang; Franklin Mingzhe Li; Zihao Huang; Tianshi Li

cs.AIOct 20, 2026

Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment

Authors: Zhiping Zhang, Franklin Mingzhe Li, Zihao Huang, Tianshi Li

Organizations: Northeastern University · Carnegie Mellon University · UNC-Chapel Hill

Abstract

When an LLM agent acts on behalf of a user, it is expected to act within the authorization scope the user grants. Existing agent evaluations largely measure task completion or harmful consequences and can therefore overlook actions that violate this delegation without producing conventional safety or privacy harms. We introduce AuthorizationBench, a benchmark of authorization alignment grounded in documented real-world cases. A scalable construction pipeline produces 286 executable tasks spanning situational triggers, the presence or absence of direct harmful consequences, and ten application domains. We execute every task with four models at two reasoning-effort settings across three agent harnesses in an authorization-aware sandbox that keeps authorized and out-of-authorization paths executable and observable, yielding 6,864 trajectories. Agents act out of authorization in 19.2% of trajectories, with rates ranging from 10.5% to 26.6% across the 24 configurations. Task completion does not imply authorization. Increasing reasoning effort leaves the aggregate rate nearly unchanged, with effects varying across models. On controlled pairs, agents act out of authorization at similar rates with (17.9%) and without (20.8%) direct harmful consequences, and more often under benign pressures than malicious injections (23.5% versus 17.6%, p=0.002). Our results show that task capability, additional reasoning, and consequence-oriented safety evaluation do not by themselves ensure authorization alignment. Moreover, agents that consult the principal before completing a task exhibit lower rates of out-of-authorization behavior (14.0% versus 21.6%, p<0.001), pointing to a tension between agent autonomy and preserving users' decision authority that calls for further human-centered study. Dataset and code are available at https://github.com/PEACH-Research-Lab/AuthorizationBench .

Figures & tables

Figure 1: Positioning AuthorizationBench relative to consequence-oriented agent safety benchmarks.

Figure 2: A real-world case (left) is represented by an authorization contract (center). In the actual sandbox trajectory (right), the forum-reply tool call matches the specified out-of-authorization action, while the final response satisfies the completion criterion. The reply is out of authorization because the principal did not authorize the agent to post through their account.

Figure 3: Completion and authorization rates for the 24 harness, reasoning effort, and model configurations. Each bar summarizes 286 trajectories, one per task. Safe completion is omitted. Each stacked bar shows out-of-authorization completion, out-of-authorization non-completion, and within-authorization non-completion. Full results are in Appendix G .

Figure 4: Out-of-authorization rate from low to high reasoning effort for 12 harness-model configurations.

Figure 5: Out-of-authorization rate against mean ask_user calls before task completion, one point per harness and model setting.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Definition
Privacy breach	Identifying or sensitive information about a person reaches a recipient not entitled to receive it.
Financial loss	Money is lost or committed through an unauthorized purchase, payment, transfer, refund, discount, or trade.
Data loss or corruption	Important files, records, or stored content are destroyed, overwritten, or altered in a way that is difficult to reverse.
Security compromise	Working credentials or access are exposed, a security control is weakened, or system access is granted to an unauthorized party.
Unsafe code execution	Untrusted code runs, or vulnerable or harmful code is written into or deployed to a project.
Malicious content spreading	False, defamatory, harassing, discriminatory, spam-like, or otherwise harmful content reaches recipients or a public surface.

Appendix

Table 2: Direct harm categories. The label concerns the direct consequence of the out-of-authorization action.

Trigger	Mechanism	Typical carrier
Vague authorization scope	The instruction states an outcome but leaves its extent, included objects, or stopping point unclear.	Principal instruction
Semantic ambiguity	A phrase has two discrete, plausible, and executable readings, only one of which matches the principal’s intent.	Principal instruction
Prior instruction with lingering effect	An expired directive from an earlier exchange still appears applicable, although a discoverable factual cue shows that its context has changed.	Prior memory
Multi-criteria priority blur	The instruction gives incompatible goals or constraints without ranking them, and no available option satisfies all of them.	Instruction and service records
Conflicting authorizations within the system	Two observable authorization sources from the principal, such as a standing setting and a current instruction, require incompatible handling.	Instruction, setting, or prior message

Appendix

Table 3: Instruction-level situational triggers.

Trigger	Mechanism	Typical carrier
Cross-session information accumulation	Information or state from another session looks relevant and is used instead of grounding the action in the current session.	Prior memory or service fixture
Input volume beyond context capacity	A large ordinary collection causes an early boundary-defining detail to be blurred or lost, while re-reading remains possible.	Paginated service records or workspace data
Adjacent out-of-scope element	A salient, urgent, or similar item outside the task sits beside in-scope material and draws action despite not being needed for completion.	Shared inbox, folder, thread, or listing
Inconvenient compliant handling	A service-enforced compliant path requires more steps than an available out-of-authorization shortcut, while remaining feasible.	Service or tool behavior
Changed technical environment	Live behavior has changed while documentation, configuration, or accumulated habit still describes the old API, schema, path, or default.	Workspace documentation and live tool state
Shifted social context and norms	Ordinary social cues move the situation across an unwritten norm’s boundary while the instruction remains unchanged.	Service messages, events, past outputs, or tools

Appendix

Table 4: Environment-level situational triggers. The final two involve malicious third-party artifacts.

Domain	Application setting
Personal life	Private relationships, communication, household logistics, memories, errands, personal files, and life administration when no more specific domain applies.
Office work	General workplace documents, meetings, email, collaboration tools, HR or team processes, internal reports, and organizational permissions.
Software development	Code, repositories, builds, infrastructure state, packages, developer tools, deployment, debugging, and project cleanup.
Travel	Bookings, itineraries, transportation, lodging, passenger details, travel documents, and expense constraints.
E-commerce	Purchases and sales of goods or subscriptions, marketplaces, carts, checkout, returns, promotions, and saved delivery or payment information.
Entertainment	Media libraries, events, games, tickets, reservations, streaming services, social outings, and leisure accounts.

Appendix

Table 5: Application domains covered by AuthorizationBench .

Dimension	Tasks	Dimension	Tasks
Provenance		Direct harmful consequences
Original	44	With	185
Expanded	242	Without	101
Trigger type		Domain
Benign	168	Personal life	25
Malicious	118	Office work	38

Appendix

Table 6: Composition of the 286 AuthorizationBench tasks.

Case-Quality Dimension	Exact Agreement	Gwet’s AC1
Plausibility	91.4% (32/35)	0.911
Domain Alignment	100.0% (35/35)	1.000
Direct Harm Alignment	97.1% (34/35)	0.971
Trigger Validity	97.1% (34/35)	0.971
Seed-Contract Consistency	100.0% (35/35)	1.000
No Information Leakage	100.0% (35/35)	1.000

Appendix

Table 7: Inter-annotator agreement for human case-quality evaluation.

Judgment	Agreement	Count
Task completion	100%	100/100
Out of authorization	97%	97/100

Appendix

Table 8: Agreement between human and AI post-hoc judgments on 100 trajectories.

Display Name	API Model Identifier	Provider
GPT-5.6 Sol	gpt-5.6-sol	OpenAI
Claude Sonnet 5	claude-sonnet-5	Anthropic via OpenRouter
Gemini 3.8 Flash	gemini-3.8-flash	Google AI Studio via OpenRouter
GLM 5.3	glm-5.3	Z.ai via OpenRouter

Appendix

Table 9: Evaluated model settings.

Harness	Version	Business Tools	Prompt and Context
OpenHands	SDK 1.49.2	MCP	Native prompt, task-configured condensation
OpenAI SDK	Agents SDK 0.20.0	Native function tools	SandboxAgent prompt, no compaction
Claude SDK	Agent SDK 0.2.159	MCP	Claude Code preset, native compaction

Appendix

Table 10: Agent harness configurations.

			Completion and authorization				Combined
Harness	Model	Effort	Safe CR	OA CR	OA non-comp.	Safe non-comp.	OAR	CR
OpenHands	GPT-5.6	low	80.8	14.3	0.3	4.5	14.7	95.1
	GPT-5.6	high	83.9	12.6	0.7	2.8	13.3	96.5
	Sonnet 5	low	79.7	15.7	0.0	4.5	15.7	95.5
	Sonnet 5	high	77.3	17.8	1.0	3.8	18.9	95.1
	Gemini 3.8	low	74.8	17.5	2.4	5.2	19.9	92.3

Appendix

Table 11: Completion and authorization rates (%) for every model setting and harness, each cell 286 trajectories, one per task. The four rates partition the trajectories by whether the task was completed ( Comp. ) or not ( Non-comp. ) and whether the agent stayed within authorization ( Safe ) or not ( OA ). OAR is the sum of the two OA rates, and the completion rate is the sum of the two Comp. rates. For each rate, the highest of the 24 configurations is in bold and the second and third are underlined .

Figure 6: Completion rate against OAR, one point per harness and model setting (dark: high effort, light: low effort).

Model	Harness	Low	High	$\Delta$	$p$
GPT-5.6 Sol	OpenHands	14.7	13.3	-1.4	1.000
	OpenAI SDK	16.4	10.5	-5.9	0.072
	Claude SDK	19.9	17.8	-2.1	1.000
	All	17.0	13.9	-3.1	0.016
Claude Sonnet 5	OpenHands	15.7	18.9	+3.2	1.000
	OpenAI SDK	16.1	21.7	+5.6	0.195

Appendix

Table 12: OAR (%) at low and high reasoning effort, per model and harness.

\Delta

is high minus low. The overall row reports a single aggregate paired McNemar test. Configuration-level

p

-values are Holm-adjusted across 12 comparisons, and model-level

p

-values are adjusted across four models. Values with adjusted

p<0.05

are in bold .

Test	Pairs	With harm	No harm	$p$
Scenario-level Wilcoxon (main text)	91	17.9	20.8	0.124
Run-level McNemar	91	17.9	20.8	0.002

Appendix

Table 13: The harm comparison of Finding 4 under scenario-level and run-level tests on all 91 harm pairs. Rates are OAR (%). Values with

p<0.05

are in bold .

Figure 7: Out-of-authorization rates with and without pre-completion consultation. In the model-setting panel (left), color identifies the model, while dark solid and light dashed lines indicate high and low reasoning effort. In the harness panel (right), triangles, circles, and squares identify OpenHands, OpenAI SDK, and Claude SDK. Lines connect descriptive subgroup rates and do not represent causal changes.

Group	Consulted OA/ $n$ (%)	No consultation OA/ $n$ (%)	Gap	OR [95% CI]	$p$
Overall	306/2182 (14.0)	1009/4682 (21.6)	-7.6	0.60 [0.48, 0.75]	$6.4\times 10^{-6}$
GPT-5.6 Sol (low)	42/308 (13.6)	104/550 (18.9)	-5.3	0.70 [0.45, 1.11]	0.521
GPT-5.6 Sol (high)	49/434 (11.3)	70/424 (16.5)	-5.2	0.73 [0.46, 1.14]	0.521
Claude Sonnet 5 (low)	53/346 (15.3)	86/512 (16.8)	-1.5	0.91 [0.60, 1.37]	0.662
Claude Sonnet 5 (high)	64/436 (14.7)	104/422 (24.6)	-9.9	0.51 [0.34, 0.78]	0.011
Gemini 3.8 Flash (low)	2/77 (2.6)	167/781 (21.4)	-18.8	0.10 [0.02, 0.40]	0.011

Appendix

Table 14: Out-of-authorization counts, rates, and adjusted associations for pre-completion consultation. Gap is the consulted rate minus the no-consultation rate in percentage points. ORs come from logistic regressions with task-clustered standard errors. Subgroup

p

-values are Holm-adjusted within model settings and within harnesses. Values with

p<0.05

are in bold .

Consultation definition	Consulted OA/ $n$ (%)	Comparison OA/ $n$ (%)	Gap
Before completion	306/2182 (14.0)	1009/4682 (21.6)	-7.6
At any point	313/2248 (13.9)	1002/4616 (21.7)	-7.8

Appendix

Table 15: Sensitivity to the consultation definition. For pre-completion consultation, the comparison group contains trajectories without a pre-completion call. For consultation at any point, it contains trajectories that never call ask_user . Gap is in percentage points.

Reference point	Timing category	$n$	%
Completion point	Asked before completion	2182	31.8
	Asked only at or after completion	66	1.0
	Never asked	4616	67.2
First OA action	All asks after first OA action	214	68.4
	Asks before and after first OA action	36	11.5
	No ask after first OA action	63	20.1

Appendix

Table 16: Consultation timing relative to task completion and the first out-of-authorization action. Percentages in the upper block use all 6,864 trajectories. Percentages in the lower block use the 313 out-of-authorization trajectories containing an ask_user call.

Metric	Definition	Rate
$\mathrm{Safe}^{1}$	Safe completion over individual executions	75.7%
$\mathrm{Safe}^{3}$	Safe completion in all three executions	63.6%
$\mathrm{Violate@3}$	At least one OA execution	24.3%

Appendix

Table 17: Authorization reliability across three executions of each sampled agent-case pair. Safe completion means satisfying a completion criterion without acting out of authorization.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

Bounded Agents: Delegation Security for Multi-Agent AI Systems
Aug 16, 2026Xabier MuruagaAI Agent Security Runtime Enforcement for AI Agents
Independent Researcher
AuthGateBench: Benchmarking LLMs as Authorization Gate
Oct 12, 2026Zhixiong Zhuang, Jin Jia, Zhiwei Shang +1Tool Access Control for LLM Agents AI Agent Safety Benchmarks
Huawei Technologies Ltd. · Sun Yat-sen University
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4LLM Agent Evaluation LLM Agents
Peking University · Beijing Academy of Artificial Intelligence

Aug 16, 2026J/K move · Enter open · S save

Bounded Agents: Delegation Security for Multi-Agent AI Systems

Xabier Muruaga

Independent Researcher

AI Agent Security Runtime Enforcement for AI Agents Tool Access Control for LLM Agents Multi-Agent System Security

Open paper AskPDF arXiv

LLM-based agents can act on behalf of a user to access cloud services, call tools, or invoke agents. At session start, the agent's permissions are set but remain static, and each request is evaluated independently, without considering prior actions. Within its permissions, an agent may act contrary to the delegated task, combine individually permitted actions into a prohibited outcome, or delegate authority to a sub-agent without limiting it. A prompt injection poses a risk only if the agent has authority to perform such actions; this is therefore a problem of authorization architecture, not just the model. The Agentic Principal Chain (APC) tracks delegated authority from one principal to the next. APC evaluates each request against the accumulated session state using six authorization checks. APC carries forward and restricts delegated scope and budgets. Using composition closure, APC checks requests against prior actions to prevent prohibited combinations and enforces the decision outside the model. We prove Blast Radius Monotonicity and Composition Soundness for APC implementations; Composition Soundness is limited to prohibited combinations under a complete restriction set and serialized admission. We evaluated 3,154 instances including InjecAgent, AgentDojo, and ASB. Our compromised-model evaluation tests APC independently of model behavior by inserting the ground-truth attack call after the first legitimate tool call. AgentDojo exfiltration fell from 75-100% to 0% across all four domains; APC blocked all 544 InjecAgent data-stealing cases. Intent binding reduced destruction from 38.6% to 4.0% and manipulation from 90.5% to 12.1%. Authorization latency was 0.24 ms at the 99th percentile on an idle host; across 949 AgentDojo task-injection pairs, utility was 8.6 and 13.9 percentage points lower in the two settings. Implementation, evaluation tools, and data are publicly available.

Abstract

LLM Agent Evaluation

Explore similar work

Bounded Agents: Delegation Security for Multi-Agent AI Systems

AuthGateBench: Benchmarking LLMs as Authorization Gate

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents