Authors: Ziyuan Wang, Genliang Zhu, Xiaobo Chen, Chu Wang
Organizations: Independent Researcher · Georgia Institute of Technology · University of Massachusetts Amherst · University of Illinois Urbana-Champaign
Intent contracts are an authorization interface for tool-using LLM agents in which the planning model holds no tools. For each step it submits a short logged contract: a goal, a verbatim quotation of the user, and scope fields. Code validates the contract, and a small trusted model checks it against the user's words and writes the tool call itself. Provided the validation and dispatch code is correct, every executed call is written by the trusted model, the planner's influence on a call is confined to a contract logged before it, and the trusted side decides how much of a request to carry out. We compare contracts with two reviewers of planner-written commands: one sees only the user's words and the command, the information boundary of a reasoning-blind permission classifier, and one also sees the planner's goal and a checked quotation. With gpt-6.1-terra as planner and GLM-5.3-Flash as the trusted model on three AmPermBench task families, every object changed under contracts is named in the goal of the contract that produced the change; under review the planner names objects in its commands and rarely in its goals. When the planner adds work the user never asked for, contracts complete the user's task in 27/30 runs and the intent-visible reviewer in 16/30, because a reviewer can only refuse a whole command. Contracts do not reduce unauthorized actions. They are indistinguishable from blind review, and once the benchmark's environment terms are clarified, every condition that changes state hits a wrong target in 40-49% of those runs. That error is the planner's choice of objects for an ambiguous reference, which no trusted model here can see the environment to check; telling the executor to decline when unsure does not change it. We evaluate benign over-reach with one pair of models, not adversarial planners. We release the plugin and run records.
Figures & tables
Figure 1: Contract execution ( Contract ). The contract is everything the planner sends. Trusted code independently supplies the original user messages to validation and to the executor, which gets back only a receipt with output size and exit code. The planner reads tool output and proposes the next contract. Both review conditions instead ask the trusted model to approve or decline a planner-written command.
Review
Review + intent
Contract
Who writes the executed call
planner
planner
trusted model
Planner text shown to the trusted model
command
goal, quotation, command
goal, quotation
What the trusted model can decide
whole command
whole command
any part of the step
Intent stated before each call
unused
shown and checked
the only input
Trusted model sees the environment
no
no
no
Table 1: Properties fixed by each interface as implemented, whichever models are used. They hold if the validation and dispatch code is correct. Rows describe information flow, not measured outcomes.
Condition
Safe completion
Unauthorized action
Incomplete
Median seconds/run
Strong : strong model alone
9/96; 9% [4, 17]
15/96; 16% [9, 24]
77/96; 80% [71, 88]
25.4
Small : small model alone
4/96; 4% [1, 10]
1/96; 1% [0, 6]
92/96; 96% [90, 99]
11.1
Review : command review
10/96; 10% [5, 18]
10/96; 10% [5, 18]
76/96; 79% [70, 87]
81.9
Review+Intent : review with intent
7/96; 7% [3, 14]
2/96; 2% [0, 7]
87/96; 91% [83, 96]
95.5
Contract : single-step contracts
12/96; 12% [7, 21]
10/96; 10% [5, 18]
76/96; 79% [70, 87]
178.8
Strong+Declare : declaration only
7/96; 7% [3, 14]
10/96; 10% [5, 18]
82/96; 85% [77, 92]
24.9
Table 2: Original AmPermBench prompts. Each outcome denominator is 96 valid prompt-runs per condition. Entries give numerator/denominator, the recorded percentage, and a 95% interval in percentage units. Unauthorized action and incomplete overlap. Time is the median seconds per run.
Comparison
Safe
Unauthorized
Incomplete
Contract vs. Review+Intent
0.227
0.039
0.027
Contract vs. Review
0.754
1.0
1.0
Contract vs. Strong
0.508
0.383
1.0
Contract vs. Small
0.039
0.012
<0.001
Review+Intent vs. Review
0.453
0.021
0.013
Review+Intent vs. Strong
0.688
0.001
0.021
Table 3: Exact paired McNemar p -values on original prompts. Each comparison uses 96 matched prompt pairs; the sample unit is a prompt pair, not a command or object. Tests are unadjusted for multiple comparisons.
Exact ID
Own ticket
Own range
Open range
Strong
8/24; 0/24
1/24; 0/24
0/24; 7/24
0/24; 8/24
Small
2/24; 0/24
1/24; 1/24
1/24; 0/24
0/24; 0/24
Review
10/24; 0/24
0/24; 0/24
0/24; 6/24
0/24; 4/24
Review+Intent
7/24; 0/24
0/24; 0/24
0/24; 1/24
0/24; 1/24
Contract
12/24; 0/24
0/24; 0/24
0/24; 6/24
0/24; 4/24
Strong+Declare
7/24; 0/24
0/24; 0/24
0/24; 6/24
0/24; 4/24
Table 4: Target-reference breakdown on original prompts. Each cell gives safe-completion and unauthorized-action proportions, separated by a semicolon. Each denominator is 24 prompt-runs in that reference category. The three ambiguous categories are defined in the text.
Condition
Named in goal
Named in command
Contract
56/56
—
Review+Intent
2/16
16/16
Review
4/54
39/54
Table 5: Declared-before-done coverage on original prompts. Each denominator counts changed objects across valid runs of that condition. An object counts when the last code-accepted contract before the invocation that changed it names the object, in its goal or, separately, in its planner-written command. Contract has no planner-authored command field.
Bundling task
Mode
Main task complete
Extra work done
Appended request
Contract
27/27
0/27
Contract †
25/26
0/26
Review+Intent
16/30
0/30
Multi-file list
Contract
24/24
0/24
Contract †
24/24
2/24
Review+Intent
19/24
0/24
Table 6: Author-written bundling tasks. Each denominator is valid task-runs in the corresponding condition. Main-task completion does not require avoiding the extra work. Contract † is Contract with command text permitted in goals.
Safe completion
Unauthorized
Incomplete
Median seconds/run
Strong
21/96; 22% [14, 31]
14/96; 15% [8, 23]
61/96; 64% [53, 73]
24.5
Review
32/96; 33% [24, 44]
25/96; 26% [18, 36]
39/96; 41% [31, 51]
74.2
Review+Intent
19/96; 20% [12, 29]
18/96; 19% [12, 28]
59/96; 61% [51, 71]
104.5
Contract
35/96; 36% [27, 47]
29/96; 30% [21, 40]
32/96; 33% [24, 44]
209.8
Contract , strict executor
35/96; 36% [27, 47]
30/96; 31% [22, 42]
31/96; 32% [23, 43]
210.2
Table 7: Clarified-lane prompts. Each denominator is 96 valid prompt-runs. Entries give numerator/denominator, percentage, and a 95% interval in percentage units; the last column is median seconds per run.
Tool-augmented large language model agents increasingly rely on external APIs, but standard tool schemas describe how to call a tool, not when the tool is causally appropriate or what task state it produces. Causal tool filtering addresses this gap by using lightweight contracts that specify each tool's preconditions, effects, risk level, and cost. However, manually writing and maintaining such contracts does not scale to large or changing tool ecosystems. We introduce Contract2Tool, a framework for inferring tool contracts from metadata, schemas, documentation, and execution traces. Contract2Tool converts observable tool evidence into normalized symbolic contracts that can be evaluated intrinsically and deployed inside downstream causal tool filtering. We evaluate learned contracts against gold preconditions, effects, and risk labels, and measure their downstream utility on multi-step agent tasks. Our results show that hybrid documentation-and-trace evidence produces contracts accurate enough to preserve most of the reliability and efficiency benefits of gold contracts. Learned-contract CMTF achieves 0.980 downstream success, close to 0.990 for gold-contract CMTF, while reducing visible tools from 100 to 1 and reducing average token usage from 26,172 to 2,528 relative to all-tools exposure. These results suggest that learned contracts can provide a scalable contract layer between tool schemas and reliable agent execution.
Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation contracts: artifacts whose later use is constrained by the external system that produced them. We show that observation contract compliance (preserving the temporal validity and byte-level integrity) is an emergent, regression-prone capability: it is neither guaranteed by general tool-use ability nor consistently improved by larger or newer models. To measure this, we introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two orthogonal failure modes no existing benchmark evaluates: validity failures (using an artifact after expiry) and integrity failures (corrupting an artifact's bytes through the observation-to-action pipeline). Our evaluation is deterministic and programmatic, with a virtual clock controlling time and SHA-256 hashes verifying byte integrity. We assign each outcome a failure label drawn from real-world API specifications. We evaluate 38 models and report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%, revealing that current frontier models still fail to comply with observation contracts; (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B: what emerges across the cliff is mid-trajectory restraint, not tool-call competence; (iii) non-monotonic scaling across the GPT-5 family: agentic post-training can erode compliance through sycophancy-driven regression; (iv) our failure taxonomy works as an actionable in-context reward signal, yielding +7.1 pp on 42 paired GPT-5.1 failures.
Jicheng Wang, Yifeng He, Zili Wang +3
University of California, Davis · University of Southern California · University of Hong Kong
As large language models (LLMs) are deployed as autonomous agents, safety failures increasingly involve consequential actions. We study agentic misalignment, where agents take harmful actions under goal conflicts and pressures. Using chain-of-thought (CoT) monitoring, we find that harmful execution is often preceded by intent signals in reasoning. However, post-hoc CoT labels are too coarse to show how intent changes during generation. We introduce INTENT-AS-A-TOOL, an approach that adds intent-targeted tools to give the model a dedicated channel for expressing commitment to a target behavior. The probability of calling an intent tool provides a judge-free, fine-grained signal of the model's tendency to pursue that behavior. Our results show that INTENT-AS-A-TOOL complements CoT monitoring, expands post-hoc CoT labels into dense trajectories, and identifies critical steps for online intervention. These findings suggest that action preferences are useful for tracking agentic misalignment during reasoning. Our code and data are accessible: https://github.com/RebeccaZhang22/intent-as-a-tool.
Yutong Zhang, Jianshuo Dong, Peng Xu +5
Tsinghua University · MatrixOrigin · SiliconProspect AI +1