Proof-Carrying Contracts: Commitments AI Agents Cannot Break
Organizations: AI Centre, Computer Science Department, University College London, London, UK · Praxis Lab, The Italian Institute of Artificial Intelligence (AI4I), Turin, Italy · SDA Bocconi School of Management. Milano, Italy · Inria d’Université Côte d’Azur, France
Abstract
AI agents increasingly act on behalf of people or other agents, purchasing services, exchanging information and delegating work. What an agent must respect is written in instructions, policies or agreements, but text alone cannot make it comply. It may make mistakes, be redirected by what it reads, or pursue different interests. For an economy of such agents to function, parties need agreements they can rely on without inspecting or trusting one another's agents. We introduce Proof-Carrying Contracts (PC2), a framework for making agreed constraints enforceable throughout an agent's interactions with a system. Before a requested operation can run, the agent must supply a machine-checkable proof that it preserves the agreed constraints in every case allowed by its model. An execution system called the Airlock checks the proof and controls access to the operation. The parties specify the conditions to preserve, while the agent supplies the reasoning needed to justify its decisions and remains free to adapt its strategy. This avoids having to design a separate permission policy and prove that every decision it permits is safe. We prove that the constraints hold at every instant during execution, even for an adversarial agent, and identify when this implies safety in the world. Reusable capability modules provide tools and guarantees from which contracts can be constructed. PC2 provides a foundation for an agent economy whose agreements protect both those delegating work and those carrying it out.
Figures & tables
| Module element | Custody (Ex.: payment instructions) | Formal interface (Ex.: booking responses) | Domain invariant (Ex.: mailbox retention) |
| World | Payment service and history of submitted instructions. | Booking service and history of issued responses. | Mail service, including delivery and acknowledgement history. |
| State | Register of payment instructions. | Log of response events. | Current mailbox contents . |
| Correspondence | Every submitted instruction is registered. | Recorded responses were issued and are permitted. | The reading is current and pending mail remains available. |
| Tools | records an instruction, then submits it. | requests, authenticates and records a response. | Read or process messages, with models covering changes to mailbox contents. |
| Idle | Register unchanged. Earlier instructions may still produce charges. | Log unchanged. The service may continue to evolve. | Mailbox contents may change, while pending messages remain available. |
| Supporting conditions | Protect register updates Item H1 . Register before submitting and restrict issuance Item H2 . | Permitted responses Item F1 , authenticated and recorded faithfully Item F2 . | Readings agree with the world Item M1 . Routines and outside changes preserve Item M2 . |
| Interaction with the world | What models cover | Contract response | Related approaches |
|---|---|---|---|
| Protected from outside changes E.g. an instruction register | No outside changes to the protected fields. | Rely on these records when authorizing later operations. | Complete mediation and custody ( Saltzer and Schroeder, 1975 ) |
| History only grows E.g. a log of service responses | Additions to the history, with past events retained. | Use facts about the past that survive further additions. | History variables ( Abadi and Lamport, 1991 ) |
| Changes within an agreed bound E.g. withdrawals by other account holders | Every debit permitted by the account’s shared withdrawal limit. | Reserve enough margin for those debits. | Rely–guarantee reasoning ( Jones, 1983 ) |
| Changes at a bounded rate E.g. charges for a metered service | Change bounded by a rate and the duration of the operation. | Keep a margin and intervene before it is exhausted. | Controlled invariants, barrier functions and Simplex ( Blanchini, 1999 ; Ames et al., 2019 ; Sha, 2001 ) |
| Holds until expiry E.g. a temporary hotel reservation | The commitment remains valid until its expiry. | Complete the operation in time, renew, or release. | Leases ( Gray and Cheriton, 1989 ) |
| May change beyond our control E.g. a flight cancellation | Each possible outcome and its consequences. | Allow a compliant way to continue or terminate after each outcome. | Demonic nondeterminism ( Dijkstra, 1975 ) |
| Question | Prompts and learned monitors | Policy enforcement | Model-based shields | PC2 |
|---|---|---|---|---|
| Examples | Prompt delimiters and attack detectors evaluated in AgentDojo ( Debenedetti et al., 2024 ) | CaMeL ( Debenedetti et al., 2025 ) and FORGE ( Palumbo et al., 2026 ) | Shield synthesis ( Bloem et al., 2015 ) and shielded RL ( Alshiekh et al., 2018 ) | This work |
| What allows a tool call to run? | The model follows instructions, or a learned monitor accepts the input or call | An author-supplied rule permits the call, given its arguments, history or data sources | An analysis of possible system behaviour identifies the action as safe | A supplied proof establishes the obligation derived from the contract and tool models |
| How are requirements written? | Instructions in ordinary language or criteria learned from examples | Executable rules on calls, past actions and information flow | Safety formulas supported by the chosen system model and analysis | State constraints from which per-call obligations are derived, even when those obligations are undecidable |
| What is established about the call’s effects? | No formal guarantee covering all possible outcomes of an accepted call | Calls and recorded events satisfy the policy. Safety of their consequences needs a further argument | Every behaviour covered by the safety analysis satisfies the specified safety property | Every modelled effect preserves the contract, despite hidden state and permitted changes before execution |
| Are restrictions guaranteed even for a malicious agent? | ✗ Resistance to attacks is empirical | ✓ The specified policy is enforced | ✓ Safety holds within the analysed model | ✓ The contract holds throughout execution |
| Does each requested call carry a separately checkable proof? | ✗ | ✗ | ✗ The shield itself may be verified | ✓ The proof refers to the contract, tool and published information |
Runtime Enforcement for AI Agents
Runtime enforcement for AI agents checks and constrains actions as they execute, helping prevent unsafe or unauthorized effects.
See all 273 papers