Human oversight is a pillar of AI safety. It is often exercised by a human in the loop (HITL) who approves critical actions taken by an AI agent. AI agents' increasing ability to bypass guardrails, though, raises the question of how to enforce HITL robustly. We introduce the concept of inalienable human-in-the-loop (I-HITL). An I-HITL protocol ensures that an agent can initiate an action only with explicit human approval, even when the human interacts with a system (e.g., a local agentic harness) compromised by the agent. A strong I-HITL protocol additionally ensures such approval even when an adversarial human attempts to delegate approvals to an agent. To motivate I-HITL, we report on experiments with a popular agentic wallet platform in which raising an agent's spending cap requires HITL approval via emailed one-time passcodes (OTPs). Given access to the user's email and a task blocked by the cap, we find that ten different frontier models will often autonomously raise the cap, harvesting OTPs to bypass a HITL guardrail and surfacing other vulnerabilities. We formally define I-HITL using the concept of communication topologies and prove necessary and sufficient conditions for its realization. We show that an I-HITL protocol requires a trusted entity - a trusted computing device (or equivalent) - and characterize which such devices can and cannot realize an I-HITL protocol. We present a prototype I-HITL system called Countersign that can plug into and strengthen existing HITL systems using off-the-shelf consumer mobile devices.
Figures & tables
Fig. 1 : Two HITL configurations. Red shading indicates resources accessible by the agent; green indicates the HITL gate. In Example 1 (panel \scriptsizeA⃝), when an agent-requested action triggers the HITL protocol, the service sends a passcode to an email inbox that the agent can access. In an I-HITL protocol (panel \scriptsizeB⃝), the approval channel must terminate at a trusted entity T that the agent cannot access or forge, so only the human can approve.
Fig. 2 : Agent self-escalation behavior with Circle’s agent wallet, across twenty runs per model/experiment pair. Models are grouped by family (Qwen, DeepSeek, GPT, Gemini, Claude) and, within each family, ordered left to right from least to most capable. Each bar shows the fraction of runs in which the agent completed the self-escalation, reading the OTP from its inbox and applying a policy with a higher daily cap. Bars at 0% are drawn as thin slivers for visibility.
Fig. 3 : The plain topology τ0 . Dashed edges represent insecure channels. The double node P is adversarial while the dashed node H may only run programs from a restricted class.
Fig. 4 : The supergraph of topologies involving a trusted entity T . The four channels on which T can send/receive each carry a label ℓ∈L∪{∅} .
H
Key
Honest
Delegating
Unattested, extractable
(ins\mathchar24891ins)
(ins\mathchar24891ins)
Unattested, encumbered
(ins\mathchar24891ins)
(ins\mathchar24891ins)
Attested, extractable
(auth\mathchar24891conf)
(ins\mathchar24891ins)
Attested, encumbered
(auth\mathchar24891conf)
(auth\mathchar24891conf)
TABLE I : Channel labels (ℓTS\mathchar24891ℓST) established by enrollment for each combination of properties of the key k held by T , under an honest and a delegating H ; shaded cells satisfy condition (ii) of Theorem 7 .
User input
Display
Untrusted
Confirmation
Keypad
Untrusted, exposed
(ins\mathchar24891ins)
(ins\mathchar24891ins)
(ins\mathchar24891auth)
Untrusted, private
(conf\mathchar24891ins)
(conf\mathchar24891ins)
(conf\mathchar24891auth)
Trusted, exposed
(auth\mathchar24891ins)
(auth\mathchar24891auth)
(auth\mathchar24891auth)
Trusted, private
(sec\mathchar24891ins)
(sec\mathchar24891auth)
(sec\mathchar24891auth)
TABLE II : Channel labels (ℓTH\mathchar24891ℓHT) established by each combination of display and user input. Shaded cells satisfy condition (i) of Theorem 7 .
Display
Input
Key
Trusted entity T
trusted
private
trusted
keypad
attested
encumbered
I-HITL
Standalone hardware
Wultra Talisman WAU1.1
✓
✓
✓
✗
✓
✓
●
Ledger Nano X
✓
✓
✓
✗
✓
✓
●
Ledger Flex or Stax
✓
✓
✓
✓
✓
✓
●
Ledger Nano X, seed-derived key
✓
✓
✓
✗
✓
✗
◐
TABLE III : Trusted entity T classified by the properties of Sections VI-B and VI-A . Display and input columns determine condition (i) via Table II ; key columns determine condition (ii) via Table I . ● strong I-HITL; ◐ weak I-HITL; ○ neither. Appendix E justifies each classification.
Fig. 6 : The Countersign topology. A user’s mobile phone serves as T and establishes channels to H and S . Channels are labeled with the mechanism they use to establish authenticity. On the phone, the green lock represents that the enclosed functionalities are run by on-device secure hardware, the application itself is run by the OS.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 10 : Screenshot of the warning added to Circle’s agent-wallet documentation following our disclosure. The notice identifies the inbox as the resource that opens the gate and asks the user to keep it out of the agent’s reach.
Fig. 11 : Hardware devices evaluated as trusted entities in Table III (not to scale). The Talisman and Ledger devices have a trusted display and input, whereas a FIDO2 key has only a confirmation button and no display.
AI agents are increasingly deployed to execute tasks and make decisions within agentic workflows, introducing new requirements for safe and controlled autonomy. Prior work has established the importance of human oversight for ensuring transparency, accountability, and trustworthiness in such systems. However, existing implementations of Human-in-the-Loop (HITL) mechanisms are typically embedded within application logic, limiting reuse, consistency, and scalability across multi-agent environments. This paper presents a decoupled HITL system architecture that treats human oversight as an independent system component within the agent operating environment. The proposed design separates human interaction management from application workflows through explicit interfaces and a structured execution model. In addition, a design framework is introduced to formalize HITL integration along four dimensions: intervention conditions, role resolution, interaction semantics, and communication channel. This framework enables selective and context-aware human involvement while maintaining system-level consistency. The approach supports alignment with emerging agent communication protocols, allowing HITL to be implemented as a protocol-level concern. By externalizing HITL and structuring its integration, the system provides a foundation for scalable governance and progressive autonomy in agentic workflows.
Agentic AI systems increasingly execute consequential actions on behalf of human principals, delegating tasks through multi-step chains of autonomous agents. No existing standard addresses a fundamental accountability gap: verifying that terminal actions in a delegation chain were genuinely authorized by a human principal, through what chain of delegation, and under what scope. This paper presents the Human Delegation Provenance (HDP) protocol, a lightweight token-based scheme that cryptographically captures and verifies human authorization context in multi-agent systems. An HDP token binds a human authorization event to a session, records each agent's delegation action as a signed hop in an append-only chain, and enables any participant to verify the full provenance record using only the issuer's Ed25519 public key and the current session identifier. Verification is fully offline, requiring no registry lookups or third-party trust anchors. We situate HDP within the existing landscape of delegation protocols, identify its distinct design point relative to OAuth 2.0 Token Exchange (RFC 8693), JSON Web Tokens (RFC 7519), UCAN, and the Intent Provenance Protocol (draft-haberkamp-ipp-00), and demonstrate that existing standards fail to address the multi-hop, append-only, human-provenance requirements of agentic systems. HDP has been published as an IETF Internet-Draft (draft-helixar-hdp-agentic-delegation-00) and a reference TypeScript SDK is publicly available.
AI agents pose significant risks as they are granted increasing autonomy. A commonly proposed solution is human oversight and keeping a ''human in the loop'', but this is not a simple solution: Not only do current approaches to AI agent design impede effective human oversight, but the cognitive capacities required for it are also themselves degraded by extended use of AI systems. This position paper argues that current approaches to the development and deployment of AI agent systems do not support effective human oversight -- they contribute to its degradation. To address this, a top priority in the advancement of AI agents should be supporting the situated goals and cognitive requirements of effective human oversight, treating the human needs of overseers at the same level of importance as AI agent capability. To put this idea into practice, we connect work on automation and human-computer interaction to AI agent processes, outlining design-level affordances and organizational protocols that (1) support overseers in exercising critical judgement and (2) counteract the skill atrophy that arises from extended use of automation. We urge developers and deployers to adopt these or similar approaches. Without explicit support for the cognitive demands of effective human-agent interaction, AI agent systems will continue to passively incentivize the degradation of the very human skills they rely on.