Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents
Organizations: Cherkasy State Business College, Cherkasy, Ukraine · State Scientific Research Institute of Armament and Military Equipment Testing and Certification, Cherkasy, Ukraine · healthPrecision, New York, NY, USA
Abstract
Benchmarks, audits, and agent protocols describe performance, permissions, and repair, but not how observed evidence should change an agent's authority during a consequential task. We call this the assurance-transition gap. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants, and illustrate them in clinical, industrial, and judicial failure probes. We then report a deterministic failure-injection study in agentic coding: 280 constructed cases evaluated by a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 80 of 100 block-required injections and all 40 review-required injections. Tuned in hindsight, it matches the conjunction on this corpus. For positive weights, a positive threshold, binary risk signals, zero-signal controls, and an injected case firing each signal alone, we show that exact agreement holds if and only if the threshold does not exceed the smallest weight. A separate set of 18 hand-authored traces checks version-pinned evidence and review transitions against simpler policy variants. In a further prospective synthetic holdout of 24 episodes, two blinded LLM judges assign identical labels to all 72 action attempts; RAC and a separately implemented full stateful baseline both match these labels. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.
Figures & tables
| Approach | Measures outcomes | Defines permissions | Defines repair | Binds human capacity | Evidence authority |
| Model evaluation / calibration | yes | no | no | no | no |
| Governance proxy / audit | yes | indirect | optional | optional | indirect |
| Agent protocol interface | not inherent | yes | yes | optional | configured |
| Assurance case / AMLAS | argument | constraints | mitigation | optional | argument-level |
| Selective prediction | model risk | no | reject | optional | model-level |
| Runtime enforcement / RTA | safety state | yes | fallback | typically no | guarded transition |
| Gate | Passing condition | Failure transition |
| G_BOUNDS | boundary allows action or required review is approved | escalate / defer / stop |
| G_CONTEXT | required context fresh and traceable | retry / defer / stop |
| G_EVIDENCE | event fields complete and source-valid | verify / retry / stop |
| G_ELIGIBILITY | for task/version/domain | switch / stop |
| G_SUPERVISION | role, load, queue, deadline admissible | defer / stop |
| G_RESOLUTION | no unresolved failure on action path | retry / escalate / stop |
| Probe | Baseline state | Failed gate | Prohibited action | Required transition | Persisted evidence |
| Stale clinical context | benchmark passed; call well-formed | G_ CONTEXT | commit order draft | refresh and retry ; else defer | stale field, gate, context version, , reason |
| Judicial draft with missing provenance | score above threshold | G_ EVIDENCE | release recommendation | verify ; if policy marks high consequence, escalate | missing field, gate, , reviewer |
| Operator queue over capacity | call permitted; review required | G_ SUPER VISION | approve setpoint change | safe hold with defer ; else stop | load, deadline, gate, , reason |
| Outcome | Arm | Count | Denominator |
| unauthorized action | RAC | 0 | 100 |
| score (vendor) | 80 | 100 | |
| score (best) | 0 | 100 | |
| protocol | 56 | 100 | |
| admitted injections | RAC | 0 | 140 |
| score (vendor) | 120 | 140 |
| Class | Gate | Req. | RAC | score (vend.) | score (best) | protocol |
| base64-secret | G_EVIDENCE | block | 0 | 20 | 0 | 20 |
| deny-relative | G_BOUNDS | block | 0 | 20 | 0 | 0 |
| path-traversal | G_BOUNDS | block | 0 | 20 | 0 | 16 |
| symlink-escape | G_BOUNDS | block | 0 | 20 | 0 | 20 |
| test-rename | G_BOUNDS | block | 0 | 0 | 0 | 0 |
| test-moved-in-scope | G_SUPERVISION | review | 0 | 20 | 0 | 20 |