Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents

Serhii Zabolotnii

cs.AISep 30, 2026

Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents

Authors: Serhii Zabolotnii

Organizations: Cherkasy State Business College, Cherkasy, Ukraine · State Scientific Research Institute of Armament and Military Equipment Testing and Certification, Cherkasy, Ukraine · healthPrecision, New York, NY, USA

Abstract

Benchmarks, audits, and agent protocols describe performance, permissions, and repair, but not how observed evidence should change an agent's authority during a consequential task. We call this the assurance-transition gap. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants, and illustrate them in clinical, industrial, and judicial failure probes. We then report a deterministic failure-injection study in agentic coding: 280 constructed cases evaluated by a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 80 of 100 block-required injections and all 40 review-required injections. Tuned in hindsight, it matches the conjunction on this corpus. For positive weights, a positive threshold, binary risk signals, zero-signal controls, and an injected case firing each signal alone, we show that exact agreement holds if and only if the threshold does not exceed the smallest weight. A separate set of 18 hand-authored traces checks version-pinned evidence and review transitions against simpler policy variants. In a further prospective synthetic holdout of 24 episodes, two blinded LLM judges assign identical labels to all 72 action attempts; RAC and a separately implemented full stateful baseline both match these labels. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.

Figures & tables

Approach	Measures outcomes	Defines permissions	Defines repair	Binds human capacity	Evidence $\to$ authority
Model evaluation / calibration	yes	no	no	no	no
Governance proxy / audit	yes	indirect	optional	optional	indirect
Agent protocol interface	not inherent	yes	yes	optional	configured
Assurance case / AMLAS	argument	constraints	mitigation	optional	argument-level
Selective prediction	model risk	no	reject	optional	model-level
Runtime enforcement / RTA	safety state	yes	fallback	typically no	guarded transition

Table 1: Typical scope of representative assurance approaches. The entries are qualified comparisons, not claims that every implementation has the same boundary. AMLAS denotes Assurance of Machine Learning in Autonomous Systems; RTA, runtime assurance; MAPE-K, monitor–analyze–plan–execute over shared knowledge; and RAC, the proposed Runtime Assurance Contract.

Figure 1: RAC state machine. An output does not authorize action directly. Failed or unknown mandatory gates fail closed; unavailable mandatory review yields defer or stop . Each consequential transition emits

e_{t}

(Eq. 3 ); Table 2 specifies failure transitions.

Gate	Passing condition	Failure transition
G_BOUNDS	boundary allows action or required review is approved	escalate / defer / stop
G_CONTEXT	required context fresh and traceable	retry / defer / stop
G_EVIDENCE	event fields complete and source-valid	verify / retry / stop
G_ELIGIBILITY	$\mathcal{K}=1$ for task/version/domain	switch / stop
G_SUPERVISION	role, load, queue, deadline admissible	defer / stop
G_RESOLUTION	no unresolved failure on action path	retry / escalate / stop

Table 2: Minimal hard gates and required failure transitions.

Probe	Baseline state	Failed gate	Prohibited action	Required transition	Persisted evidence
Stale clinical context	benchmark passed; call well-formed	G_ CONTEXT	commit order draft	refresh and retry ; else defer	stale field, gate, context version, $\delta_{t}$ , reason
Judicial draft with missing provenance	score above threshold	G_ EVIDENCE	release recommendation	verify ; if policy marks high consequence, escalate	missing field, gate, $\delta_{t}$ , reviewer
Operator queue over capacity	call permitted; review required	G_ SUPER VISION	approve setpoint change	safe hold with defer ; else stop	load, deadline, gate, $\delta_{t}$ , reason

Table 3: Illustrative failure probes and mandatory transitions.

Outcome	Arm	Count	Denominator
unauthorized action	RAC	0	100
	score (vendor)	80	100
	score (best)	0	100
	protocol	56	100
admitted injections	RAC	0	140
	score (vendor)	120	140

Table 4: Admitted constructed cases by arm (lower is better on injected cases). The score arm is shown at the published example configuration of Section 4 ,

w=(0.45,0.30,0.25)

\tau=0.6

, and at its best operating point chosen in hindsight over the whole grid,

w=(0.25,0.30,0.45)

\tau=0.05

Figure 2: Failure rates of the score-only arm against the refusal threshold, at the most favorable weight permutation for each

\tau

. The clean region ends exactly at

\tau=\min_{i}w_{i}=0.25

on this positive grid; automatic admission of review-required cases jumps to

1.0

immediately above it. The published vendor threshold lies well outside.

Class	Gate	Req.	RAC	score (vend.)	score (best)	protocol
base64-secret	G_EVIDENCE	block	0	20	0	20
deny-relative	G_BOUNDS	block	0	20	0	0
path-traversal	G_BOUNDS	block	0	20	0	16
symlink-escape	G_BOUNDS	block	0	20	0	20
test-rename	G_BOUNDS	block	0	0	0	0
test-moved-in-scope	G_SUPERVISION	review	0	20	0	20

Table 5: Injected cases passed by each arm, out of 20 per class. Lower is better; the required transition is block or human review .

Explore similar work

Aug 11, 2026cs.CR

Agent Safety Should Be a Runtime Contract

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

Albus W. Ng, Yi Han, Jusheng Zhang +1

1Vast Intelligence Lab · 2Southwest University · 3Sun Yat-sen University

Artificial Intelligence Safety Agentic Deployments Contracts Trajectory Generation

Ask PDF

Apr 18, 2026cs.SE

Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

Agentic AI systems plan, use tools, maintain state, and act across multi-step workflows with external effects, meaning trustworthy deployment can no longer be judged by task completion alone. The current literature remains fragmented across benchmark-centered evaluation, standards-based governance, orchestration architectures, and runtime assurance mechanisms. This paper contributes a bounded evidence synthesis across a manually coded corpus of twenty-four recent sources. The core finding is a governance-to-action closure gap: evaluation tells us whether outcomes were good, governance defines what should be allowed, but neither identifies where obligations bind to concrete actions or how compliance can later be proven. To close that gap, the paper introduces three linked artifacts: (1) a four-layer framework spanning evaluation, governance, orchestration, and assurance; (2) an ODTA runtime-placement test based on observability, decidability, timeliness, and attestability; and (3) a minimum action-evidence bundle for state-changing actions. Across sources, evaluation papers identify safety, robustness, and trajectory-level measurement as open gaps; governance frameworks define obligations but omit execution-time control logic; orchestration research positions the control plane as the locus of policy mediation, identity, and telemetry; runtime-governance work shows path-dependent behavior cannot be governed through prompts or static permissions alone; and action-safety studies show text alignment does not reliably transfer to tool actions. A worked enterprise procurement-agent scenario illustrates how these artifacts consolidate existing evidence without introducing new experimental data.

Christopher Koch, Joshua Andreas Wellbrock

Independent Researcher

Artificial Intelligence Governance Governance Governed Orchestration

Ask PDF

Sep 28, 2026cs.CL

Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance

We propose RegLLM, a diagnostic harness for bounded autonomy in regulated agentic workflows. It instruments six trustworthiness signals: citation validity, source grounding, schema compliance, escalation correctness, constitutional alignment, and unsafe-action rate. Signals are distinguished by their source of supervision: programmatic verifiers, task-level escalation labels, or AI-judge scores. A deterministic runtime supervisor blocks ungrounded answers and forces escalation, logging interventions. The same domain constitution informs evaluation, training rewards, and serving guardrails. Task-level should-escalate labels make the act-versus-defer decision a measurable training signal. We demonstrate the harness at smoke scale. An offline reference run (n=12) lifts escalation recall from 0 to 0.67 and reduces unsafe-action rate from 0.33 to 0.08 when governance is enabled. Two single-GPU Qwen2.5-3B LoRA/DPO pilots (n=8, same seed and evaluation split) expose substantial variation: nominally identical RL-base configurations yield task success of 0.25 versus 0.12 and escalation recall of 1.0 versus 0.5. An answer-quality adapter changes recall from 1.0 to 0.5 in Run A, but from 0.5 to 1.0 in Run B. An escalation-aware variant produces no measurable change in Run B. These small pilots do not establish reliable adapter effects or production readiness. Their contribution is diagnostic: configuration variance can overwhelm apparent tuning effects on bounded-autonomy metrics, motivating larger evaluation sets and repeated runs.

Dipankar Sarkar

Skelf Research

Escalation AI Trustworthiness Diagnosis

Ask PDF