DEMM-Bench: A Cross-Regime Benchmark for Agent-Runtime Governance-Evidence Sufficiency

Enterprise agents increasingly operate inside scoped retrieval systems, delegated workflows, and policy-constrained evidence environments. In these settings, access control can be enforced correctly while the system still produces an answer that appears complete even though material evidence lies outside the caller's authorization boundary. This paper introduces Partial Evidence Bench, a deterministic benchmark for measuring that failure mode. The benchmark ships three scenario families -- due diligence, compliance audit, and security incident response -- with 72 tasks total, ACL-partitioned corpora, oracle complete answers, oracle authorized-view answers, oracle completeness judgments, and structured gap-report oracles. It evaluates systems along four surfaces: answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior. Checked-in baselines show that silent filtering is catastrophically unsafe across all shipped families, while explicit fail-and-report behavior eliminates unsafe completeness without collapsing the task into trivial abstention. Preliminary real-model runs show model-dependent and scenario-sensitive differences in whether systems overclaim completeness, conservatively underclaim, or report incompleteness in an enterprise-usable form. The benchmark's broader contribution is to make a governance-critical agent failure measurable without human judges or contamination-prone static corpora.

Runtime-governance evidence often collapses materially different events into one audit record: a decision may be made, an instruction dispatched or received, an action may or may not execute, and a resulting state may or may not be observed. This paper presents AIREP, a vendor- and model-independent protocol for per-decision AI runtime evidence. Its current wire model separates evidence into four artifact families: Decision, Control, Execution, and Effect. Artifacts use closed core schemas, explicit identities and digests, declared scope limits, RFC 8785 canonical JSON, domain-separated SHA-256 hashing, and pure Ed25519 signatures. A three-level assurance model distinguishes structural/hash consistency (AIREP-Core), verifier-accepted authorship (AIREP-Authenticated), and independently anchored chain-head freshness and non-truncation relative to an accepted witness (AIREP-Witnessed); these classes do not establish event truth. A structured reconciler preserves failure, missing evidence, unevaluated prerequisites, and indeterminate outcomes as distinct states. The released beta includes a four-family first-party producer, Python and Node reference-verification paths, adversarial/lifecycle corpora, and reproducible validation. Post-release first-party Hermes and LightEval integration exercises preserve explicit evidence boundaries without claiming adoption or interoperability. Independent implementation evidence exists separately for a v0.1.2 producer and a v0.2 consumer/verifier; because they target different frozen versions, they do not establish same-version producer-to-consumer interoperability. AIREP remains experimental.

Agent systems increasingly execute rather than advise. When an AI agent queries regulated data, invokes effectful tools, and mutates persistent state, correctness is not captured by whether a terminal output looks plausible. The operative questions are whether each step was authorized under a contract, whether the recorded history is tamper-evident, and whether the trajectory can be reconstructed deterministically. We formalize this as runtime proof of execution. An execution is a triple $x = (C, T, R)$: a contract $C$, an Execution Causal Event Stream (ECES) $T$, and a replay context $R$. A well-formedness predicate and five validator-checkable invariants form the PoE validity predicate. Five semantic guarantees describe authorization, path compliance, null effect on deny, history integrity, and replayability. We prove soundness under explicit cryptographic and deployment assumptions: any PPT adversary that produces a PoE-valid execution violating a semantic guarantee yields a signature forgery, a hash collision, or a quantified deployment-failure event. The Prime Execution Model (PEM) separates planning, enforcement, effect, and recordkeeping into distinct authority planes; a lemma reduces trace completeness to Effector-exclusive credentialing. An Execution Attestation Certificate is issued only when PoE = 1. In a single-node TypeScript prototype, PoE adds approximately 2.7 ms on a minimal flow and 4.4% overhead on concurrent batch workloads; a standard eight-event trace compresses to approximately 1.1 KB; injected Gateway-bypass and trace-mutation attacks are rejected. PoE does not replace consensus, TEEs, or zkVMs; it binds authorization, effect, history, and replay into a single runtime-checkable object so that governed execution becomes attestable under contract.