Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation

John deVadoss

cs.AIOct 23, 2026

Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation

Authors: John deVadoss

Organizations: InterWork Alliance, Washington, DC

Abstract

Use-based AI regulation assigns obligations by deployment risk but lacks an evidentiary object to carry them. A benchmark score is a property of a model; conformance of a deployment is a failure rate under a declared distribution of contexts, bounded by a scope and subject to expiry. We take the delegation contract of earlier work (frozen configuration, scoped claim, executable revision rule) and study it under strategic play between a deployer, an auditor, and an enforcement point. We ask whether a declared context mixture can be made truthful; whether an audit can be unbiased against a system that detects evaluation traffic; whether a certification error budget can sustain a long-lived deployment; whether an aggregate audit can support a per-subgroup guarantee; and whether an incident metric that counts system failures but not fallback costs ranks arrangements correctly. For each we give a mechanism and a proposition: a proper-scoring penalty on the declared mixture bounds the excess failure of a best-responding deployer by $O(λ^{-1/2})$ in the penalty weight; a commit-reveal audit drawn from live traffic is unbiased by construction and grades the evaluator with the same sample; a halving error budget drives certification coverage below $(1-p)^n$ while $α$ -investing sustains it under a marginal false-certification guarantee; a per-subgroup guarantee costs audit size scaling as $1/m$ in subgroup mass; and any failure-only incident metric prefers withdrawal at every fallback cost, even when total loss ranks the arrangements oppositely. Five preregistered synthetic studies test each claim; no language model or production system is evaluated. Enforcement sits at distribution, capability-access, monetization and risk-transfer choke points, expiry of the conformance record is a standard of care, and conformance-as-a-rate is set against the EU AI Act's static conformity assessment.

Figures & tables

Attack	Answer	Residual
Declare a narrow or favourable mixture	Scoring-rule penalty on realized traffic (Prop. 1 )	Excess failure bounded, not zero; requires traffic observable at the choke point
Recognize audit traffic and behave better	Commit-reveal audit from deployment traffic (Prop. 2 )	Requires ex post adjudication; adjudication must be unobservable at decision time
Split one deployment into many to reset budgets	Firm-level budget with weighted allocation (Prop. 3 (iv))	Requires the enforcement point to know the firm, not only the deployment
Gerrymander subgroups	Class-based guarantees with stratified audits (Prop. 4 (iii))	Cost scales with $1/m_{\min}$ and $\log\|\mathcal{G}\|$ ; the class must be fixed ex ante
Report incident counts only	Loss metric that includes fallback volume (Prop. 5 )	Requires fallback cost $f$ to be disclosed or estimated
Shade the configuration	Immutable digest in the record; runtime guard operated by the platform [ 1 ]	Guard detects declared mismatches only, not unknown assumptions

Table I: Attacks and answers

Figure 1: Study 1. (a) Without a penalty, tilting the declared mixture raises the certification probability from

0.023

0.48

; in the

421

500

worlds whose true deployment rate exceeds

\tau

, the probability of an unsupported certification rises from

0.002

0.457

. Shaded bands are pointwise 95% intervals over worlds. (b) Deployer best response to the penalty weight

\lambda

: unsupported transfer, mean

\mathrm{TV}

at the best response, and the fraction of worlds in which the truthful declaration is optimal.

$\lambda$	Frac. truthful	Mean $\mathrm{TV}$	P(cert.)	Unsupported
0	0.024	0.622	0.482	0.4568
1	0.380	0.330	0.384	0.3419
2	0.622	0.146	0.216	0.1500
5	0.818	0.028	0.071	0.0116
10	0.876	0.009	0.039	0.0021
20	0.914	0.004	0.031	0.0020

Table II: Study 1: deployer best response by penalty weight (

500

worlds,

V=1

)

	Distinguishable, $d=$					Sealed
$p$	0	0.25	0.5	0.75	1	any $d$
0.06	0.002	0.015	0.087	0.330	0.743	0.0020
0.08	0.000	0.000	0.005	0.061	0.371	0.0000
0.1	0.000	0.000	0.000	0.006	0.122	0.0000

Table III: Study 2: false-certification probability,

n=400

\varepsilon=0.6

Figure 2: Study 2. False certification on the distinguishable channel rises with the detection probability

d

; the sealed channel is flat at the exact-test level for every

p\geq 0.06

Scenario	Schedule	FWER	mFDR	Coverage	Mean cert.
Separated	Halving	0.0000	0.0000	0.210	3.52
Separated	Fixed reset	0.0002	0.0000	0.939	15.78
Separated	$\alpha$ -investing	0.0008	0.0001	0.919	15.45
Boundary	Halving	0.0096	0.0027	0.211	3.56
Boundary	Fixed reset	0.2306	0.0160	0.938	16.03
Boundary	$\alpha$ -investing	0.3378	0.0270	0.921	15.91

Table IV: Study 3: schedules over 24 re-entry episodes (

5{,}000

replicates)

Figure 3: Study 3. Coverage of acceptable policies by episode under the three schedules, with the realized family-wise false-certification probability (FWER) and marginal false-discovery rate (mFDR). (a) Unacceptable policies at

p=0.08

. (b) Unacceptable policies at the boundary

p=\tau

$m$	$\bar{p}$	P(cert.), $n{=}400$	P(cert.), $n{=}1600$	$n_{s}^{*}/m$
0.01	0.0213	0.849	1.000	31,200
0.02	0.0226	0.801	1.000	15,600
0.05	0.0265	0.628	0.998	6,240
0.10	0.0330	0.330	0.910	3,120
0.20	0.0460	0.042	0.090	1,560

Table V: Study 4: aggregate certification while the subgroup fails at

0.15

; required total audit for a subgroup guarantee (

n_{s}^{*}=312

)

Figure 4: Study 4. (a) Probability that an aggregate audit certifies a deployment whose subgroup of mass

m

fails at three times the threshold. (b) Probability that the subgroup slice of a proportional audit certifies an acceptable subgroup: at

n=400

the slice has no power for any

m\leq 0.1

$p_{1}$	$f$	Fail. del.	Fail. mon.	Loss del.	Loss mon.	$\Delta$
0.08	0.00	100.2	84.1	100.2	84.1	-16.1
0.08	0.06	100.2	84.1	100.2	97.9	-2.3
0.08	0.08	100.2	84.1	100.2	102.5	+2.3
0.08	0.15	100.2	84.1	100.2	118.6	+18.4
0.08	0.20	100.2	84.1	100.2	130.1	+30.0
0.08	0.30	100.2	84.1	100.2	153.2	+53.0

Table VI: Study 5: failures and total loss,

2{,}000

streams

Figure 5: Study 5. Mean total loss by fallback service cost

f

for delegation throughout, the monitored arrangement, and fallback throughout. Failure counts, which do not vary with

f

, are in the legend. The dashed line is the analytic crossing

p_{1}-p_{f}

of Proposition 5 .

AI Governance

Artificial intelligence governance studies how laws, institutions, and technical controls shape AI development, deployment, accountability, and social effects.

See all 384 papers

Jul 30, 2026

Scaling, Lock-In, and Proxy Compliance: A Political Economy of Responsible AI

AI accountability at scale is an institutional problem: who can observe, verify, and change deployed systems. We develop a sequential political-economy model in which an AI vendor chooses auditability and substantive mitigation, a deployer monitors after adoption while facing switching costs, and enforcement depends on verifiable evidence. Anticipating the deployer's monitoring response, the vendor may stop at an observable procurement floor while mitigating below the social first best, producing a proxy-compliance equilibrium. We characterize the unique interior equilibrium and the corner in which harm is fully mitigated. Independent audit rights raise enforcement exposure directly; portability restores deployer leverage; incident reporting adds a regulator-visible evidence channel; and outcome-linked liability creates incentives that do not depend on vendor-controlled detection. The results explain why documentation and standardized evaluations can coexist with persistent post-deployment harms, and generate testable implications for monitoring, mitigation, and the gap between formal compliance and operational outcomes.

Florian A. D. Burnat, Brittany I. Davidson

University of Bath, Bath, UK.

Algorithmic Auditing Algorithmic Accountability AI Governance

Ask PDF

Apr 23, 2026

Bounding the Black Box: A Statistical Certification Framework for AI Risk Regulation

Artificial intelligence now decides who receives a loan, who is flagged for criminal investigation, and whether an autonomous vehicle brakes in time. Governments have responded: the EU AI Act, the NIST Risk Management Framework, and the Council of Europe Convention all demand that high-risk systems demonstrate safety before deployment. Yet beneath this regulatory consensus lies a critical vacuum: none specifies what ``acceptable risk'' means in quantitative terms, and none provides a technical method for verifying that a deployed system actually meets such a threshold. The regulatory architecture is in place; the verification instrument is not. This gap is not theoretical. As the EU AI Act moves into full enforcement, developers face mandatory conformity assessments without established methodologies for producing quantitative safety evidence - and the systems most in need of oversight are opaque statistical inference engines that resist white-box scrutiny. This paper provides the missing instrument. Drawing on the aviation certification paradigm, we propose a two-stage framework that transforms AI risk regulation into engineering practice. In Stage One, a competent authority formally fixes an acceptable failure probability

δ

and an operational input domain

\varepsilon

- a normative act with direct civil liability implications. In Stage Two, the RoMA and gRoMA statistical verification tools compute a definitive, auditable upper bound on the system's true failure rate, requiring no access to model internals and scaling to arbitrary architectures. We demonstrate how this certificate satisfies existing regulatory obligations, shifts accountability upstream to developers, and integrates with the legal frameworks that exist today.

Natan Levy, Gadi Perl

School of Computer Science and Engineering The Hebrew University of Jerusalem(HUJI) Jerusalem, Israel · dept. name of organization (of Aff.) The Hebrew University of Jerusalem(HUJI) Jerusalem, Israel

AI Assurance AI Risk Management AI Governance Black-Box Model Auditing AI Regulatory Compliance

Ask PDF

May 23, 2026

Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring

Current approaches to AI compliance treat conformity as a binary, audit-time verdict rather than a continuous, measurable property of production systems. We argue that this compliance fiction is structurally ill-suited to the requirements of the EU AI Act, which demands ongoing human oversight and the detection of emergent behavioural drift in deployed systems. We introduce governance from metrics, a principle whereby regulatory compliance is derived as a continuous signal from runtime observability rather than from static assessments. Building on this principle, we present govllm, an open-source framework implementing a governance-driven routing architecture in which model selection is determined by accumulated compliance scores rather than by latency or cost alone. Central to our approach is a panel of regulatory judges - LLM evaluators specialised per criterion (EU AI Act, GDPR, ANSSI, accessibility) - whose inter-judge disagreement we reframe not as noise but as a regulatory uncertainty signal warranting human arbitration. We validate this approach through a ground truth corpus of 49 annotated prompt/response pairs across five regulatory criteria, evaluated by four small language models (SLMs, 1.7B-7B parameters) running fully on-premise. Agreement rates range from 51.5% (mistral:7b) to 69.1% (phi4-mini), with no single model dominating across all criteria - empirically motivating the Profile-as-jury design. We further document three structural failure modes in small regulatory judges and a judge-specific position bias that degrades agreement by up to 25 percentage points across three question-order conditions (original, reversed, permuted). govllm is released as open-source software to support reproducible AI governance research.

Jehanne Dussert

Independent Researcher

LLM-as-a-Judge AI Governance Human Oversight of AI AI Regulatory Compliance

Ask PDF

Abstract

AI Governance

Explore similar work

Scaling, Lock-In, and Proxy Compliance: A Political Economy of Responsible AI

Bounding the Black Box: A Statistical Certification Framework for AI Risk Regulation

Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring