Convex AI Compositionality and the Governance of AI System Populations

Andrea Ferrario

cs.AISep 21, 2026

Convex AI Compositionality and the Governance of AI System Populations

Authors: Andrea Ferrario

Organizations: Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland · SUPSI, Dalle Molle Institute for Artificial Intelligence (IDSIA), Lugano, Switzerland · ETH Zürich, Zürich, Switzerland

Abstract

AI governance increasingly requires providers and public authorities to reason about multiple AI instantiations, alternative versions, and deployment configurations of multiple AI systems. Yet current regulation remains predominantly single-system-centric, acknowledging such multiplicity only sparsely without treating collections of related AI systems as governance objects. This creates an AI population governance problem: determining which instantiations can be meaningfully considered together and how their changing configurations can be represented and monitored. The first requirement has recently been addressed through trustworthiness-based accounts of AI identity. We address the second by introducing convex AI compositionality: a formal representation of the configurations generated by finite AI system populations that uses convex spaces. The core idea is that convex compositions of the operational states that a population of AI system instantiations may occupy over time are compatible with lifecycle reachability across the population and can preserve the formal identity relations between these systems. Well-known statistical and geometric constructions, such as weighted state distributions and convex hulls, become AI governance tools for distinguishing operational states, population weights, heterogeneity, and AI configuration change across different governance modes while remaining compatible, under stated conditions, with lifecycle reachability and AI identity. We illustrate our AI population governance framework through distributed healthcare deployments and controlled deployment of recruitment AI variants.

Figures & tables

Panel of Figure 1 — Figure 1: Examples of functions $L_{P}$ . (a) A one-dimensional stepwise example with profile coordinate $x_{1}$ and thresholds at $0.57$ and $0.84$ . (b) A two-dimensional example represented as a stepwise-constant graph over $[0,1]^{2}$ , with four rectangular level regions. (c) A three-dimensional example represented by the partition of the cube $[0,1]^{3}$ into six rectangular level regions $C_{1},\dots,C_{6}$ .

Panel of Figure 2 — Figure 2: Diagrammatic summary of the categorical construction in ( Ferrario, 2026a ) . (a) The type datum $\diamondsuit=(F,P,L_{P})$ specifies the techno-function, trustworthiness profile, and trustworthiness level function. (b) The state space $S_{\diamondsuit}$ is partitioned into level fibers $S_{\diamondsuit,k}$ , with primitive lifecycle transformations represented by relations between states, possibly across fibers. (c) The thin category $\mathsf{Sys}_{\diamondsuit}$ retains only trustworthiness-preserving reachability; the thicker arrows indicate the structure underlying weak and strong AI state identity.

Panel of Figure 3 — Figure 3: (a) Partition of the cube $[0,1]^{3}$ into six regions $C_{1},\dots,C_{6}$ . (b) Schematic migration of a population support over time: the quantified profiles of anchor states move from $C_{5}$ at $t_{2}$ to $C_{3}$ at $t_{3}$ , thereby changing the convex hull from $H_{t_{2}}$ to $H_{t_{3}}$ . The stars denote the associated barycenters. (c) Governance example with four fixed anchor states in the same trustworthiness region. At $t_{0}$ and $t_{1}$ , the anchors—and therefore the convex hull—remain unchanged, while the population allocation $\lambda(t)=(\lambda_{1}(t),\dots,\lambda_{4}(t))$ shifts across them, moving the barycenter and changing heterogeneity. The weights represent allocation frequencies, $\lambda_{i}(t)=m_{i}(t)/m$ , so the AI population may contain arbitrarily many AI instantiations; only their relative allocation is shown. In the example, the weighted variance increases by approximately 55%, while $\|\lambda(t_{1})-\lambda(t_{0})\|_{1}=\sum_{i=1}^{N}\left|\lambda_{i}(t_{1})-\lambda_{i}(t_{0})\right|$ quantifies the magnitude of the redistribution.

Panel of Figure 4 — Figure 4: Two-step AI population governance workflow. Step I fixes and operationalizes the comparison domain $\diamondsuit$ , while Step II selects governance-relevant anchor states and constructs and monitors the corresponding population-level governance objects. The workflow is common to the governance modes summarized in Table 1 . Abbreviation: TW, trustworthiness.

Mode	Purpose	Evidence and anchors	Procedure	Signals and actions	Risks
Retrospective	$\bullet$ Audit, incident analysis $\bullet$ Reassessment	$\bullet$ Archived states and weights $\bullet$ Historical anchors	$\bullet$ Reconstruct $H_{t}(\mathcal{P}),\mu_{t}(\mathcal{P})$ over time and anchor transformations in $\mathcal{U}_{\diamondsuit}$ $\bullet$ Trace state, weight, barycenter and variance changes	$\bullet$ TW level transitions, heterogeneity changes $\bullet$ Anchor occupancy drops to zero $\bullet$ Incident audit or reassessment	$\bullet$ Missing or changed records $\bullet$ Selection or protocol bias
Operational	$\bullet$ Lifecycle monitoring $\bullet$ Post-market control	$\bullet$ Current states and weights $\bullet$ Occupied anchors and provenance	$\bullet$ Update anchors, $H_{t}(\mathcal{P}),\mu_{t}(\mathcal{P})$ $\bullet$ Track TW level and heterogeneity over time	$\bullet$ Crossings, new states, weight shifts $\bullet$ Review, restrict, or rollback	$\bullet$ Noisy or delayed evidence $\bullet$ Aggregate or anchor masking
Prospective	$\bullet$ Pre-deployment scenarios $\bullet$ Population update assessment	$\bullet$ Evidence, constraints, assumptions $\bullet$ Candidate or counterfactual anchors	$\bullet$ Generate states and weights over time across scenarios $\bullet$ Compare resulting $H_{t^{\prime}}(\mathcal{P}),\mu_{t^{\prime}}(\mathcal{P})$ and their metrics	$\bullet$ Crossings or critical configurations $\bullet$ Adapt or block deployment	$\bullet$ Scenario misspecification $\bullet$ Distribution shift

Table 1: Three modes of population-level AI governance. All modes use the workflow in Figure 4 . Abbreviation: TW, trustworthiness.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Role in the formalization
$F$	Techno-function : the designed technical capability that fixes the AI system kind at the selected level of abstraction.
$P=\{p_{1},\ldots,p_{n}\}$	Trustworthiness profile : the selected dimensions together with their requirements, measurement conventions, evidential conditions, normalization rules, and aggregation procedures.
$q\in I_{P}=[0,1]^{n}$	Quantified profile : a particular assessment produced under $P$ . Unlike $F,P,L_{P}$ , it varies across states, deployments, copies, variants, and times.
$L_{P}:I_{P}\to[K]$	Trustworthiness level function : the auditable rule assigning quantified profiles to a finite collection of governance-relevant levels ( Ferrario, 2026b ) .
$x=(q_{x},L_{P}(q_{x}))\in S_{\diamondsuit}$	$\diamondsuit$ -relative state : the abstract state determined by a quantified profile and the level assigned to it.
$\mathcal{U}_{\diamondsuit}$	Primitive lifecycle vocabulary : the admissible elementary transformation types considered under the fixed datum $\diamondsuit$ , each represented by a relation on $S_{\diamondsuit}$ .

Appendix

Table 2: Primitive components used in the present paper. Adapted from Ferrario (2026a) ; notation and descriptions are aligned with the present construction.

TW dimension	Representative evaluation measures / strategies	Representation in $P$
Technical robustness & safety	$\bullet$ Accuracy, AUROC $\bullet$ Brier score, NLL, ECE $\bullet$ Performance under distribution shift $\bullet$ Certified accuracy $\bullet$ Attack success and fooling rates $\bullet$ OOD detection confidence scores	Higher-is-better performance measures are normalized directly; losses, errors, and attack rates are reversed. $P$ fixes the test population, shift/attack regime, and evaluation protocol.
Diversity, non-discrimination & fairness	$\bullet$ Demographic/statistical parity difference $\bullet$ Disparate impact $\bullet$ Equal opportunity $\bullet$ Equalized odds $\bullet$ Predictive parity $\bullet$ Generalized Entropy Index	Fairness measures are generally target-based. $P$ fixes protected groups, thresholds, targets, and the evaluation population.
Transparency & explainability	$\bullet$ Faithfulness and fidelity $\bullet$ Explanation robustness $\bullet$ Sparsity and complexity $\bullet$ Information loss and gain $\bullet$ Human task accuracy $\bullet$ Task time and satisfaction	Quantitative and human-evaluation measures are normalized according to their direction. Ordinal or subjective scales require a documented numerical encoding.
Privacy & data governance	$\bullet$ $k$ -anonymity, $l$ -diversity, $t$ -closeness $\bullet$ Differential-privacy $\epsilon,\delta$ $\bullet$ Membership-inference attack success rate and TPR at low FPR $\bullet$ Model-extraction fidelity/performance $\bullet$ Model-inversion performance	$P$ fixes the privacy mechanism, threat model, and admissible ranges. Lower-is-better privacy-loss and attack measures are reversed.
Human agency & oversight	$\bullet$ Human error rate $\bullet$ Task-completion time $\bullet$ Cognitive load $\bullet$ Number of user actions $\bullet$ Compliance rate $\bullet$ Trust, reliance, and trust calibration $\bullet$ User satisfaction	Performance and usability measures are oriented consistently; reliance and trust may require target-based or psychometric encoding.
Accountability & auditability	$\bullet$ Attribution analysis $\bullet$ Algorithmic auditing $\bullet$ Legal and regulatory compliance assessment $\bullet$ Remediation and redress evaluation $\bullet$ Impact assessments	These are primarily qualitative, categorical, or procedural evaluations; few established quantitative metrics exist. Numerical inclusion in $P$ therefore requires an explicit operationalization.

Appendix

Table 3: Representative trustworthiness evaluation measures and strategies that may be used to construct the quantified profile

q

under protocol

P

. The first seven rows draw primarily on the quantitative evaluation catalogue of Kemmerzell et al. (2025) , with broader trustworthiness framing and additional examples from Kaur et al. (2022) , Li et al. (2023) , and Díaz-Rodríguez et al. (2023) . The final four rows summarize the AI-agent reliability dimensions and metrics of Rabanser et al. (2026) .

Panel of Figure 5 — Figure 5: A trustworthiness level function $L_{P}:[0,1]^{2}\to[3]$ with non-convex fibers. Both $C_{1}$ and $C_{2}$ are L-shaped and therefore non-convex. In particular, $C_{2}$ is not closed under convex combinations: two points in $C_{2}$ may have a midpoint in $C_{3}$ .

AI Governance

Artificial intelligence governance studies how laws, institutions, and technical controls shape AI development, deployment, accountability, and social effects.

See all 386 papers

Apr 30, 2026

The Two Boundaries: Why Behavioral AI Governance Fails Structurally

Every system that performs effects has two boundaries: what it can do (expressiveness) and what governance covers (governance). In nearly all deployed AI systems, these boundaries are defined independently, creating three regions: governed capabilities (the only useful region), ungoverned capabilities (risk), and governance policies that address non-existent capabilities (theater). Two of the three regions are failure modes. We focus on the governance of effects: actions that AI systems perform in the world (API calls, database writes, tool invocations). This is distinct from the governance of model outputs (content quality, bias, fairness), which operates at a different level and requires different mechanisms. We present a formal framework for analyzing this structural gap. Rice's theorem (1953) proves the gap is undecidable in the general case for any Turing-complete architecture that attempts to govern effects behaviorally: no algorithm can decide non-trivial semantic properties of arbitrary programs, including the property "this program's effects comply with the governance policy." We define coterminous governance: a system property where the expressivenessboundary equals the governance boundary. We show that coterminous governance requires an architectural decision (separatingcomputation from effect) rather than a governance layer added after the fact. We show that structural governance under this separation subsumes separate governance infrastructure: governance checks become part of the execution pipeline rather than a second system running alongside it. We propose coterminous governance as the testable criterion for any AI governance system: either the two boundaries are provably identical, or risk and theater are structurally inevitable. Proofs are mechanized in Coq (454 theorems, 36 modules, 0 admitted).

Alan L. McCann

Mashin, Inc.

Capability-Based Security AI Agent Governance AI Governance

Ask PDF

May 23, 2026

Is Decentralized AI Governable? From Regulative Policy to Constitutive Protocol

Every major framework for governing artificial intelligence presupposes an identifiable entity -- a developer, deployer, or operator -- who can be held responsible and compelled to comply. Decentralized AI (DeAI) dissolves this presupposition. We analyze DeAI as a six-layer decentralizing stack -- model, training, compute, harness, identity, and ownership -- and show how partial decentralization across layers compounds into what we call the governance vacuum: a condition in which AI systems are consequential enough to require governance but lack the properties that existing frameworks presuppose in their targets. This vacuum takes two analytically distinct forms: an accountability gap, where no addressable principal can be identified, and an incapacitation gap, where even an identified principal cannot alter the running system. We demonstrate that these failures are not merely jurisdictional but defeat every presupposition of governance through normative address -- the communication of rules to a comprehending, responsive agent. Drawing on Lessig's modalities of regulation and Searle's distinction between regulative and constitutive rules, we argue for a shift in the locus of governance from policy to protocol, from normative address to architectural constraint. Protocol-based constitutive governance does not address the agents operating within a system but shapes the substrate that determines what kinds of actions are possible within it. We identify four ethical conditions -- legitimacy, contestability, transparency, and non-domination -- that such governance must satisfy to avoid degenerating into unaccountable technocratic power, and we argue that the central political challenge of governing AI in a decentralized world is reconstructing forms of democratic authorization for architectural choices that persist after the ordinary chain of policy has broken down.

Botao Amber Hu, Helena Rong

University of Oxford, UK. · New York University Shanghai, China.

AI Accountability AI Governance

Ask PDF

Apr 7, 2026

Who Governs the Machine? A Machine Identity Governance Taxonomy (MIGT) for AI Systems Operating Across Enterprise and Geopolitical Boundaries

The governance of artificial intelligence has a blind spot: the machine identities that AI systems use to act. AI agents, service accounts, API tokens, and automated workflows now outnumber human identities in enterprise environments by ratios exceeding 80 to 1, yet no integrated framework exists to govern them. A single ungoverned automated agent produced $5.4-10 billion in losses in the 2024 CrowdStrike outage; nation-state actors including Silk Typhoon and Salt Typhoon have operationalized ungoverned machine credentials as primary espionage vectors against critical infrastructure. This paper makes four original contributions. First, the AI-Identity Risk Taxonomy (AIRT): a comprehensive enumeration of 37 risk sub-categories across eight domains, each grounded in documented incidents, regulatory recognition, practitioner prevalence data, and threat intelligence. Second, the Machine Identity Governance Taxonomy (MIGT): an integrated six-domain governance framework simultaneously addressing the technical governance gap, the regulatory compliance gap, and the cross-jurisdictional coordination gap that existing frameworks address only in isolation. Third, a foreign state actor threat model for enterprise identity governance, establishing that Silk Typhoon, Salt Typhoon, Volt Typhoon, and North Korean AI-enhanced identity fraud operations have already operationalized AI identity vulnerabilities as active attack vectors. Fourth, a cross-jurisdictional regulatory alignment structure mapping enterprise AI identity governance obligations under EU, US, and Chinese frameworks simultaneously, identifying irreconcilable conflicts and providing a governance mechanism for managing them. A four-phase implementation roadmap translates the MIGT into actionable enterprise programs.

Andrew Kurtz, Klaudia Krawiecka

AI Governance Cybersecurity AI Agent Security AI Risk Management

Ask PDF

Abstract

AI Governance

Explore similar work

The Two Boundaries: Why Behavioral AI Governance Fails Structurally

Is Decentralized AI Governable? From Regulative Policy to Constitutive Protocol

Who Governs the Machine? A Machine Identity Governance Taxonomy (MIGT) for AI Systems Operating Across Enterprise and Geopolitical Boundaries