Convex AI Compositionality and the Governance of AI System Populations

Andrea Ferrario

cs.AISep 21, 2026

Convex AI Compositionality and the Governance of AI System Populations

Authors: Andrea Ferrario

Organizations: Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland · SUPSI, Dalle Molle Institute for Artificial Intelligence (IDSIA), Lugano, Switzerland · ETH Zürich, Zürich, Switzerland

Abstract

AI governance increasingly requires providers and public authorities to reason about multiple AI instantiations, alternative versions, and deployment configurations of multiple AI systems. Yet current regulation remains predominantly single-system-centric, acknowledging such multiplicity only sparsely without treating collections of related AI systems as governance objects. This creates an AI population governance problem: determining which instantiations can be meaningfully considered together and how their changing configurations can be represented and monitored. The first requirement has recently been addressed through trustworthiness-based accounts of AI identity. We address the second by introducing convex AI compositionality: a formal representation of the configurations generated by finite AI system populations that uses convex spaces. The core idea is that convex compositions of the operational states that a population of AI system instantiations may occupy over time are compatible with lifecycle reachability across the population and can preserve the formal identity relations between these systems. Well-known statistical and geometric constructions, such as weighted state distributions and convex hulls, become AI governance tools for distinguishing operational states, population weights, heterogeneity, and AI configuration change across different governance modes while remaining compatible, under stated conditions, with lifecycle reachability and AI identity. We illustrate our AI population governance framework through distributed healthcare deployments and controlled deployment of recruitment AI variants.

Figures & tables

Panel of Figure 1 — Figure 1: Examples of functions $L_{P}$ . (a) A one-dimensional stepwise example with profile coordinate $x_{1}$ and thresholds at $0.57$ and $0.84$ . (b) A two-dimensional example represented as a stepwise-constant graph over $[0,1]^{2}$ , with four rectangular level regions. (c) A three-dimensional example represented by the partition of the cube $[0,1]^{3}$ into six rectangular level regions $C_{1},\dots,C_{6}$ .

Panel of Figure 2 — Figure 2: Diagrammatic summary of the categorical construction in ( Ferrario, 2026a ) . (a) The type datum $\diamondsuit=(F,P,L_{P})$ specifies the techno-function, trustworthiness profile, and trustworthiness level function. (b) The state space $S_{\diamondsuit}$ is partitioned into level fibers $S_{\diamondsuit,k}$ , with primitive lifecycle transformations represented by relations between states, possibly across fibers. (c) The thin category $\mathsf{Sys}_{\diamondsuit}$ retains only trustworthiness-preserving reachability; the thicker arrows indicate the structure underlying weak and strong AI state identity.

Panel of Figure 3 — Figure 3: (a) Partition of the cube $[0,1]^{3}$ into six regions $C_{1},\dots,C_{6}$ . (b) Schematic migration of a population support over time: the quantified profiles of anchor states move from $C_{5}$ at $t_{2}$ to $C_{3}$ at $t_{3}$ , thereby changing the convex hull from $H_{t_{2}}$ to $H_{t_{3}}$ . The stars denote the associated barycenters. (c) Governance example with four fixed anchor states in the same trustworthiness region. At $t_{0}$ and $t_{1}$ , the anchors—and therefore the convex hull—remain unchanged, while the population allocation $\lambda(t)=(\lambda_{1}(t),\dots,\lambda_{4}(t))$ shifts across them, moving the barycenter and changing heterogeneity. The weights represent allocation frequencies, $\lambda_{i}(t)=m_{i}(t)/m$ , so the AI population may contain arbitrarily many AI instantiations; only their relative allocation is shown. In the example, the weighted variance increases by approximately 55%, while $\|\lambda(t_{1})-\lambda(t_{0})\|_{1}=\sum_{i=1}^{N}\left|\lambda_{i}(t_{1})-\lambda_{i}(t_{0})\right|$ quantifies the magnitude of the redistribution.

Panel of Figure 4 — Figure 4: Two-step AI population governance workflow. Step I fixes and operationalizes the comparison domain $\diamondsuit$ , while Step II selects governance-relevant anchor states and constructs and monitors the corresponding population-level governance objects. The workflow is common to the governance modes summarized in Table 1 . Abbreviation: TW, trustworthiness.

Mode	Purpose	Evidence and anchors	Procedure	Signals and actions	Risks
Retrospective	$\bullet$ Audit, incident analysis $\bullet$ Reassessment	$\bullet$ Archived states and weights $\bullet$ Historical anchors	$\bullet$ Reconstruct $H_{t}(\mathcal{P}),\mu_{t}(\mathcal{P})$ over time and anchor transformations in $\mathcal{U}_{\diamondsuit}$ $\bullet$ Trace state, weight, barycenter and variance changes	$\bullet$ TW level transitions, heterogeneity changes $\bullet$ Anchor occupancy drops to zero $\bullet$ Incident audit or reassessment	$\bullet$ Missing or changed records $\bullet$ Selection or protocol bias
Operational	$\bullet$ Lifecycle monitoring $\bullet$ Post-market control	$\bullet$ Current states and weights $\bullet$ Occupied anchors and provenance	$\bullet$ Update anchors, $H_{t}(\mathcal{P}),\mu_{t}(\mathcal{P})$ $\bullet$ Track TW level and heterogeneity over time	$\bullet$ Crossings, new states, weight shifts $\bullet$ Review, restrict, or rollback	$\bullet$ Noisy or delayed evidence $\bullet$ Aggregate or anchor masking
Prospective	$\bullet$ Pre-deployment scenarios $\bullet$ Population update assessment	$\bullet$ Evidence, constraints, assumptions $\bullet$ Candidate or counterfactual anchors	$\bullet$ Generate states and weights over time across scenarios $\bullet$ Compare resulting $H_{t^{\prime}}(\mathcal{P}),\mu_{t^{\prime}}(\mathcal{P})$ and their metrics	$\bullet$ Crossings or critical configurations $\bullet$ Adapt or block deployment	$\bullet$ Scenario misspecification $\bullet$ Distribution shift

Table 1: Three modes of population-level AI governance. All modes use the workflow in Figure 4 . Abbreviation: TW, trustworthiness.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Role in the formalization
$F$	Techno-function : the designed technical capability that fixes the AI system kind at the selected level of abstraction.
$P=\{p_{1},\ldots,p_{n}\}$	Trustworthiness profile : the selected dimensions together with their requirements, measurement conventions, evidential conditions, normalization rules, and aggregation procedures.
$q\in I_{P}=[0,1]^{n}$	Quantified profile : a particular assessment produced under $P$ . Unlike $F,P,L_{P}$ , it varies across states, deployments, copies, variants, and times.
$L_{P}:I_{P}\to[K]$	Trustworthiness level function : the auditable rule assigning quantified profiles to a finite collection of governance-relevant levels ( Ferrario, 2026b ) .
$x=(q_{x},L_{P}(q_{x}))\in S_{\diamondsuit}$	$\diamondsuit$ -relative state : the abstract state determined by a quantified profile and the level assigned to it.
$\mathcal{U}_{\diamondsuit}$	Primitive lifecycle vocabulary : the admissible elementary transformation types considered under the fixed datum $\diamondsuit$ , each represented by a relation on $S_{\diamondsuit}$ .

Appendix

Table 2: Primitive components used in the present paper. Adapted from Ferrario (2026a) ; notation and descriptions are aligned with the present construction.

TW dimension	Representative evaluation measures / strategies	Representation in $P$
Technical robustness & safety	$\bullet$ Accuracy, AUROC $\bullet$ Brier score, NLL, ECE $\bullet$ Performance under distribution shift $\bullet$ Certified accuracy $\bullet$ Attack success and fooling rates $\bullet$ OOD detection confidence scores	Higher-is-better performance measures are normalized directly; losses, errors, and attack rates are reversed. $P$ fixes the test population, shift/attack regime, and evaluation protocol.
Diversity, non-discrimination & fairness	$\bullet$ Demographic/statistical parity difference $\bullet$ Disparate impact $\bullet$ Equal opportunity $\bullet$ Equalized odds $\bullet$ Predictive parity $\bullet$ Generalized Entropy Index	Fairness measures are generally target-based. $P$ fixes protected groups, thresholds, targets, and the evaluation population.
Transparency & explainability	$\bullet$ Faithfulness and fidelity $\bullet$ Explanation robustness $\bullet$ Sparsity and complexity $\bullet$ Information loss and gain $\bullet$ Human task accuracy $\bullet$ Task time and satisfaction	Quantitative and human-evaluation measures are normalized according to their direction. Ordinal or subjective scales require a documented numerical encoding.
Privacy & data governance	$\bullet$ $k$ -anonymity, $l$ -diversity, $t$ -closeness $\bullet$ Differential-privacy $\epsilon,\delta$ $\bullet$ Membership-inference attack success rate and TPR at low FPR $\bullet$ Model-extraction fidelity/performance $\bullet$ Model-inversion performance	$P$ fixes the privacy mechanism, threat model, and admissible ranges. Lower-is-better privacy-loss and attack measures are reversed.
Human agency & oversight	$\bullet$ Human error rate $\bullet$ Task-completion time $\bullet$ Cognitive load $\bullet$ Number of user actions $\bullet$ Compliance rate $\bullet$ Trust, reliance, and trust calibration $\bullet$ User satisfaction	Performance and usability measures are oriented consistently; reliance and trust may require target-based or psychometric encoding.
Accountability & auditability	$\bullet$ Attribution analysis $\bullet$ Algorithmic auditing $\bullet$ Legal and regulatory compliance assessment $\bullet$ Remediation and redress evaluation $\bullet$ Impact assessments	These are primarily qualitative, categorical, or procedural evaluations; few established quantitative metrics exist. Numerical inclusion in $P$ therefore requires an explicit operationalization.

Appendix

Table 3: Representative trustworthiness evaluation measures and strategies that may be used to construct the quantified profile

q

under protocol

P

. The first seven rows draw primarily on the quantitative evaluation catalogue of Kemmerzell et al. (2025) , with broader trustworthiness framing and additional examples from Kaur et al. (2022) , Li et al. (2023) , and Díaz-Rodríguez et al. (2023) . The final four rows summarize the AI-agent reliability dimensions and metrics of Rabanser et al. (2026) .

Panel of Figure 5 — Figure 5: A trustworthiness level function $L_{P}:[0,1]^{2}\to[3]$ with non-convex fibers. Both $C_{1}$ and $C_{2}$ are L-shaped and therefore non-convex. In particular, $C_{2}$ is not closed under convex combinations: two points in $C_{2}$ may have a midpoint in $C_{3}$ .

AI Governance

Artificial intelligence governance studies how laws, institutions, and technical controls shape AI development, deployment, accountability, and social effects.

See all 386 papers

The Two Boundaries: Why Behavioral AI Governance Fails Structurally
Apr 30, 2026Alan L. McCannCapability-Based Security AI Agent Governance
Mashin, Inc.
Is Decentralized AI Governable? From Regulative Policy to Constitutive Protocol
May 23, 2026Botao Amber Hu, Helena RongAI Accountability AI Governance
University of Oxford, UK. · New York University Shanghai, China.
Who Governs the Machine? A Machine Identity Governance Taxonomy (MIGT) for AI Systems Operating Across Enterprise and Geopolitical Boundaries
Apr 7, 2026Andrew Kurtz, Klaudia KrawieckaAI Governance Cybersecurity

Apr 30, 2026J/K move · Enter open · S save

The Two Boundaries: Why Behavioral AI Governance Fails Structurally

Alan L. McCann

Mashin, Inc.

Capability-Based Security AI Agent Governance AI Governance

Open paper AskPDF arXiv

Every system that performs effects has two boundaries: what it can do (expressiveness) and what governance covers (governance). In nearly all deployed AI systems, these boundaries are defined independently, creating three regions: governed capabilities (the only useful region), ungoverned capabilities (risk), and governance policies that address non-existent capabilities (theater). Two of the three regions are failure modes. We focus on the governance of effects: actions that AI systems perform in the world (API calls, database writes, tool invocations). This is distinct from the governance of model outputs (content quality, bias, fairness), which operates at a different level and requires different mechanisms. We present a formal framework for analyzing this structural gap. Rice's theorem (1953) proves the gap is undecidable in the general case for any Turing-complete architecture that attempts to govern effects behaviorally: no algorithm can decide non-trivial semantic properties of arbitrary programs, including the property "this program's effects comply with the governance policy." We define coterminous governance: a system property where the expressivenessboundary equals the governance boundary. We show that coterminous governance requires an architectural decision (separatingcomputation from effect) rather than a governance layer added after the fact. We show that structural governance under this separation subsumes separate governance infrastructure: governance checks become part of the execution pipeline rather than a second system running alongside it. We propose coterminous governance as the testable criterion for any AI governance system: either the two boundaries are provably identical, or risk and theater are structurally inevitable. Proofs are mechanized in Coq (454 theorems, 36 modules, 0 admitted).

Abstract

AI Governance

Explore similar work

The Two Boundaries: Why Behavioral AI Governance Fails Structurally

Is Decentralized AI Governable? From Regulative Policy to Constitutive Protocol

Who Governs the Machine? A Machine Identity Governance Taxonomy (MIGT) for AI Systems Operating Across Enterprise and Geopolitical Boundaries