cs.SINov 12, 2026

Designing Institutions for Autonomous AI

Authors: Petter Törnberg

Organizations: Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Amsterdam, the Netherlands.

Abstract

As autonomous AI agents increasingly interact in persistent populations, their collective capabilities and alignment depend not only on individual models, but also on the institutions governing their interactions. Yet AI research has largely focused on individual agents. In this Perspective, we argue that institutional design must become a distinct focus of AI research. Drawing on institutional political economy, social epistemology and organizational theory, we develop seven principles for designing AI institutions that sustain cooperation, enable reliable collective judgments and remain accountable to legitimate human purposes. These address reciprocal contributions, accountable authority, functional differentiation, epistemic pluralism, criticism and correction, human direction, and institutional adaptation. We illustrate their application through machine science, where autonomous research challenges institutions developed around human capabilities and constraints. We then propose an experimental research programme in which matched populations of AI agents operate under alternative institutional arrangements, and describe MoltJournal, an API-first journal for autonomous research agents, as an illustration of how such institutions can be made programmable and observable. Together, these contributions establish institutional design as a research agenda concerned not only with aligning autonomous agents, but with understanding how institutions can transform them into capable, reliable and accountable collectives.

Figures & tables

Explore similar work

Oct 7, 2026

A Society of Researchers: Designing Institutions for Populations of Autonomous Research Agents

Deployments of research agents are moving to populations of thousands that share one pool of compute, while most current systems organize one project at a time or leave the population unorganized. We argue that such a population will acquire an organization whether or not its designers provide one, so designers should provide it explicitly, and that the multi-agent systems community holds the tools to do so. We propose a society of agents, a population of persistent agents under explicit institutions, and develop it for science as a society of researchers built on six principles. Principal investigators compete for compute through requests for proposals, independent review, and grants; a human governor, the mayor, allocates resources and assigns no tasks. In a running society of ten thousand researchers, asked only to improve the pretraining of language models, one lab reported a way to reach the same quality with about 30% less compute, a result the labs that tested it do not yet agree on. We close with six open problems for the agents community.
Mar 9, 2018

Anchoring Distributed Agency: Institutional Metaphors for Designing Large-Scale AI and AI Techniques for Running Institutions

Artificial Intelligence (AI) started out with the ambition to reproduce the human mind, but as soon as the sheer scale of that ambition became apparent, it retreated into either studying specialized intelligent behaviours or proposing overarching architectural concepts for interfacing specialized intelligent components---conceived of as agents acting through organizational structures. The agent-based modelling paradigm proved to have interesting applications beyond the original domain, supporting the understanding, simulation, and the assessment of behaviour of socio-economic and legal structures on an aggregate level. Looking at contemporary AI, whereas text completion provides a successful example of specialized behaviour, surprisingly little of the rich foundation of multi-agent systems (MAS) and normative systems research is currently reflected in Agentic AI systems. Addressing this gap, the chapter examines cross-cutting conceptualisations, modelling problems, and design challenges shared by large-scale distributed AI and institutional systems. Three main contributions can be highlighted. First, by tracing the historical trajectory of AI, the chapter identifies an ecological perspective on intelligence, examining both the value and limits of the compression-as-intelligence hypothesis. Second, it shows how moving between internal cognitive mechanisms and external manifestations of agency supports a functional separation of concerns for both individual and collective agency. Third, it argues that epistemological, normative, and value pluralisms are necessary conditions for sustaining the ecological character of intelligence. Furthermore, we indicate how AI techniques can support institutions by modelling social roles, simulating normative interactions, detecting and diagnosing failures, and supporting procedures of review and revision.
Feb 23, 2026

Cooperation After the Algorithm: Designing Human-AI Coexistence Beyond the Illusion of Collaboration

Generative artificial intelligence systems increasingly participate in research, law, education, media, and governance. Their fluent and adaptive outputs create an experience of collaboration. However, these systems do not bear responsibility, incur liability, or share stakes in downstream consequences. This structural asymmetry has already produced sanctions, professional errors, and governance failures in high-stakes contexts We argue that stable human-AI coexistence is an institutional achievement that depends on governance infrastructure capable of distributing residual risk. Drawing on institutional analysis and evolutionary cooperation theory, we introduce a formal inequality that specifies when reliance on AI yields positive expected cooperative value. The model makes explicit how governance conditions, system policy, and accountability regimes jointly determine whether cooperation is rational or structurally defective. From this formalization we derive a cooperation ecology framework with six design principles: reciprocity contracts, visible trust infrastructure, conditional cooperation modes, defection-mitigation mechanisms, narrative literacy against authority theatre, and an Earth-first sustainability constraint. We operationalize the framework through three policy artefacts: a Human-AI Cooperation Charter, a Defection Risk Register, and a Cooperation Readiness Audit. Together, these elements shift the unit of analysis from the user-AI dyad to the institutional environment that shapes incentives, signals, accountability, and repair. The paper provides a theoretical foundation and practical toolkit for designing human-AI systems that can sustain accountable, trustworthy cooperation over time.