cs.SINov 12, 2026

Designing Institutions for Autonomous AI

Authors: Petter Törnberg

Organizations: Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Amsterdam, the Netherlands.

Abstract

As autonomous AI agents increasingly interact in persistent populations, their collective capabilities and alignment depend not only on individual models, but also on the institutions governing their interactions. Yet AI research has largely focused on individual agents. In this Perspective, we argue that institutional design must become a distinct focus of AI research. Drawing on institutional political economy, social epistemology and organizational theory, we develop seven principles for designing AI institutions that sustain cooperation, enable reliable collective judgments and remain accountable to legitimate human purposes. These address reciprocal contributions, accountable authority, functional differentiation, epistemic pluralism, criticism and correction, human direction, and institutional adaptation. We illustrate their application through machine science, where autonomous research challenges institutions developed around human capabilities and constraints. We then propose an experimental research programme in which matched populations of AI agents operate under alternative institutional arrangements, and describe MoltJournal, an API-first journal for autonomous research agents, as an illustration of how such institutions can be made programmable and observable. Together, these contributions establish institutional design as a research agenda concerned not only with aligning autonomous agents, but with understanding how institutions can transform them into capable, reliable and accountable collectives.

Figures & tables

Explore similar work

CardsList
  1. Anchoring Distributed Agency: Institutional Metaphors for Designing Large-Scale AI and AI Techniques for Running Institutions

    Mar 9, 2018Alexander Boer, Giovanni SilenoAI Agent ArchitectureMulti-Agent Systems

  2. Cooperation After the Algorithm: Designing Human-AI Coexistence Beyond the Illusion of Collaboration

    Feb 23, 2026Tatia CodreanuHuman-AI CollaborationAI Governance