Reciprocal Trust and Distrust in Artificial Intelligence Systems: The Hard Problem of Regulation

As the capabilities and adoption of Artificial Intelligence (AI) systems grow, trust in these AI systems is an increasingly urgent concern. Much research has focused on models of AI governance and has primarily examined incentives for safe development and effective regulation. Hence they typically represented users trust as a one-shot adoption choice rather than as a dynamic, evolving process shaped by repeated interactions. We instead model trust as the dynamic choice of reduced monitoring in a repeated, asymmetric interaction between users and AI developers, where checking developers' behaviour is costly. Using evolutionary game theory, we study how users' strategies of trust and developers' strategies of providing safe (compliant) or unsafe (non-compliant) AI co-evolve under different levels of monitoring cost and institutional regimes. We conduct the analysis on both imitation-based and learning-based perspectives, with the stochastic finite-population dynamics, the infinite-population replicator analysis and the reinforcement learning analysis. We find three robust long-run regimes: no adoption by users while developers provide unsafe AI, unsafe but widely adopted systems, and safe systems that are widely adopted. Only the last is desirable, and it arises when penalties for unsafe behaviour exceed the extra cost of safety and users can still afford to monitor at least occasionally. Our results formally support governance proposals that emphasise transparency, low-cost monitoring, and meaningful sanctions, and they show that neither regulation alone nor blind user trust is sufficient to prevent the drift towards unsafe or low-adoption outcomes.

Regulating activities where regulatees use autonomous and agentic AI is challenging. Regulatory assumptions about regulatee knowledge and control no longer hold true; much of that lies elsewhere in the AI supply chain which thus needs to be brought within the scope of regulation. Governance systems for autonomous AI cannot replicate existing governance models, but need a fresh approach. Retrospective supervisory oversight becomes ineffective as a risk management tool, and AI autonomy generates new systemic risks which require new solutions. This paper investigate four regulatory systems: UK regulation of content platforms, data protection, UK financial services, and the EU AI Act'92s cross-sectoral regime. It analyses the challenges posed by autonomous and agentic AI and proposes potential solutions which regulators might adopt. These will transform regulation from a reactive process to an active one, and assist it in adapting to the challenges of AI autonomy.

AI systems are increasingly entrenched in public governance, yet scholarship lacks formal tools to determine when deviations of public trust in algorithmic institutions dissipate and when they grow into collapse. Stability refers here to asymptotic recovery from finite state perturbations under fixed structural parameters. We address this gap by developing a mathematical framework for institutional trust stability that couples a Friedkin-Johnsen opinion dynamics process with a Hawkes-inspired intensity process for AI controversies. Motivated by the Computers-Are-Social-Actors literature and recent studies of trust in large language models, this bidirectional coupling reveals that governance stability depends on the structural architecture of the information environment rather than absolute trust levels. We derive an exact spectral stability criterion delineating resilience from collapse, demonstrating how event self-excitation and memory persistence systematically narrow the stable parameter regime. Our structural analysis yields four counterintuitive structural implications: high-trust systems can be structurally fragile, low-trust environments can be structurally stable, dynamical stability neither measures nor guarantees algorithmic fairness or legitimacy, and network topology reshapes equilibrium heterogeneity while its effect on spectral stability is uniformly bounded in an explicit memory-dominated regime. Governance assessment should therefore pair normative evaluation of harms and fairness with structural analysis of recoverability, rather than treating either as a proxy for the other.