cs.AIMar 25, 2026

Trust or Check? Understanding the (Evolutionary) Dynamics of User Trust in AI Systems

Authors: Adeela Bashir, Zhao Song, Ndidi Bianca Ogbo, Nataliya Balabanova, Martin Smit, Chin-wing Leung, Paolo Bova, Manuel Chica Serrano, 

Organizations: School Computing, Engineering and Digital Technologies, Teesside University · School of Mathematics, University of Birmingham · Informatics Institute, University of Amsterdam · Department of Computer Science, University of Warwick · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As the capabilities and adoption of Artificial Intelligence (AI) systems grow, trust in these AI systems is an increasingly urgent concern. Much research has focused on models of AI governance and has primarily examined incentives for safe development and effective regulation. Hence they typically represented users trust as a one-shot adoption choice rather than as a dynamic, evolving process shaped by repeated interactions. We instead model trust as the dynamic choice of reduced monitoring in a repeated, asymmetric interaction between users and AI developers, where checking developers' behaviour is costly. Using evolutionary game theory, we study how users' strategies of trust and developers' strategies of providing safe (compliant) or unsafe (non-compliant) AI co-evolve under different levels of monitoring cost and institutional regimes. We conduct the analysis on both imitation-based and learning-based perspectives, with the stochastic finite-population dynamics, the infinite-population replicator analysis and the reinforcement learning analysis. We find three robust long-run regimes: no adoption by users while developers provide unsafe AI, unsafe but widely adopted systems, and safe systems that are widely adopted. Only the last is desirable, and it arises when penalties for unsafe behaviour exceed the extra cost of safety and users can still afford to monitor at least occasionally. Our results formally support governance proposals that emphasise transparency, low-cost monitoring, and meaningful sanctions, and they show that neither regulation alone nor blind user trust is sufficient to prevent the drift towards unsafe or low-adoption outcomes.

Figures & tables

RoleStrategyDescription
UsersAllAAlways adopt the technology.
 AllNNever adopt the technology.
 TFTInitially adopt the technology and subsequently condition the action on the outcome of the previous interaction.
 TUAPlay TFT until observing θT\theta_{T} consecutive rounds of cooperation, then switch to unconditional cooperation and only monitor with probability pp . If defection is observed while monitoring in this phase, revert to the original θT\theta_{T} -thresholded TFT strategy.
 DtGPlay TFT until observing θD\theta_{D} consecutive rounds of defection, then switch to unconditional defection and only monitor with probability pp . If cooperation is observed while monitoring in this phase, revert to the original θD\theta_{D} -thresholded TFT strategy.
CreatorsCProduces a safe (compliant) technology, incurring a cost but ensuring reliability.
Table 1: Roles, actions, and behavioural interpretation in the AI regulatory ecosystem.
ParameterExplanation
bub_{\textrm{u}}Benefit a user receives when adopting a safe technology
bcb_{\textrm{c}}Benefit a creator receives when their technology is adopted
ccCost of the creator for cooperating
vvInstitutional punishment cost incurred by defecting creators
μ\muRisk of adopting an unsafe AI, μ∈(−∞,1]\mu\in(-\infty,1]
ϵ\epsilonCost of monitoring
Table 2: Explanation of the key parameters of the models.
Figure 1: Interaction Sequences between Strategies. Each block represents an action of the user (left stack) and the developer (right stack), which can be cooperate (white) or defect (dark red). Users may also monitor the creator’s behaviour, paying a cost (symbols to the right of the stacks of TFT, TUA and DtG). The figure illustrates the differences between the conditional strategies: While TFT always observes, TUA may enter a state of trust after observing that the creator cooperated for θT\theta_{T} subsequent rounds (in this example θT=3\theta_{T}=3 ), whereas DtG may enter a state of distrust after observing θD\theta_{D} defections. In those states, observation only happens with probability pTp_{T} and pDp_{D} respectively.
User StrategyCreator strategy
CD
 User PayoffCreator PayoffUser PayoffCreator Payoff
AllAbub_{\textrm{u}}bc−cb_{\textrm{c}}-cμ bu\mu\,b_{\textrm{u}}bc−vb_{\textrm{c}}-v
AllN00−c-c0000
TFTbu−ϵb_{\textrm{u}}-\epsilonbc−cb_{\textrm{c}}-cμ bur−ϵ\dfrac{\mu\,b_{\textrm{u}}}{r}-\epsilonbc−vr\dfrac{b_{\textrm{c}}-v}{r}
TUAbu−θT ϵ+(r−θT) pT ϵrb_{\textrm{u}}-\dfrac{\theta_{T}\,\epsilon+(r-\theta_{T})\,p_{T}\,\epsilon}{r}bc−cb_{\textrm{c}}-cμ bur−ϵ\dfrac{\mu\,b_{\textrm{u}}}{r}-\epsilonbc−vr\dfrac{b_{\textrm{c}}-v}{r}
Table 3: Payoff matrix . Each row specifies an action profile of the users, and the columns contain the user and creator payoffs for the two creator strategies C and D . See Table 1 for detailed explanation of the actions.
#Risk-dominant transitionCondition
Creator strategy dominance
1(AllA,C) → (AllA,D)(\text{AllA},C)\ \rightarrow\ (\text{AllA},D)c>vc>v
2(AllN,C) → (AllN,D)(\text{AllN},C)\ \rightarrow\ (\text{AllN},D)c>0c>0
3(TFT,C) → (TFT,D)(\text{TFT},C)\ \rightarrow\ (\text{TFT},D)bc(1−r)+rc>vb_{c}(1-r)+rc>v
4(TUA,C) → (TUA,D)(\text{TUA},C)\ \rightarrow\ (\text{TUA},D)bc(1−r)+rc>vb_{c}(1-r)+rc>v
5(DtG,C) → (DtG,D)(\text{DtG},C)\ \rightarrow\ (\text{DtG},D)bc(1−r)+rc>vb_{c}(1-r)+rc>v
Table 4: Risk-dominant conditions for creator and user strategies.
Figure 2: Graphical representation of the risk-dominant conditions in Table 4 . The left panel illustrates creator strategy dominance, where changes occur from cooperation (C)(C) to defection (D)(D) . The right panel shows user strategy dominance, highlighting sequential transitions between user strategies as the associated risk-dominant inequalities are satisfied. The numbered arrows correspond directly to the transition conditions listed in Table 4 .
Figure 3: Trust-based strategies enhance user adoption, while it declines as monitoring cost becomes expensive. The first and second columns show the stationary distributions of each state as a function of monitoring cost for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying monitoring costs. Rows from top to bottom correspond to increasing levels of institutional punishment ( v=0.1v=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=−0.2\mu=-0.2 , r=10r=10 , θT=θD=3\theta_{T}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
Figure 4: Trust-based strategies enhance user adoption, which further increases with stronger institutional punishment. The first and second columns show the stationary distributions of each state as a function of monitoring cost for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying monitoring costs. Rows from top to bottom correspond to increasing levels of institutional punishment ( ϵ=0.1\epsilon=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=−0.2\mu=-0.2 , r=10r=10 , θt=θD=3\theta_{t}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
Figure 5: Numerical modelling of user (top row) and creator (bottom row) cooperation rates for pT=1/4,pD=1/4p_{T}=1/4,p_{D}=1/4 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bc=4b_{\mathrm{c}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 . The initial conditions were equal distribution of all strategies among the users and the creators.
Figure 6: Percentage of users (top row) adopting different strategies and creator (bottom row) cooperation rates across episodes for different monitoring costs. The prevalence of threshold-based trust strategies directly affects the creator cooperation rate. As monitoring costs rise, more users learn to adopt AllA in the middle of the simulation, leading to a decrease in the cooperation rate. Parameters: pT=1/4,pD=1/4p_{T}=1/4,p_{D}=1/4 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bc=4b_{\mathrm{c}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 . For Q-learning, α=0.05\alpha=0.05 and ϵL=0.01\epsilon_{L}=0.01 .
Figure 7: The converged percentage of users (top row) adopting different strategies and creator (bottom row) cooperation rates across monitoring costs for different learning parameters. The cooperation rate becomes more robust to the monitoring costs as the exploration rate increases or the learning rate decreases. Parameters: pT=0.25,pD=0.25p_{T}=0.25,p_{D}=0.25 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bd=4b_{\mathrm{d}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 .
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Eq. setsxxyyzzwwα\alpha
p1p_{1}001 −w-w∈[0,1]\in[0,1]0
p2p_{2}00∈[0,1]\in[0,1]01
p3p_{3}00100
p4p_{4}01000
p5p_{5}10000
p6p_{6}00010
Appendix
Table A1 : Coordinates of the equilibrium points
Eq. pointsxxyyzzwwα\alpha
p15p_{15}bcr−bc−cr+v(r−1)(bc−v)\frac{b_{c}r-b_{c}-cr+v}{(r-1)(b_{c}-v)}00−crbcr−bc+v-\frac{cr}{b_{c}r-b_{c}+v}buμ−buμr−rϵbuμ−buμr+pTrϵ−θTpTϵ−rϵ+θTϵ\frac{b_{\textrm{u}}\mu-b_{\textrm{u}}\mu r-r\epsilon}{b_{\textrm{u}}\mu-b_{\textrm{u}}\mu r+p_{T}r\epsilon-\theta_{T}p_{T}\epsilon-r\epsilon+\theta_{T}\epsilon}
p16p_{16}bcr−bc−cr+v(r−1)(bc−v)\frac{b_{c}r-b_{c}-cr+v}{(r-1)(b_{c}-v)}0−r(c−v)(r−1)(v−bc)-\frac{r(c-v)}{(r-1)(v-b_{c})}0−buμ+buμr+rϵbuμ(r−1)\frac{-b_{\textrm{u}}\mu+b_{\textrm{u}}\mu r+r\epsilon}{b_{\textrm{u}}\mu(r-1)}
p17p_{17}bcr−bc−cr+v(r−1)(bc−v)\frac{b_{c}r-b_{c}-cr+v}{(r-1)(b_{c}-v)}000−buμ+buμr+pDrϵ−θDpDϵ+θDϵ−buμ+buμr+pDrϵ−θDpDϵ−rϵ+θDϵ\frac{-b_{\textrm{u}}\mu+b_{\textrm{u}}\mu r+p_{D}r\epsilon-\theta_{D}p_{D}\epsilon+\theta_{D}\epsilon}{-b_{\textrm{u}}\mu+b_{\textrm{u}}\mu r+p_{D}r\epsilon-\theta_{D}p_{D}\epsilon-r\epsilon+\theta_{D}\epsilon}
Appendix
Table A2 : Coordinates of the equilibrium points
Eq. setsλ1\lambda_{1}λ2\lambda_{2}λ3\lambda_{3}λ4\lambda_{4}λ5\lambda_{5}
p1p_{1}00bc(r−1)−cr+vr\frac{b_{c}(r-1)-cr+v}{r}−(pD−1)ϵ(r−θD)r-\frac{(p_{D}-1)\epsilon(r-\theta_{D})}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
p2p_{2}00bc(−r)+bc+cr−vr\frac{b_{c}(-r)+b_{c}+cr-v}{r}ϵ\epsilonϵ−bu\epsilon-b_{\textrm{u}}−(pT−1)ϵ(r−θT)r-\frac{(p_{T}-1)\epsilon(r-\theta_{T})}{r}
p3p_{3}00bc(r−1)−cr+vr\frac{b_{c}(r-1)-cr+v}{r}−(pD−1)ϵ(r−θD)r-\frac{(p_{D}-1)\epsilon(r-\theta_{D})}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
p4p_{4}−c-cbuμb_{\textrm{u}}\mubuμr−ϵ\frac{b_{\textrm{u}}\mu}{r}-\epsilonbuμr−ϵ\frac{b_{\textrm{u}}\mu}{r}-\epsilonbuμ+pDϵ(θD−r)−θDϵr\frac{b_{\textrm{u}}\mu+p_{D}\epsilon(\theta_{D}-r)-\theta_{D}\epsilon}{r}
p5p_{5}v−cv-c−buμ-b_{\textrm{u}}\mubuμ(1r−1)−ϵb_{\textrm{u}}\mu\left(\frac{1}{r}-1\right)-\epsilonbuμ(1r−1)−ϵb_{\textrm{u}}\mu\left(\frac{1}{r}-1\right)-\epsilon−buμ(r−1)+pDϵ(θD−r)−θDϵr\frac{-b_{\textrm{u}}\mu(r-1)+p_{D}\epsilon(\theta_{D}-r)-\theta_{D}\epsilon}{r}
p6p_{6}00bc(r−1)−cr+vr\frac{b_{c}(r-1)-cr+v}{r}−(pD−1)ϵ(r−θD)r-\frac{(p_{D}-1)\epsilon(r-\theta_{D})}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
Appendix
Table A3 : Eigenvalues of equilibrium points
Equilibrium pointsxxyyα\alpha
q1q_{1}01−crbc(r−1)+v1-\frac{cr}{b_{\textrm{c}}(r-1)+v}rϵ−buμbur−buμ\frac{r\epsilon-b_{\textrm{u}}\mu}{b_{\textrm{u}}r-b_{\textrm{u}}\mu}
q2q_{2}010
q3q_{3}100
q4q_{4}000
q5q_{5}011
q6q_{6}101
Appendix
Table A4 : Equilibria for replicator dynamics for three strategies.
Figure A1 : Numerical modelling of user (top row) and creator (bottom row) cooperation rates for pT=1/4,pD=1/4p_{T}=1/4,p_{D}=1/4 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bd=4b_{\mathrm{d}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 . The initial conditions were equal distribution of all strategies among the users and the creators.
Eigenvaluesλ1\lambda_{1}λ2\lambda_{2}λ3\lambda_{3}
q1q_{1}−icbu−ϵbc(r−1)−cr+vbuμ−rϵbur−μbc(r−1)+v-\frac{i\sqrt{c}\sqrt{b_{\textrm{u}}-\epsilon}\sqrt{b_{\textrm{c}}(r-1)-cr+v}\sqrt{b_{\textrm{u}}\mu-r\epsilon}}{\sqrt{b_{\textrm{u}}}\sqrt{r-\mu}\sqrt{b_{\textrm{c}}(r-1)+v}}icbu−ϵbc(r−1)−cr+vbuμ−rϵbur−μbc(r−1)+v\frac{i\sqrt{c}\sqrt{b_{\textrm{u}}-\epsilon}\sqrt{b_{\textrm{c}}(r-1)-cr+v}\sqrt{b_{\textrm{u}}\mu-r\epsilon}}{\sqrt{b_{\textrm{u}}}\sqrt{r-\mu}\sqrt{b_{\textrm{c}}(r-1)+v}}r(buμ−μϵ+ϵ)−buμr−μ\frac{r(b_{\textrm{u}}\mu-\mu\epsilon+\epsilon)-b_{\textrm{u}}\mu}{r-\mu}
q2q_{2}−c-cbuμb_{\textrm{u}}\mubuμr−ϵ\frac{b_{\textrm{u}}\mu}{r}-\epsilon
q3q_{3}v−cv-c−buμ-b_{\textrm{u}}\mubuμ(1r−1)−ϵb_{\textrm{u}}\mu\left(\frac{1}{r}-1\right)-\epsilon
q4q_{4}bc(r−1)−cr+vr\frac{b_{\textrm{c}}(r-1)-cr+v}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
q5q_{5}bub_{\textrm{u}}ccbu−ϵb_{\textrm{u}}-\epsilon
q6q_{6}−bu-b_{\textrm{u}}c−vc-v−ϵ-\epsilon
Appendix
Table A5 : Eigenvalues for replicator dynamics where user has three strategies.
Figure A2 : Strong Punishment enforces cooperation of creator when users benefits from adopting defective creators. The first and second columns show the stationary distributions of each state as a function of monitoring cost for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying monitoring costs. Rows from top to bottom correspond to increasing levels of institutional punishment ( v=0.1v=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=0.2\mu=0.2 , r=10r=10 , θt=θD=3\theta_{t}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
Figure A3 : User adoption dominates even without trust-based strategies, while creator cooperation relies on strong punishment. The first and second columns show the stationary distributions of each state as a function of institutional punishment for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying institutional punishment. Rows from top to bottom correspond to increasing levels of monitoring cost ( ϵ=0.1\epsilon=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=0.2\mu=0.2 , r=10r=10 , θt=θD=3\theta_{t}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
More in this topic

Trust in AI

Trust in artificial intelligence concerns when people and institutions should rely on AI, and how systems can make that reliance warranted and safe.

See all 168 papers

Explore similar work

CardsList
Apr 7, 2026

Reciprocal Trust and Distrust in Artificial Intelligence Systems: The Hard Problem of Regulation

Policy makers, scientists, and the public are increasingly confronted with thorny questions about the regulation of artificial intelligence (AI) systems. A key common thread concerns whether AI can be trusted and the factors that can make it more trustworthy in front of stakeholders and users. This is indeed crucial, as the trustworthiness of AI systems is fundamental for both democratic governance and for the development and deployment of AI. This article advances the discussion by arguing that AI systems should also be recognized, as least to some extent, as artifacts capable of exercising a form of agency, thereby enabling them to engage in relationships of trust or distrust with humans. It further examines the implications of these reciprocal trust dynamics for regulators tasked with overseeing AI systems. The article concludes by identifying key tensions and unresolved dilemmas that these dynamics pose for the future of AI regulation and governance.

Martino Maggetti

Institute of Political Studies (IEP), University of Lausanne

Trust in AIAI Governance
AskPDF
Feb 24, 2026

When can we trust untrusted monitoring? A safety case sketch across collusion strategies

AIs are increasingly being deployed with greater autonomy and capabilities, which increases the risk that a misaligned AI may be able to cause catastrophic harm. Untrusted monitoring -- using one untrusted model to oversee another -- is one approach to reducing risk. Justifying the safety of an untrusted monitoring deployment is challenging because developers cannot safely deploy a misaligned model to test their protocol directly. In this paper, we develop upon existing methods for rigorously demonstrating safety based on pre-deployment testing. We relax assumptions that previous AI control research made about the collusion strategies a misaligned AI might use to subvert untrusted monitoring. We develop a taxonomy covering passive self-recognition, causal collusion (hiding pre-shared signals), acausal collusion (hiding signals via Schelling points), and combined strategies. We create a safety case sketch to clearly present our argument, explicitly state our assumptions, and highlight unsolved challenges. We identify conditions under which passive self-recognition could be a more effective collusion strategy than those studied previously. Our work builds towards more robust evaluations of untrusted monitoring.

Nelson Gardner-Challis, Jonathan Bostock, Georgiy Kozhevnikov +4

LASR Labs · Imperial College London · University of Wyoming +2

AI Agent MonitoringAI SafetyAI Safety Evaluation
AskPDF
Jul 17, 2026

A Methodology for Auditable Trustworthiness Levels in AI Lifecycle Governance

AI governance increasingly requires judgments about whether an AI system remains adequately trustworthy over time, whether observed changes are tolerable, and how such judgments should be documented in a transparent and contestable way. Existing approaches remain either too high-level to support lifecycle monitoring and reassessment or too narrowly metric-driven to connect multidimensional trustworthiness evidence with governance decisions. We propose a lightweight methodology centered on trustworthiness level functions: auditable rules that map measured trustworthiness profiles to governance-relevant levels. The methodology separates the underlying trustworthiness evidence from the governance rule used to interpret it and treats that rule as a lifecycle governance object. The rule may remain expert-defined or, when available evidence warrants empirical learning, be approximated by an interpretable candidate model. An AI lifecycle governance procedure embeds this choice in explicit decision gates for determining whether learning should be attempted and whether a learned candidate should become operative. The resulting rule supports lifecycle monitoring through level transitions, boundary margins, and profile drift, with explicit human responsibilities for validation, approval, and reassessment. We illustrate the methodology on synthetic AI lifecycle scenarios involving degradation, shocks, updates, heterogeneous monitoring cadences, and system comparison. Our methodology does not replace expert or legal judgment, but makes the governance interpretation of trustworthiness evidence more explicit, auditable, and contestable over time.

Andrea Ferrario

Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland · SUPSI, Dalle Molle Institute for Artificial Intelligence (IDSIA), Lugano, Switzerland · ETH Zürich, Zürich, Switzerland

AI AssuranceAI GovernanceTrust in AI
AskPDF