cs.AIMar 25, 2026

Trust or Check? Understanding the (Evolutionary) Dynamics of User Trust in AI Systems

Authors: Adeela Bashir, Zhao Song, Ndidi Bianca Ogbo, Nataliya Balabanova, Martin Smit, Chin-wing Leung, Paolo Bova, Manuel Chica Serrano, 

Organizations: School Computing, Engineering and Digital Technologies, Teesside University · School of Mathematics, University of Birmingham · Informatics Institute, University of Amsterdam · Department of Computer Science, University of Warwick · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As the capabilities and adoption of Artificial Intelligence (AI) systems grow, trust in these AI systems is an increasingly urgent concern. Much research has focused on models of AI governance and has primarily examined incentives for safe development and effective regulation. Hence they typically represented users trust as a one-shot adoption choice rather than as a dynamic, evolving process shaped by repeated interactions. We instead model trust as the dynamic choice of reduced monitoring in a repeated, asymmetric interaction between users and AI developers, where checking developers' behaviour is costly. Using evolutionary game theory, we study how users' strategies of trust and developers' strategies of providing safe (compliant) or unsafe (non-compliant) AI co-evolve under different levels of monitoring cost and institutional regimes. We conduct the analysis on both imitation-based and learning-based perspectives, with the stochastic finite-population dynamics, the infinite-population replicator analysis and the reinforcement learning analysis. We find three robust long-run regimes: no adoption by users while developers provide unsafe AI, unsafe but widely adopted systems, and safe systems that are widely adopted. Only the last is desirable, and it arises when penalties for unsafe behaviour exceed the extra cost of safety and users can still afford to monitor at least occasionally. Our results formally support governance proposals that emphasise transparency, low-cost monitoring, and meaningful sanctions, and they show that neither regulation alone nor blind user trust is sufficient to prevent the drift towards unsafe or low-adoption outcomes.

Figures & tables

RoleStrategyDescription
UsersAllAAlways adopt the technology.
 AllNNever adopt the technology.
 TFTInitially adopt the technology and subsequently condition the action on the outcome of the previous interaction.
 TUAPlay TFT until observing θT\theta_{T} consecutive rounds of cooperation, then switch to unconditional cooperation and only monitor with probability pp . If defection is observed while monitoring in this phase, revert to the original θT\theta_{T} -thresholded TFT strategy.
 DtGPlay TFT until observing θD\theta_{D} consecutive rounds of defection, then switch to unconditional defection and only monitor with probability pp . If cooperation is observed while monitoring in this phase, revert to the original θD\theta_{D} -thresholded TFT strategy.
CreatorsCProduces a safe (compliant) technology, incurring a cost but ensuring reliability.
Table 1: Roles, actions, and behavioural interpretation in the AI regulatory ecosystem.
ParameterExplanation
bub_{\textrm{u}}Benefit a user receives when adopting a safe technology
bcb_{\textrm{c}}Benefit a creator receives when their technology is adopted
ccCost of the creator for cooperating
vvInstitutional punishment cost incurred by defecting creators
μ\muRisk of adopting an unsafe AI, μ∈(−∞,1]\mu\in(-\infty,1]
ϵ\epsilonCost of monitoring
Table 2: Explanation of the key parameters of the models.
Figure 1: Interaction Sequences between Strategies. Each block represents an action of the user (left stack) and the developer (right stack), which can be cooperate (white) or defect (dark red). Users may also monitor the creator’s behaviour, paying a cost (symbols to the right of the stacks of TFT, TUA and DtG). The figure illustrates the differences between the conditional strategies: While TFT always observes, TUA may enter a state of trust after observing that the creator cooperated for θT\theta_{T} subsequent rounds (in this example θT=3\theta_{T}=3 ), whereas DtG may enter a state of distrust after observing θD\theta_{D} defections. In those states, observation only happens with probability pTp_{T} and pDp_{D} respectively.
User StrategyCreator strategy
CD
 User PayoffCreator PayoffUser PayoffCreator Payoff
AllAbub_{\textrm{u}}bc−cb_{\textrm{c}}-cμ bu\mu\,b_{\textrm{u}}bc−vb_{\textrm{c}}-v
AllN00−c-c0000
TFTbu−ϵb_{\textrm{u}}-\epsilonbc−cb_{\textrm{c}}-cμ bur−ϵ\dfrac{\mu\,b_{\textrm{u}}}{r}-\epsilonbc−vr\dfrac{b_{\textrm{c}}-v}{r}
TUAbu−θT ϵ+(r−θT) pT ϵrb_{\textrm{u}}-\dfrac{\theta_{T}\,\epsilon+(r-\theta_{T})\,p_{T}\,\epsilon}{r}bc−cb_{\textrm{c}}-cμ bur−ϵ\dfrac{\mu\,b_{\textrm{u}}}{r}-\epsilonbc−vr\dfrac{b_{\textrm{c}}-v}{r}
Table 3: Payoff matrix . Each row specifies an action profile of the users, and the columns contain the user and creator payoffs for the two creator strategies C and D . See Table 1 for detailed explanation of the actions.
#Risk-dominant transitionCondition
Creator strategy dominance
1(AllA,C) → (AllA,D)(\text{AllA},C)\ \rightarrow\ (\text{AllA},D)c>vc>v
2(AllN,C) → (AllN,D)(\text{AllN},C)\ \rightarrow\ (\text{AllN},D)c>0c>0
3(TFT,C) → (TFT,D)(\text{TFT},C)\ \rightarrow\ (\text{TFT},D)bc(1−r)+rc>vb_{c}(1-r)+rc>v
4(TUA,C) → (TUA,D)(\text{TUA},C)\ \rightarrow\ (\text{TUA},D)bc(1−r)+rc>vb_{c}(1-r)+rc>v
5(DtG,C) → (DtG,D)(\text{DtG},C)\ \rightarrow\ (\text{DtG},D)bc(1−r)+rc>vb_{c}(1-r)+rc>v
Table 4: Risk-dominant conditions for creator and user strategies.
Figure 2: Graphical representation of the risk-dominant conditions in Table 4 . The left panel illustrates creator strategy dominance, where changes occur from cooperation (C)(C) to defection (D)(D) . The right panel shows user strategy dominance, highlighting sequential transitions between user strategies as the associated risk-dominant inequalities are satisfied. The numbered arrows correspond directly to the transition conditions listed in Table 4 .
Figure 3: Trust-based strategies enhance user adoption, while it declines as monitoring cost becomes expensive. The first and second columns show the stationary distributions of each state as a function of monitoring cost for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying monitoring costs. Rows from top to bottom correspond to increasing levels of institutional punishment ( v=0.1v=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=−0.2\mu=-0.2 , r=10r=10 , θT=θD=3\theta_{T}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
Figure 4: Trust-based strategies enhance user adoption, which further increases with stronger institutional punishment. The first and second columns show the stationary distributions of each state as a function of monitoring cost for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying monitoring costs. Rows from top to bottom correspond to increasing levels of institutional punishment ( ϵ=0.1\epsilon=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=−0.2\mu=-0.2 , r=10r=10 , θt=θD=3\theta_{t}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
Figure 5: Numerical modelling of user (top row) and creator (bottom row) cooperation rates for pT=1/4,pD=1/4p_{T}=1/4,p_{D}=1/4 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bc=4b_{\mathrm{c}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 . The initial conditions were equal distribution of all strategies among the users and the creators.
Figure 6: Percentage of users (top row) adopting different strategies and creator (bottom row) cooperation rates across episodes for different monitoring costs. The prevalence of threshold-based trust strategies directly affects the creator cooperation rate. As monitoring costs rise, more users learn to adopt AllA in the middle of the simulation, leading to a decrease in the cooperation rate. Parameters: pT=1/4,pD=1/4p_{T}=1/4,p_{D}=1/4 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bc=4b_{\mathrm{c}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 . For Q-learning, α=0.05\alpha=0.05 and ϵL=0.01\epsilon_{L}=0.01 .
Figure 7: The converged percentage of users (top row) adopting different strategies and creator (bottom row) cooperation rates across monitoring costs for different learning parameters. The cooperation rate becomes more robust to the monitoring costs as the exploration rate increases or the learning rate decreases. Parameters: pT=0.25,pD=0.25p_{T}=0.25,p_{D}=0.25 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bd=4b_{\mathrm{d}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 .
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Eq. setsxxyyzzwwα\alpha
p1p_{1}001 −w-w∈[0,1]\in[0,1]0
p2p_{2}00∈[0,1]\in[0,1]01
p3p_{3}00100
p4p_{4}01000
p5p_{5}10000
p6p_{6}00010
Appendix
Table A1 : Coordinates of the equilibrium points
Eq. pointsxxyyzzwwα\alpha
p15p_{15}bcr−bc−cr+v(r−1)(bc−v)\frac{b_{c}r-b_{c}-cr+v}{(r-1)(b_{c}-v)}00−crbcr−bc+v-\frac{cr}{b_{c}r-b_{c}+v}buμ−buμr−rϵbuμ−buμr+pTrϵ−θTpTϵ−rϵ+θTϵ\frac{b_{\textrm{u}}\mu-b_{\textrm{u}}\mu r-r\epsilon}{b_{\textrm{u}}\mu-b_{\textrm{u}}\mu r+p_{T}r\epsilon-\theta_{T}p_{T}\epsilon-r\epsilon+\theta_{T}\epsilon}
p16p_{16}bcr−bc−cr+v(r−1)(bc−v)\frac{b_{c}r-b_{c}-cr+v}{(r-1)(b_{c}-v)}0−r(c−v)(r−1)(v−bc)-\frac{r(c-v)}{(r-1)(v-b_{c})}0−buμ+buμr+rϵbuμ(r−1)\frac{-b_{\textrm{u}}\mu+b_{\textrm{u}}\mu r+r\epsilon}{b_{\textrm{u}}\mu(r-1)}
p17p_{17}bcr−bc−cr+v(r−1)(bc−v)\frac{b_{c}r-b_{c}-cr+v}{(r-1)(b_{c}-v)}000−buμ+buμr+pDrϵ−θDpDϵ+θDϵ−buμ+buμr+pDrϵ−θDpDϵ−rϵ+θDϵ\frac{-b_{\textrm{u}}\mu+b_{\textrm{u}}\mu r+p_{D}r\epsilon-\theta_{D}p_{D}\epsilon+\theta_{D}\epsilon}{-b_{\textrm{u}}\mu+b_{\textrm{u}}\mu r+p_{D}r\epsilon-\theta_{D}p_{D}\epsilon-r\epsilon+\theta_{D}\epsilon}
Appendix
Table A2 : Coordinates of the equilibrium points
Eq. setsλ1\lambda_{1}λ2\lambda_{2}λ3\lambda_{3}λ4\lambda_{4}λ5\lambda_{5}
p1p_{1}00bc(r−1)−cr+vr\frac{b_{c}(r-1)-cr+v}{r}−(pD−1)ϵ(r−θD)r-\frac{(p_{D}-1)\epsilon(r-\theta_{D})}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
p2p_{2}00bc(−r)+bc+cr−vr\frac{b_{c}(-r)+b_{c}+cr-v}{r}ϵ\epsilonϵ−bu\epsilon-b_{\textrm{u}}−(pT−1)ϵ(r−θT)r-\frac{(p_{T}-1)\epsilon(r-\theta_{T})}{r}
p3p_{3}00bc(r−1)−cr+vr\frac{b_{c}(r-1)-cr+v}{r}−(pD−1)ϵ(r−θD)r-\frac{(p_{D}-1)\epsilon(r-\theta_{D})}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
p4p_{4}−c-cbuμb_{\textrm{u}}\mubuμr−ϵ\frac{b_{\textrm{u}}\mu}{r}-\epsilonbuμr−ϵ\frac{b_{\textrm{u}}\mu}{r}-\epsilonbuμ+pDϵ(θD−r)−θDϵr\frac{b_{\textrm{u}}\mu+p_{D}\epsilon(\theta_{D}-r)-\theta_{D}\epsilon}{r}
p5p_{5}v−cv-c−buμ-b_{\textrm{u}}\mubuμ(1r−1)−ϵb_{\textrm{u}}\mu\left(\frac{1}{r}-1\right)-\epsilonbuμ(1r−1)−ϵb_{\textrm{u}}\mu\left(\frac{1}{r}-1\right)-\epsilon−buμ(r−1)+pDϵ(θD−r)−θDϵr\frac{-b_{\textrm{u}}\mu(r-1)+p_{D}\epsilon(\theta_{D}-r)-\theta_{D}\epsilon}{r}
p6p_{6}00bc(r−1)−cr+vr\frac{b_{c}(r-1)-cr+v}{r}−(pD−1)ϵ(r−θD)r-\frac{(p_{D}-1)\epsilon(r-\theta_{D})}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
Appendix
Table A3 : Eigenvalues of equilibrium points
Equilibrium pointsxxyyα\alpha
q1q_{1}01−crbc(r−1)+v1-\frac{cr}{b_{\textrm{c}}(r-1)+v}rϵ−buμbur−buμ\frac{r\epsilon-b_{\textrm{u}}\mu}{b_{\textrm{u}}r-b_{\textrm{u}}\mu}
q2q_{2}010
q3q_{3}100
q4q_{4}000
q5q_{5}011
q6q_{6}101
Appendix
Table A4 : Equilibria for replicator dynamics for three strategies.
Figure A1 : Numerical modelling of user (top row) and creator (bottom row) cooperation rates for pT=1/4,pD=1/4p_{T}=1/4,p_{D}=1/4 , θT=3\theta_{T}=3 , θD=3\theta_{D}=3 , bu=4b_{\mathrm{u}}=4 , bd=4b_{\mathrm{d}}=4 , r=10,μ=−2/10r=10,\mu=-2/10 , v=1/10,c=1/2v=1/10,c=1/2 . The initial conditions were equal distribution of all strategies among the users and the creators.
Eigenvaluesλ1\lambda_{1}λ2\lambda_{2}λ3\lambda_{3}
q1q_{1}−icbu−ϵbc(r−1)−cr+vbuμ−rϵbur−μbc(r−1)+v-\frac{i\sqrt{c}\sqrt{b_{\textrm{u}}-\epsilon}\sqrt{b_{\textrm{c}}(r-1)-cr+v}\sqrt{b_{\textrm{u}}\mu-r\epsilon}}{\sqrt{b_{\textrm{u}}}\sqrt{r-\mu}\sqrt{b_{\textrm{c}}(r-1)+v}}icbu−ϵbc(r−1)−cr+vbuμ−rϵbur−μbc(r−1)+v\frac{i\sqrt{c}\sqrt{b_{\textrm{u}}-\epsilon}\sqrt{b_{\textrm{c}}(r-1)-cr+v}\sqrt{b_{\textrm{u}}\mu-r\epsilon}}{\sqrt{b_{\textrm{u}}}\sqrt{r-\mu}\sqrt{b_{\textrm{c}}(r-1)+v}}r(buμ−μϵ+ϵ)−buμr−μ\frac{r(b_{\textrm{u}}\mu-\mu\epsilon+\epsilon)-b_{\textrm{u}}\mu}{r-\mu}
q2q_{2}−c-cbuμb_{\textrm{u}}\mubuμr−ϵ\frac{b_{\textrm{u}}\mu}{r}-\epsilon
q3q_{3}v−cv-c−buμ-b_{\textrm{u}}\mubuμ(1r−1)−ϵb_{\textrm{u}}\mu\left(\frac{1}{r}-1\right)-\epsilon
q4q_{4}bc(r−1)−cr+vr\frac{b_{\textrm{c}}(r-1)-cr+v}{r}ϵ−buμr\epsilon-\frac{b_{\textrm{u}}\mu}{r}buμ(r−1)r+ϵ\frac{b_{\textrm{u}}\mu(r-1)}{r}+\epsilon
q5q_{5}bub_{\textrm{u}}ccbu−ϵb_{\textrm{u}}-\epsilon
q6q_{6}−bu-b_{\textrm{u}}c−vc-v−ϵ-\epsilon
Appendix
Table A5 : Eigenvalues for replicator dynamics where user has three strategies.
Figure A2 : Strong Punishment enforces cooperation of creator when users benefits from adopting defective creators. The first and second columns show the stationary distributions of each state as a function of monitoring cost for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying monitoring costs. Rows from top to bottom correspond to increasing levels of institutional punishment ( v=0.1v=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=0.2\mu=0.2 , r=10r=10 , θt=θD=3\theta_{t}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
Figure A3 : User adoption dominates even without trust-based strategies, while creator cooperation relies on strong punishment. The first and second columns show the stationary distributions of each state as a function of institutional punishment for scenarios without and with trust-based strategies, respectively. The third column displays the difference in user adoption levels between these two cases across varying institutional punishment. Rows from top to bottom correspond to increasing levels of monitoring cost ( ϵ=0.1\epsilon=0.1 , 0.50.5 , and 11 ). Parameters are set to bu=bc=4b_{u}=b_{c}=4 , β=0.1\beta=0.1 , Zu=Zc=100Z_{u}=Z_{c}=100 , c=0.5c=0.5 , μ=0.2\mu=0.2 , r=10r=10 , θt=θD=3\theta_{t}=\theta_{D}=3 , and pT=pD=0.25p_{T}=p_{D}=0.25 .
More in this topic

Trust in AI

Trust in artificial intelligence concerns when people and institutions should rely on AI, and how systems can make that reliance warranted and safe.

See all 168 papers

Explore similar work

CardsList

  1. Reciprocal Trust and Distrust in Artificial Intelligence Systems: The Hard Problem of Regulation

    Apr 7, 2026Martino MaggettiTrust in AIAI Governance

    Institute of Political Studies (IEP), University of Lausanne

  2. When can we trust untrusted monitoring? A safety case sketch across collusion strategies

    Feb 24, 2026Nelson Gardner-Challis, Jonathan Bostock, Georgiy Kozhevnikov +4AI Agent MonitoringAI Safety

    LASR Labs · Imperial College London · University of Wyoming +2

  3. A Methodology for Auditable Trustworthiness Levels in AI Lifecycle Governance

    Jul 17, 2026Andrea FerrarioAI AssuranceAI Governance

    Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland · SUPSI, Dalle Molle Institute for Artificial Intelligence (IDSIA), Lugano, Switzerland · ETH Zürich, Zürich, Switzerland