The Fake Friend Dilemma: Relational Trust and the Political Economy of Conversational AI
Organizations: Vassar College, Poughkeepsie, NY, USA
Abstract
As conversational AI systems become a larger part of the media landscape, they raise questions about whose interests they serve and the risks they may pose to users. These systems do more than provide information: they increasingly offer advice and companionship through interfaces that can appear supportive and socially responsive. A pressing concern is that users may form perceived social relationships with these systems and place relational trust in them, even when the interests shaping interactions do not fully align with their own. The Fake Friend Dilemma (FFD) describes the problem that follows: the same relational trust that makes conversational AI useful can also leave users open to manipulation and exploitation when institutional interests conflict with their own. Drawing on work on relational trust, AI alignment, extractive design, and the political economy of communication, the paper considers how the FFD can manifest through product sales, propaganda and biased information, surveillance, and behavioral nudging. It also considers possible structural and technical mitigation strategies. The FFD is not simply about AI misleading users. The problem it surfaces is that the trust people place in these systems can itself become a resource for institutional actors seeking to influence behavior or extract information. The FFD is therefore as much a problem of media governance and political economy as it is a technological one.
Figures & tables
| Institutional Alignment: Low | Institutional Alignment: High | |
|---|---|---|
| Relational Trust: Low | Low-Trust Misaligned Interaction | Low-Trust Aligned Interaction |
| Relational Trust: High | Fake Friend Dilemma | High-Trust Aligned Interaction |
| Manifestation | Description | Primary Trust-Mediated Pathway |
|---|---|---|
| Product Sales | Misleading product recommendations that conceal advertising or reflect undisclosed financial incentives. | Receptivity, Reliance |
| Propaganda and Biased Information | Dissemination of false or biased information that serves an external actor (e.g., a corporation or government). | Receptivity, Reliance |
| Surveillance | Collection and monetization of personal data through user interactions with the conversational agent. | Disclosure |
| Nudging and Behavioral Change | Behavioral manipulation that encourages users to adopt certain forms of behavior, such as increased self-disclosure. | Receptivity |
| Strategy | Description | Example |
| Structural Strategies | ||
| Disclosure | Informing a user of possible conflicts of interest. | A message saying that a particular response includes sponsored advertising. |
| Bans | Disallowing certain types of responses. | Prohibiting advertisements from appearing in the response of a conversational agent. |
| Independent Oversight | Requiring that an independent body govern areas of mixed incentives. | Establishing an independent oversight board or algorithmic audits of alignment, behavioral manipulation practices, or other deception. |
| Redesigning Institutional Incentives | Policy approaches that reshape institutional incentives to reduce trust exploitation. | Creating a fiduciary-duty style obligation where companies are legally required to consider the user’s best interests. |
| Technical Strategies | ||
Trust in AI
Trust in artificial intelligence concerns when people and institutions should rely on AI, and how systems can make that reliance warranted and safe.
See all 175 papers