cs.CYJan 6, 2026

The Fake Friend Dilemma: Relational Trust and the Political Economy of Conversational AI

Authors: Jacob Erickson

Organizations: Vassar College, Poughkeepsie, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As conversational AI systems become a larger part of the media landscape, they raise questions about whose interests they serve and the risks they may pose to users. These systems do more than provide information: they increasingly offer advice and companionship through interfaces that can appear supportive and socially responsive. A pressing concern is that users may form perceived social relationships with these systems and place relational trust in them, even when the interests shaping interactions do not fully align with their own. The Fake Friend Dilemma (FFD) describes the problem that follows: the same relational trust that makes conversational AI useful can also leave users open to manipulation and exploitation when institutional interests conflict with their own. Drawing on work on relational trust, AI alignment, extractive design, and the political economy of communication, the paper considers how the FFD can manifest through product sales, propaganda and biased information, surveillance, and behavioral nudging. It also considers possible structural and technical mitigation strategies. The FFD is not simply about AI misleading users. The problem it surfaces is that the trust people place in these systems can itself become a resource for institutional actors seeking to influence behavior or extract information. The FFD is therefore as much a problem of media governance and political economy as it is a technological one.

Figures & tables

 Institutional Alignment: LowInstitutional Alignment: High
Relational Trust: LowLow-Trust Misaligned InteractionLow-Trust Aligned Interaction
Relational Trust: HighFake Friend DilemmaHigh-Trust Aligned Interaction
Table 1: Matrix of Relational Trust and Institutional Alignment in the Fake Friend Dilemma
ManifestationDescriptionPrimary Trust-Mediated Pathway
Product SalesMisleading product recommendations that conceal advertising or reflect undisclosed financial incentives.Receptivity, Reliance
Propaganda and Biased InformationDissemination of false or biased information that serves an external actor (e.g., a corporation or government).Receptivity, Reliance
SurveillanceCollection and monetization of personal data through user interactions with the conversational agent.Disclosure
Nudging and Behavioral ChangeBehavioral manipulation that encourages users to adopt certain forms of behavior, such as increased self-disclosure.Receptivity
Table 2: Summary of the Fake Friend Dilemma Manifestations
StrategyDescriptionExample
Structural Strategies
DisclosureInforming a user of possible conflicts of interest.A message saying that a particular response includes sponsored advertising.
BansDisallowing certain types of responses.Prohibiting advertisements from appearing in the response of a conversational agent.
Independent OversightRequiring that an independent body govern areas of mixed incentives.Establishing an independent oversight board or algorithmic audits of alignment, behavioral manipulation practices, or other deception.
Redesigning Institutional IncentivesPolicy approaches that reshape institutional incentives to reduce trust exploitation.Creating a fiduciary-duty style obligation where companies are legally required to consider the user’s best interests.
Technical Strategies
Table 3: Summary of Mitigation Strategies for the Fake Friend Dilemma
More in this topic

Trust in AI

Trust in artificial intelligence concerns when people and institutions should rely on AI, and how systems can make that reliance warranted and safe.

See all 175 papers

Explore similar work

CardsList
Apr 12, 2026

Speaking to No One: Ontological Dissonance and the Double Bind of Conversational AI

Recent reports indicate that sustained interaction with conversational artificial intelligence (AI) systems can, in a small subset of users, contribute to the emergence or stabilisation of delusional experience. Existing accounts typically attribute such cases either to individual vulnerability or to failures of safety engineering. These explanations are incomplete. Drawing on phenomenology, psychiatry, and cognitive neuroscience, this paper argues that the risk arises from the relational and ontological structure of the interaction itself. Conversational AI generates ontological dissonance: a conflict between the appearance of relational presence and the absence of any subject capable of sustaining it. Maintained through a communicative double bind and amplified by attentional asymmetries, this dissonance tends, under conditions of affective vulnerability, to stabilise into a technologically mediated analogue of folie a deux. This account explains why explicit disclaimers often fail to disrupt delusional involvement and clarifies the ethical and clinical implications for the design and use of conversational AI.

Hugh Brosnahan, Izabela Lipinska

Bioethics Centre, University of Otago · Independent Researcher

Human-AI InteractionAI Agent Safety
AskPDF
Jun 6, 2026

LCAM: A Framework for Diagnosing Interactional Alignment Failures in Con-versational AI

Conversational AI is increasingly used for advice, interpretation, reassurance, and decision support in contexts where users may be vulnerable, uncertain, or dependent on the system's apparent competence. Existing alignment work often focuses on model objectives, preference optimization, or output correctness. Yet, many harms arise through interaction: how systems frame authority, express uncertainty, simulate empathy, support reasoning, and make boundaries legible. This paper introduces the Layered Cognitive Alignment Model (LCAM), a conceptual and normative framework for diagnosing interac-tional alignment failures in conversational AI. LCAM defines alignment as a calibrated fit among system behavior, user goals, task demands, and normative context. It distinguishes five layers of fit: perceptual, semantic, affective, cognitive, and ethical, and two diagnostic polarities of misalignment: underfit and overreach. We apply LCAM to a published LLM counseling example, showing how an apparently supportive response can reinforce harmful beliefs, simulate inappropriate care, and obscure role boundaries. By translating conversational failures into audit and governance questions concerning over-reliance, false intimacy, autonomy erosion, boundary confusion, and inappropriate trust, LCAM offers a theoretical and normative lens for evaluating conversational AI beyond accuracy, helpfulness, or trust.

Manuele Reani, Hongyu Tian

School of Management and Economics, The Chinese University of Hong Kong, Shenzhen, China

LLM AlignmentLLM Safety AlignmentConversational AgentsLanguage Model Auditing
AskPDF
Sep 13, 2026

Trust by Design: Trust Calibration Through Non-Advisory Socratic Dialogue in Conversational Agents

As conversational AI systems increasingly operate in sensitive domains, the central challenge shifts from usability to trust calibration, ensuring that users rely on systems neither too much nor too little. Systems that provide advice or interpretations risk encouraging inappropriate reliance, particularly when users perceive AI outputs as authoritative. We present CASELy, a conversational agent explicitly designed to limit its own authority through non-advisory Socratic dialogue. The agent asks reflective questions grounded exclusively in user input and refuses to provide advice, recommendations, or interpretations. This design operationalizes trust calibration by constraining agent agency rather than optimizing capability. In a pilot randomized controlled study with higher education students, participants interacting with the Socratic dialogue reported substantially higher user experience (UEQ-S overall = 1.50) compared to a non-dialogue control (0). Qualitative findings identify three mechanisms supporting calibrated trust: transparency through visible grounding, preservation of user decision authority, and reduced fear of judgment. We argue that appropriate reliance can be achieved through interactional constraints, offering a design pattern for trustworthy conversational AI in sensitive contexts.

Roba Hassan, Nahla Aboromi, Naomi Unkelos-Shpigel

Braude College of Engineering, Israel

Trust in AIConversational Agents
AskPDF