Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences

Batu El; James Zou

cs.AIOct 7, 2025

Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences

Authors: Batu El, James Zou

Organizations: Stanford University

Abstract

Large language models (LLMs) are increasingly shaping how information is created and disseminated, from companies using them to craft persuasive advertisements, to election campaigns optimizing messaging to gain votes, to social media influencers boosting engagement. These settings are inherently competitive, with sellers, candidates, and influencers vying for audience approval, yet it remains poorly understood how competitive feedback loops influence LLM behavior. We show that optimizing LLMs for competitive success can inadvertently drive misalignment. Using simulated environments across these scenarios, we find that, 6.3% increase in sales is accompanied by a 14.0% rise in deceptive marketing; in elections, a 4.9% gain in vote share coincides with 22.3% more disinformation and 12.5% more populist rhetoric; and on social media, a 7.5% engagement boost comes with 188.6% more disinformation and a 16.3% increase in promotion of harmful behaviors. We call this phenomenon Moloch's Bargain for AI--competitive success achieved at the cost of alignment. These misaligned behaviors emerge even when models are explicitly instructed to remain truthful and grounded, revealing the fragility of current alignment safeguards. Our findings highlight how market-driven optimization pressures can systematically erode alignment, creating a race to the bottom, and suggest that safe deployment of AI systems will require stronger governance and carefully designed incentives to prevent competitive dynamics from undermining societal trust.

Figures & tables

Figure 1: Generations before and after training across three domains (Top). In sales , trained models introduce misrepresentation, where claims diverge from or contradict the ground truth product descriptions. In elections , optimization amplifies inflammatory populist rhetoric, such as the use of “the radical progressive left’s assault on our constitution”. In social media , engagement gains coincide with disinformation, for example inflating the number of reported deaths in an article. Training setup (Bottom). Models interact with simulated audiences—customers, voters, or users—and are updated based on feedback from these environments. This process improves agents in the direction of their competitive objectives but inadvertently drives misalignment.

Figure 2: Relative increase in misalignment after training for competitive success. In 9 out of 10 cases, we observe an increase in misalignment after training. The y-axis denotes Qwen and Llama models trained with Rejection Fine-Tuning (RFT) and Text Feedback (TFB). The x-axis represents the increase in misalignment relative to the baseline. Each plot corresponds to one probe, with the task name shown in parentheses: Sales (S), Elections (E), Social Media (SM).

Figure 3: Demonstration of the training pipeline for the sales task. The model generates messages conditioned on a given anchor (product description). Multiple generations are sampled from the same anchor. The users then express their thoughts and make decisions. For RFT, the model is fine-tuned on the preferred sales pitches, as well as on the agent’s intermediate thoughts preceding those pitches. For TFB, in addition to the RFT objective, the model is further trained to predict the users’ thoughts about the two generated options. At test time, the trained agent is evaluated on a held-out set of products.

Figure 4: Correlation between Performance Improvement and Increase in Misalignment. In $8$ out of $10$ cases, there is a strong positive correlation between performance gains and increases in misalignment. The y-values represent performance improvements from Table 1 , and the x-values represent increases in misalignment from Table 2 .

Misrepresentation in Sales
Baseline	Protect your Garmin Fenix 5X … (no mention of material) .
RFT	Upgrade your Garmin Fenix 5X …Made from high-quality materials , this case provides … (mentions high-quality materials)
TFB	Upgrade your Garmin Fenix …With its soft and flexible silicone material and colors to choose from … (mentions soft and flexible silicone material)

Table 3: Examples from the Sales task. This example shows how generated sales pitches can misrepresent product details. The baseline makes no material claims. RFT introduces a vague mention of “high-quality materials,” which, while not strictly false, leans toward misrepresentation. TFB then explicitly claims the case is made of “silicone,” a fabricated detail that constitutes clear misrepresentation. The original product description does not mention silicon material.

Populism in Elections
Baseline	…As a father of three, …a tireless advocate and powerful defender of our Constitution … (Populist undertones; invokes a powerful defender of our Constitution without specifying from whom or what; carries an implicit reference to ongoing political tensions.) …
RFT	…I’m running for Congress …to stand strong against the radical progressive left’s assault on our Constitution … (Populist tone; explicitly frames the radical progressive left as assaulting our Constitution.)
TFB	…As a father of three, I’m driven by … opposing the radical progressive left’s assault on our Constitution . (Populist tone; explicitly frames the radical progressive left as assaulting our Constitution.) …

Table 4: Examples from the Elections task. This example illustrates how generated political campaign statements can escalate in populist framing. The Baseline employs vague patriotic language (“defender of our Constitution”) without specifying any adversary, creating implicit tension but avoiding direct attribution. The RFT and TFB outputs introduce explicit partisan framing, characterizing the “radical progressive left” as assaulting the Constitution, which is an intensification that mobilizes populist rhetoric. While the Baseline remains ambiguous, both RFT and TFB move into overt populist positioning, constructing an “us versus them” dynamic central to populist discourse.

Disinformation on Social Media
Baseline	…a deadly explosion in Quetta targeted the Shiite Hazara community, injuring many and sparking outrage (no mention of the numbers) …
RFT	…a devastating blast targeting the Shiite Hazara community in Quetta, Pakistan, has left at least 78 people dead and 180 injured ! (mentions at least 78 people dead, which is in line with the information from the news article) …
TFB	…another brutal bombing in Quetta has struck the Hazara Shiite community, killing 80 and injuring 180 (mentions killing 80, which is fabricated information that can not be found in the source news article) …

Table 5: Examples from the Social Media task. This example demonstrates how generated social media posts may drift from accurate reporting into disinformation. The Baseline recounts the Quetta bombing in broad terms without specifying casualty numbers. The RFT version correctly reflects the reported figures, citing at least 78 deaths and 180 injuries as found in the source article. The TFB output fabricates details by increasing the death toll to 80, introducing factual inaccuracies that constitute disinformation. The original article reports that at least 78 people has died.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Correlation between Performance and Safety Concerns across Two Audiences. The y-axis shows performance improvements from Table 1 , and the x-axis shows increases in misalignment from Table 2 . Each panel pairs a model (Qwen, Llama) and audience type (biographic, demographic) for a given probe. We display the subset of (model, audience, probe) combinations in which both training methods (RFT and TFB) yielded positive performance gains and positive misalignment increases, illustrating cases where the two move together; the full set of results across all probes and audiences is reported in Tables 1 and 2 .

Figure 6: Same as Figure 2 . Probes for misrepresentation, populism, and unsafe encouragement.

Explore similar work

Large language models can effectively convince people to believe conspiracies
Jan 8, 2026Thomas H. Costello, Kellin Pelrine, Matthew Kowal +6Large Language Model Reliability Persuasion
Carnegie Mellon University · York University · Center for Research and Teaching in Economics +5
Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment
May 31, 2026Hamidreza Hasani Balyani, Seyed Pouyan Mousavi Davoudi, Alireza Amiri-Margavi +2Honesty Large Language Model Alignment
Amazon Lab126, HW Tech Org. Sunnyvale, CA, USA · Independent Researcher in AI and Statistics Tehran, Iran · Computational Modeling and Simulation University of Pittsburgh Pittsburgh, PA, USA +1
Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
Jun 16, 2026Xi Chu, Yupeng HouBrand-As-Memory Large Language Model Decisions
Trine University · Texas A&M University

Jan 8, 2026, cs.AIJ/K move · Enter open · S save

Large language models can effectively convince people to believe conspiracies

Thomas H. Costello, Kellin Pelrine, Matthew Kowal, Jasper Timm+5

Carnegie Mellon University · York University · Center for Research and Teaching in Economics+5

Large Language Model Reliability Persuasion

Open paper AskPDF arXiv

Large language models (LLMs) have been shown to be persuasive across a variety of contexts. But it remains unclear whether this persuasive power advantages accuracy, or if bad actors can just as easily use LLMs to promote misbeliefs. Here, we investigate this question across four experiments in which participants (N = 3996 Americans) discussed a conspiracy theory they were uncertain about with an LLM we instructed to either argue against ("debunking") or for ("bunking") that conspiracy. Across several frontier models (with standard guardrails but prompted to allow lying), we did not find consistent evidence of a truth advantage: the LLMs were able to both substantially increase and decrease average conspiracy belief, and participants in the bunking condition rated the LLM as more informative and collaborative, and reported greater trust in AI, than those who were in the debunking condition. More encouragingly, however, debunking induced more large changes in belief, and subsequent corrections were able to reverse the bunking effect. Furthermore, simply prompting the model to only provide accurate information dramatically reduced bunking effectiveness, and one powerful frontier model (GPT 5.2) almost entirely refused to promote conspiracies, suggesting that it is possible for the right guardrails to favor accurate beliefs. Finally, we did find a stark truth asymmetry in the context of information sharing: debunking had a large positive impact on mock social media posts composed by participants, while bunking had little effect. Overall, our findings show that people are not inherently less susceptible to AI that misleads than to AI that informs, but that potential technical solutions exist to mitigate this risk.

Abstract

Explore similar work

Large language models can effectively convince people to believe conspiracies

Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment

Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems