Meta-Learning for Repeated Bayesian Persuasion

Bayesian persuasion studies how an informed sender can influence the behavior of a receiver through strategic information disclosure. Standard models assume the sender is the receiver's only source of information, yet in many applications receivers also consult external sources the sender can neither observe nor control. We study an online Bayesian persuasion problem in which a binary-action receiver has access to a fixed signaling scheme that is unknown to the sender. Over $T$ rounds, the sender commits to a signaling scheme and sends a signal; the receiver combines it with its private signal and acts, while the sender observes only the action. We design a learning algorithm that achieves regret $\widetilde{O}(T^{3/4})$ relative to the optimal scheme of a sender who knows the private signaling scheme of the receiver, with polynomial dependence on the sizes of the state space and the receiver's signal alphabet. Our key insight is reducing the problem of learning the exponentially large belief-space partitioning induced by the private scheme to a one-dimensional change-point detection problem.

Large language models (LLMs) still struggle with strategic persuasion, largely because existing approaches either neglect information asymmetry or rely on unrealistic pre-commitment assumptions. We introduce a type-induced commitment-communication mechanism that grounds Bayesian Persuasion (BP) in natural language dialogue without pre-commitment: the persuader narrates their potential types (e.g., honest vs. dishonest) to dynamically construct an information schema, enabling the persuadee to perform Bayesian belief updates within the conversation itself. We implement two variants: Semi-Formal-Natural-Language (SFNL) and Fully-Natural-Language (FNL), evaluating them against strong baselines across multiple LLMs and human judges. BP strategies consistently outperform baselines: SFNL excels in logical credibility, while FNL shows superior robustness and emotional resonance. We verify that gains stem from genuine Bayesian reasoning rather than superficial formatting, and we further show that supervised fine-tuning enables small models to match the persuasive performance of much larger ones.

Frontier models are now trained for complex multi-agent tasks, from running a company to engaging in negotiations. These tasks necessitate training models to be persuasive. But what does this optimization pressure mean for model alignment? In this work, we study how reinforcement learning (RL) training for persuasion affects models' use of misaligned strategies such as evidence fabrication and belief coercion, and what techniques can reduce such misalignment. We find that optimizing sender models solely for receiver belief shifts causes the average number of distinct misaligned strategies used per game to increase by more than 25%, and fabrications per game increase to more than 4 times relative to pre-RL models. We compare various mitigation approaches, including stronger strategy guidance, clean demonstrations, penalizing misaligned strategy usage, and improving the model's ability to detect fabrications. We find that navigating the persuasiveness-misalignment trade-off in this setting is challenging, with the best trade-offs coming from combining guidance on aligned persuasion strategies with reward penalties on misalignment. Overall, our work provides a testbed for assessing misalignment stemming from RL-based persuasion training and shows concrete paths for reducing misalignment. Importantly, it highlights that without direct intervention, persuasion-involved RL environments are likely to naturally lead to large increases in the use of misaligned strategies.