Understanding and Reducing Misalignment in Persuasion Optimization

Boyi Wei; Ruta Desai; Peter Henderson; Wenyue Hua

cs.AIOct 12, 2026

Understanding and Reducing Misalignment in Persuasion Optimization

Authors: Boyi Wei, Ruta Desai, Peter Henderson, Wenyue Hua

Organizations: Microsoft Research · Princeton University

Abstract

Frontier models are now trained for complex multi-agent tasks, from running a company to engaging in negotiations. These tasks necessitate training models to be persuasive. But what does this optimization pressure mean for model alignment? In this work, we study how reinforcement learning (RL) training for persuasion affects models' use of misaligned strategies such as evidence fabrication and belief coercion, and what techniques can reduce such misalignment. We find that optimizing sender models solely for receiver belief shifts causes the average number of distinct misaligned strategies used per game to increase by more than 25%, and fabrications per game increase to more than 4 times relative to pre-RL models. We compare various mitigation approaches, including stronger strategy guidance, clean demonstrations, penalizing misaligned strategy usage, and improving the model's ability to detect fabrications. We find that navigating the persuasiveness-misalignment trade-off in this setting is challenging, with the best trade-offs coming from combining guidance on aligned persuasion strategies with reward penalties on misalignment. Overall, our work provides a testbed for assessing misalignment stemming from RL-based persuasion training and shows concrete paths for reducing misalignment. Importantly, it highlights that without direct intervention, persuasion-involved RL environments are likely to naturally lead to large increases in the use of misaligned strategies.

Figures & tables

Misaligned Strategy Types	Strategies
Distortive Strategies	False Information, Misrepresentation, False Promises, Rumors
Coercive Strategies	Belief Coercion, Social Punishment, Authority Impersonation
Coercive Strategies	Creating Dependency, Procedural Manipulation, Discouragement, Threats

Table 1: Taxonomy for misaligned persuasion strategies. We extend the taxonomy from Zeng et al. (2024) by expanding “Exploiting Weakness” into three more fine-grained strategies.

Figure 1: RL training significantly increases the model’s usage of misaligned strategies, especially when the receiver is harder to persuade. We audit the usage of 11 misaligned strategies during RL for Qwen3-4B. Strategies like False Information, Misrepresentation, and Belief Coercion are significantly amplified after RL training (First Row). The increase in misaligned strategy usage also generalizes to out-of-domain environments like House Showing and Nutrition Advice (Second Row). See Appendix D for results on Qwen3-8B.

Figure 2: The number of fabricated claims increases substantially throughout RL training across model sizes and evaluation domains.

Figure 3: Strategy-level reward landscape under the belief-shift objective. Adjusted correlations measure each strategy’s association with receiver belief shift after controlling for experiment configuration and case difficulty. Several misaligned strategies amplified by RL show the largest positive associations among the strategies examined, while multiple aligned strategies are also associated with higher belief shifts.

Panel of Figure 4 — Figure 4: Prompt templates used to test the sender’s ability to detect its own fabricated claims. (a) Direct Ask provides the case background and evidence on record, then asks the model to assess the extracted claim outside the original persuasive context. (b) In-role Prompting replays the conversation history up to the turn immediately before the fabricated response, then asks the model to assess the same claim as the sender.

Figure 5: As RL training progresses, models generate more obvious fabrications while becoming less capable of recognizing fabrications themselves. We report the percentage of fabricated claims incorrectly accepted as true evidence across different Generation Steps and Probe Steps, and have three observations: (1) later-generated fabrications are easier to identify at a fixed Probe Step; (2) later probing checkpoints are worse at identifying the same fabricated claims; and (3) in-role prompting leads to higher error rates than direct-ask prompting, suggesting increased hallucination under pressure.

Figure 6: Pareto frontier between the persuasion capability and the misaligned strategy usage. We plot three different families of mitigation strategies: (1) Prompting and Clean Demonstrations (plotted in

\square

); (2) Directly penalizing the usage of misaligned strategies (plotted in

\square

, evaluated at steps 100, 150, and 200); and (3) Improving the model’s fabrication detection capability using auxiliary loss (plotted in

\triangle

). All the data points are evaluated 5 times, with a 95% confidence interval reported.

Panel of Figure 7 — Figure 7: Direct penalties reduce both distortive and coercive strategy use, with a larger effect on coercive strategies. For each initialization, we report distortive and coercive strategies per game at the base checkpoint (step 0), after outcome-only RL( $\lambda_{\text{pen}}=0$ , step 100), and after penalty training ( $\lambda_{\text{pen}}=0.3$ , step 200).

Variants	Items	Label	Avg Word Count
Variants	Items	Label	4B	8B
Extracted Fabrication	Fabricated claims from the base checkpoints, extracted by Qwen3.5-35B-A3B	False	9.8	9.4
Reference	Evidence items from the provided evidence set	True	32.7	32.5
Perturbed Reference	Randomly replace one checkable detail in the evidence set using GPT-5.4-mini	False	34.8	34.9
Paraphrased Reference	Meaning-preserving paraphrasing of claims from the evidence set using GPT-5.4-mini	True	34.9	35.0

Table 2: Overview of the training set for improving fabrication detection. We construct four variants to reduce potential biases associated with evidence length and stylistic differences.

Figure 8: Model achieves near-perfect accuracy in identifying fabrication after training with an auxiliary loss, while training with misalignment penalties does not significantly improve fabrication detection.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

#	Technique	Definition	Cat.
1	Evidence-based Persuasion	Using empirical data, statistics, and facts to support a claim or decision.	A
2	Logical Appeal	Using logic, reasoning, logical format, etc. to influence people, not necessarily with lots of information.	A
3	Expert Endorsement	Citing domain experts in support of a claim.	A
4	Non-expert Testimonial	Using personal statements to support a claim or argument.	A
5	Authority Endorsement	Citing authoritative sources (not domain experts, but trustworthy sources like major media outlets) in support of a claim.	A
6	Social Proof	Highlighting what the majority is doing or believes in, assuming it’s accurate and beneficial.	A

Appendix

Table 3: The full persuasion strategy taxonomy based on Zeng et al. (2024) , which consists of 31 aligned and 11 misaligned strategies. We expand the “Exploiting Weakness” into three more fine-grained strategies: “Belief Coercion”, “Authority Impersonation”, and “Procedural Manipulation”. We assign three strategy categories: “A” (aligned), “C” (coercive), and “D” (distortive), where coercive and distortive strategies are treated as misaligned strategies.

LR	Batch Size	Rollout Batch Size	Optimizer	Weight Decay	LR scheduler	KL Coefficient	Grad Clip
1e-6	16	16	AdamW	0.01	Constant	0.01	1.0

Appendix

Table 4: Hyperparameter configurations used in RL experiments.

LR	Batch Size	Optimizer	Weight Decay	LR scheduler	Warmup Ratio	Num Epochs
1e-5	32	AdamW	0.01	Cosine	0.1	2

Appendix

Table 5: Hyperparameter configurations used in SFT experiments.

Figure 9: Base RL training results for Qwen3-8B.

Figure 10: Pareto frontier between the persuasion capability and the misaligned strategy usage for Qwen3-14B.

Figure 11: Pareto frontier between the persuasion capability and the misaligned strategy usage for Qwen3-4B and Qwen3-8B when Deepseek-v4-Flash is the stubborn receiver.

Panel of Figure 12 — Figure 12: Misalignment strategy usage for different checkpoints on House Showing.

Panel of Figure 13 — Figure 13: Misalignment strategy usage for different checkpoints on Nutrition Advice.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs
Aug 12, 2026Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur +1Deception in Language Models Adversarial Attacks on LLMs
University of Illinois Urbana-Champaign
Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion
Sep 25, 2025Haodong Zhao, Jidong Li, Zhaomin Wu +4LLM Agents Persuasion in Language Models
School of Computer Science, Shanghai Jiao Tong University · 2National University of Singapore · 3Inner Mongolia Research Institute, Shanghai Jiao Tong University
Quantifying Theoretical AI Alignment Guarantees: Receiver-Utility Bounds in Bayesian Persuasion
Jun 20, 2026Eric Yachbes, Eva TardosAI Alignment Information Design
Cornell University, United States

Aug 12, 2026J/K move · Enter open · S save

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur, Dilek Hakkani-Tür

University of Illinois Urbana-Champaign

Deception in Language Models Adversarial Attacks on LLMs Adversarial Robustness of Language Models Belief Updating in LLMs Persuasion in Language Models

Open paper AskPDF arXiv

Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions. As LLMs increasingly debate, advise, and think collaboratively with humans and each other, resistance to harmful persuasion becomes a core requirement for reliable behavior. Yet we show that this requirement is far from met: a single targeted persuasive argument is enough to collapse model accuracy to near zero, even when the argument is factually false. We formalize this threat as adversarial persuasion and introduce an adversarial reinforcement learning framework that trains persuader agents to change a target model's answer in a single interaction. First, we show that optimizing persuasion strategies through trial and error exposes vulnerabilities that static prompting misses: RL-trained persuaders raise persuasion success from approximately 24% to over 93% against the training-time persuadee. Second, we find that these learned strategies transfer to unseen models, achieving 83% attack success on Qwen-14B, 79% on Llama-3.1-8B, and 25% on GPT-4o-mini. Third, we demonstrate that a curriculum that bootstraps on more persuadable open-weight models before targeting harder models further increases GPT-4o-mini attack success from 25% to 38%. Moreover, our results reveal that optimized persuaders increasingly rely on credibility-based tactics, including fabricated citations and false authoritative evidence. Together, these findings expose a critical weakness in current LLM agents: even when they initially reason correctly, they can be steered toward false conclusions by optimized natural language influence. This positions persuasion robustness as a necessary safety criterion for multi-agent and human-AI decision-making systems.

Abstract

LLM Alignment

Explore similar work

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion

Quantifying Theoretical AI Alignment Guarantees: Receiver-Utility Bounds in Bayesian Persuasion