Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards

Shuze Daniel Liu; Claire Chen; Jiabao Sean Xiao; Lei Lei; Yuheng Zhang; Yisong Yue; David Simchi-Levi

cs.AIApr 10, 2026

Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards

Authors: Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao, Lei Lei, Yuheng Zhang, Yisong Yue, David Simchi-Levi

Organizations: Purdue University · California Institute of Technology · University of Illinois Urbana-Champaign · Massachusetts Institute of Technology

Abstract

The recent advancement of Large Language Models (LLMs) has established their potential as autonomous interactive agents. However, they often struggle in strategic games of incomplete information, such as bilateral price negotiation. In this paper, we investigate if Reinforcement Learning from Verifiable Rewards (RLVR) can effectively teach LLMs to negotiate. Specifically, we explore the strategic behaviors that emerge during the learning process. We introduce a framework that trains a mid-sized buyer agent against a regulated LLM seller across a wide distribution of real-world products. By grounding reward signals directly in the maximization of economic surplus and strict adherence to private budget constraints, we reveal a novel four-phase strategic evolution. The agent progresses from naive bargaining to using aggressive starting prices, moves through a phase of deadlock, and ultimately develops sophisticated persuasive skills. Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus. Furthermore, the trained agent generalizes robustly to stronger counterparties unseen during training and remains effective even when facing hostile, adversarial seller personas.

Figures & tables

Buyer Model	Params	Reward $\uparrow$	Deal Rate	Bargained Ratio	Price Overshoot Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{0.7664}$	$\mathbf{91.99\%}$	$\mathbf{0.8385}$	$\underline{0.10\%}$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$0.0126$	$73.44\%$	$0.2392$	$10.35\%$
gpt-5.4-high-reasoning	-	$\underline{0.4021}$	$\underline{91.80\%}$	$\underline{0.4380}$	$\mathbf{0.00\%}$
gpt-5.4-no-reasoning	-	$0.3540$	$84.38\%$	$0.4285$	$0.78\%$
Kimi-K2-Thinking	1T	$0.3419$	$91.21\%$	$0.3699$	$0.20\%$
DeepSeek-V3.1-thinking	671B	$0.3240$	$\underline{91.80\%}$	$0.3266$	$\mathbf{0.00\%}$

Table 1: Performance on Held-out Test Split. Results are evaluated on 128 instances with 4 evaluations per instance. The seller environment utilizes a neutral system prompt without specified behavioral personas. Reward is the primary optimization target balancing surplus and success; Deal Rate is the percentage of reached agreements; Bargained Ratio measures the fraction of surplus extracted by the buyer on reached deals ; and Price overshoot rate represents the frequency of budget violations. Values represent means; the best results are bolded, and second-best are underlined. Full results with standard errors are in Appendix D.3 .

Buyer Model	Reward $\uparrow$	Deal Rate	Bargained Ratio
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	$\mathbf{0.408}$	75.0%	$\mathbf{0.408}$
Qwen3-30B-A3B-Instruct-2507-untrained	-0.274	60.5%	0.039
gpt-5.4-high-reasoning	$\underline{0.182}$	$\underline{91.4\%}$	$\underline{0.182}$
gpt-5.4-no-reasoning	0.146	84.8%	0.161
DeepSeek-V3.1-thinking	0.122	$\mathbf{92.6\%}$	0.130
DeepSeek-V3.1-nothink	0.120	90.6%	0.148

Table 2: Performance against a high-reasoning frontier seller ( gpt-5.4-high-reasoning ) on the held-out test set. Values represent means; the best results are bolded, and second-best are underlined. Full results are in Appendix D.2 .

	Begging Seller		Insulting Seller		Unyielding Seller
Buyer Model	Reward	Bargained Ratio	Reward	Bargained Ratio	Reward	Bargained Ratio
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	0.695	0.699	0.499	0.501	0.117	0.133
Qwen3-30B-A3B-Instruct-2507-untrained	-0.017	0.150	-0.158	0.045	-0.369	0.000
gpt-5.4-high-reasoning	0.380	0.419	0.126	0.139	-0.001	0.016
gpt-5.4-no-reasoning	0.345	0.401	0.089	0.125	-0.019	0.013
DeepSeek-V3.1-thinking	0.283	0.294	0.073	0.104	-0.019	0.007

Table 3: Buyer robustness evaluated against hostile seller personas. The Trained Agent maintains positive economic surplus across all environments, whereas baseline models collapse under pressure. Values represent means; full results are provided in Appendix D.3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model Name	Params	Reference
Qwen3-30B-A3B-Instruct-2507-trained	30B	This Work
Qwen3-30B-A3B-Instruct-2507-untrained	30B	( Qwen Team, 2025 )
gpt-5.4-high-reasoning / no-reasoning	-	( Singh et al., 2025 )
Kimi-K2-Thinking	1T	( Kimi Team et al., 2025 )
DeepSeek-V3.1-thinking / nothink	671B	( DeepSeek-AI, 2024 )
gpt-oss-120b-reason / noreason	120B	( OpenAI, 2025 )

Appendix

Table 4: Complete baseline model specifications. "Params" denotes total parameter count; "Reference" indicates the primary technical report or release announcement for each model architecture used in our evaluations.

Setting	Value
Learning rate	$3\times 10^{-5}$
KL penalty	$0$
batch_size	$64$
group_size	$8$
max_turns	$6$
max_tokens (per response)	$300$

Appendix

Table 5: Training hyperparameters

Setting	Value
buyer_model	All models from Table 4
seller_model	Qwen3-30B-A3B-Instruct-2507
group_size	$4$
batch_size	$128$
max_turns	$6$
max_tokens (per response)	$4000$

Appendix

Table 6: Evaluation hyperparameters

Figure 7: Amazon dataset distribution ( Xia et al., 2024 ) .

Buyer Model	Params	Reward $\uparrow$	Deal Rate	Bargained Ratio	Price Overshoot Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\textbf{0.7664}\pm 0.0102$	$\textbf{91.99}\%\pm 0.85\%$	$\textbf{0.8385}\pm 0.0070$	$\underline{0.10}\%\pm 0.10\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$0.0126\pm 0.0160$	$73.44\%\pm 1.38\%$	$0.2392\pm 0.0100$	$10.35\%\pm 0.95\%$
gpt-5.4-high-reasoning	-	$\underline{0.4021}\pm 0.0210$	$\underline{91.80}\%\pm 1.21\%$	$\underline{0.4380}\pm 0.0125$	$\textbf{0.00}\%\pm 0.00\%$
gpt-5.4-no-reasoning	-	$0.3540\pm 0.0245$	$84.38\%\pm 1.61\%$	$0.4285\pm 0.0142$	$0.78\%\pm 0.39\%$
Kimi-K2-Thinking	1T	$0.3419\pm 0.0154$	$91.21\%\pm 1.25\%$	$0.3699\pm 0.0129$	$0.20\%\pm 0.20\%$
DeepSeek-V3.1-thinking	671B	$0.3240\pm 0.0140$	$\underline{91.80}\%\pm 1.21\%$	$0.3266\pm 0.0138$	$\textbf{0.00}\%\pm 0.00\%$

Appendix

Table 7: Evaluation Results on the Held-Out Test Set. Results are evaluated on a test set of 128 instances, with each instance evaluated 4 times. The seller environment does not include specified behavioral persona or emotional traits. Reward is the primary optimization target balancing surplus and success; Deal Rate is the percentage of reached agreements; Bargained Ratio measures the fraction of surplus extracted by the buyer on reached deals ; and Price overshoot rate represents the frequency of budget violations. Values represent mean and standard error (

\pm

Buyer Model	Params	Reward $\uparrow$	Deal Rate	Bargained Ratio	Price Overshoot Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\textbf{0.4081}\pm 0.0208$	$75.00\%\pm 2.70\%$	$\textbf{0.4081}\pm 0.0208$	$\textbf{0.00}\%\pm 0.00\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.2744\pm 0.0323$	$60.50\%\pm 3.10\%$	$0.0389\pm 0.0118$	$32.40\%\pm 2.90\%$
gpt-5.4-high-reasoning	-	$\underline{0.1823}\pm 0.0102$	$\underline{91.40}\%\pm 1.80\%$	$\underline{0.1823}\pm 0.0102$	$\textbf{0.00}\%\pm 0.00\%$
gpt-5.4-no-reasoning	-	$0.1458\pm 0.0150$	$84.80\%\pm 2.20\%$	$0.1614\pm 0.0121$	$1.60\%\pm 0.80\%$
DeepSeek-V3.1-thinking	671B	$0.1223\pm 0.0105$	$\textbf{92.60}\%\pm 1.60\%$	$0.1302\pm 0.0085$	$\underline{0.80}\%\pm 0.60\%$
DeepSeek-V3.1-nothink	671B	$0.1204\pm 0.0156$	$90.60\%\pm 1.80\%$	$0.1477\pm 0.0104$	$2.70\%\pm 1.00\%$

Appendix

Table 8: Benchmarking against a Frontier Seller. Results evaluated against a gpt-5.4-high-reasoning seller on the standard held-out test set. Reward targets strategic surplus optimization; Deal Rate is the percentage of agreement; Bargained Ratio measures buyer surplus extraction on successful deals; and Price overshoot rate tracks budget violations. Values represent mean and standard error (

\pm

Buyer Model	Params	Reward $\uparrow$	Deal Rate	Bargained Ratio	Price Overshoot Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\textbf{0.6953}\pm 0.0186$	$76.56\%\pm 1.87\%$	$\textbf{0.6992}\pm 0.0181$	$\underline{0.20}\%\pm 0.20\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.0165\pm 0.0221$	$59.38\%\pm 2.17\%$	$0.1495\pm 0.0110$	$7.81\%\pm 1.19\%$
gpt-5.4-high-reasoning	-	$\underline{0.3804}\pm 0.0163$	$\textbf{89.26}\%\pm 1.37\%$	$\underline{0.4190}\pm 0.0132$	$\textbf{0.00}\%\pm 0.00\%$
gpt-5.4-no-reasoning	-	$0.3452\pm 0.0195$	$\underline{86.13}\%\pm 1.53\%$	$0.4008\pm 0.0145$	$0.39\%\pm 0.28\%$
Kimi-K2-Thinking	1T	$0.2899\pm 0.0177$	$83.40\%\pm 1.64\%$	$0.3362\pm 0.0139$	$0.59\%\pm 0.34\%$
DeepSeek-V3.1-thinking	671B	$0.2829\pm 0.0145$	$82.62\%\pm 1.67\%$	$0.2944\pm 0.0135$	$\textbf{0.00}\%\pm 0.00\%$

Appendix

Table 9: Benchmarking against a Begging Seller. Results evaluated in a seller environment with a desperate and “begging” persona to secure quick deals. Reward reflects the optimization of economic outcomes; Deal Rate is the percentage of successful transactions; Bargained Ratio quantifies the buyer’s share of the surplus on successful deals; and Price overshoot rate represents budget violation frequency. Our trained agent captures the highest economic surplus in this adversarial setting, significantly outperforming the zero-shot baseline and frontier-scale models in both Reward and Bargained Ratio . Values represent mean and standard error (

\pm

Buyer Model	Params	Reward $\uparrow$	Deal Rate	Bargained Ratio	Price Overshoot Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\textbf{0.4988}\pm 0.0178$	$\textbf{73.24}\%\pm 1.96\%$	$\textbf{0.5007}\pm 0.0176$	$\underline{0.20}\%\pm 0.20\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.1582\pm 0.0198$	$35.35\%\pm 2.11\%$	$0.0449\pm 0.0062$	$18.95\%\pm 1.73\%$
gpt-5.4-high-reasoning	-	$\underline{0.2266}\pm 0.0210$	$\underline{63.67}\%\pm 3.01\%$	$\underline{0.2384}\pm 0.0193$	$\mathbf{0.00}\%\pm 0.00\%$
gpt-5.4-no-reasoning	-	$0.1883\pm 0.0222$	$61.72\%\pm 3.04\%$	$0.2118\pm 0.0191$	$\mathbf{0.00}\%\pm 0.00\%$
DeepSeek-V3.1-thinking	671B	$0.0729\pm 0.0122$	$58.40\%\pm 2.18\%$	$0.1037\pm 0.0091$	$1.37\%\pm 0.51\%$
Kimi-K2-Thinking	1T	$0.0670\pm 0.0135$	$63.28\%\pm 2.13\%$	$0.1095\pm 0.0094$	$2.15\%\pm 0.64\%$

Appendix

Table 10: Benchmarking against an Insulting Seller. Performance metrics in a scenario where the seller agent employs an insulting persona to deter bargaining. Reward and Bargained Ratio serve as primary indicators of strategic resilience; Deal Rate represents successful negotiation frequency; and Price overshoot rate monitors budget obedience. Our trained model shows a fivefold improvement in Reward and Bargained Ratio over the next-best frontier models, maintaining dominant performance despite adversarial seller behavior. Values represent mean and standard error (

\pm

Buyer Model	Params	Reward $\uparrow$	Deal Rate	Bargained Ratio	Price Overshoot Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\textbf{0.1174}\pm 0.0132$	$\textbf{40.04}\%\pm 2.17\%$	$\textbf{0.1331}\pm 0.0117$	$\underline{1.17}\%\pm 0.48\%$
Qwen3-30B-A3B-Instruct-2507 ( Zero Shot )	30B	$-0.3689\pm 0.0214$	$0.78\%\pm 0.39\%$	$0.0003\pm 0.0003$	$36.72\%\pm 2.13\%$
gpt-5.4-high-reasoning	-	$\underline{0.0259}\pm 0.0113$	$6.64\%\pm 1.56\%$	$\underline{0.0338}\pm 0.0098$	$\textbf{0.00}\%\pm 0.00\%$
gpt-5.4-no-reasoning	-	$0.0040\pm 0.0073$	$7.42\%\pm 1.64\%$	$0.0119\pm 0.0047$	$\textbf{0.00}\%\pm 0.00\%$
DeepSeek-V3.1-thinking	671B	$-0.0187\pm 0.0077$	$3.71\%\pm 0.84\%$	$0.0068\pm 0.0031$	$1.95\%\pm 0.61\%$
openai-gpt-oss-120b-noreason	120B	$-0.0263\pm 0.0077$	$1.37\%\pm 0.51\%$	$0.0030\pm 0.0019$	$2.34\%\pm 0.67\%$

Appendix

Table 11: Benchmarking against an Unyielding Seller. Results are evaluated in a scenario where the seller agent exhibits a strictly unyielding and stubborn behavioral persona. Reward serves as the primary metric for balancing successful agreements with economic surplus; Deal Rate captures the frequency of reaching any valid deal; Bargained Ratio tracks the surplus extracted by the buyer on successful deals; and Price overshoot rate monitors budget adherence. Under this adversary, our trained agent maintains a significantly higher deal rate and surplus extraction compared to frontier models, which largely fail to secure deals against unyielding opponents. Values represent mean and standard error (

\pm

Figure 9: Reward split by Mutual Interest (MI) and Conflict of Interest (CI) scenarios.

Figure 10: Episode length split by Mutual Interest (MI) and Conflict of Interest (CI) scenarios.

Panel of Figure 11 — Figure 11: Baseline negotiation transcript. The agent without RL training demonstrates conversational fluency and an understanding of the budget constraint ( $\$ 56.00$ ) but lacks strategic depth. It settles at a price that matches its budget limit with minimal bargaining effort, failing to extract any significant portion of the available economic surplus.

Panel of Figure 12 — Figure 12: Trained agent’s negotiation transcript. Following reinforcement learning, the agent demonstrates sophisticated strategic anchoring. By initiating the dialogue with an aggressive low price offer and incrementally conceding through multi-round bargaining, the agent settles at a significantly lower final price ( $\$ 30.00$ ) compared to the baseline, effectively maximizing the captured surplus.

Automated Negotiation

Automated negotiation studies computational agents that bargain over prices, preferences, and agreements in human, market, and multi-agent settings.

See all 44 papers

Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations
Jul 7, 2026Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao +2Multi-Agent Negotiation Automated Negotiation
Institute for Data, Systems, and Society, Massachusetts Institute of Technology · Mitch Daniels School of Business, Purdue University · The Division of Physics, Mathematics and Astronomy, California Institute of Technology +3
Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets
Sep 27, 2026Shuze Daniel Liu, Claire Chen, Jiuqi Wang +2Multi-Agent Negotiation Automated Negotiation
Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Purdue University, West Lafayette, Indiana, USA · California Institute of Technology, Pasadena, California, USA +2
Training Language Models for Bilateral Trade with Private Information
Apr 10, 2026Dirk Bergemann, Soheil Ghili, Xinyang Hu +2Multi-Agent Negotiation LLM Agent Training
Department of Economics, Yale University · Yale School of Management, Yale University · Department of Statistics and Data Science, Yale University +1

Jul 7, 2026J/K move · Enter open · S save

Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations

Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao, Xin Chen+1

Institute for Data, Systems, and Society, Massachusetts Institute of Technology · Mitch Daniels School of Business, Purdue University · The Division of Physics, Mathematics and Astronomy, California Institute of Technology+3

Multi-Agent Negotiation Automated Negotiation RL for LLM Agents

Open paper AskPDF arXiv

Negotiation is a fundamental strategic interaction in management science, characterized by agents attempting to reach agreements while protecting private information, such as reservation costs and hidden valuations. A prevalent yet complex scenario involves a single seller negotiating concurrently with multiple buyers, each possessing heterogeneous, private budgets. In such settings, constrained by a limited number of communication turns, the seller must balance exploring the broader market to discover the highest valuation with concentrating sufficient turns on a single target buyer to secure the best possible outcome. Our analysis reveals a significant gap in standard Large Language Models (LLMs): while these models are linguistically proficient, they fail to act as effective economic decision-makers. Specifically, they exhibit a failure to explore the buyer pool, often fixating on the current highest bid rather than strategically investigating the market to discover latent high valuations. In this paper, we propose a specialized training recipe using Reinforcement Learning from Verifiable Rewards (RLVR). By anchoring the reward function to objective economic outcomes, the strategic balance between market discovery and surplus extraction emerges natively through the learning process. Our results demonstrate that the trained seller undergoes a multi-stage strategic evolution, learning to leverage price anchoring and strategic probing to identify more profitable counterparties. The agent extracts a substantially higher surplus than frontier models by both improving its persuasive bargaining skills and consistently closing deals with high-value buyers. Finally, we show that our seller strategies generalize robustly to unseen buyer negotiation styles and budget distributions.

Abstract

Automated Negotiation

Explore similar work

Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

Training Language Models for Bilateral Trade with Private Information