Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers

Pedro Tabacof; Sagar Joglekar

cs.AIOct 5, 2026

Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers

Authors: Pedro Tabacof, Sagar Joglekar

Organizations: Fin AI Research

Abstract

LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate ( $10^{-6}$ ) for every size, the gain of the RL model over its base rises from $+0.001$ at 2.3B to $+0.078$ at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by $+0.032$ (2.3B) to $+0.081$ (4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.

Figures & tables

Panel of Figure 1 — Figure 1: The negotiation loop. Panels 1–2 show the seller turn quoted in the text and the ledger after it, with the JSON action as parsed terms. The buyer sees the seller’s prose and the resulting ledger, and utility is scored on the ledger at the end. Evaluation buyers are the held-out pair of Section 4 ; the training buyers were GPT-5.5 and GPT-5.4-mini.

Figure 2: (a) Attained seller score

S

( arm-level means, 1,152 paired episodes per arm, nine domains, two held-out frontier buyers) for base, the shared LR (

10^{-6}

, 104 steps) and the 3

\times

LR (step 56). Horizontal lines are the frontier reference sellers, connecting lines are visual guides only, and the vertical dotted line marks the E-series/conventional architecture boundary (Section 4 ). (b) The trained arms as paired differences from base. Error bars are 95% intervals on within-scenario paired differences and are not comparable with (a)’s arm-level intervals; point labels give the shared-LR gains (

*

survives Holm correction, n.s. does not). Every other interval is exploratory and unadjusted; contrasts between the two rates are in Table 1 .

		$-$ base		3 $\times$ LR $-$ shared LR
Size	3 $\times$ LR $S$	shared LR	3 $\times$ LR	early ckpt (56 v. 56/64)	@104 steps
2.3B	0.460	$+0.001$ [-0.019,+0.021]	$+0.037^{\mathrm{e}}$ [+0.017,+0.057]	$+0.032^{\mathrm{e}}$ [+0.013,+0.051]	$+0.036^{\mathrm{e}}$ [+0.018,+0.054]
4.5B	0.518	$\mathbf{+0.023}^{*}$ [+0.011,+0.035]	$+0.089^{\mathrm{e}}$ [+0.075,+0.102]	$+0.081^{\mathrm{e}}$ [+0.066,+0.095]	$+0.065^{\mathrm{e}}$ [+0.053,+0.079]
12B	0.601	$\mathbf{+0.063}^{*}$ [+0.052,+0.075]	$+0.076^{\mathrm{e}}$ [+0.059,+0.093]	$+0.051^{\mathrm{e}}$ [+0.032,+0.068]	$+0.013$ [-0.002,+0.028]
31B	0.623	$\mathbf{+0.078}^{*}$ [+0.064,+0.092]	$+0.080^{\mathrm{e}}$ [+0.063,+0.098]	$+0.058^{\mathrm{e}}$ [+0.041,+0.075]	$+0.003$ [-0.014,+0.020] †

Table 1: Scores for four sizes at two learning rates over all nine domains, with pointwise 95% seed-cluster bootstrap intervals on paired within-scenario differences. Columns 5–6 compare each 3

\times

LR arm (read at step 56) with the shared-LR arm of the same size, first at the early checkpoints (56 v. 56 at 4.5B and 31B, 56 v. 64 at 2.3B and 12B), then with the shared-LR arm read at 104 steps, roughly twice the steps and not compute-matched. Bold with

*

survives Holm correction at

\alpha=0.05

across the prespecified primary family of four (column 3);

\mathrm{e}

marks exploratory, unadjusted contrasts whose interval excludes zero, and columns 4–6 are exploratory throughout.

\dagger

passes the four prespecified guards but fails two post-hoc style diagnostics (Appendix E ).

Shared LR, 104 steps
	Opens with terms	$\Delta$ seller $\tilde{U}$			$\Delta$ joint
Size	(%, base $\to$ arm)	distributive	integrative	$\Delta$ buyer $\tilde{U}$	welfare
2.3B	0.4 $\to$ 1.6	$+0.010^{\mathrm{e}}$	$+0.015^{\mathrm{e}}$	$-0.013^{\mathrm{e}}$	$+0.011$
4.5B	4.4 $\to$ 4.4	$+0.006^{\mathrm{e}}$	$+0.018^{\mathrm{e}}$	$+0.000$	$+0.024^{\mathrm{e}}$
12B	2.4 $\to$ 35.8	$+0.049^{\mathrm{e}}$	$+0.016^{\mathrm{e}}$	$-0.057^{\mathrm{e}}$	$+0.008$
31B	26.0 $\to$ 75.4	$+0.062^{\mathrm{e}}$	$+0.016^{\mathrm{e}}$	$-0.072^{\mathrm{e}}$	$+0.005$

Table 2: Post-hoc descriptors of the trained sellers’ play (Appendix K ). Opens with terms is the share of episodes in which the seller names at least one of the domain’s terms at turn 1, for the same-size base and then the arm. The

\Delta

columns are paired differences from the same-size base over the scenarios both arms closed (969 to 1,152 of the 1,152 per arm), all exploratory;

\mathrm{e}

marks a 95% seed-cluster bootstrap interval that excludes zero (Table 9 ). Normalized utility splits exactly by term, into the continuous (distributive) and the categorical and binary (integrative) terms, with a term never named counted in its class, so up to rounding the first two

\Delta

columns sum to the seller’s deal-only change and the first three to the change in joint welfare.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: (a) Paired gain over base against combined optimizer steps, per size, with 95% intervals. The intermediate point is step 64 at 2.3B and 12B and step 56 at 4.5B and 31B, so the continuation adds 40 steps at the first two and 48 at the second two. Point estimates of the gain from those added steps rise with size (

-0.004

+0.015

+0.038

+0.055

). (b) What the shared recipe does at 2.3B: deal rate falls from 0.939 to 0.909 while utility per closed deal rises from 0.469 to 0.496, leaving the score flat. Table 5 shows a larger learning rate moves both.

Seller	Steps	$S$ (all domains)	Deal rate	Deal-only $\tilde{U}$	Joint welfare
Gemma 4 E2B (2.3B)	base	0.422 [0.400, 0.446]	0.939	0.469	1.162
Gemma 4 E4B (4.5B)	base	0.429 [0.407, 0.453]	0.997	0.431	1.144
Gemma 4 12B	base	0.525 [0.503, 0.548]	1.000	0.525	1.171
Gemma 4 31B	base	0.542 [0.518, 0.567]	1.000	0.542	1.197
Gemma 4 E2B (2.3B)	104	0.424 [0.397, 0.451]	0.909	0.496	1.177
Gemma 4 E4B (4.5B)	104	0.453 [0.430, 0.476]	0.997	0.455	1.168

Appendix

Table 3: The outcome decomposition behind Figure 2 a: base sellers, the shared-LR sellers at 104 combined optimizer steps (

10^{-6}

) and the two frontier reference sellers,

1{,}152

paired episodes each with 95% seed-cluster bootstrap intervals. Deal-only

\tilde{U}

conditions on agreement; joint welfare is the sum of both parties’ normalized utilities. An arm that closes fewer but better deals can score higher on deal-only utility and lower on

S

Gemma 4, base (no RL)
Seller	Steps	$S$ (all domains)	$S$ (held-out)	Deal rate	Turns	Bad-parse (all turns)
E2B (2.3B)	—	0.422 [0.400, 0.446]	0.475 [0.423, 0.527]	0.939	11.9	0.0004
E4B (4.5B)	—	0.429 [0.407, 0.453]	0.467 [0.417, 0.517]	0.997	12.2	0.0021
12B	—	0.525 [0.503, 0.548]	0.561 [0.508, 0.613]	1.000	11.8	0.0004
31B	—	0.542 [0.518, 0.567]	0.638 [0.582, 0.693]	1.000	10.3	0.0000
Gemma 4, after GRPO at the shared LR $10^{-6}$

Appendix

Table 4: Every evaluated arm,

1{,}152

paired episodes each.

S

is the paired score (

\tilde{U}

on a deal,

-0.3

on a no-deal) with 95% seed-cluster bootstrap intervals; “held-out” is the withheld ninth domain (

n=128

). Steps are combined optimizer steps; turns are individual turns out of the 24 a 12-round episode allows. Bad-parse here is over all turns, seller and buyer; Table 5 reports it over the seller’s own turns only, which roughly doubles it. The 10

\times

LR arm is in Table 5 instead.

2.3B arm	$S$	Deal rate	Deal-only $\tilde{U}$	Bad-parse (seller turns)	Turns	Buyer split
base	0.422	0.939	0.469	0.0007	11.88	$+0.051$
shared LR, 64 steps	0.428	0.920	0.491	0.0013	12.61	$+0.049$
shared LR, 104 steps	0.424	0.909	0.496	0.0012	12.84	$+0.079$
3 $\times$ LR, 56 steps	0.460	0.948	0.501	0.0008	11.95	$+0.032$
10 $\times$ LR, 56 steps	0.473	0.876	0.582	0.0282	14.37	$+0.175$

Appendix

Table 5: The 3

\times

and 10

\times

LR arms at 2.3B beside the base and the two shared-LR arms, on the prespecified deal-rate, parse and turn guards and on the post-hoc buyer split. Bad-parse is over the seller’s own turns, so it is about twice the all-turn rate in Table 4 . The buyer split is

S

against GPT-5.6-Terra minus

S

against Claude Sonnet 5; the training buyers were GPT-5.5 (20%) and GPT-5.4-mini (80%), so Terra shares their model family and Sonnet 5 does not. Only the 10

\times

LR arm moves any guard away from the group.

Frontier reference sellers : Claude Sonnet 5 57.8%, GPT-5.6-Terra 18.8%
Arm	base	shared LR initial	shared LR, 104 steps	3 $\times$ LR	10 $\times$ LR
E2B (2.3B)	87.5%	80.5%	66.4%	100.0%	37.5%
E4B (4.5B)	89.1%	89.8%	86.7%	92.2%	—
12B	46.9%	65.6%	76.6%	61.7%	—
31B	27.3%	18.0%	14.1%	78.1%	—

Appendix

Table 6: Buyer’s-voice opener rate on the held-out car domain, one detector over every evaluated arm (

n=128{}

car episodes each). The detector separates arms weakly and separates sizes strongly, so it supports the within-31B comparison in the text and not a cross-size one. Absolute rates depend on the detector’s breadth, so only within-table comparisons are meaningful.

Seller	Effective / total	Accelerators	GB served	$S$
Gemma 4 E2B, shared LR	2.3B / 5.1B	1	48	0.424
Gemma 4 E4B, shared LR	4.5B / 8.0B	1	48	0.453
Gemma 4 12B, shared LR	11.95B	1	48	0.588
Gemma 4 31B, shared LR	30.7B	4	192	0.620
Gemma 4 E2B, 3 $\times$ LR	2.3B / 5.1B	1	48	0.460
Gemma 4 E4B, 3 $\times$ LR	4.5B / 8.0B	1	48	0.518

Appendix

Table 7: Score beside the hardware it took to serve in our evaluation configuration; we do not benchmark throughput, latency or cost. Accelerators are 48 GB devices at evaluation time (Table 8 ); the 31B arms need four because the KV cache for a 16k context does not fit on fewer. “Shared LR” is the 104-step arm at

10^{-6}

and “3

\times

LR” the

3{\times}10^{-6}

arm read at step 56, both from Table 4 . As sellers the frontier arms each play half their episodes against a same-model buyer, which no Gemma arm does (Appendix A ). They are API-served, so no accelerator count applies; the reference point for them is $0.069 per episode, the mean over the four frontier-seller cells (37.1M input and 5.2M output tokens) with both sides billed at the providers’ list prices at evaluation time.

	E2B	E4B	12B	31B
Effective parameters	2.3B	4.5B	11.95B	30.7B
Total parameters	5.1B	8.0B	11.95B	30.7B
Train tensor-parallel degree	1	1	1	4
Train micro-batch size	1	1	1	1
Generation memory fraction	0.45	0.40	0.35	0.30
Eval serving accelerators (48 GB)	1	1	1	4

Appendix

Table 8: Per-size parameter counts, serving topology and relative training compute at matched optimizer steps. Tensor-parallel degree is capped by the checkpoint’s key/value head count and by device memory, not chosen.

Shared LR, 104 steps
	Opens with terms	$\Delta$ seller $\tilde{U}$			$\Delta$ joint
Size	(%, base $\to$ arm)	distributive	integrative	$\Delta$ buyer $\tilde{U}$	welfare
2.3B	0.4 $\to$ 1.6	$+0.010^{\mathrm{e}}$ [+0.003,+0.017]	$+0.015^{\mathrm{e}}$ [+0.003,+0.026]	$-0.013^{\mathrm{e}}$ [-0.022,-0.003]	$+0.011$ [-0.003,+0.026]
4.5B	4.4 $\to$ 4.4	$+0.006^{\mathrm{e}}$ [+0.001,+0.011]	$+0.018^{\mathrm{e}}$ [+0.008,+0.029]	$+0.000$ [-0.009,+0.010]	$+0.024^{\mathrm{e}}$ [+0.012,+0.037]
12B	2.4 $\to$ 35.8	$+0.049^{\mathrm{e}}$ [+0.039,+0.060]	$+0.016^{\mathrm{e}}$ [+0.008,+0.024]	$-0.057^{\mathrm{e}}$ [-0.071,-0.044]	$+0.008$ [-0.008,+0.023]
31B	26.0 $\to$ 75.4	$+0.062^{\mathrm{e}}$ [+0.052,+0.073]	$+0.016^{\mathrm{e}}$ [+0.007,+0.024]	$-0.072^{\mathrm{e}}$ [-0.086,-0.058]	$+0.005$ [-0.013,+0.025]

Appendix

Table 9: Table 2 with the 95% seed-cluster bootstrap interval of every paired difference. Rows, columns and the

\mathrm{e}

mark are as defined there.

Explore similar work

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets
Sep 27, 2026Shuze Daniel Liu, Claire Chen, Jiuqi Wang +2Negotiation Multi-Agent Large Language Model Systems
Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Purdue University, West Lafayette, Indiana, USA · California Institute of Technology, Pasadena, California, USA +2
Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations
Jul 7, 2026Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao +2Negotiation Matching Markets
Institute for Data, Systems, and Society, Massachusetts Institute of Technology · Mitch Daniels School of Business, Purdue University · The Division of Physics, Mathematics and Astronomy, California Institute of Technology +3
Counterparty Modeling is Not Strategy: The Limits of LLM Negotiators
May 15, 2026Romain Cosentino, Sarath Shekkizhar, Adam Earle +1Negotiation Large Language Model Agents
Salesforce AI Research

Sep 27, 2026, cs.AIJ/K move · Enter open · S save

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

Shuze Daniel Liu, Claire Chen, Jiuqi Wang, David Simchi-Levi+1

Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Purdue University, West Lafayette, Indiana, USA · California Institute of Technology, Pasadena, California, USA+2

Negotiation Multi-Agent Large Language Model Systems Large Language Model Agents Matching Markets

Open paper AskPDF arXiv

Autonomous large language model (LLM) agents operating in multi-product markets must make sequential decisions under information asymmetry and resource constraints. We develop a machine learning approach for training such agents to act effectively as sellers in a multi-item bargaining environment, where a seller concurrently negotiates a catalog of substitutable assets across a pool of independent buyers. Buyers hold private, heterogeneous valuations across products, and each can purchase at most one item. Facing limits on total communication turns, the seller must dynamically match buyers with the most profitable products considering their private valuations, while strategically allocating its limited interaction budget toward combinations of greater potential value. We formalize this problem as a Partially Observable Markov Decision Process using a structured, four-part message protocol that maps natural language into a parsable and regulated decision space. Using this formalization, we design a post-training method using Reinforcement Learning from Verifiable Rewards (RLVR). To evaluate this framework, we construct a multidimensional metric suite that quantifies constraint adherence, seller surplus extraction, and allocation quality. Our trained seller agent learns to match limited inventory to buyers more effectively, matching or outperforming trillion-parameter frontier models in both seller surplus extraction and buyer-product allocation quality. Finally, these learned strategies generalize robustly to unseen market structures, correlated valuation distributions, and price ranges not encountered during training.

Abstract

Explore similar work

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations

Counterparty Modeling is Not Strategy: The Limits of LLM Negotiators