Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

Shuze Daniel Liu; Claire Chen; Jiuqi Wang; David Simchi-Levi; Thorsten Joachims

cs.AISep 27, 2026

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

Authors: Shuze Daniel Liu, Claire Chen, Jiuqi Wang, David Simchi-Levi, Thorsten Joachims

Organizations: Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Purdue University, West Lafayette, Indiana, USA · California Institute of Technology, Pasadena, California, USA · University of Virginia, Charlottesville, Virginia, USA · Cornell University, Ithaca, New York, USA

Abstract

Autonomous large language model (LLM) agents operating in multi-product markets must make sequential decisions under information asymmetry and resource constraints. We develop a machine learning approach for training such agents to act effectively as sellers in a multi-item bargaining environment, where a seller concurrently negotiates a catalog of substitutable assets across a pool of independent buyers. Buyers hold private, heterogeneous valuations across products, and each can purchase at most one item. Facing limits on total communication turns, the seller must dynamically match buyers with the most profitable products considering their private valuations, while strategically allocating its limited interaction budget toward combinations of greater potential value. We formalize this problem as a Partially Observable Markov Decision Process using a structured, four-part message protocol that maps natural language into a parsable and regulated decision space. Using this formalization, we design a post-training method using Reinforcement Learning from Verifiable Rewards (RLVR). To evaluate this framework, we construct a multidimensional metric suite that quantifies constraint adherence, seller surplus extraction, and allocation quality. Our trained seller agent learns to match limited inventory to buyers more effectively, matching or outperforming trillion-parameter frontier models in both seller surplus extraction and buyer-product allocation quality. Finally, these learned strategies generalize robustly to unseen market structures, correlated valuation distributions, and price ranges not encountered during training.

Figures & tables

Figure 1 . Structural design of the bargaining environment for substitutable product portfolios. The seller agent manages inventory across independent, sealed buyer channels subject to global and local communication constraints. A schematic of the market: one seller at the centre holds a catalog of substitutable items and private per-item costs, connected by separate sealed channels to several buyers, each holding private per-item valuations. Labels on the channels mark the per-buyer turn limit

K

, and a label on the seller marks the global budget

T

, with

T

smaller than the number of buyers times

K

Panel of Figure 2 — Figure 2 . First two seller turns of a trained-seller episode on a three-item video-recorder catalog . The table above the transcript gives the episode’s catalog (list price, seller cost) and the hidden per-item buyer valuations (bold marks each item’s top valuation). All three buyers open on the cheapest item with bids below even that item’s cost; the seller rejects Buyer 1’s opening bid, quotes near-list prices on the two high-margin items, and closes Item 1 one turn later at $140. Talk and Action lines are verbatim; buyer Thought fields are omitted, and the seller Thought is abridged. The full transcript and the corresponding untrained-base-model transcript appear in Appendix H . A table of the three-item catalog with list prices, seller costs and the hidden buyer valuation matrix, above a transcript of the first two seller turns. Each turn is shown as a labelled message box with the seller's Thought, Target, Talk and Action fields, alternating with the addressed buyer's reply.

Figure 3 . Composition of the substitutable-product dataset. The outer ring shows the

13

Amazon broad categories (

100

clusters each); the inner ring groups them into five themes; the center reports the

1{,}300

-cluster total. A two-level ring chart. The outer ring is divided into 13 equal segments, one per Amazon broad category, each holding 100 clusters; the inner ring groups those segments into five themes of unequal size; the centre reports the 1,300-cluster total.

Figure 4 . Schematic Architecture of the Multidimensional Evaluation Framework. The evaluation pipeline computes performance metrics across five phases: (1) a Sampled Episode sets global and local turn constraints; (2) the Seller Agent negotiates with several buyer agents over the public catalog under asymmetric information; (3) a Realized Outcome captures the negotiation transcript, deal prices, and allocation; (4) a Centralized Assignment Optimization determines the maximum possible surplus from the private buyer valuations; and (5) the Metric Computation engine overlays realized results against this optimum to measure Episode Outcome , Constraint Adherence , Surplus Extraction , and Allocation Quality . A pipeline diagram in five stages: a sampled episode fixing the global and per-buyer turn limits; the seller agent negotiating with several buyer agents over the public catalog; the resulting transcripts and closed deals; an oracle that computes the surplus-maximizing assignment; and a final block of metrics grouped into episode outcome, constraint adherence, surplus extraction, and allocation quality.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.407}$	$\mathbf{+0.425}$	$64.7\%$	$\mathbf{0.687}$	$\mathbf{0.844}$	$1.3\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.329$	$+0.184$	$66.2\%$	$0.544$	$0.554$	$43.3\%$
Kimi-K2.6-thinking	1T	$\underline{+0.379}$	$+0.379$	$73.0\%$	$0.566$	$0.559$	$\mathbf{0.0\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.369$	$\underline{+0.392}$	$73.4\%$	$0.612$	$0.592$	$1.7\%$
DeepSeek-V4-Pro-nothink	1.6T	$+0.361$	$+0.378$	$\underline{76.2\%}$	$0.596$	$0.640$	$1.3\%$
DeepSeek-V4-Flash-thinking	284B	$+0.349$	$+0.366$	$71.0\%$	$0.568$	$0.566$	$1.3\%$

Table 1. In-distribution held-out evaluation (

3

buyers,

3

items,

T=7

). The trained checkpoint ( Ours ) and the untrained base above the rule; frontier baselines below. Baselines sorted by Reward (descending); best per column in bold, second-best underlined ( Instruction Violation Rate is lower-is-better). Metric definitions in Section 6 . Standard errors in Table 11 .

OOD Environment	Buyers $m$	Items $n$	Turns $T$	Buyer Model	Valuation Structure	Product Category	Core Stress Test
In-Distribution Reference
	3	3	7	Fixed iter- 60	Baseline ( 2 )	12 Train Categories	Training configuration
Scale Generalization
1. Expanded Standard	5	5	12	iter- {50, 60, 70}	Baseline ( 2 )	12 Train Categories	Scale & horizon expansion
2. Expanded Musical	5	5	12	iter- {50, 60, 70}	Baseline ( 2 )	Held-out Musical Instruments	Scale & unseen price range
Asymmetric Market Structures

Table 2. Experimental evaluation matrix. The top row establishes the baseline in-distribution reference configuration. Environments 1–6 represent out-of-distribution (OOD) configurations; all of these environments share the dynamic buyer checkpoint pool

\{\text{iter-}50,\text{iter-}60,\text{iter-}70\}

, while each individual environment introduces a unique structural configuration as specified in the table. For the correlated settings (Environments 5 and 6), the respective valuation multipliers follow

\mu_{j},\gamma_{i}\sim\mathcal{U}[0.5,2]

Figure 11 . Out-of-distribution generalization — Reward. Mean episode reward

\mathcal{R}

of the RL-trained seller against the entire frontier-baseline field, one row per market distribution (the in-distribution reference plus six out-of-distribution shifts). Grey band: minimum-to-maximum range of the eight baselines; grey diamond: best baseline (named); blue circle: trained seller; hollow circle: untrained base model. The right column reports the improvement over the best baseline. Error bars denote the standard error. A horizontal forest plot with seven rows: an in-distribution reference row, separated by a rule, above six out-of-distribution rows (Expanded Standard, Expanded Musical, Buyer Heavy, Item Heavy, Item Correlation, Buyer Correlation). Each row plots mean episode reward on a shared axis from

-0.4

0.6

: a hollow circle for the untrained base model, a grey band spanning the minimum-to-maximum range of the eight baselines, a grey diamond for the best baseline, and a filled blue circle for the trained seller, all with standard-error bars. The untrained model is negative in every row. The trained seller lies to the right of the best baseline in all seven rows, and the best baseline is Kimi-K2.6-thinking throughout. The gap is narrowest in the in-distribution row and wider in each out-of-distribution row; the right-hand column lists it as

+0.028

in-distribution and between

+0.089

and

+0.135

across the six shifts.

Figure 12 . Out-of-distribution generalization — Seller Surplus Extraction Ratio. The seller’s share of the catalog-wide list-price margin captured (aggregated over valid episodes), with the same rows and marks as Figure 11 . Error bars denote the standard error. A horizontal forest plot with the same seven rows and the same marks as the reward figure, plotting the Seller Surplus Extraction Ratio on an axis from

0.1

0.7

. The untrained base model sits between roughly

0.18

and

0.35

, the baseline band spans the middle of the axis, and the trained seller lies to the right of the best baseline in every row. DeepSeek-V4-Pro-thinking is the best baseline in all seven rows. The right-hand column reports the improvement as

+0.033

in-distribution and between

+0.054

and

+0.099

across the six out-of-distribution shifts, so the margin is again smallest in-distribution.

Figure 13 . Out-of-distribution generalization — Optimal-Pair Offer Coverage Rate. How often the seller proposes the oracle-optimal buyer–item pairings (aggregated over valid episodes), with the same rows and marks as Figure 11 . Error bars denote the standard error. A horizontal forest plot with the same seven rows and the same marks as the reward figure, plotting the Optimal-Pair Offer Coverage Rate on an axis from

0.3

0.9

. The trained seller lies well to the right of the best baseline in every row, and the separation is visibly larger than in the reward and surplus figures. Unlike those two, the best baseline changes across rows: Kimi-K2.6-nothink in-distribution, DeepSeek-V4-Pro-thinking, DeepSeek-V4-Pro-nothink, and GPT-5.4-high-reasoning in three rows. The right-hand column reports improvements between

+0.174

and

+0.255

, with

+0.189

in-distribution.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Name	Description
Episode parameters and indices (§ 3 )
$n$ , $m$	Item and buyer counts	Number of items and buyer slots per episode.
$j$ , $i$	Item and buyer indices	Item index $j\in\{1,\ldots,n\}$ ; buyer index $i\in\{1,\ldots,m\}$ .
$t$	Turn index	Seller turn index $t$ .
$T$ , $K$	Communication budget and turn limit	Global communication budget ( $T=7$ in training) and per-buyer turn limit ( $K=3$ ); $T<mK$ .
Prices, costs, and valuations (§ 3 , § 5.2 , § 9 )

Appendix

Table 3 . Notation used throughout the paper, grouped by the part of the framework each symbol belongs to.

Panel of Figure 14 — Figure 14 . In-distribution performance benchmarking ( $3$ buyers, $3$ items, $T=7$ ). Per-metric seller rankings on byte-identical in-distribution held-out instances. The trained model ( Ours ) and its untrained base appear above the rule, the frontier baselines below. Error bars show the standard error. Values match Table 1 . Four horizontal bar charts, one per metric: Reward, Seller Surplus Extraction Ratio, Optimal-Pair Offer Coverage Rate and Deal on Top Buyer Rate. Each chart is a single series with one bar per model, sorted by value. In every panel the trained seller and its untrained base appear above a rule and the eight frontier baselines below it. The trained seller has the longest bar on all four metrics, while the untrained base is the shortest on Reward.

Role	Base model	Provenance / checkpoint
Seller Starting Checkpoint	Qwen/Qwen3-30B-A3B- Instruct-2507 ( Qwen Team, 2025 )	Original checkpoint from ( Qwen Team, 2025 ) ; fine-tuned as the seller.
Buyers in Training	Qwen/Qwen3-30B-A3B- Instruct-2507	Frozen buyer checkpoint iteration $60$ (iter- $60$ ), trained via RLVR on a bilateral negotiation task ( Liu et al., 2026a ) .
Buyers in Evaluation	Qwen/Qwen3-30B-A3B- Instruct-2507	In-distribution evaluation: fixed iter- $60$ . Out-of-distribution evaluation: per-buyer uniform over the pool $\{\text{iter-}50,\ \text{iter-}60,\ \text{iter-}70\}$ .

Appendix

Table 4. Models used by the multi-item-seller pipeline. The seller is the trained agent; the buyers are frozen checkpoints from Liu et al. (2026a) .

Model Name	Parameter Count	Reference
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	This Work
Qwen3-30B-A3B-Instruct-2507-untrained	30B	Qwen Team (2025)
GPT-5.4-high-reasoning	closed-source	Singh et al. (2025)
GPT-5.4-mini-high-reasoning	closed-source	Singh et al. (2025)
DeepSeek-V4-Pro-thinking / nothink	1.6T	DeepSeek-AI (2026)
DeepSeek-V4-Flash-thinking / nothink	284B	DeepSeek-AI (2026)

Appendix

Table 5. Seller models evaluated in the held-out benchmark, with total parameter counts. Three open-weight models are run in both a thinking and a no-thinking ( nothink ) mode, giving a

10

-model panel. DeepSeek-V4-Pro is a mixture-of-experts model with

1.6

T total /

49

B active parameters; DeepSeek-V4-Flash has

284

B total /

13

B active parameters. The buyer is held fixed across the panel (Table 4 ).

Setting	Value
loss function	cispo
CISPO clip low / high	$0.8\ /\ 6.0$
Learning rate	$3\times 10^{-5}$
KL penalty (kl coef)	$0$
batch size	$64$
group size	$8$

Appendix

Table 6. Optimization hyperparameters for the RL training run.

Setting	Value
num buyers $m$	$3$
num items $n$	$3$
total turns $T$	$7$
per-buyer turn limit $K$	$3$
valuation ratio distribution	$\mathcal{U}[0.40,\,1.15]$
first-offer-noise probability	$0.5$

Appendix

Table 7. Environment hyperparameters for training. Each buyer valuation entry is

\mathcal{B}_{i,j}=r_{i,j}\,P_{\text{list},j}

with

r_{i,j}\sim\mathcal{U}[0.40,1.15]

drawn i.i.d. across all

(i,j)

pairs. First-offer noise is applied independently to each buyer with probability

0.5

: at episode start, a buyer selected for first-offer noise has its opening offer on every chosen item resampled into

[0.25,0.30]\cdot P_{\text{list},j}

, decoupling the opening from the buyer’s private valuation; otherwise the buyer generates its opening offer freely subject to its private valuation constraint.

Setting	Value
seller model	All models from Table 5
buyer model	Qwen3-30B-A3B-Instruct-2507 iter- $60$ (Table 4 )
Test split size	$240$ held-out clusters
seller temperature (eval)	$0.7$
buyer temperature (eval)	$0.7$

Appendix

Table 8. Evaluation hyperparameters for the held-out test split. The

240

test clusters are the

20\%

cluster-level hold-out from the

12

in-distribution categories ( Section 4 ); product contexts are unseen during training.

Environment	Buyers $m$	Items $n$	Turns $T$	Valuation	Catalog
In-distribution (reference)	$3$	$3$	$7$	uniform	$12$ categories
Env. 1: Expanded Standard	$5$	$5$	$12$	uniform	$12$ categories
Env. 2: Expanded Musical	$5$	$5$	$12$	uniform	Musical Instruments
Env. 3: Buyer Heavy	$5$	$3$	$12$	uniform	$12$ categories
Env. 4: Item Heavy	$3$	$5$	$7$	uniform	$12$ categories
Env. 5: Item Correlation	$5$	$5$	$12$	uniform $\times\,\mu_{j}$	$12$ categories

Appendix

Table 9. Per-episode environment knobs for the in-distribution reference and the out-of-distribution evaluation environments. “uniform” denotes the in-distribution valuation

\mathcal{B}_{i,j}=r_{i,j}\,P_{\text{list},j}

with

r_{i,j}\sim\mathcal{U}[0.40,1.15]

. Env. 5 (Item Correlation) draws

\mu_{j}\sim\mathcal{U}[0.5,2]

once per item with

\gamma_{i}=1

, while Env. 6 (Buyer Correlation) draws

\gamma_{i}\sim\mathcal{U}[0.5,2]

once per buyer with

\mu_{j}=1

. The in-distribution reference uses the fixed iter-

60

buyer; every OOD environment samples each buyer slot uniformly from the pool

\{\text{iter-}50,\text{iter-}60,\text{iter-}70\}

. First-offer noise (Table 7 ) is applied in every environment.

Broad Category	Mean Distinct Brands	Median List Price	Median Cluster Min Price	Median Cluster Max Price	Median Price Spread
Appliances	4.3	$109.98	$79.99	$159.99	1.71 $\times$
Automotive	4.3	$45.86	$31.49	$59.99	2.00 $\times$
Baby	4.5	$34.99	$28.24	$48.97	1.61 $\times$
Beauty & Personal Care	4.5	$29.99	$22.49	$43.98	1.79 $\times$
Clothing, Shoes & Jewelry	4.4	$36.49	$29.95	$49.99	1.73 $\times$
Electronics	3.9	$139.99	$99.79	$199.99	1.95 $\times$

Appendix

Table 10 . Per-category statistics of the substitutable-product dataset. Every category contributes

100

clusters of

5

substitutable products (

500

products per category;

1{,}300

clusters and

6{,}500

products in total). Brand counts are means over a category’s

100

clusters; prices are medians; the within-cluster price spread

s=P_{\max}/P_{\min}

is computed over each cluster’s

5

list prices and reported as the per-category median.

Figure 15 . List price distribution of the dataset’s

6{,}500

products on a log scale: Musical Instruments (

N{=}500

, red) versus the

12

remaining categories pooled (

N{=}6{,}000

, blue). Musical Instruments concentrates in a visibly higher price range, motivating its selection as the held-out out-of-distribution category. Two overlaid filled density curves of product list prices on a logarithmic price axis. The blue curve is the

12

pooled in-distribution categories (

N=6{,}000

); the red curve is Musical Instruments (

N=500

). The Musical Instruments density is shifted toward substantially higher prices, with most of its mass to the right of the pooled curve.

Figure 16 . Distributions over the

1{,}300

clusters of (left) the within-cluster price spread

s=P_{\max}/P_{\min}

, with the

24

clusters whose spread exceeds

6\times

folded into the last bin, and (right) the number of distinct brands among each cluster’s

5

products. Two bar charts over the

1{,}300

clusters. Left, the within-cluster price spread

P_{\max}/P_{\min}

is right-skewed: most clusters fall between roughly

1.5\times

and

2.4\times

, with a sparse tail out to larger spreads and the

24

clusters above

6\times

folded into the last bin. Right, the number of distinct brands among each cluster's five products piles up at the top of its range rather than trailing off: about

59\%

of clusters contain five distinct brands and roughly

24\%

contain four, while clusters with two or three brands are uncommon.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.407\pm 0.018}$	$\mathbf{+0.425\pm 0.015}$	$64.7\%\pm 1.6\%$	$\mathbf{0.687\pm 0.022}$	$\mathbf{0.844\pm 0.014}$	$1.3\%\pm 0.7\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.329\pm 0.039$	$+0.184\pm 0.015$	$66.2\%\pm 2.4\%$	$0.544\pm 0.030$	$0.554\pm 0.026$	$43.3\%\pm 3.2\%$
Kimi-K2.6-thinking	1T	$\underline{+0.379\pm 0.013}$	$+0.379\pm 0.013$	$73.0\%\pm 1.5\%$	$0.566\pm 0.022$	$0.559\pm 0.020$	$\mathbf{0.0\%\pm 0.0\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.369\pm 0.018$	$\underline{+0.392\pm 0.015}$	$73.4\%\pm 1.6\%$	$0.612\pm 0.021$	$0.592\pm 0.020$	$1.7\%\pm 0.8\%$
DeepSeek-V4-Pro-nothink	1.6T	$+0.361\pm 0.017$	$+0.378\pm 0.014$	$\underline{76.2\%\pm 1.6\%}$	$0.596\pm 0.021$	$0.640\pm 0.019$	$1.3\%\pm 0.7\%$
DeepSeek-V4-Flash-thinking	284B	$+0.349\pm 0.017$	$+0.366\pm 0.014$	$71.0\%\pm 1.7\%$	$0.568\pm 0.022$	$0.566\pm 0.020$	$1.3\%\pm 0.7\%$

Appendix

Table 11. In-distribution held-out evaluation, full results (

3

buyers,

3

items,

T=7

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.444}$	$\mathbf{+0.493}$	$70.8\%$	$\mathbf{0.594}$	$\mathbf{0.699}$	$3.3\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.187$	$+0.240$	$74.1\%$	$0.384$	$0.378$	$34.5\%$
Kimi-K2.6-thinking	1T	$\underline{+0.322}$	$+0.345$	$75.9\%$	$0.398$	$0.402$	$\underline{1.7\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.308$	$+0.347$	$81.5\%$	$0.417$	$0.426$	$2.9\%$
DeepSeek-V4-Flash-thinking	284B	$+0.268$	$+0.353$	$77.1\%$	$0.388$	$0.398$	$6.3\%$
GPT-5.4-high-reasoning	closed-source	$+0.265$	$+0.265$	$\mathbf{82.8\%}$	$0.363$	$0.439$	$\mathbf{0.0\%}$

Appendix

Table 12. Out-of-distribution Env. 1, Expanded Standard (

5

buyers,

5

items,

T=12

, uniform valuation) with the diverse buyer pool. Baselines sorted by Reward (descending); best per column in bold, second-best underlined ( Instruction Violation Rate is lower-is-better). Metric definitions in Section 6 . Standard errors in Table 13 .

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.444\pm 0.022}$	$\mathbf{+0.493\pm 0.013}$	$70.8\%\pm 1.3\%$	$\mathbf{0.594\pm 0.017}$	$\mathbf{0.699\pm 0.014}$	$3.3\%\pm 1.2\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.187\pm 0.039$	$+0.240\pm 0.012$	$74.1\%\pm 1.7\%$	$0.384\pm 0.020$	$0.378\pm 0.018$	$34.5\%\pm 3.1\%$
Kimi-K2.6-thinking	1T	$\underline{+0.322\pm 0.016}$	$+0.345\pm 0.012$	$75.9\%\pm 1.4\%$	$0.398\pm 0.016$	$0.402\pm 0.015$	$\underline{1.7\%\pm 0.8\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.308\pm 0.018$	$+0.347\pm 0.011$	$81.5\%\pm 1.1\%$	$0.417\pm 0.016$	$0.426\pm 0.015$	$2.9\%\pm 1.1\%$
DeepSeek-V4-Flash-thinking	284B	$+0.268\pm 0.024$	$+0.353\pm 0.011$	$77.1\%\pm 1.3\%$	$0.388\pm 0.017$	$0.398\pm 0.015$	$6.3\%\pm 1.6\%$
GPT-5.4-high-reasoning	closed-source	$+0.265\pm 0.010$	$+0.265\pm 0.010$	$\mathbf{82.8\%\pm 1.1\%}$	$0.363\pm 0.016$	$0.439\pm 0.015$	$\mathbf{0.0\%\pm 0.0\%}$

Appendix

Table 13. Out-of-distribution (

5

buyers,

5

items,

T=12

, uniform valuation), full results.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.438}$	$\mathbf{+0.483}$	$69.5\%$	$\mathbf{0.550}$	$\mathbf{0.699}$	$3.0\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.129$	$+0.253$	$75.4\%$	$0.349$	$0.329$	$30.5\%$
Kimi-K2.6-thinking	1T	$\underline{+0.303}$	$+0.337$	$74.8\%$	$0.418$	$0.406$	$\underline{2.5\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.281$	$+0.371$	$\mathbf{81.9\%}$	$0.433$	$\underline{0.459}$	$6.5\%$
GPT-5.4-high-reasoning	closed-source	$+0.260$	$+0.260$	$\underline{81.5\%}$	$0.378$	$0.458$	$\mathbf{0.0\%}$
DeepSeek-V4-Flash-nothink	284B	$+0.238$	$+0.310$	$78.2\%$	$0.367$	$0.419$	$5.5\%$

Appendix

Table 14. Out-of-distribution Env. 2, Expanded Musical (held-out Musical Instruments category) (

5

buyers,

5

items,

T=12

, uniform valuation). Baselines sorted by Reward (descending); best per column in bold, second-best underlined ( Instruction Violation Rate is lower-is-better). Metric definitions in Section 6 . Standard errors in Table 15 .

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.438\pm 0.023}$	$\mathbf{+0.483\pm 0.015}$	$69.5\%\pm 1.5\%$	$\mathbf{0.550\pm 0.019}$	$\mathbf{0.699\pm 0.015}$	$3.0\%\pm 1.2\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.129\pm 0.042$	$+0.253\pm 0.013$	$75.4\%\pm 1.7\%$	$0.349\pm 0.021$	$0.329\pm 0.018$	$30.5\%\pm 3.3\%$
Kimi-K2.6-thinking	1T	$\underline{+0.303\pm 0.020}$	$+0.337\pm 0.014$	$74.8\%\pm 1.6\%$	$0.418\pm 0.018$	$0.406\pm 0.016$	$\underline{2.5\%\pm 1.1\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.281\pm 0.027$	$+0.371\pm 0.013$	$\mathbf{81.9\%\pm 1.2\%}$	$0.433\pm 0.018$	$\underline{0.459\pm 0.017}$	$6.5\%\pm 1.7\%$
GPT-5.4-high-reasoning	closed-source	$+0.260\pm 0.012$	$+0.260\pm 0.012$	$\underline{81.5\%\pm 1.2\%}$	$0.378\pm 0.017$	$0.458\pm 0.017$	$\mathbf{0.0\%\pm 0.0\%}$
DeepSeek-V4-Flash-nothink	284B	$+0.238\pm 0.025$	$+0.310\pm 0.013$	$78.2\%\pm 1.3\%$	$0.367\pm 0.018$	$0.419\pm 0.017$	$5.5\%\pm 1.6\%$

Appendix

Table 15. Out-of-distribution on held-out Musical Instruments (

5

5

T=12

), full results.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.559}$	$\mathbf{+0.586}$	$80.6\%$	$\mathbf{0.534}$	$\mathbf{0.727}$	$1.7\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.065$	$+0.313$	$89.5\%$	$0.318$	$0.450$	$28.7\%$
Kimi-K2.6-thinking	1T	$\underline{+0.443}$	$+0.449$	$90.7\%$	$0.403$	$0.508$	$\underline{0.4\%}$
GPT-5.4-high-reasoning	closed-source	$+0.400$	$+0.400$	$\mathbf{95.0\%}$	$0.412$	$\underline{0.553}$	$\mathbf{0.0\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.367$	$\underline{+0.505}$	$93.3\%$	$0.420$	$0.534$	$9.2\%$
DeepSeek-V4-Pro-nothink	1.6T	$+0.337$	$+0.414$	$\underline{94.6\%}$	$0.394$	$0.504$	$5.4\%$

Appendix

Table 16. Out-of-distribution Env. 3, Buyer Heavy (

5

buyers,

3

items,

T=12

) , uniform valuation. Baselines sorted by Reward (descending); best per column in bold, second-best underlined ( Instruction Violation Rate is lower-is-better). Metric definitions in Section 6 . Standard errors in Table 17 .

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.559\pm 0.022}$	$\mathbf{+0.586\pm 0.018}$	$80.6\%\pm 1.5\%$	$\mathbf{0.534\pm 0.021}$	$\mathbf{0.727\pm 0.017}$	$1.7\%\pm 0.8\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.065\pm 0.040$	$+0.313\pm 0.015$	$89.5\%\pm 1.6\%$	$0.318\pm 0.022$	$0.450\pm 0.022$	$28.7\%\pm 2.9\%$
Kimi-K2.6-thinking	1T	$\underline{+0.443\pm 0.015}$	$+0.449\pm 0.014$	$90.7\%\pm 1.3\%$	$0.403\pm 0.019$	$0.508\pm 0.019$	$\underline{0.4\%\pm 0.4\%}$
GPT-5.4-high-reasoning	closed-source	$+0.400\pm 0.014$	$+0.400\pm 0.014$	$\mathbf{95.0\%\pm 0.9\%}$	$0.412\pm 0.019$	$\underline{0.553\pm 0.019}$	$\mathbf{0.0\%\pm 0.0\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.367\pm 0.031$	$\underline{+0.505\pm 0.014}$	$93.3\%\pm 1.0\%$	$0.420\pm 0.020$	$0.534\pm 0.020$	$9.2\%\pm 1.9\%$
DeepSeek-V4-Pro-nothink	1.6T	$+0.337\pm 0.025$	$+0.414\pm 0.015$	$\underline{94.6\%\pm 1.0\%}$	$0.394\pm 0.019$	$0.504\pm 0.019$	$5.4\%\pm 1.5\%$

Appendix

Table 17. Out-of-distribution (

5

buyers,

3

items,

T=12

), full results.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.326}$	$\mathbf{+0.343}$	$46.4\%$	$\mathbf{0.728}$	$\mathbf{0.817}$	$1.3\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.089$	$+0.189$	$46.8\%$	$0.568$	$0.377$	$23.3\%$
Kimi-K2.6-thinking	1T	$\underline{+0.238}$	$+0.248$	$49.6\%$	$0.573$	$0.465$	$\underline{0.8\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.213$	$+0.239$	$\mathbf{51.7\%}$	$0.580$	$0.520$	$2.1\%$
Kimi-K2.6-nothink	1T	$+0.211$	$+0.253$	$\underline{51.4\%}$	$0.596$	$0.527$	$3.3\%$
GPT-5.4-high-reasoning	closed-source	$+0.195$	$+0.195$	$50.4\%$	$0.513$	$0.503$	$\mathbf{0.0\%}$

Appendix

Table 18. Out-of-distribution Env. 4, Item Heavy (

3

buyers,

5

items,

T=7

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.326\pm 0.015}$	$\mathbf{+0.343\pm 0.012}$	$46.4\%\pm 1.0\%$	$\mathbf{0.728\pm 0.019}$	$\mathbf{0.817\pm 0.015}$	$1.3\%\pm 0.7\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.089\pm 0.034$	$+0.189\pm 0.011$	$46.8\%\pm 1.2\%$	$0.568\pm 0.024$	$0.377\pm 0.021$	$23.3\%\pm 2.7\%$
Kimi-K2.6-thinking	1T	$\underline{+0.238\pm 0.012}$	$+0.248\pm 0.009$	$49.6\%\pm 0.9\%$	$0.573\pm 0.020$	$0.465\pm 0.019$	$\underline{0.8\%\pm 0.6\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.213\pm 0.015$	$+0.239\pm 0.010$	$\mathbf{51.7\%\pm 0.8\%}$	$0.580\pm 0.020$	$0.520\pm 0.019$	$2.1\%\pm 0.9\%$
Kimi-K2.6-nothink	1T	$+0.211\pm 0.017$	$+0.253\pm 0.009$	$\underline{51.4\%\pm 0.9\%}$	$0.596\pm 0.020$	$0.527\pm 0.019$	$3.3\%\pm 1.2\%$
GPT-5.4-high-reasoning	closed-source	$+0.195\pm 0.009$	$+0.195\pm 0.009$	$50.4\%\pm 0.9\%$	$0.513\pm 0.020$	$0.503\pm 0.019$	$\mathbf{0.0\%\pm 0.0\%}$

Appendix

Table 19. Out-of-distribution (

3

buyers,

5

items,

T=7

), full results.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.452}$	$\mathbf{+0.497}$	$69.9\%$	$\mathbf{0.366}$	$\mathbf{0.737}$	$3.0\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.215$	$+0.260$	$74.4\%$	$0.276$	$0.446$	$37.7\%$
Kimi-K2.6-thinking	1T	$\underline{+0.317}$	$+0.334$	$73.6\%$	$0.289$	$0.481$	$\underline{1.3\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.293$	$+0.337$	$77.8\%$	$\underline{0.313}$	$0.515$	$3.3\%$
GPT-5.4-high-reasoning	closed-source	$+0.272$	$+0.278$	$\mathbf{79.8\%}$	$0.311$	$\underline{0.547}$	$\mathbf{0.4\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.251$	$\underline{+0.405}$	$75.9\%$	$0.288$	$0.519$	$10.9\%$

Appendix

Table 20. Out-of-distribution Env. 5, Item Correlation: per-item valuation multiplier

\mu_{j}\sim\mathcal{U}[0.5,2]

(

5

buyers,

5

items,

T=12

). Baselines sorted by Reward (descending); best per column in bold, second-best underlined ( Instruction Violation Rate is lower-is-better). Metric definitions in Section 6 . Standard errors in Table 21 .

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.452\pm 0.022}$	$\mathbf{+0.497\pm 0.015}$	$69.9\%\pm 1.3\%$	$\mathbf{0.366\pm 0.017}$	$\mathbf{0.737\pm 0.014}$	$3.0\%\pm 1.1\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.215\pm 0.040$	$+0.260\pm 0.014$	$74.4\%\pm 1.8\%$	$0.276\pm 0.019$	$0.446\pm 0.019$	$37.7\%\pm 3.1\%$
Kimi-K2.6-thinking	1T	$\underline{+0.317\pm 0.015}$	$+0.334\pm 0.011$	$73.6\%\pm 1.5\%$	$0.289\pm 0.016$	$0.481\pm 0.015$	$\underline{1.3\%\pm 0.7\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.293\pm 0.019$	$+0.337\pm 0.011$	$77.8\%\pm 1.2\%$	$\underline{0.313\pm 0.015}$	$0.515\pm 0.015$	$3.3\%\pm 1.2\%$
GPT-5.4-high-reasoning	closed-source	$+0.272\pm 0.012$	$+0.278\pm 0.011$	$\mathbf{79.8\%\pm 1.3\%}$	$0.311\pm 0.015$	$\underline{0.547\pm 0.016}$	$\mathbf{0.4\%\pm 0.4\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.251\pm 0.030$	$\underline{+0.405\pm 0.012}$	$75.9\%\pm 1.3\%$	$0.288\pm 0.016$	$0.519\pm 0.016$	$10.9\%\pm 2.0\%$

Appendix

Table 21. Out-of-distribution with per-item valuation multiplier

\mu_{j}\sim\mathcal{U}[0.5,2]

, full results.

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.449}$	$\mathbf{+0.499}$	$70.7\%$	$\mathbf{0.372}$	$\mathbf{0.739}$	$3.3\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.150$	$+0.299$	$74.1\%$	$0.289$	$0.409$	$34.6\%$
Kimi-K2.6-thinking	1T	$\underline{+0.349}$	$+0.372$	$74.5\%$	$\underline{0.292}$	$0.500$	$\underline{1.7\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.304$	$+0.361$	$\underline{80.0\%}$	$0.263$	$0.482$	$4.2\%$
GPT-5.4-high-reasoning	closed-source	$+0.297$	$+0.297$	$79.5\%$	$0.283$	$\underline{0.536}$	$\mathbf{0.0\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.293$	$\underline{+0.444}$	$79.8\%$	$0.274$	$0.512$	$10.5\%$

Appendix

Table 22. Out-of-distribution Env. 6, Buyer Correlation: per-buyer generosity multiplier

\gamma_{i}\sim\mathcal{U}[0.5,2]

(

5

buyers,

5

items,

T=12

Model	Params	Reward $\uparrow$	Seller Surplus Extraction Ratio	Item Deal Rate	Deal on Top Buyer Rate	Optimal-Pair Offer Coverage Rate	Instruction Violation Rate
Qwen3-30B-A3B-Instruct-2507-trained ( Ours )	30B	$\mathbf{+0.449\pm 0.022}$	$\mathbf{+0.499\pm 0.015}$	$70.7\%\pm 1.4\%$	$\mathbf{0.372\pm 0.017}$	$\mathbf{0.739\pm 0.014}$	$3.3\%\pm 1.2\%$
Qwen3-30B-A3B-Instruct-2507-untrained	30B	$-0.150\pm 0.041$	$+0.299\pm 0.015$	$74.1\%\pm 1.6\%$	$0.289\pm 0.019$	$0.409\pm 0.019$	$34.6\%\pm 3.1\%$
Kimi-K2.6-thinking	1T	$\underline{+0.349\pm 0.017}$	$+0.372\pm 0.012$	$74.5\%\pm 1.4\%$	$\underline{0.292\pm 0.015}$	$0.500\pm 0.016$	$\underline{1.7\%\pm 0.8\%}$
DeepSeek-V4-Pro-nothink	1.6T	$+0.304\pm 0.021$	$+0.361\pm 0.011$	$\underline{80.0\%\pm 1.2\%}$	$0.263\pm 0.015$	$0.482\pm 0.016$	$4.2\%\pm 1.3\%$
GPT-5.4-high-reasoning	closed-source	$+0.297\pm 0.012$	$+0.297\pm 0.012$	$79.5\%\pm 1.2\%$	$0.283\pm 0.015$	$\underline{0.536\pm 0.016}$	$\mathbf{0.0\%\pm 0.0\%}$
DeepSeek-V4-Pro-thinking	1.6T	$+0.293\pm 0.031$	$\underline{+0.444\pm 0.013}$	$79.8\%\pm 1.2\%$	$0.274\pm 0.015$	$0.512\pm 0.016$	$10.5\%\pm 2.0\%$

Appendix

Table 23. Out-of-distribution with per-buyer generosity multiplier

\gamma_{i}\sim\mathcal{U}[0.5,2]

, full results.

	Item 1 (ANNKE)	Item 2 (Lorex)	Item 3 (ZOSI)
List price $P_{\text{list},j}$	$169.99	$249.99	$55.99
Seller cost $\mathcal{C}_{j}$	$55.99	$100.00	$42.07
Untrained outcome ( Section H.2 )	Buyer 2 @ $100.00	unsold	Buyer 1 @ $45.66
Trained outcome ( Section H.3 )	Buyer 1 @ $140.00	Buyer 2 @ $240.00	Buyer 3 @ $43.99

Appendix

Table 24 . Catalog economics and realized allocations for the appendix instance (the hidden valuation matrix appears with each transcript, Section H.2 and Section H.3 ). Buyer 2 holds the top valuation on both high-value items but can close at most one deal, so the surplus-maximizing assignment routes Item 1 to Buyer 1, Item 2 to Buyer 2, and Item 3 to Buyer 3—exactly the allocation the trained seller realizes. The untrained seller instead spends Buyer 1 on the cheapest item and Buyer 2 on the mid item, after which no remaining buyer can afford the highest-margin Lorex.

Explore similar work

Jul 7, 2026cs.LG

Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations

Negotiation is a fundamental strategic interaction in management science, characterized by agents attempting to reach agreements while protecting private information, such as reservation costs and hidden valuations. A prevalent yet complex scenario involves a single seller negotiating concurrently with multiple buyers, each possessing heterogeneous, private budgets. In such settings, constrained by a limited number of communication turns, the seller must balance exploring the broader market to discover the highest valuation with concentrating sufficient turns on a single target buyer to secure the best possible outcome. Our analysis reveals a significant gap in standard Large Language Models (LLMs): while these models are linguistically proficient, they fail to act as effective economic decision-makers. Specifically, they exhibit a failure to explore the buyer pool, often fixating on the current highest bid rather than strategically investigating the market to discover latent high valuations. In this paper, we propose a specialized training recipe using Reinforcement Learning from Verifiable Rewards (RLVR). By anchoring the reward function to objective economic outcomes, the strategic balance between market discovery and surplus extraction emerges natively through the learning process. Our results demonstrate that the trained seller undergoes a multi-stage strategic evolution, learning to leverage price anchoring and strategic probing to identify more profitable counterparties. The agent extracts a substantially higher surplus than frontier models by both improving its persuasive bargaining skills and consistently closing deals with high-value buyers. Finally, we show that our seller strategies generalize robustly to unseen buyer negotiation styles and budget distributions.

Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao +2

Institute for Data, Systems, and Society, Massachusetts Institute of Technology · Mitch Daniels School of Business, Purdue University · The Division of Physics, Mathematics and Astronomy, California Institute of Technology +3

Negotiation Matching Markets Verifiable Rewards

Ask PDF

Oct 5, 2026cs.AI

Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers

LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate (

10^{-6}

) for every size, the gain of the RL model over its base rises from

+0.001

at 2.3B to

+0.078

at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by

+0.032

(2.3B) to

+0.081

(4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.

Pedro Tabacof, Sagar Joglekar

Fin AI Research

Negotiation

Ask PDF

May 11, 2026cs.AI

Strategic Exploitation in LLM Agent Markets: A Simulation Framework for E-Commerce Trust

Agent-based modeling (ABM) has long been used in economics to study human behavior, and large language model (LLM) agents now enable new forms of social and economic simulation. While prior work has discovered strategic deception by LLM agents in financial trading and auction markets, e-commerce remains underexplored despite its distinctive information asymmetry: sellers privately observe product quality, whereas buyers rely on advertised claims and reputation signals. We introduce TruthMarketTwin, a controlled simulation framework for studying LLM-agent behavior in e-commerce markets. The framework is one of the first to model bilateral trade under asymmetric information sharing, where agents make strategic listing, purchasing, rating, and recourse-related decisions to optimize seller profit and buyer utility. We find that LLM agents released into traditional markets autonomously exploit weaknesses in reputation-based governance, while warrant enforcement reduces deception and reshapes strategic reasoning. Our results position LLM-agent simulation as a tool for studying institution-governed autonomous markets.

Shijun Lei, Quang Nguyen, Swapneel S Mehta +7

Northwestern Polytechnical University · Boston University · Fudan University +3

Multi-Agent Simulations Personalized Large Language Model Agents User Simulation Reputation Deception AI Trustworthiness+1

Ask PDF