cs.GTApr 10, 2026

Training Language Models for Bilateral Trade with Private Information

Authors: Dirk Bergemann, Soheil Ghili, Xinyang Hu, Chuanhao Li, Zhuoran Yang

Organizations: Department of Economics, Yale University · Yale School of Management, Yale University · Department of Statistics and Data Science, Yale University · Department of Industrial Engineering, Tsinghua University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Bilateral bargaining under incomplete information provides a controlled testbed for evaluating large language model (LLM) agent capabilities. Bilateral trade demands individual rationality, strategic surplus maximization, and cooperation to realize gains from trade. We develop a structured bargaining environment where LLMs negotiate via tool calls within an event-driven simulator, separating binding offers from natural-language messages to enable automated evaluation. The environment serves two purposes: as a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning. In benchmark experiments, a round-robin tournament among five frontier models (15,000 negotiations) reveals that effective strategies implement price discrimination through sequential offers. Aggressive anchoring, calibrated concession, and temporal patience correlate with the highest surplus share and deal rate. Accommodating strategies that concede quickly disable price discrimination in the buyer role, yielding the lowest surplus capture and deal completion. Stronger models scale their behavior proportionally to item value, maintaining performance across price tiers; weaker models perform well only when wide zones of possible agreement offset suboptimal strategies. In training experiments, we fine-tune Qwen3 (8B, 14B) via supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) against a fixed frontier opponent. These stages optimize competing objectives: SFT approximately doubles surplus share but reduces deal rates, while RL recovers deal rates but erodes surplus gains, reflecting the reward structure. SFT also compresses surplus variation across price tiers, which generalizes to unseen opponents, suggesting that behavioral cloning instills proportional strategies rather than memorized price points.

Figures & tables

Figure 1: The diagram illustrates the bilateral bargaining space defined by the overlap between the seller’s reservation price ( ss ) and the buyer’s reservation price ( bb ). The ZOPA represents the total available economic surplus, calculated as b−sb-s . Within this range, any agreed settlement price ( pp ) facilitates allocative efficiency by distributing the surplus between the two parties.
Tool callDescription
make_offer(agent, price, side_offer)Propose a price for the item.
respond_to_offer(agent, response)Accept or reject opponent’s offer.
send_message(agent, content)Send a negotiation message to opponent.
search_price()Retrieve historical market reference prices.
quit_negotiation(agent)Terminal tool; end negotiation without deal.
wait_for_response(agent)Terminal tool; wait for opponent’s action.
Table 1: Structured action space: tool calls available to each agent.
Figure 2: Hierarchical structure of the agent system prompt. The prompt integrates private economic constraints ( s,bs,b ), strategic bargaining directives, and the “Thought-Action” protocol governing agent behavior within the simulation.
Figure 3: This is the system architecture of the bilateral bargaining simulation environment. Agents are initialized with reservation prices ( ss for the seller, bb for the buyer). The negotiation phase is governed by an Event Manager that enforces a protocol of alternating turns, a limit of max rounds, and a maximum of three structured tool calls per turn from the predefined Tool Space. Agent outputs have private ”Thoughts” (internal reasoning) and ”Code” (external tool-based actions). The cycle concludes in the Outcome phase, where the environment evaluates deal status and calculates resulting utilities ( UB=b−pU_{B}=b-p ; US=p−sU_{S}=p-s ) based on the final settlement price ( pp ).
MetricDescription
Violation rateFraction of negotiations where an agent accepts a price yielding negative utility ( p>bp>b for the buyer or p<sp<s for the seller). A no-deal outcome (utility =0=0 ) is not a violation. In GFT scenarios, violations arise when a deal is struck outside the ZOPA; in NGFT scenarios, every deal violates IR for at least one party, so the violation rate coincides with the NGFT deal rate (see Table 4 ). The two metrics are otherwise distinct: violation rate measures individual constraint adherence, while deal rate measures pair-level agreement realization.
Table 2: Dimension 1 metrics: individual rationality (IR).
MetricDescription
Average utility (all)Mean UBU_{B} and USU_{S} across all negotiations in a given regime (GFT or NGFT), including no-deal outcomes (utility =0=0 ).
Average utility (deals)Mean UBU_{B} and USU_{S} restricted to negotiations reaching agreement. Isolates agreement quality from agreement frequency.
Surplus shareShare of surplus captured in both-valid deals: (b−p)/(b−s)(b-p)/(b-s) for the buyer, (p−s)/(b−s)(p-s)/(b-s) for the seller ( Definition 5 ). Defined only for GFT scenarios.
Table 3: Dimension 2 metrics: strategic effectiveness (surplus maximization).
MetricDescription
Deal rateFraction of negotiations reaching agreement ( Definition 6 ). Higher is better in GFT; lower is better in NGFT.
Table 4: Dimension 3 metrics: allocative efficiency (gains-from-trade realization).
MetricDescription
Seller initial aggressivenessRatio of the seller’s opening offer p0sellerp_{0}^{\text{seller}} to the seller’s reservation price: αseller=p0seller/s\alpha^{\text{seller}}=p_{0}^{\text{seller}}\big/s . Higher values indicate a more aggressive opening position (asking further above cost).
Buyer initial aggressivenessMeasured in two variants (higher == more aggressive): • Gap closure: αbuyer,1=(p0seller−p0buyer)/p0seller,\alpha^{\text{buyer},1}=\bigl(p_{0}^{\text{seller}}-p_{0}^{\text{buyer}}\bigr)\big/p_{0}^{\text{seller}}, fraction of the seller’s asking price cut by the buyer’s first counter-offer. • Reservation ratio: αbuyer,2=(b−p0buyer)/b,\alpha^{\text{buyer},2}=\bigl(b-p_{0}^{\text{buyer}}\bigr)\big/b, fraction of the buyer’s reservation price retained as surplus in the opening bid.
Concession rateFraction of remaining surplus conceded per round (computed for valid deals only). • Seller: ctseller=(ptseller−pt+1seller)/(ptseller−s).c_{t}^{\text{seller}}=\bigl(p_{t}^{\text{seller}}-p_{t+1}^{\text{seller}}\bigr)\big/\bigl(p_{t}^{\text{seller}}-s\bigr). • Buyer: ctbuyer=(pt+1buyer−ptbuyer)/(b−ptbuyer).c_{t}^{\text{buyer}}=\bigl(p_{t+1}^{\text{buyer}}-p_{t}^{\text{buyer}}\bigr)\big/\bigl(b-p_{t}^{\text{buyer}}\bigr). Higher values indicate aggressive bargaining. The sign is used as a Dimension 1 diagnostic ( Table 2 ).
Temporal patienceMean number of rounds to termination. Measures willingness to sustain dialogue rather than terminating prematurely.
Table 5: Behavioral driver metrics: mechanisms underlying strategic effectiveness and allocative efficiency outcomes.
ModelProviderCost ($/1M tokens, in / out)
DeepSeek-V3-0324DeepSeek$0.27 /$1.10
Gemini-2.5-FlashGoogle$0.30 /$2.50
Gemini-2.5-ProGoogle$1.25 /$10.00
GPT-4.1OpenAI$2.00 /$8.00
o3 †OpenAI$2.00 /$8.00
Table 6: Frontier models evaluated in the round-robin benchmark. Prices reflect API list rates at the time of experiments. † o3 is a reasoning model; internal reasoning tokens are billed as output tokens, increasing effective cost.
Figure 4: IR compliance for frontier models. Self: the model’s own violation rate (deals yielding negative utility for itself). Opp.: the opponent’s violation rate when facing this model. All values averaged over all five opponents. Green/red borders highlight best/worst values in each column. In NGFT, any deal is a violation for at least one party; o3 as buyer induces 5.00% opponent violations despite only 0.10% self-violations. Full table in Table 14 .
Figure 5: NGFT pairwise breakdown for o3 as buyer. The irrationality comes almost entirely from opponents: GPT-4.1 and Gemini-2.5-Flash violate at 11.5% and 9.5%, while o3 itself violates at most 0.5%. Full table in Table 15 .
Figure 6: Surplus share and deal rate for frontier models (GFT scenarios), averaged over all five opponents. B = buyer, S = seller. Surplus: fraction of ZOPA captured conditional on agreement. Deal rate: fraction of negotiations reaching agreement. Full table in Table 16 .
Figure 7: Behavioral drivers for frontier models (GFT scenarios), averaged over all five opponents. B = buyer, S = seller. Init. Aggr. (buyer): fraction of the seller’s asking price cut by the buyer’s first counter-offer (higher = more aggressive). Init. Aggr. (seller): ratio of first offer to reservation price (higher = more aggressive). Conc. Rate: fraction of remaining surplus conceded per round. Patience: mean number of rounds to termination (max 10). Full table in Table 16 .
Figure 8: Surplus share by reservation price quintile, averaged across all 5 opponents (GFT). Left panel: buyer surplus by buyer-value bracket. Right panel: seller surplus by seller-cost bracket. Strong models (o3, Gem.-Pro) show 4–6 pp spreads; weak buyers (Gem.-Flash) show 13 pp. Full table in Table 19 .
Figure 9: Deal rates by reservation price quintile, averaged across all 5 opponents (GFT). Left panel: buyer deal rate by buyer-value bracket. Right panel: seller deal rate by seller-cost bracket. Flash as buyer shows the largest spread (9.7 pp); all other models stay within 1–5 pp. Full table in Table 20 .
Figure 10: Combined violation rates (either party) by reservation price quintile, averaged across all 5 opponents (GFT). All rates are below 2% with no systematic bracket trend. Gemini-2.5-Pro as seller: 0% in every quintile. Full table in Table 21 .
Figure 11: Buyer performance profile. Six axes: IR compliance, surplus share, deal rate, anchoring power (fraction cut from asking price), price-tier consistency, and opponent pressure resistance. All min-max normalized; higher = better. Full table in Table 17 .
Figure 12: Seller performance profile. Same six axes as the buyer plot, with anchoring power measured as the ratio of first offer to reservation price. o3 dominates on anchoring and surplus; GPT-4.1 leads on consistency. Full table in Table 18 .
Figure 13: Turn-level decomposition of a negotiation trajectory into autoregressive training samples.
Figure 14: Reasoning-token masking: <think> blocks are masked from input context turns, so the model conditions only on observable agent responses.
ComponentDefinitionRange
RparsingR_{\text{parsing}}Fraction of agent messages containing a valid tool-call code block: (# parseable messages)/(# all messages){\text{(\# parseable messages)}}/{\text{(\# all messages)}}[0,1][0,1]
RexecutionR_{\text{execution}}Fraction of parseable code blocks that execute successfully: (# successfully executed blocks)/(# parseable blocks){\text{(\# successfully executed blocks)}}/{\text{(\# parseable blocks)}}[0,1][0,1]
RconstraintsR_{\text{constraints}}Behavioral consistency, averaged over two checks: 0.5 (1−1{accepted_worse_offer_later})+0.5 (1−1{proposed_worse_than_rejected})0.5\,(1-\mathbf{1}\{\texttt{accepted\_worse\_offer\_later}\})+0.5\,(1-\mathbf{1}\{\texttt{proposed\_worse\_than\_rejected}\}) . Penalizes accepting a deal worse than a previously rejected offer, or proposing terms worse than what the opponent already rejected.{0,0.5,1}\{0,0.5,1\}
RutilityR_{\text{utility}}Surplus share ( Definition 5 ) if a deal is reached; 00 otherwise[0,1][0,1]
Table 8: Reward components in the composite reward ( Equation 8 ). Protocol components ( RparsingR_{\text{parsing}} , RexecutionR_{\text{execution}} ) reward well-formed tool usage; outcome components reward behavioral consistency ( RconstraintsR_{\text{constraints}} ) and surplus capture ( RutilityR_{\text{utility}} ).
Figure 15: NGFT violation rates across training stages. Self: the Qwen model strikes a deal in a no-gains-from-trade scenario (irrational for at least one party). Opp.: the frontier counterpart does so when facing Qwen. Full table in Table 34 .
Figure 16: Strategic effectiveness and allocative efficiency across training stages (GFT scenarios). Deal rate: fraction of negotiations reaching agreement. Surplus share: fraction of ZOPA captured conditional on agreement. GPT-4.1 is the RL training opponent (in-distribution); o3-mini is unseen (out-of-distribution). Full table in Table 35 .
Figure 17: Combined violation rate by reservation-price quintile and training stage, split by opponent. Top row: Qwen3-8B. Bottom row: Qwen3-14B. 4.1 = GPT-4.1; o3 = o3-mini. Violations show no systematic Q1 →\to Q5 trend. Full table in Table 36 .
Figure 18: Surplus share by reservation-price quintile and training stage, split by opponent. Top row: Qwen3-8B. Bottom row: Qwen3-14B. 4.1 = GPT-4.1 (RL training opponent); o3 = o3-mini (unseen). SFT compresses buyer-side spreads to 7–9pp; Base and RL show 15–25pp Q1 →\to Q5 spreads. Full table in Table 37 .
Figure 19: Deal rate by reservation-price quintile and training stage, split by opponent. Top row: Qwen3-8B. Bottom row: Qwen3-14B. 14B-SFT as seller shows the largest bracket spread (26.3pp vs. GPT-4.1). Base and RL deal rates are high and value-invariant. Full table in Table 38 .
Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 20: Example listing used in the negotiation simulation environment.
ToolParametersEffect
make_offerprice , side_offer (opt.)Proposes a formal price; becomes pending offer that the counterpart must respond to. Price must be positive.
respond_to_offerresponse (Boolean)Accepts ( True ) or rejects ( False ) the pending offer. Acceptance concludes the negotiation at the offered price.
send_messagecontent (string)Sends a free-form natural-language message to the counterpart. No direct effect on negotiation state.
search_price(none)Returns historical high and low prices for the item. Information only.
quit_negotiation(none)Exercises the outside option (BATNA). Negotiation terminates immediately with no deal; both utilities are zero.
wait_for_response(none)Yields the turn to the counterpart. Must be the last tool called in a turn.
Appendix
Table 9: Complete tool specification for the bilateral bargaining environment. Each turn permits at most 3 tool calls. The main text ( Table 1 ) lists the six primary tools; wait_for_time_period is an additional tool used to model tactical delays.
RoundAgentAction
1Sellermake_offer(1400)
2Buyersend_message(‘‘Your price is too high for a used laptop’’) ; make_offer(950)
3Sellersend_message(‘‘I can’t go that low, but I can offer a discount’’) ; make_offer(1150)
4Buyermake_offer(1050)
5Sellerrespond_to_offer(True)
Appendix
Table 10: GFT example: deal at p^{*}=\1{,}050.Buyerutility. Buyer utilityU_{B}=150,sellerutility, seller utilityU_{S}=150$ . Surplus shares: 50/50.
RoundAgentAction
1Sellermake_offer(1300)
2Buyermake_offer(800)
3Sellersend_message(‘‘I can’t go below $1,100’’)
4Buyersend_message(‘‘I can’t go above $850’’)
5Buyerquit_negotiation()
Appendix
Table 11: NGFT example: no deal (rational outcome). Both utilities are zero. The buyer correctly identifies the infeasible negotiation and exercises the outside option.
Parameter8B Model14B Model
Optimization
Learning rate2×10−52\times 10^{-5}1×10−51\times 10^{-5}
LR scheduleCosineCosine
Warmup ratio0.10.1
Number of epochs33
Weight decay0.010.01
Appendix
Table 12: SFT training hyperparameters for 8B and 14B models. Both models are trained with full parameter updates (no LoRA). The effective batch size is per-device batch×accumulation steps×GPUs\text{per-device batch}\times\text{accumulation steps}\times\text{GPUs} : 32 for 8B, 16 for 14B.
Parameter8B Model14B Model
Optimization
Learning rate3×10−63\times 10^{-6}2×10−62\times 10^{-6}
Warmup steps2020
Max training steps200200
Per-device batch size11
Gradient accumulation steps1616
Appendix
Table 13: GRPO training hyperparameters for 8B and 14B models. The 8B model is trained with full parameters; the 14B model uses LoRA (rank 64, <1%{<}1\% trainable parameters) due to memory constraints.
  GFT Violation (%)NGFT Violation (%)
ModelRoleSelfOpp.SelfOpp.
DeepSeek-V3Buyer0.450.150.100.40
Seller0.700.450.600.00
Gem.-2.5-FlashBuyer0.200.250.400.00
Seller0.100.101.900.00
Gem.-2.5-ProBuyer0.100.200.000.00
Appendix
Table 14: IR compliance for frontier models. Self: the model’s own violation rate (deals yielding negative utility for itself). Opp.: the opponent’s violation rate when facing this model. All values averaged over all five opponents. Bold highlights best and worst values. In NGFT, any deal is a violation for at least one party; o3 as buyer induces 5.00% opponent violations despite only 0.10% self-violations.
o3 as Buyer vs.o3 Self-Viol. (%)Seller Opp.-Viol. (%)
GPT-4.10.511.5
Gem.-2.5-Flash0.09.5
Gem.-2.5-Pro0.03.5
DeepSeek-V30.00.5
o30.00.0
Appendix
Table 15: NGFT pairwise breakdown for o3 as buyer. Self-violation = o3’s own irrational deals; opponent-violation = seller’s irrational deals when facing o3. The irrationality comes almost entirely from opponents: GPT-4.1 and Gemini-2.5-Flash violate at 11.5% and 9.5%, while o3 itself violates at most 0.5%.
 Surplus (%)Deal Rate (%)Init. Aggr.Conc. RatePatience
ModelBSBSBSBSBS
DeepSeek-V335.149.694.492.10.341.970.520.506.655.98
Gem.-2.5-Flash24.357.686.494.90.372.140.670.545.726.02
Gem.-2.5-Pro44.972.696.791.70.402.060.510.437.077.06
GPT-4.134.157.794.095.20.362.070.530.526.076.08
o347.277.098.095.60.393.040.470.556.506.86
Appendix
Table 16: Outcome metrics and behavioral drivers for frontier models (GFT scenarios), averaged over all five opponents. B = buyer, S = seller. Surplus: fraction of ZOPA captured conditional on agreement. Deal rate: fraction of negotiations reaching agreement. Init. Aggr. (buyer): fraction of the seller’s asking price cut by the buyer’s first counter-offer (higher = more aggressive). Init. Aggr. (seller): ratio of first offer to reservation price (higher = more aggressive). Conc. Rate: fraction of remaining surplus conceded per round. Patience: mean number of rounds to termination (max 10).
ModelIR Compl.SurplusDeal RateAnchor.ConsistencyPressure Res.
 (%)(%)(%) (%)(%)
o399.9547.298.00.3993.999.90
Gem.-2.5-Pro99.9044.996.70.4094.0100.00
GPT-4.199.7534.194.00.3687.599.80
DeepSeek-V399.5535.194.40.3493.699.90
Gem.-2.5-Flash99.8024.386.40.3787.299.60
Appendix
Table 17: Buyer radar-plot profile for frontier models. IR Compl. =100−GFT self-violation rate=100-\text{GFT self-violation rate} . Surplus: buyer surplus share (GFT). Deal Rate: buyer GFT deal rate. Anchor.: fraction of the seller’s asking price cut by the buyer’s first counter-offer. Consistency =100−quintile surplus spread (pp)=100-\text{quintile surplus spread (pp)} . Pressure Res. =100−NGFT self-violation rate=100-\text{NGFT self-violation rate} . All axes: higher is better.
ModelIR Compl.SurplusDeal RateAnchor.ConsistencyPressure Res.
 (%)(%)(%) (%)(%)
o399.9577.095.63.0496.0100.00
Gem.-2.5-Pro100.0072.691.72.0694.099.30
GPT-4.199.9057.795.22.0796.297.70
DeepSeek-V399.3049.692.11.9790.099.40
Gem.-2.5-Flash99.9057.694.92.1493.498.10
Appendix
Table 18: Seller radar-plot profile for frontier models. IR Compl. =100−GFT self-violation rate=100-\text{GFT self-violation rate} . Surplus: seller surplus share (GFT). Deal Rate: seller GFT deal rate. Anchor.: ratio of first offer to reservation price. Consistency =100−quintile surplus spread (pp)=100-\text{quintile surplus spread (pp)} . Pressure Res. =100−NGFT self-violation rate=100-\text{NGFT self-violation rate} . All axes: higher is better.
Figure 21: Violation rate heatmaps for all 25 buyer–seller pairings. Left: GFT scenarios (violations arise from deals struck outside the ZOPA). Right: NGFT scenarios (every deal is a violation for at least one party, so violation rate == deal rate).
Figure 22: Average buyer utility in GFT scenarios. Left: averaged over all negotiations (including no-deal outcomes at utility =0=0 ). Right: conditional on agreement.
Figure 23: Average seller utility in GFT scenarios. Left: averaged over all negotiations. Right: conditional on agreement.
Figure 24: Surplus share heatmaps for all 25 buyer–seller pairings (GFT scenarios). Left: buyer surplus share. Right: seller surplus share. The two shares sum to one for each cell.
Figure 25: Deal rate for all 25 buyer–seller pairings (GFT scenarios). NGFT deal rates equal violation rates ( Figure 21 ) and are omitted.
Figure 26: Seller initial aggressiveness (first offer / reservation price). Higher values indicate more aggressive opening offers.
Figure 27: Buyer initial aggressiveness. Left: V1 (gap closure). Right: V2 (reservation ratio). Lower values indicate more aggressive opening offers.
Figure 28: Per-round concession rate. Left: buyer concession rate. Right: seller concession rate. Higher values indicate faster concessions.
Figure 29: Average negotiation rounds for all 25 buyer–seller pairings (GFT scenarios). The protocol allows a maximum of 10 rounds.
 Buyer Surplus by Buyer Value (%)Seller Surplus by Seller Cost (%)
ModelQ1Q2Q3Q4Q5SpreadQ1Q2Q3Q4Q5Spread
o344.549.949.648.243.86.179.376.475.575.378.74.0
Gem.-2.5-Pro42.845.047.647.141.66.075.069.673.469.675.66.0
GPT-4.126.134.134.538.636.912.559.755.956.358.258.53.8
DeepSeek-V331.635.938.033.636.36.454.950.851.046.344.910.0
Gem.-2.5-Flash16.723.025.626.829.512.859.656.554.855.961.46.6
Appendix
Table 19: Surplus share by reservation price quintile, averaged across all 5 opponents (GFT). Left panel: buyer surplus by buyer-value bracket. Right panel: seller surplus by seller-cost bracket. Spread = max −- min across quintiles. Strong models (o3, Gem.-Pro) show 4–6 pp spreads; weak buyers (Gem.-Flash) show 13 pp.
 Buyer Deal Rate by Buyer Value (%)Seller Deal Rate by Seller Cost (%)
ModelQ1Q2Q3Q4Q5SpreadQ1Q2Q3Q4Q5Spread
o397.398.897.298.398.51.696.094.595.796.595.02.0
Gem.-2.5-Pro97.096.096.396.897.31.389.589.593.893.592.04.3
GPT-4.194.091.394.294.595.84.592.392.597.097.097.04.7
DeepSeek-V392.395.095.795.893.03.590.890.594.292.092.83.7
Gem.-2.5-Flash87.579.887.887.589.59.792.094.395.096.896.84.8
Appendix
Table 20: Deal rates by reservation price quintile, averaged across all 5 opponents (GFT). Left panel: buyer deal rate by buyer-value bracket. Right panel: seller deal rate by seller-cost bracket. Flash as buyer shows the largest spread (9.7 pp); all other models stay within 1–5 pp.
 As Buyer (%)As Seller (%)
ModelQ1Q2Q3Q4Q5SpreadQ1Q2Q3Q4Q5Spread
o30.500.250.250.250.000.500.250.750.000.500.250.75
Gem.-2.5-Pro0.500.000.250.250.500.500.000.000.000.000.000.00
GPT-4.10.500.000.500.750.250.750.500.250.250.250.250.25
DeepSeek-V31.000.500.500.750.250.751.250.251.511.751.001.50
Gem.-2.5-Flash1.000.500.000.750.001.000.250.500.000.250.000.50
Appendix
Table 21: Combined violation rates (either party) by reservation price quintile, averaged across all 5 opponents (GFT). Left panel: by buyer-value bracket (model as buyer). Right panel: by seller-cost bracket (model as seller). All rates are below 2% with no systematic bracket trend. Gemini-2.5-Pro as seller: 0% in every quintile.
vs. SellerQ1Q2Q3Q4Q5Avg
o326.837.135.034.528.032.3
GPT-4.154.660.655.754.452.255.5
Gem.-Pro27.737.637.436.228.333.4
Gem.-Flash59.752.055.352.545.453.0
DeepSeek53.762.264.563.665.261.8
Appendix
Table 22: o3 as buyer: surplus share (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o328.028.329.933.724.128.8
GPT-4.153.652.349.249.644.849.9
Gem.-Pro30.438.142.040.831.636.6
Gem.-Flash50.650.956.851.743.050.6
DeepSeek51.655.660.359.564.758.3
Appendix
Table 23: Gemini-2.5-Pro as buyer: surplus share (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o312.918.417.825.623.319.6
GPT-4.132.138.342.239.842.238.9
Gem.-Pro12.823.725.328.826.323.4
Gem.-Flash35.742.740.648.738.241.2
DeepSeek36.947.346.550.354.747.1
Appendix
Table 24: GPT-4.1 as buyer: surplus share (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o319.620.826.123.522.722.5
GPT-4.142.343.243.530.642.640.4
Gem.-Pro18.526.830.630.126.526.5
Gem.-Flash33.841.040.636.134.937.3
DeepSeek43.647.549.147.554.948.5
Appendix
Table 25: DeepSeek-V3 as buyer: surplus share (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o36.09.910.714.017.511.6
GPT-4.120.929.925.428.629.226.8
Gem.-Pro9.415.717.420.221.516.8
Gem.-Flash28.126.335.828.430.329.8
DeepSeek19.033.338.942.748.836.5
Appendix
Table 26: Gemini-2.5-Flash as buyer: surplus share (%) by value quintile and opponent. Flash shows the strongest Q1 →\to Q5 increase, especially vs. DeepSeek (+29.8 pp).
vs. SellerQ1Q2Q3Q4Q5Avg
o3100.0100.097.5100.098.899.3
GPT-4.198.898.897.598.897.598.3
Gem.-Pro96.3100.0100.098.8100.099.0
Gem.-Flash93.897.598.897.5100.097.5
DeepSeek97.597.592.496.396.396.0
Appendix
Table 27: o3 as buyer: deal rate (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o3100.098.897.598.898.898.8
GPT-4.198.897.598.8100.097.598.5
Gem.-Pro95.098.898.896.393.896.5
Gem.-Flash97.591.396.396.397.595.8
DeepSeek93.893.890.092.598.893.8
Appendix
Table 28: Gemini-2.5-Pro as buyer: deal rate (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o398.891.396.397.5100.096.8
GPT-4.193.892.595.096.397.595.0
Gem.-Pro92.588.891.390.098.892.3
Gem.-Flash96.396.394.996.393.895.5
DeepSeek88.887.593.892.588.890.3
Appendix
Table 29: GPT-4.1 as buyer: deal rate (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o390.091.395.095.086.391.5
GPT-4.197.598.898.898.8100.098.8
Gem.-Pro83.888.891.390.083.887.5
Gem.-Flash93.897.595.098.8100.097.0
DeepSeek96.398.898.796.395.097.0
Appendix
Table 30: DeepSeek-V3 as buyer: deal rate (%) by value quintile and opponent.
vs. SellerQ1Q2Q3Q4Q5Avg
o397.587.591.391.390.091.5
GPT-4.178.878.887.587.593.885.3
Gem.-Pro88.870.082.582.591.383.0
Gem.-Flash87.581.392.590.093.889.0
DeepSeek85.081.385.086.378.883.3
Appendix
Table 31: Gemini-2.5-Flash as buyer: deal rate (%) by value quintile and opponent. Flash shows the strongest bracket sensitivity, with Q1 →\to Q5 gains of up to 15 pp (vs. GPT-4.1).
 Deal Rate (%)Buyer Surplus Share (%)
Quintileo3ProDeepSeekFlasho3ProDeepSeekFlash
Q1 (lowest)98.898.897.578.854.653.642.320.9
Q298.897.598.878.860.652.343.229.9
Q397.598.898.887.555.749.243.525.4
Q498.8100.098.887.554.449.630.628.6
Q5 (highest)97.597.5100.093.852.244.842.629.2
Appendix
Table 32: Per-model price-tier decomposition: buyer deal rate and buyer surplus share vs. GPT-4.1 (GFT scenarios), partitioned by buyer value quintile. Strong negotiators (o3, Pro) show flat performance across brackets; weaker negotiators (Flash) show bracket-dependent deal rates.
   vs GPT-4.1 (%)vs o3-mini (%)Average (%)
ModelStageRoleSelfOpp.SelfOpp.SelfOpp.
Qwen3-8BBaseBuyer0.500.001.000.000.750.00
Seller3.750.251.000.002.380.13
SFTBuyer0.000.501.250.750.630.63
Seller0.250.750.500.250.380.50
RLBuyer0.250.000.750.250.500.13
Appendix
Table 33: GFT violation rates across training stages. Self: the Qwen model accepts a deal yielding negative utility. Opp.: the frontier counterpart accepts such a deal when facing Qwen. GPT-4.1 is the RL training opponent (in-distribution); o3-mini is unseen (out-of-distribution). Bold highlights key self-violation values discussed in text.
   vs GPT-4.1 (%)vs o3-mini (%)Average (%)
ModelStageRoleSelfOpp.SelfOpp.SelfOpp.
Qwen3-8BBaseBuyer0.000.000.250.000.130.00
Seller4.000.004.000.004.000.00
SFTBuyer0.250.500.250.000.250.25
Seller0.500.000.250.000.380.00
RLBuyer0.000.000.000.000.000.00
Appendix
Table 34: NGFT violation rates across training stages. Self: the Qwen model strikes a deal in a no-gains-from-trade scenario (irrational for at least one party). Opp.: the frontier counterpart does so when facing Qwen. Bold highlights key self-violation values discussed in text. Dash indicates missing data.
   Deal Rate (%)Surplus Share (%)
ModelStageRolevs 4.1vs o3vs 4.1vs o3
Qwen3-8BBaseBuyer70.0066.7522.4111.16
Seller80.0070.2532.5324.80
SFTBuyer86.2579.2540.7736.12
Seller70.2556.5060.5664.79
RLBuyer69.7569.5020.3911.60
Appendix
Table 35: Strategic effectiveness and allocative efficiency across training stages (GFT scenarios). Deal rate: fraction of negotiations reaching agreement. Surplus share: fraction of ZOPA captured conditional on agreement. GPT-4.1 is the RL training opponent (in-distribution); o3-mini is unseen (out-of-distribution).
  Buyer Violation by Buyer Value (%)Seller Violation by Seller Cost (%)
Model–StageOpp.Q1Q2Q3Q4Q5SpreadQ1Q2Q3Q4Q5Spread
14B-Base4.10.000.001.250.000.001.252.500.002.501.251.252.50
 o30.000.001.250.000.001.251.250.001.250.000.001.25
14B-SFT4.10.000.000.000.000.000.000.001.250.000.001.251.25
 o31.251.253.750.001.253.751.251.250.000.000.001.25
14B-RL4.10.000.000.001.250.001.252.501.251.253.750.003.75
Appendix
Table 36: Combined violation rate by reservation-price quintile and training stage, split by opponent. Bold highlights 8B seller violations exceeding 5%.
  Buyer Surplus by Buyer Value (%)Seller Surplus by Seller Cost (%)
Model–StageOpp.Q1Q2Q3Q4Q5SpreadQ1Q2Q3Q4Q5Spread
14B-Base4.118.924.318.720.034.115.442.838.439.037.040.05.8
 o34.77.912.414.026.521.835.032.131.337.530.27.3
14B-SFT4.149.348.940.743.342.58.664.251.458.163.156.012.8
 o330.737.738.737.739.38.660.166.459.456.861.69.6
14B-RL4.114.020.318.122.934.720.742.237.444.033.843.410.2
Appendix
Table 37: Surplus share by reservation-price quintile and training stage, split by opponent. SFT compresses buyer-side spreads to 7–9pp; Base and RL show 15–25pp Q1 →\to Q5 gradients.
  Buyer Deal Rate by Buyer Value (%)Seller Deal Rate by Seller Cost (%)
Model–StageOpp.Q1Q2Q3Q4Q5SpreadQ1Q2Q3Q4Q5Spread
14B-Base4.197.597.596.395.0100.05.096.397.597.598.896.32.5
 o396.397.596.395.091.36.293.893.895.093.896.32.5
14B-SFT4.173.862.577.571.373.815.048.856.345.035.061.326.3
 o362.572.555.062.568.817.536.342.545.036.345.08.7
14B-RL4.196.396.396.393.895.02.5100.098.8100.097.598.82.5
Appendix
Table 38: Deal rate by reservation-price quintile and training stage, split by opponent. 14B-SFT as seller shows the largest bracket spread (26.3pp vs. GPT-4.1).
More in this topic

Multi-Agent Negotiation

Multi-agent negotiation studies how autonomous agents bargain, coordinate, and resolve conflicts over resources, contracts, and shared decisions.

See all 53 papers

Explore similar work

CardsList
Apr 10, 2026

Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards

The recent advancement of Large Language Models (LLMs) has established their potential as autonomous interactive agents. However, they often struggle in strategic games of incomplete information, such as bilateral price negotiation. In this paper, we investigate if Reinforcement Learning from Verifiable Rewards (RLVR) can effectively teach LLMs to negotiate. Specifically, we explore the strategic behaviors that emerge during the learning process. We introduce a framework that trains a mid-sized buyer agent against a regulated LLM seller across a wide distribution of real-world products. By grounding reward signals directly in the maximization of economic surplus and strict adherence to private budget constraints, we reveal a novel four-phase strategic evolution. The agent progresses from naive bargaining to using aggressive starting prices, moves through a phase of deadlock, and ultimately develops sophisticated persuasive skills. Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus. Furthermore, the trained agent generalizes robustly to stronger counterparties unseen during training and remains effective even when facing hostile, adversarial seller personas.

Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao +4

Purdue University · California Institute of Technology · University of Illinois Urbana-Champaign +1

Automated NegotiationMulti-Agent NegotiationRL for LLM AgentsReinforcement Learning with Verifiable Rewards
AskPDF
Oct 5, 2026

Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers

LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate (10−610^{-6}) for every size, the gain of the RL model over its base rises from +0.001+0.001 at 2.3B to +0.078+0.078 at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by +0.032+0.032 (2.3B) to +0.081+0.081 (4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.

Pedro Tabacof, Sagar Joglekar

Fin AI Research

RL for Language ModelsMulti-Agent NegotiationSmall Language ModelsLanguage Model Scaling
AskPDF
May 14, 2026

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining

We introduce \textsc{Cattle Trade, a multi-agent benchmark for evaluating large language models (LLMs) as agents in strategic reasoning under imperfect information, adversarial interaction, and resource constraints. The benchmark combines auctions, hidden-offer trade challenges (TCs), bargaining, bluffing, opponent modeling, and resource allocation within a single long-horizon game lasting 50--60 turns. Unlike prior agent benchmarks that test these abilities in isolation, Cattle Trade evaluates whether agents integrate them across a competitive, multi-agent economic game with conflicting incentives. The benchmark logs every bid, TC offer, counteroffer, and card selection, enabling behavioural analysis beyond final scores or win rates. We evaluate seven cost-efficient language models and three deterministic code agents across 242 games. Strategic coherence, in particular spending efficiency, resource discipline, and phase-adaptive bidding, is associated with rank more strongly than spending volume or any single subskill. Two heuristic code agents outperform most tested LLMs, and behavioural traces surface recurring LLM failure modes including overbidding, self-bidding, bankrupt TC initiation, and weak opponent-state adaptation. Evaluating agentic competence requires benchmarks that test the joint deployment of multiple capabilities in multi-agent environments with conflicting incentives, uncertainty, and economic dynamics.

Robert Müller, Clemens Müller

Multi-Agent NegotiationGame-Playing AgentsLLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent Benchmarks
AskPDF