PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios
Organizations: Yale University · UC Berkeley · BloomBerg · Rutgers University
Abstract
We present an in-depth evaluation of LLMs' ability to negotiate, a central business task that requires strategic reasoning and theory of mind to create and claim economic value. To do so, we introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions over realistic scenarios adapted from MBA negotiation courses at an elite business school. We evaluate language agents across three pairing regimes: mirror-play, cross-play, and human-LM play. We develop a ranking model for continuous negotiation payoffs that yields order-invariant, uncertainty-quantified leaderboards while correcting for systematic experimental asymmetries. We further study the effects of shared-intentionality agentic scaffolding and find asymmetric gains, with large improvements for mid- and lower-tier LMs and diminishing returns for frontier LMs. As calibration anchors, we collect human-human and human-LM negotiation data from trained business school students, finding that a representative frontier language agent matches or exceeds this human baseline in our evaluation settings. Beyond deal outcomes, PieArena provides a multi-dimensional behavioral profile that reveals cross-model heterogeneity in instruction compliance, computation accuracy, as well as judge-assessed deception and reputation, illustrating the value of evaluation beyond outcome-only leaderboards.
Figures & tables
| Model | Base | Pro | ||
|---|---|---|---|---|
| Mean | Sig. | Mean | Sig. | |
| GPT-5 | 0.960 | 0.970 | ||
| GPT-5.2 | 0.924 | — | 0.941 | |
| Grok-3 | 0.854 | — | 0.962 | |
| Grok-4.1-Fast | 0.842 | — | 0.934 | — |
| ERNIE-4.5 | 0.814 | 0.891 | — | |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | CV MSE | CV MAE |
|---|---|---|
| Sequential Elo | ||
| Logistic BTL | ||
| Calibrated LogBTL | ||
| BBTL | ||
| Full BBTL |
| LM | Coverage | LM | Coverage | LM | Coverage |
|---|---|---|---|---|---|
| GPT-5 | 0.968 | Claude-Opus-4.5 | 0.992 | Grok-3 | 0.926 |
| GPT-5.2 | 0.978 | Claude-Sonnet-4.5 | 0.968 | Grok-3-Mini | 0.980 |
| GPT-4.1 | 0.992 | Gemini-2.5-Flash | anchor | Grok-4 | 0.996 |
| o4-Mini-High | 0.964 | Gemini-3-Pro | 0.994 | Grok-4.1-Fast | 0.982 |
| ERNIE-4.5 | 0.934 |
| Slice / Mode | |||||||
| Integrated, Base | (0.013) | (0.038) | (0.020) | (0.019) | |||
| Integrated, Pro | (0.015) | (0.039) | (0.021) | (0.023) | |||
| Multi-Issue, Base | (0.017) | (0.037) | (0.020) | — | — | ||
| Multi-Issue, Pro | (0.018) | (0.038) | (0.020) | — | — | ||
| Single-Issue, Base | (0.018) | — | — | (0.018) | — | — | |
| Single-Issue, Pro | (0.021) | — | — | (0.021) | — | — |
| Scenario | Regime | Used For |
|---|---|---|
| SnyderMed | multi-issue positive-sum | mirror-play (H–H, LM–LM) |
| Hire Talent | multi-issue positive-sum | human–LM play and LM cross-play |
| IP License | multi-issue positive-sum | LM cross-play |
| Service Quote | single-issue zero-sum | LM cross-play |
| Main Street | single-issue zero-sum | human–LM play |
| Site Purchase | no ZOPA | LM screening (mirror-play) |
| Model | n deals | Mean CI | Median | Diff (Mean CI) | Cliff’s | Sig. | |
| (A) Base Mode | |||||||
| Gemini-3-Pro | 20 | 0.965 | 0.617 | 0.002 | |||
| GPT-5 | 19 | 0.976 | 0.588 | 0.003 | |||
| Grok-4 | 18 | 0.965 | 0.506 | 0.010 | |||
| GPT-5.2 | 19 | 0.965 | 0.359 | 0.059 | — | ||
| Grok-3-Mini | 16 | 0.938 | 0.207 | 0.284 | — | ||
| Model | Pro | Base | Pro (Mean CI) | Base (Mean CI) | Diff (Mean CI) | Cliff’s | Sig. | |
| Significant Scaffolding Benefit ( ) | ||||||||
| Grok-3 | 15 | 19 | 0.778 | 0.000 | ||||
| Grok-4.1-Fast | 18 | 18 | 0.765 | 0.000 | ||||
| Claude-Opus-4.5 | 20 | 19 | 0.711 | 0.000 | ||||
| Gemini-2.5-Flash | 12 | 20 | 0.647 | 0.002 | ||||
| GPT-4.1 | 16 | 18 | 0.555 | 0.004 | ||||
| Model | 95% CI | Rank | |
|---|---|---|---|
| Integrated (Base Mode) | |||
| GPT-5 | - 0.2220 | [ - 0.1276, - 0.3164] | 1 |
| Grok-4.1-Fast | - 0.1301 | [ - 0.0381, - 0.2221] | 2 |
| GPT-5.2 | - 0.0095 | [-0.0837, - 0.1028] | 3 |
| Gemini-2.5-Flash | - 0.0000 | [ - 0.0000, - 0.0000] | 4 |
| Grok-3 | -0.0009 | [-0.0941, - 0.0923] | 5 |
| Model | 95% CI | Rank | |
|---|---|---|---|
| Multi-Issue (Base Mode) | |||
| Claude-Opus-4.5 | - 0.1794 | [ - 0.0570, - 0.3019] | 1 |
| Grok-4.1-Fast | - 0.1467 | [ - 0.0248, - 0.2687] | 2 |
| Grok-3 | - 0.0696 | [-0.0549, - 0.1941] | 3 |
| GPT-5 | - 0.0158 | [-0.1083, - 0.1399] | 4 |
| Gemini-2.5-Flash | - 0.0000 | [ - 0.0000, - 0.0000] | 5 |
| Model | 95% CI | Rank | |
|---|---|---|---|
| Single-Issue (Base Mode) | |||
| GPT-5 | - 0.4603 | [ - 0.3372, - 0.5834] | 1 |
| Grok-4.1-Fast | - 0.1255 | [ - 0.0035, - 0.2475] | 2 |
| GPT-5.2 | - 0.0976 | [-0.0284, - 0.2237] | 3 |
| GPT-4.1 | - 0.0299 | [-0.0988, - 0.1586] | 4 |
| Grok-4 | - 0.0042 | [-0.1171, - 0.1256] | 5 |
| Model | 95% CI | Rank | |
|---|---|---|---|
| Hire Talent (Base Mode) | |||
| Claude-Opus-4.5 | - 0.6827 | [ - 0.4323, - 0.9332] | 1 |
| Gemini-3-Pro | - 0.6047 | [ - 0.3542, - 0.8553] | 2 |
| GPT-5 | - 0.4641 | [ - 0.2169, - 0.7112] | 3 |
| Claude-Sonnet-4.5 | - 0.3966 | [ - 0.1441, - 0.6491] | 4 |
| Grok-3 | - 0.2348 | [-0.0202, - 0.4897] | 5 |
| Model | 95% CI | Rank | |
|---|---|---|---|
| IP License (Base Mode) | |||
| Grok-4.1-Fast | - 0.1290 | [-0.0062, - 0.2642] | 1 |
| Claude-Opus-4.5 | - 0.0413 | [-0.0932, - 0.1758] | 2 |
| Grok-3 | - 0.0344 | [-0.1032, - 0.1721] | 3 |
| Gemini-2.5-Flash | - 0.0000 | [ - 0.0000, - 0.0000] | 4 |
| GPT-5.2 | -0.0307 | [-0.1653, - 0.1039] | 5 |
| Model | Pie | Deal | Valid | BATNA | Lie | Rep | Acc |
|---|---|---|---|---|---|---|---|
| Integrated Cross-Play (Base Mode) | |||||||
| GPT-5 | 0.582 | 0.961 | 0.986 | 1.000 | 0.226 | 0.641 | 0.996 |
| [0.561, 0.603] | [0.942, 0.977] | [0.973, 0.995] | [1.000, 1.000] | [0.190, 0.262] | [0.635, 0.647] | [0.989, 1.000] | |
| Grok-4.1-Fast | 0.553 | 0.965 | 0.998 | 1.000 | 0.182 | 0.605 | 0.800 |
| [0.533, 0.573] | [0.947, 0.981] | [0.993, 1.000] | [1.000, 1.000] | [0.152, 0.214] | [0.600, 0.610] | [0.753, 0.847] | |
| Gemini-3-Pro | 0.547 | 0.958 | 0.988 | 1.000 | 0.399 | 0.637 | 0.946 |
| Model | Pie | Deal | Valid | BATNA | Lie | Rep |
|---|---|---|---|---|---|---|
| Single-Issue Cross-Play (Base Mode) | ||||||
| GPT-5 | 0.631 | 0.986 | 1.000 | 1.000 | 0.289 | 0.578 |
| [0.605, 0.655] | [0.965, 1.000] | [1.000, 1.000] | [1.000, 1.000] | [0.233, 0.346] | [0.572, 0.585] | |
| Grok-4.1-Fast | 0.557 | 0.986 | 1.000 | 1.000 | 0.227 | 0.568 |
| [0.535, 0.579] | [0.965, 1.000] | [1.000, 1.000] | [1.000, 1.000] | [0.176, 0.280] | [0.562, 0.574] | |
| Grok-4 | 0.528 | 0.972 | 1.000 | 1.000 | 0.206 | 0.591 |
| Model | Pie | Deal | Valid | BATNA | Lie | Rep | Acc |
|---|---|---|---|---|---|---|---|
| Multi-Issue Cross-Play (Base Mode) | |||||||
| Claude-Opus-4.5 | 0.589 | 0.941 | 0.554 | 1.000 | 0.234 | 0.659 | 0.830 |
| [0.566, 0.614] | [0.913, 0.965] | [0.494, 0.613] | [1.000, 1.000] | [0.190, 0.280] | [0.653, 0.665] | [0.782, 0.875] | |
| Gemini-3-Pro | 0.561 | 0.955 | 0.996 | 1.000 | 0.479 | 0.663 | 0.946 |
| [0.535, 0.586] | [0.931, 0.976] | [0.989, 1.000] | [1.000, 1.000] | [0.410, 0.552] | [0.658, 0.668] | [0.917, 0.971] | |
| GPT-5 | 0.557 | 0.948 | 0.978 | 1.000 | 0.196 | 0.672 | 0.996 |
| Fractional Logit Coefficient | Average Marginal Effect (pp) | AME per SD | |||||
|---|---|---|---|---|---|---|---|
| Coef. | SE | -value | AME | SE | -value | (pp) | |
| State Tracking only vs. Base | Full Harness vs. Base | |||||
|---|---|---|---|---|---|---|
| Model | Sig. | Sig. | ||||
| Grok-3 | ||||||
| Grok-4.1-Fast | ||||||
| Claude-Opus-4.5 | ||||||
| Gemini-2.5-Flash | ||||||
| GPT-4.1 | ||||||
| Short name | OpenRouter ID | Provider model ID |
|---|---|---|
| Claude-Opus-4.5 | anthropic/claude-opus-4.5 | claude-opus-4-5-20251101 |
| Claude-Sonnet-4.5 | anthropic/claude-sonnet-4.5 | claude-sonnet-4-5-20250929 |
| ERNIE-4.5 | baidu/ernie-4.5-vl-424b-a47b | baidu/ERNIE-4.5-VL-424B-A47B-PT § |
| Gemini-2.5-Flash | google/gemini-2.5-flash | gemini-2.5-flash ‡ |
| Gemini-3-Pro | google/gemini-3-pro-preview | gemini-3-pro-preview ‡ |
| GPT-4.1 | openai/gpt-4.1 | gpt-4.1-2025-04-14 |
| Metric | Description (Per Negotiation Episode) |
|---|---|
| Total Pie | Joint economic surplus created by the negotiation, measured as the sum of both parties’ utilities above their BATNAs. |
| Pie Share | Each party’s share of the total pie. |
| Output Validity | Whether the final response strictly conforms to the required structured-output schema without invoking post-hoc JSON repair. |
| Verified Agreement | Whether the negotiation terminates in a full agreement (exactly matching issue-level terms across parties) and the agreed outcome satisfies all scenario-defined hard constraints. |
| BATNA Compliance | Whether the final agreed outcome yields utility at least as high as the party’s own BATNA. |
| Computation Accuracy | Whether a party correctly computes and reports its own utility from the final agreement; measured only in multi-issue negotiation scenarios. |
| Condition | Full cleaned | Double-balanced |
|---|---|---|
| (n=89) | (n=68) | |
| First mover | ||
| LM first | 50 | 34 |
| Student first | 39 | 34 |
| LM mode | ||
| Pro | 41 | 34 |
| Characteristic | Full cleaned | Double-balanced |
|---|---|---|
| (n=89) | (n=68) | |
| Gender | ||
| Male | 46 | 31 |
| Female | 29 | 24 |
| Non-binary | 1 | 1 |
| Missing | 13 | 12 |
| Metric | Value | 95% CI |
|---|---|---|
| Human mean surplus | 79,449 | [70,934, 88,338] |
| LM mean surplus | 120,551 | [111,662, 129,066] |
| Total pie | 200,000 | — |
| Human share of pie | 39.7% | [35.5%, 44.2%] |
| Human median surplus | 75,000 | [70,000, 85,000] |
| LM median surplus | 125,000 | [115,000, 130,000] |
| Comparison | Group A Mean | Group B Mean | Test (H1) | / Effect | |
|---|---|---|---|---|---|
| Human vs LM (Base) | 34 | 83,721 | 116,279 | Wilcoxon (two-sided) | ; |
| Human vs LM (Pro) | 34 | 75,176 | 124,824 | Wilcoxon (two-sided) | ; |
| LM (Pro) vs LM (Base) | 34/34 | 124,824 | 116,279 | MWU (one-sided; Pro Base) | ; |
| Condition | Full cleaned | Double-balanced |
|---|---|---|
| (n=55) | (n=40) | |
| Student role | ||
| Recruiter (side1) | 29 | 20 |
| Candidate (side2) | 26 | 20 |
| First mover | ||
| LM first | 29 | 20 |
| Characteristic | Full cleaned | Double-balanced |
|---|---|---|
| (n=55) | (n=40) | |
| Gender | ||
| Male | 29 | 24 |
| Female | 20 | 13 |
| Missing | 6 | 3 |
| Academic program | ||
| Metric | Value | 95% CI |
|---|---|---|
| Human mean surplus | 28,388 | [23,025, 33,588] |
| LM mean surplus | 22,682 | [18,400, 27,015] |
| Total pie (mean) | 51,070 | [48,630, 53,370] |
| Human share of pie | 53.8% | [44.5%, 62.9%] |
| Human median surplus | 28,500 | [21,250, 35,000] |
| LM median surplus | 22,750 | [18,000, 27,750] |
| Comparison | Mean A | Mean B | Test (H1) | / Effect | |
|---|---|---|---|---|---|
| Human vs LM | 40 | 28,388 | 22,682 | Wilcoxon (two-sided) | ; |
| Human share vs 50% | 40 | 53.8% | 50.0% | Wilcoxon (two-sided) | ; |
| Scenario | Cohen’s | Power / | Bayes factor | |
|---|---|---|---|---|
| Main Street (1 issue) | at observed | (LM) | ||
| Hire Talent (7 issues) | at ; at | (parity) |
| Judge | Lie | Reputation |
|---|---|---|
| GPT-5 | ||
| Grok-4.1-Fast | ||
| GPT-5.2 | ||
| Gemini-2.5-Pro | ||
| Gemini-3-Flash | ||
| Claude-Opus-4.5 |
| own-GPT | own-Grok | own-Gemini | own-Claude | |
|---|---|---|---|---|
| Reputation | ||||
| Lie rate |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,286 papers