Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition

Yushuo Zheng; Huiyu Duan; Zicheng Zhang; Yucheng Zhu; Xiongkuo Min; Guangtao Zhai

cs.AIApr 7, 2026

Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition

Authors: Yushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai

Organizations: Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

The ability of large language models (LLMs) to manage and acquire economic resources remains unclear. In this paper, we introduce Market-Bench, a comprehensive benchmark that evaluates the capabilities of LLMs in economically-relevant tasks through economic and trade competition. Specifically, we construct a configurable multi-agent supply chain economic model where LLMs act as retailer agents responsible for procuring and retailing merchandise. In the procurement stage, LLMs bid for limited inventory in budget-constrained auctions. In the retail stage, LLMs set retail prices, generate marketing slogans, and provide them to buyers through a role-based attention mechanism for purchase. Market-Bench logs complete trajectories of bids, prices, slogans, sales, and balance-sheet states, enabling automatic evaluation with economic, operational, and semantic metrics. Benchmarking on 20 open- and closed-source LLM agents reveals significant performance disparities and winner-take-most phenomenon, i.e., only a small subset of LLM retailers can consistently achieve capital appreciation, while many hover around the break-even point despite similar semantic matching scores. Market-Bench provides a reproducible testbed for studying how LLMs interact in competitive markets.

Figures & tables

Figure 1: Existing LLM benchmarks focus on either semantic complexity or quantitative competition, but rarely both simultaneously under economic scarcity. Thus, we propose Market-Bench, coupling marketing slogans and operations to jointly evaluate mathematical optimization and language comprehension.

Figure 2: Overview of the Market-Bench environment. Agents operate in a competitive supply chain economy, making procurement, pricing, and marketing decisions under scarcity conditions. The Persona-Gated Attention mechanism requires agents to align their marketing slogans with latent buyer personas to convert demand into sales. Performance is evaluated across economic, supply chain, and cognitive metrics.

Work	Competitive Interaction	Hard Scarcity	Closed-Loop Economy	Free-Form Language	Text $\to$ Payoff
Agent Societies
Park et al. (2023)	✗	✗	✗	✓	✗
Zhou et al. (2024)	✗	✗	✗	✓	✗
Piao et al. (2025)	✗	✗	✗	✓	✗
Strategic Games
Zhu et al. (2025)	✓	✗	✗	✓	✗

Table 1: Comparison of representative LLM-agent benchmarks and economic simulators. Market-Bench is the only benchmark providing both a closed-loop multi-agent market under hard scarcity and free-form language that directly affects payoff via Persona-Gated Attention.

Mode Type	Model	Economics				Operation				Cognitive
Mode Type	Model	NPM $\uparrow$	Pi $\uparrow$	RAR $\uparrow$	IEI $\uparrow$	Stockout Rate $\downarrow$	Bid Eff. $\uparrow$	OSI $\uparrow$	Fill Rate $\uparrow$	MMS $\uparrow$
Closed Source Model	Gemini 2.5 Pro Gemini Team (2025)	0.167	36589	2.751	0.861	0.765	0.792	0.777	0.221	0.691
	Gemini 2.5 Flash Gemini Team (2025)	0.190	26104	1.569	0.645	0.791	0.795	0.820	0.194	0.675
	O3 El-Kishky et al. (2025)	0.097	13368	1.016	0.901	0.573	0.699	1.000	0.418	0.690
	Sonnet 4.5 Anthropic (2025a)	0.158	10619	1.438	0.780	0.892	0.411	0.603	0.097	0.671
	GPT-4o OpenAI (2024)	0.117	7619	0.712	0.742	0.911	0.450	0.640	0.079	0.669

Table 2: Performance Metrics of Different Models. For each metric, an arrow indicates whether higher (

\uparrow

) or lower (

\downarrow

) values are better. The best and second-best performances for each metric are highlighted in red and blue, respectively.

Figure 3: Market-level indices computed from logged trajectories in the default setting. Inequality increases (Gini, Theil, CV); the top-4 share rises (CR4) while concentration remains competitive (HHI); and the Active Ratio declines.

Figure 4: Temporal economic outcomes (mean across 10 runs). From left to right: net profit margin (NPM), per-step profit, and end-of-step funds. A small set of models, led by Gemini 2.5 Pro and Gemini 2.5 Flash, consistently earn high profits and compound capital, while many models remain near break-even.

Figure 5: Operational dynamics over time (mean across 10 runs). Left: Order Stability Index (OSI). Middle: procurement BidEfficiency. Right: downstream FillRate. Agents with sustained auction success (high BidEfficiency) also achieve higher FillRate and tend to maintain higher OSI.

Figure 6: Slogan-persona similarity dynamics (mean $\pm$ std across 10 runs). Each line is an agent’s average cosine similarity to each buyer tribe persona (Thrifty, Ethical, Hype, Quality) and the resulting weighted MMS. Across agents, similarities shift most between steps 0 and 1 and then stabilize.

Figure 7: Evolution of slogan embedding clusters across steps (PCA projection of tribe-similarity features; mean across 10 runs). Agents start dispersed but rapidly converge to a tight region by step 5, suggesting that competitive feedback induces similar messaging strategies.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Tribe	Weight	$\lambda$	Keywords
Thrifty	0.4	0.2	(price-focused)
Ethical	0.3	0.8	green, fair, eco
Hype	0.2	0.9	exclusive, limited
Quality	0.1	0.5	quality, craft

Appendix

Table 3: Default buyer persona distribution.

\lambda

denotes slogan sensitivity weight.

Parameter	Value
Number of agents	20
Number of steps	6
Bidding rounds	2
Buyers per round	200
Initial funds	22,500
Item categories	8

Appendix

Table 4: Default experimental configuration.

Item	Category	Qty	Base Price
item1	Commodity	200	50
item2	Commodity	200	50
item3	Standard	133	150
item4	Standard	133	150
item5	Standard	134	150
item6	Luxury	75	800

Appendix

Table 5: Supplier offer structure for 20-agent experiments.

Category	Models
Closed-source	Gemini 2.5 Pro, Gemini 2.5 Flash, O3, Claude Sonnet 4.5, GPT-4o
Open-weight	Phi-4, Qwen2.5 VL 72B, Llama 3.1 70B, QwenLong L1 32B, Qwen2.5 32B, Gemma 3 27B, Qwen2.5 VL 32B, ERNIE 4.5 300B, InternLM2.5 20B, InternLM3 8B, DeepSeek V3.2, Qwen2.5 72B, Hunyuan A13B, Qwen3 30B-A3B, ERNIE 4.5 21B

Appendix

Table 6: LLM agents evaluated in Market-Bench experiments.

Explore similar work

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining
May 14, 2026Robert Müller, Clemens MüllerMulti-Agent Negotiation Game-Playing Agents
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
Jul 30, 2026Shimaa Ahmed, Yiwei Cai, Mohsen Minaei +1LLM Agent Evaluation AI Agent Benchmarks
Visa Research
E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
Aug 31, 2026Wei Fan, Xinjie Shen, Xudong Guo +8Long-Horizon Agent Evaluation AI Agent Benchmarks
1Qwen Team, Alibaba Group · Department of Computer Science and Engineering, HKUST · 2Taobao & Tmall Group, Alibaba Group

May 14, 2026J/K move · Enter open · S save

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining

Robert Müller, Clemens Müller

Multi-Agent Negotiation Game-Playing Agents LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

We introduce \textsc{Cattle Trade, a multi-agent benchmark for evaluating large language models (LLMs) as agents in strategic reasoning under imperfect information, adversarial interaction, and resource constraints. The benchmark combines auctions, hidden-offer trade challenges (TCs), bargaining, bluffing, opponent modeling, and resource allocation within a single long-horizon game lasting 50--60 turns. Unlike prior agent benchmarks that test these abilities in isolation, Cattle Trade evaluates whether agents integrate them across a competitive, multi-agent economic game with conflicting incentives. The benchmark logs every bid, TC offer, counteroffer, and card selection, enabling behavioural analysis beyond final scores or win rates. We evaluate seven cost-efficient language models and three deterministic code agents across 242 games. Strategic coherence, in particular spending efficiency, resource discipline, and phase-adaptive bidding, is associated with rank more strongly than spending volume or any single subskill. Two heuristic code agents outperform most tested LLMs, and behavioural traces surface recurring LLM failure modes including overbidding, self-bidding, bankrupt TC initiation, and weak opponent-state adaptation. Evaluating agentic competence requires benchmarks that test the joint deployment of multiple capabilities in multi-agent environments with conflicting incentives, uncertainty, and economic dynamics.

Abstract

Explore similar work

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining

Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation