Modern information access ecosystems consist of mixtures of systems, such as retrieval systems and large language models, and increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems inherent to deployment. In such settings, outcomes are shaped not only by benchmark quality but also by competitive pressure, including user switching, routing decisions, and operational constraints. Yet evaluation is still largely conducted on static benchmarks with accuracy-focused measures that assume systems operate in isolation. This mismatch makes it difficult to predict post-deployment success and obscures competitive effects such as early-adoption advantages and market dominance. We introduce Marketplace Evaluation, a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace. By simulating repeated interactions and evolving user and agent preferences, the framework enables longitudinal evaluation and marketplace-level metrics, such as retention and market share, that complement and can extend beyond traditional accuracy-based metrics. We formalize the framework and outline a research agenda, motivated by business and economics, around marketplace simulation, metrics, optimization, and adoption in evaluation campaigns like TREC.
Figures & tables
Figure 1. Comparison of evaluation paradigms for information access systems. (a) Cranfield : a multi-user single-system setting where all queries are routed to a fixed system, eliminating competition and user choice. (b) Arena : a multi-user multi-system setting where a central platform mediates anonymous pairwise comparisons, precluding persistent user preferences or switching behavior. (c) Marketplace : a multi-user multi-system setting where users actively select among competing providers without requiring a central mediator, enabling competitive dynamics, behavioral adaptation, and the emergence of market share over time.
Table 2
Figure 2. Marketplace dynamics illustrating fluctuating system rankings and market share following the entry of a new model. The first 100 simulation steps were conducted with six models, after which a seventh model was introduced into the warm started marketplace.
Figure 3. An example snapshot of a RAG marketplace simulation. Nodes represent agents grouped by stakeholder role: users, generators , retrievers , and routers to generators and retrievers. Arrow thickness reflects selection preference intensity, node size reflects accumulated market share, and arrow opacity indicates whether a connection was selected or not selected in the current snapshot.
Stakeholder
Goal
Adaptive Behavior in the Marketplace
Evaluation Focus
Users
Sustained utility from the marketplace over time
Adapt preferences over generators based on past satisfaction, cost, latency, or trust
Table 4. Prominent stakeholders in information access agent marketplaces, defined by functional roles. Each stakeholder adapts its behavior based on feedback from the simulated marketplace, and evaluation is conducted using longitudinal and market-level signals that complement traditional effectiveness metrics.
Markets are a promising way to coordinate AI agent activity for similar reasons to those used to justify markets more broadly. In order to effectively participate in markets, agents need to have informative signals of their own ability to successfully complete a task and the cost of doing so. We propose MarketBench, a benchmark for assessing whether AI agents have these capabilities. We use a 93-task subset of SWE-bench Lite, a software engineering benchmark, with six recently released LLMs as a demonstration. These LLMs are miscalibrated on both success probability and token usage, and auctions built from these self-reports diverge from a full-information allocation. A follow-up intervention where we add information about capabilities from prior experiments to the context improves calibration, but only modestly narrows the gap to a full-information benchmark. We also document the performance of a market-based scaffolding with these LLMs. Our results point to self-assessment as a key bottleneck for market-style coordination of AI agents.
Andrey Fradkin, Rohit Krishnan
Boston University and the MIT Initiative on the Digital Economy. · Independent Researcher.
AI evaluation shapes the decisions of model providers, users, funders, and regulators. We argue that designing valid benchmarks requires contextualizing design choices in the dynamics of this ecosystem of actors. We develop a simulation architecture that combines rule-based market dynamics with LLM-driven strategic actors, building on advances in Generative Agent-Based Modeling (GABM). We model benchmarks, consumer needs, and provider capabilities as vectors over a six-dimensional capability space (reasoning, coding, knowledge, safety, communication, agentic), with structural information partitions across actors. As a case study, we apply this stylized simulation to explore benchmark holdout design. We find that moving from public benchmarks to private holdout benchmarks shrinks the gap between benchmark scores and user satisfaction on most benchmarks but widens it on a few, depending on where holdout weights shift scoring credit. We stress-test our findings at both the instrument and case-study level, drawing on the V&V framework of Sargent (2013) and GABM-specific evidence criteria. Beyond holdout design, our simulation is a hypothesis-generating sandbox for studying how evaluator and policy choices, in turn, reshape the ecosystem.
Yash Dave, Sang T. Truong, Serena Wang +1
Stanford University · University of British Columbia
The ability of large language models (LLMs) to manage and acquire economic resources remains unclear. In this paper, we introduce Market-Bench, a comprehensive benchmark that evaluates the capabilities of LLMs in economically-relevant tasks through economic and trade competition. Specifically, we construct a configurable multi-agent supply chain economic model where LLMs act as retailer agents responsible for procuring and retailing merchandise. In the procurement stage, LLMs bid for limited inventory in budget-constrained auctions. In the retail stage, LLMs set retail prices, generate marketing slogans, and provide them to buyers through a role-based attention mechanism for purchase. Market-Bench logs complete trajectories of bids, prices, slogans, sales, and balance-sheet states, enabling automatic evaluation with economic, operational, and semantic metrics. Benchmarking on 20 open- and closed-source LLM agents reveals significant performance disparities and winner-take-most phenomenon, i.e., only a small subset of LLM retailers can consistently achieve capital appreciation, while many hover around the break-even point despite similar semantic matching scores. Market-Bench provides a reproducible testbed for studying how LLMs interact in competitive markets.
Yushuo Zheng, Huiyu Duan, Zicheng Zhang +3
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory