cs.MANov 17, 2026

MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain

Authors: Sara Amar

Organizations: Department of Engineering, Bowling Green State University

Abstract

We introduce MANGO Arena, an open-source testbed in which four LLM agents each run one company in a serial supply chain adapted from MANGO, a classroom factory-management game. The environment enforces hard rules (a truck limit, a no-credit cash constraint and production capacity), logs every blocked attempt to break them, and gives agents a public message board on which side payments are explicitly forbidden. This lets one environment measure three things at once: an economic outcome (the bullwhip effect), rule adherence under pressure, and proposals to collude. In a pilot with three small reasoning models, two scripted reference policies, two conditions (with and without a supplier fire) and three demand seeds, all LLM agents amplified demand variability as it moved upstream, by 7 to 25 times at the Retailer and 13 to 179 times at the Supplier, often more than a textbook order-up-to policy. They did so largely by ordering in batches (zero orders in up to 60% of decisions). Real rule violations were rare (at most 7 of 288 decisions per model) and no agent proposed a money side deal. The pilot's main lesson concerns evaluation method. API failures (rate limits and runaway hidden reasoning) entered the metrics silently as "no order" decisions. And when we reran games with the same seeds and no failed calls, upstream bullwhip changed by up to 17 times between runs. With three seeds, conclusions about how a disruption affects a given model did not survive a rerun. We report these failures in full and propose reporting practices for multi-agent evaluations of this kind. Code, prompts and every logged decision are public.

Figures & tables

Explore similar work

CardsList
  1. Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

    Aug 9, 2026Yijun Pan, Yukun Lian, Kunyu Shi +5LLM Agent EvaluationAI Agent Benchmarks

  2. Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management

    May 16, 2026Carol Xuan Long, David Simchi-Levi, Feng Zhu +3AI Agent ReliabilityMulti-Agent Systems