Differentiable Agent-Based Modeling for Multi-Echelon Supply Chains: Gradient-Based Optimization of Heterogeneous Policies at Scale
Authors: Bulent Soykan, Murtadha Alsayegh, Sean Mondesire, Ghaith Rabadi
Organizations: Institute for Simulation and Training, University of Central Florida, Orlando, USA · School of Modeling, Simulation, and Training, University of Central Florida, Orlando, USA
Global supply chains rely on the interactions of thousands of heterogeneous agents, posing significant challenges for optimization. While traditional optimization often assumes homogeneity to ensure tractability, Reinforcement Learning struggles with sample inefficiency when scaling to high-dimensional state spaces. We propose a Differentiable Agent-Based Model for multi-echelon inventory management. By implementing stochastic dynamics via differentiable tensor operations, we enable direct gradient propagation through the simulation. We validate the framework by recovering theoretical Newsvendor optima without supervision. Crucially, we demonstrate ``Mass Personalization'' by optimizing 10,000 distinct agent policies simultaneously in seconds, achieving a 53% cost reduction compared to scalar baselines. Furthermore, we utilize gradient sensitivity analysis to generate causal heatmaps, accurately recovering inherent lead-time delays from the computation graph. This approach bridges the gap between simulation and optimization, enabling the efficient training of self-correcting Digital Twins.
Figures & tables
Figure 1: Automatic Policy Disentanglement. Starting from an identical initialization ( θ=12 ), the gradient descent algorithm automatically drives the policies of the two groups in opposite directions. Group A (Blue) converges downward to the low-demand optimum, while Group B (Red) converges upward to the high-demand optimum. The model successfully disentangles the population based solely on the cost gradients flowing from the environment.
Figure 2: Mass Personalization at Scale. (Left) The convergence curves show the Scalar policy (Gray) plateauing at a high cost, while the Heterogeneous policy (Blue) continues to improve, achieving a 53% reduction in total loss. (Right) The policy distribution explains the performance gap. The Scalar policy settles for a "least worst" average ( ≈24.2 ), largely overstocking low-demand agents to prevent costly stockouts at the high end. In contrast, the Heterogeneous policy learns a perfect linear mapping, tailoring inventory targets to the specific demand mean of each individual agent.
Figure 3: The Resilience Horizon. We induce a demand shock at t=15 (Red Dashed Line). The curve shows the gradient magnitude, indicating the impact of factory production at time t on minimizing the eventual shortage. The gradient remains high for all steps preceding the shock, confirming the model respects conservation of mass (inventory carryover). The decay after t=15 reveals the “Resilience Horizon”—the duration required for the system to recover from the deficit.
Modern supply chain operations can require coordinating replenishment across thousands of heterogeneous items under correlated stochastic demand, heterogeneous lead times, and shared fixed ordering costs, yielding observation spaces exceeding 104 dimensions. At this scale, rolling-horizon stochastic mixed-integer linear programs (MILPs) become prohibitively slow, while standard reinforcement learning (RL) methods face increasingly challenging credit assignment in high-dimensional action spaces. We introduce OR-Transformer, a deep reinforcement learning framework for joint replenishment under stochastic demand, with an item-permutation-equivariant Transformer architecture and pathwise-gradient training through the inventory dynamics. Across problem sizes up to 1,024 inventory items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines as scale grows. It also reduces online decision-making time by over 4 million times relative to MILP solvers, enabling real-time, large-scale deep RL in supply chain operations.
Shuze Daniel Liu, David Simchi-Levi, Claire Chen +2
Massachusetts Institute of Technology · Purdue University · California Institute of Technology +1
Large language model (LLM)-based multi-agent systems (MAS) achieve strong performance by employing specialized multiple agents, yet their performance depends on the prompt design of each agent. For MAS prompt optimization, textual gradient methods that guide prompt updates using natural-language feedback have emerged as a leading paradigm. In this paper, we identify limitations in two stages of existing textual gradient approaches: gradient extraction and gradient aggregation. In gradient extraction, previous works select a prompt to update without verifying whether modifying it resolves the failure, and derive gradients without agent-level supervision over the corresponding agent's intermediate output. In gradient aggregation, individual gradients are randomly grouped and concatenated, often mixing unrelated failure modes and producing prompts that fail to generalize. To address these limitations, we propose AgentGrad, a prompt optimization framework for multi-agent systems based on sequential intervention and semantic textual gradient abstraction. For each failure, sequential intervention injects a hint into one agent at a time to identify the target agent whose modification resolves the failure. The modified output of the target agent then serves as agent-level supervision for extracting a fine-grained gradient. Semantic textual gradient abstraction clusters semantically similar gradients to prevent mixing unrelated failure modes, and abstracts each cluster into a generalized gradient that captures the shared corrective pattern. Experimental results show that AgentGrad achieves state-of-the-art performance across five MAS benchmarks while reducing wall-clock optimization time by 2.5 times and optimization cost by 21.8% on average compared to the next-best baselines. Our code is available at https://github.com/mlvlab/AgentGrad.
This paper studies the performance and reliability of autonomous generative AI agents in multi-echelon supply chains using the MIT Beer Game. We examine how model choice, operational guardrails, centralized data sharing, and prompt design affect system performance. In our best-performing configuration, GenAI agents reduce total supply-chain costs by up to 80% relative to human teams. Despite strong average performance, autonomous agents can exhibit substantial run-to-run instability, generating volatile procurement decisions and large tail costs. We characterize this phenomenon as agent bullwhip, the amplification of decision instability in autonomous multi-agent systems. We show that this instability can propagate across echelons and compound over time, even when the underlying demand path is held fixed. We then evaluate two approaches for improving reliability: reinforcement-learning post-training and operational guardrails. Both reduce tail events and mitigate agent bullwhip, but they operate through different mechanisms and require different levels of information and model access. Reinforcement-learning post-training delivers the largest gains in reliability and system performance when system-level feedback is available, while guardrails provide a simple training-free alternative for constraining extreme decisions.