Differentiable Agent-Based Modeling for Multi-Echelon Supply Chains: Gradient-Based Optimization of Heterogeneous Policies at Scale
Authors: Bulent Soykan, Murtadha Alsayegh, Sean Mondesire, Ghaith Rabadi
Organizations: Institute for Simulation and Training, University of Central Florida, Orlando, USA · School of Modeling, Simulation, and Training, University of Central Florida, Orlando, USA
Global supply chains rely on the interactions of thousands of heterogeneous agents, posing significant challenges for optimization. While traditional optimization often assumes homogeneity to ensure tractability, Reinforcement Learning struggles with sample inefficiency when scaling to high-dimensional state spaces. We propose a Differentiable Agent-Based Model for multi-echelon inventory management. By implementing stochastic dynamics via differentiable tensor operations, we enable direct gradient propagation through the simulation. We validate the framework by recovering theoretical Newsvendor optima without supervision. Crucially, we demonstrate ``Mass Personalization'' by optimizing 10,000 distinct agent policies simultaneously in seconds, achieving a 53% cost reduction compared to scalar baselines. Furthermore, we utilize gradient sensitivity analysis to generate causal heatmaps, accurately recovering inherent lead-time delays from the computation graph. This approach bridges the gap between simulation and optimization, enabling the efficient training of self-correcting Digital Twins.
Figures & tables
Figure 1: Automatic Policy Disentanglement. Starting from an identical initialization ( θ=12 ), the gradient descent algorithm automatically drives the policies of the two groups in opposite directions. Group A (Blue) converges downward to the low-demand optimum, while Group B (Red) converges upward to the high-demand optimum. The model successfully disentangles the population based solely on the cost gradients flowing from the environment.
Figure 2: Mass Personalization at Scale. (Left) The convergence curves show the Scalar policy (Gray) plateauing at a high cost, while the Heterogeneous policy (Blue) continues to improve, achieving a 53% reduction in total loss. (Right) The policy distribution explains the performance gap. The Scalar policy settles for a "least worst" average ( ≈24.2 ), largely overstocking low-demand agents to prevent costly stockouts at the high end. In contrast, the Heterogeneous policy learns a perfect linear mapping, tailoring inventory targets to the specific demand mean of each individual agent.
Figure 3: The Resilience Horizon. We induce a demand shock at t=15 (Red Dashed Line). The curve shows the gradient magnitude, indicating the impact of factory production at time t on minimizing the eventual shortage. The gradient remains high for all steps preceding the shock, confirming the model respects conservation of mass (inventory carryover). The decay after t=15 reveals the “Resilience Horizon”—the duration required for the system to recover from the deficit.