cs.LGNov 11, 2026

Reinforcement Learning for Inventory Control under Regime-Switching Demand: A Controlled Comparison of LSTM, MLP, and (s,S) Policies

Authors: Alex Haines, Nathanael L. Baisa

Organizations: School of Computer Science and Informatics, De Montfort University, Leicester LE1 9BH, UK

Abstract

Classical inventory policies such as (s,S) are provably optimal under stationary demand but cannot adapt when demand switches between hidden regimes, and memoryless deep reinforcement learning policies, confined to a fixed observation window, cannot maintain a belief over latent demand structure. We ask whether a recurrent policy offers a measurable advantage over both a memoryless policy and a classical baseline under regime-switching demand, and how that advantage scales with supply-chain lead time. We build a custom Gymnasium environment in which daily demand is generated by a two-state hidden Markov process with Poisson emissions, so that the regime is known at evaluation time but hidden from the agent. An optimised (s,S) heuristic, an MLP PPO agent, and an LSTM Recurrent PPO agent are trained under matched compute and evaluated at lead times of 0, 3, 6, and 10 days over 30 deterministic episodes each. The recurrent policy outperforms both alternatives at every lead time, and its margin over the MLP widens monotonically with lead time, from parity at lead time 0 to 2.17×\times at lead time 10. Principal component analysis of the LSTM hidden state explains the widening gap: the representation shifts from encoding a binary regime indicator at short lead times to encoding multi-step pipeline state at long lead times. Recurrent memory becomes the decisive architectural factor as the credit-assignment horizon lengthens.

Figures & tables

Explore similar work

CardsList
  1. DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Management

    Mar 20, 2026Yaqi Xie, Xinru Hao, Jiaxi Liu +4Inventory ControlReinforcement Learning

  2. Reinforcement Learning in MDPs with Information-Ordered Policies

    Aug 5, 2025Zhongjun Zhang, Shipra Agrawal, Ilan Lobel +2Queueing TheoryPolicy Learning

  3. gym-invmgmt: An Open Benchmarking Framework for Inventory Management Methods

    May 12, 2026Reza Barati, Qinmin Vivian HuInventory ControlRL Benchmarks