Delayed Heuristic-Behavior-Target Coupling for Off-Policy Reinforcement Learning in Inventory Control
Organizations: Xi'an Jiaotong University · Xi’an Jiaotong University
Abstract
Many reinforcement learning methods tie data collection to the task policy, while coordinating independently optimized behavior and target policies remains challenging. Policies learned by these methods can perform unreliably in long-horizon tasks under uncertainty, such as inventory control, where failures can incur extreme costs. We introduce Heuristic Behavior Policy Optimization (HBPO), an off-policy framework that routes heuristic guidance through persistent behavior learning, coupled with target-policy, updating separately with residual-based advantage estimation and emphatic weighting. Under matched environment interaction budgets on DSIS and its risk-augmented variant DSIS-RB, HBPO recovers many high-cost initial policies and improves most non-collapse comparison groups. Cold starts achieve competitive costs without final cost collapse across all tested seeds. We identify a transmission chain in the delayed heuristic-behavior-target coupling as a key driver of recovery and improvement. Generation-level diagnostics trace heuristic uptake and subsequent selective target responses, while component ablations support the contribution of this transmission mechanism to policy recovery and improvement.
Figures & tables
| DSIS : Mean cost per period | DSIS-RB : Normalized risk-AUC cost | |||||||||||||||
| Seed cost | Across seeds | Seed cost | Across seeds | |||||||||||||
| Algorithm | 0 | 1 | 2 | 3 | 4 | Mean | Median | Std | 0 | 1 | 2 | 3 | 4 | Mean | Median | Std |
| PPO | 21.0 | 21.2 | 56.9 | 13.8k | 21.3 | 2.79k | 21.3 | 6.17k | 70.1 | 75.5 | 67.0k | 75.1 | 74.9 | 13.5k | 75.1 | 29.9k |
| H-PPO | 20.7 | 21.3 | 51.4 | 159 | 20.8 | 54.6 | 21.3 | 59.8 | 52.7 | 81.9 | 5.89k | 55.3 | 112 | 1.24k | 81.9 | 2.60k |
| A2C | 23.7 | 28.7 | 23.0k | 30.4 | 23.4 | 4.62k | 28.7 | 10.3k | 74.5 | 69.6 | 8.04k | 807 | 69.6 | 1.81k | 74.5 | 3.49k |
| H-A2C | 32.3 | 20.8 | 6.64k | 20.9 | 20.7 | 1.35k | 20.9 | 2.96k | 51.6 | 51.3 | 7.40k | 126 | 654 | 1.66k | 126 | 3.22k |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning and convention |
|---|---|
| Indices, observations, actions, and physical dynamics | |
| Learning generation; physical period; simulated period, optimization iteration, or sampled-root index, as specified locally. | |
| Critic member ; simulation-tape index; weighting role ( for target return, for transmission, and for the unweighted behavior-return role). The risk grid separately uses . | |
| Observation space; an observation and its random counterpart; successor observation and its random counterpart. In the residual theorem, is countable and closed under the frozen model’s transitions. | |
| Net inventory (negative values represent backlog) and the four-stage regular-order pipeline. | |
| Current binary regular-supply availability, its observation-coordinate accessor, and realized/random next availability. Zero means unavailable and one means available. | |
| Formula | Description | Physical or modeling role | DSIS / DSIS-RB | Baseline use |
|---|---|---|---|---|
| Physical rules and stochastic model structure | ||||
| Finite action domain. | Two ordering channels, each accepting quantities from to . | Both | All: action interface; GPS: model branches. | |
| Deterministic inventory update. | Inventory conservation, scheduled arrivals, and immediate, reliable expediting. | Both | GPS: explicit state model. | |
| Pipeline shift and order gate. | Four-period regular lead time; outages reject new regular orders without cancelling existing orders. | Both | GPS: explicit state model. | |
| Known cost and reward map. | Expediting, holding, and backlog costs; a common reward scale. | Both | GPS: model rewards; all: observed rewards. | |
| Public nominal stochastic law. | Memoryless demand and uninterrupted regular supply. | DSIS | GPS: nominal model. | |
| Method | Searched dimensions |
|---|---|
| PPO, PPO-81 | Learning rate; clip ratio ; GAE ; epochs ; rollout length ; entropy coefficient . |
| A2C | Learning rate; GAE ; rollout length ; entropy coefficient . |
| Discrete SAC | Learning rate; minibatch ; target update rate ; target entropy ratio . |
| ACE | Actor and critic learning rates; ETD ; behavior temperature . |
| TRPO | KL limit: log ; value learning rate; GAE ; rollout length . |
| GRPO | Actor and value learning rates; clip ratio ; group size ; actor epochs ; KL coefficient ; RB collection multiplier . |
| Method | Trial | Selected parameters |
|---|---|---|
| PPO | 03 | Learning rate ; clip ; GAE ; 5 epochs; rollout 512; entropy 0. |
| A2C | 04 | Learning rate ; GAE ; rollout 20; entropy . |
| Discrete SAC | 11 | Learning rate ; minibatch 512; target update rate (Polyak coefficient ); entropy ratio . |
| ACE | 01 | Actor learning rate ; critic/follow-on learning rate ; ETD ; behavior temperature . |
| PPO-81 | 04 | Learning rate ; clip ; GAE ; 5 epochs; rollout 512; entropy . |
| TRPO | 03 | KL limit ; value learning rate ; GAE ; rollout 1024. |
| Method | Trial | Selected parameters |
|---|---|---|
| PPO | 08 | Learning rate ; clip ; GAE ; 10 epochs; rollout 256; entropy 0. |
| A2C | 11 | Learning rate ; GAE ; rollout 100; entropy 0. |
| Discrete SAC | 04 | Learning rate ; minibatch 128; target update rate (Polyak coefficient ); entropy ratio . |
| ACE | 10 | Actor learning rate ; critic/follow-on learning rate ; ETD ; behavior temperature 2. |
| PPO-81 | 03 | Learning rate ; clip ; GAE ; 5 epochs; rollout 512; entropy 0. |
| TRPO | 00 | KL limit ; value learning rate ; GAE ; rollout 2048. |
| Candidate | Value learning rate | Emphatic learning rate | |
|---|---|---|---|
| H0 | |||
| H1 | |||
| H2 | |||
| H3 | |||
| H4 | |||
| H5 |
| (a) Individual training seeds | |||||
|---|---|---|---|---|---|
| Policy | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 |
| PPO | 21.00574 | 21.16667 | 56.89301 | 13817.30298 | 21.33039 |
| H-PPO | 20.73032 | 21.25826 | 51.41226 | 158.87953 | 20.75846 |
| A2C | 23.65804 | 28.68256 | 23012.57348 | 30.43475 | 23.38418 |
| H-A2C | 32.28773 | 20.76730 | 6638.13434 | 20.88466 | 20.74205 |
| Discrete SAC | 27.82865 | 36.65902 | 38.56467 | 27.45829 | 345.08867 |
| (a) Individual training seeds | |||||
|---|---|---|---|---|---|
| Policy | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 |
| PPO | 70.061774 | 75.460918 | 66970.754322 | 75.131193 | 74.882238 |
| H-PPO | 52.679426 | 81.929137 | 5893.013522 | 55.296788 | 112.014382 |
| A2C | 74.484857 | 69.637995 | 8036.258056 | 807.481221 | 69.608571 |
| H-A2C | 51.574127 | 51.333329 | 7397.851829 | 125.837686 | 653.867411 |
| Discrete SAC | 1183.646256 | 10969.145360 | 71.162997 | 69.717788 | 101.754500 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,847 papersCross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
avg@32 and pass@32 performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged avg@32 and pass@32 over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.