Optimizing the consolidation process in container-based fulfillment centers requires trading off competing objectives such as processing speed, resource usage, and space utilization while adhering to a range of real-world operational constraints. This process involves moving items between containers via a combination of human and robotic workstations to free up space for inbound inventory and increase container utilization. We formulate this problem as a large-scale Multi-Objective Reinforcement Learning (MORL) task with high-dimensional state spaces and dynamic system behavior. Our method builds on recent theoretical advances in solving constrained RL problems via best-response and no-regret dynamics in zero-sum games, enabling principled minimax policy learning. Policy evaluation on realistic warehouse simulations shows that our approach effectively trades off objectives, and we empirically observe that it learns a single policy that simultaneously satisfies all constraints, even if this is not theoretically guaranteed. We further introduce a theoretical framework to handle the problem of error cancellation, where time-averaged solutions display oscillatory behavior. This method returns a single iterate whose Lagrangian value is close to the minimax value of the game. These results demonstrate the promise of MORL in solving complex, high-impact decision-making problems in large-scale industrial systems.
Figures & tables
Figure 1: Left: High-throughput container-based fulfillment center with dense human-robot collaboration. Right: Robotic consolidation station autonomously transferring items between totes.
Figure 2: Human-robot collaborative fulfillment center workflow.
Figure 3: Episodic returns for DQN in the single-objective ETPH setting, showing unnormalized performance of a best-response policy optimizing ETPH.
Figure 4: Average Lagrange multipliers λˉ over training rounds. The Nlarge and robot capacity multipliers remain at zero, while the remaining constraints exhibit significant oscillations.
Figure 5: Policy performance over repeated game rounds. (a) Performance of individual policies at each round. (b) Performance of time averaged policies. Each plot shows the global ETPH objective and four constraint values. Black horizontal lines denote constraint thresholds, red markers indicate violations, and green vertical regions indicate rounds where all constraints are simultaneously satisfied.
Figure 6: Phase diagram of ETPH versus constraint satisfaction across rounds. Color indicates training rounds, red dashed lines denote constraint thresholds. As training progresses, time averaged policies trade ETPH for improved S/D and manual capacity satisfaction.
Figure 7: Average episodic return over rounds for time averaged policies parameterized by λˉ , showing improved performance as policies are mixed across rounds.
Figure 8: Feasible stationary policies over training rounds. Although feasibility is guaranteed only in distribution, many individual policies are feasible in practice. Best, average, and worst over 20 seeds are shown.
MORL (single policy)
Random Actions
Unconstrained
Satisfies all constraints?
√
×
×
ETPH
20.52
9.19
61.81
Nlarge Slack
362.62
359.15
370.62
S/D Slack
10.81
0.49
-0.37
Manual Cap. Slack
83.21
-32.99
-563.23
Robot Cap. Slack
258.23
571.01
743.68
Table 1: Comparison of ETPH and constraint satisfaction across MORL, random, and unconstrained policies. The unconstrained policy attains higher ETPH at the cost of severe capacity violations.
Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL's application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging.
Neil Janwani, Ellen Novoseller, Vernon J. Lawhern +1
Georgia Institute of Technology, Atlanta, GA USA · DEVCOM Army Research Laboratory, MD, USA
Multi-objective reinforcement learning (MORL) is effective for multi-echelon combinatorial supply chain optimisation, where tasks involve high dimensionality, uncertainty, and competing objectives. However, its deployment in dynamic environments is hindered by the need for task-specific retraining and substantial computational cost. We introduce MIRACL (Meta multI-objective Reinforcement leArning with Composite Learning), a hierarchical Meta-MORL framework that allows for a few-shot generalisation across diverse tasks. MIRACL decomposes each task into structured subproblems for efficient policy adaptation and meta-learns a global policy across tasks using a Pareto-based adaptation strategy to encourage diversity in meta-training and fine-tuning. To our knowledge, this is the first integration of Meta-MORL with such mechanisms in combinatorial optimisation. Although validated in the supply chain domain, MIRACL is theoretically domain-agnostic and applicable to broader dynamic multi-objective decision-making problems. Empirical evaluations show that MIRACL outperforms conventional MORL baselines in simple to moderate tasks, achieving up to 10% higher hypervolume and 5% better expected utility. These results underscore the potential of MIRACL for robust, efficient adaptation in multi-objective problems.
Rifny Rachman, Josh Tingey, Richard Allmendinger +3
The University of Manchester, United Kingdom · Peak AI, Ltd, United Kingdom
Multi-Objective Reinforcement Learning (MORL) extends standard RL by optimizing policies with respect to multiple, often conflicting, objectives. While max-min MORL has emerged as an effective approach for promoting fairness, its applicability remains limited, particularly when constraints must be incorporated. In this paper, we propose a MORL framework that integrates the max-min criterion with explicit constraint satisfaction. We establish a theoretical foundation for the proposed framework and validate the resulting algorithm through convergence analysis and experiments in tabular settings. We further demonstrate the practical relevance of our approach in simulated building thermal control, multi-objective locomotion control, and greenhouse-gas-emission-aware traffic management. Across these domains, our method effectively balances fairness and constraint satisfaction in multi-objective decision-making.
Giseung Park, Hyunyoung Nam, Woohyeon Byeon +2
Robotics Institute, University of Toronto, 55 St George St, Toronto, Canada. · School of Electical Engineering, KAIST, Daejeon 34141, Republic of Korea. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 52900, Israel.