MIRACL: A Diverse Meta-Reinforcement Learning for Multi-Objective Multi-Echelon Combinatorial Supply Chain Optimisation

Rifny Rachman; Josh Tingey; Richard Allmendinger; Wei Pan; Pradyumn Shukla; Bahrul Ilmi Nasution

cs.LGMar 5, 2026

MIRACL: A Diverse Meta-Reinforcement Learning for Multi-Objective Multi-Echelon Combinatorial Supply Chain Optimisation

Authors: Rifny Rachman, Josh Tingey, Richard Allmendinger, Wei Pan, Pradyumn Shukla, Bahrul Ilmi Nasution

Organizations: The University of Manchester, United Kingdom · Peak AI, Ltd, United Kingdom

Abstract

Multi-objective reinforcement learning (MORL) is effective for multi-echelon combinatorial supply chain optimisation, where tasks involve high dimensionality, uncertainty, and competing objectives. However, its deployment in dynamic environments is hindered by the need for task-specific retraining and substantial computational cost. We introduce MIRACL (Meta multI-objective Reinforcement leArning with Composite Learning), a hierarchical Meta-MORL framework that allows for a few-shot generalisation across diverse tasks. MIRACL decomposes each task into structured subproblems for efficient policy adaptation and meta-learns a global policy across tasks using a Pareto-based adaptation strategy to encourage diversity in meta-training and fine-tuning. To our knowledge, this is the first integration of Meta-MORL with such mechanisms in combinatorial optimisation. Although validated in the supply chain domain, MIRACL is theoretically domain-agnostic and applicable to broader dynamic multi-objective decision-making problems. Empirical evaluations show that MIRACL outperforms conventional MORL baselines in simple to moderate tasks, achieving up to 10% higher hypervolume and 5% better expected utility. These results underscore the potential of MIRACL for robust, efficient adaptation in multi-objective problems.

Figures & tables

Figure 1: Meta-learning and adaptation phases in MIRACL. Each task is decomposed using several weight vectors, then PSA is applied between meta-updates to improve the task diversity.

Figure 2: Fine-tuning phase requires only a few shots training since it utilises the meta-policy as a good initialisation.

Complexities	Action	Observation	Nodes	Edges
Simple	8	20	7	8
Moderate	21	49	13	21
Complex	59	131	24	59

Table 1: SC environments simulated in our experiment. Each environment complexity corresponds with the action and observation space dimension, number of nodes, and edges.

Panel of Figure 3 — Figure 3: Normalised hypervolume comparison of MORL/D, MORL/D with SB and PSA, NSGA-II, Meta-MORL, and our proposed methods. Left panels show early time steps (proportional to NSGA-II generations), while right panels show later iterations. MIRACL consistently outperforms all baseline methods in simple ( 3(a) ) and moderate problems ( 3(b) ), but is exceeded by MORL/D in complex ones ( 3(c) ).

Algorithms	Simple	Moderate	Complex
MIRACL(MT)	8	12	14
MIRACL(FT)	14	47	77
MORL/D	84	95	111
NSGA-II	2	10	47

Table 2: Runtime comparison (in minutes) between MIRACL and traditional algorithms. Meta-training time (MT) is reported separately as a one-off overhead (using 4 parallel workers), while fine-tuning time (FT) reflects the per-task cost.

Problems	Aspects	Meta-MORL	MT	MT&FT	$\Delta$ MT	$\Delta$ MT&FT
Simple	Hypervolume	0.4333	0.4460	0.4600	2.92%	6.16% **
	Sparsity	0.0056	0.0052	0.0057	-8.03%	1.36 %
	EUM	0.7075	0.7398	0.7436	4.57%	5.10% **
Moderate	Hypervolume	0.6737	0.6853	0.7306	1.73%	8.45% ***
	Sparsity	0.0015	0.0025	0.0013	71.75 %*	-7.78%
	EUM	0.8387	0.8538	0.8394	2.75% *	1.03%

Table 3: Performance comparison between Meta-MORL and MIRACL. MIRACL is evaluated with PSA applied only in meta-training (MT) or in both meta-training and fine-tuning (MT&FT). We report hypervolume, sparsity, and EUM, with

\Delta

indicating the percentage change relative to Meta-MORL. Statistical significance is assessed via a Kruskal–Wallis test followed by Dunn’s post hoc tests with Bonferroni correction, where *, **, and *** denote

p<0.05

p<0.01

, and

p<0.001

, respectively.

Panel of Figure 4 — Figure 4: PF approximation sets of NSGA-II, MORL/D, Meta-MORL, and MIRACL across simple, moderate, and complex SC problems. Meta-learning-based methods produce more diverse solutions than NSGA-II, yet, more concentrated than MORL/D. The concentration increases with the problem complexity.

Algorithm	Hypervolume	Sparsity	EUM
mo-hopper-v4
MIRACL	0.1108*	0.0234*	0.4003*
Meta-MORL	0.0756	0.0047	0.3718
mo-halfcheetah-v4
MIRACL	0.7002	0.0143	0.6910
Meta-MORL	0.6556	0.0114	0.6655

Table 4: Cross-domain benchmarks (mean over 10 runs). The asterisk (*) denote statistically significantly different results using the Wilcoxon test on

p<0.05

, respectively against Meta-MORL.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Definition
$q^{\tau}_{tij}$	Transport quantity at period $t$ , from node $i$ to $j$
$q^{\psi}_{tk}$	Manufacturing quantity at period $t$ , at manufacturer $k$
$Prof$	Total profit of all periods
$E$	Total emission of all periods
$F$	Total SL inequality measure of all periods
$SL_{y}$	SL at node $y$ of all periods

Appendix

Table 1: The nodes

i

j

, and

k

symbolise the SC network facilities, with period

t

as the daily interval. The variables

\varsigma

\psi

, and

\tau

signify inventory, production, and transport activities.

Node	$I_{0j}$	$c_{j}^{\varsigma}$	$e^{\varsigma}_{j}$	$c_{k}^{\psi}$	$v_{k}^{\psi}$	$e^{\psi}_{k}$
3	380	0.1100	0.0002	2.0000	1.0000	5.0126
4	350	0.1300	0.0002	2.2000	1.0000	4.5754
5	400	0.1200	0.0002	NA	NA	NA
6	80	0.1500	0.0002	NA	NA	NA

Appendix

Table 2: Simple SC. Node parameter values that correspond to the facilities in Simple SC environment.

From Node	To Node	$c^{\tau}_{ij}$	$e^{\tau}_{ij}$
1	2	0.2200	0.1258
1	3	0.6900	0.3947
2	4	1.0550	0.6035
2	5	0.4300	0.2460
3	4	0.4850	0.2774
3	5	0.7500	0.4290

Appendix

Table 3: Transport parameter values that correspond to product delivery.

Node	$I_{0j}$	$c_{j}^{\varsigma}$	$e^{\varsigma}_{j}$	$c_{k}^{\psi}$	$v_{k}^{\psi}$	$e^{\psi}_{k}$
3	380	0.1100	0.0002	2.0000	1.0000	5.0126
4	350	0.1300	0.0002	2.2000	1.0000	4.5754
5	400	0.1200	0.0002	2.3000	1.0000	5.4491
6	80	0.1500	0.0002	NA	NA	NA
7	110	0.2000	0.0002	NA	NA	NA
8	100	0.2500	0.0002	NA	NA	NA

Appendix

Table 4: Moderate SC. Node parameter values that correspond to the facilities in Moderate SC environment.

From Node	To Node	$c^{\tau}_{ij}$	$e^{\tau}_{ij}$
1	3	0.2200	0.1258
	4	0.6900	0.3947
	5	0.5650	0.3232
2	3	1.0550	0.6035
	4	0.6500	0.3718
	5	0.6300	0.3604

Appendix

Table 5: Transport parameter values that correspond to product delivery.

Node	$I_{0j}$	$c_{j}^{\varsigma}$	$e^{\varsigma}_{j}$	$c_{k}^{\psi}$	$v_{k}^{\psi}$	$e^{\psi}_{k}$
4	155	0.2300	0.0002	2.0000	1.0000	5.0126
5	267	0.3500	0.0002	2.2000	1.0000	4.5754
6	342	0.2200	0.0002	2.1000	1.0000	5.4491
7	211	0.1100	0.0002	2.0000	1.0000	6.1232
8	162	0.2900	0.0002	2.3000	1.0000	5.5157
9	195	0.3700	0.0002	NA	NA	NA

Appendix

Table 6: Complex SC. Node parameter values that correspond to the facilities in Complex SC environment.

From Node	To Node	$c^{\tau}_{ij}$	$e^{\tau}_{ij}$
1	4	0.5350	0.3060
	5	0.2650	0.1516
	6	1.8450	1.0553
	7	1.6000	0.9152
	8	1.4400	0.8237
2	4	0.3600	0.2059

Appendix

Table 7: Transport parameter values that correspond to product delivery.

Hyperparameter	Value
Inner-loop learning rate $\alpha$	0.003
Outer-loop learning rate $\beta$	0.001
PPO steps per meta-update	10
Adaptation steps	4–8
Network architecture	[64, 64]
Discount factor $\gamma$	0.99

Appendix

Table 8: Hyperparameters for MAML-based meta-training.

Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Steps per update	2048
Minibatch size	64
Epochs per update	10
Discount factor $\gamma$	0.99
GAE weight $\lambda$	0.95

Appendix

Table 9: PPO hyperparameters used in fine-tuning.

Hyperparameters	Values
MORL/D
Discount factor	0.995
Population size	6
Exchange every	$5\times 10^{4}$
Total time steps	$5\times 10^{5}$
Neighbourhood size	1

Appendix

Table 10: Hyperparameters of MORL/D Algorithm with MOSAC Policy

Hyperparameters	Values
Population size	300
Offspring number	30
Cross-over operator	binary (SBX)
Cross-over probability	90%
Cross-over $\eta$	15
Mutation method	polynomial

Appendix

Table 11: Hyperparameters of NSGA-II to solve the SC problem across three network complexities: simple, moderate, and complex.

Panel of Figure 1 — Figure 1: Sparsity and EUM of NSGA-II, MORL/D, MORL/D with SB and PSA, Meta-MORL, and our proposed algorithms. The left parts are zoomed in on the early timesteps with meta-learning, while the right parts show MORL/D timesteps and NSGA-II generations. The EUM reveal that meta-learning methods outperform MORL/D in simpler problems but increasingly struggle with more complex ones. Normalised sparsity consistently shows the lowest values for meta-learning-based methods across all complexities.

Number of subproblems $K$
Setting	Hypervolume	Sparsity	EUM
$K=3$	0.3832***	0.0088	0.7156
$K=5$	0.4003*	0.0064	0.7146
$\mathbf{K=10}$	0.4387	0.0076	0.7141
PSA steps $S$
$S=3$	0.4371	0.0060	0.7055**

Appendix

Table 12: Ablation study for MIRACL on simple SC (mean over 10 runs), where *, **, *** denote statistically significant differences:

p<0.05

p<0.01

, and

p<0.001

, respectively. The Kruskal–Wallis followed with Dunn’s post hoc (Bonferroni-corrected) is used for data sets

>2

, and the Wilcoxon test for data sets

=2

. Bold text represents the baseline value of each parameter.

Panel of Figure 2 — Figure 2: PF approximation sets between algorithms in all problem complexities. In general, meta-learning-based algorithms reside in a narrower solution space. The solutions consistently spread in the range of moderate to high values of profit and GHG emissions, while low to moderate values of SL inequality in all problem complexities.

Panel of Figure 5 — Figure 5: Unmet demand resulting from the three algorithms across various problem complexities. Our proposed method mostly shows proximity values to MORL/D, which is lower than NSGA-II, especially in the simpler problems.

Explore similar work

Meta-Reinforcement Learning via Evolution for Multi-Objective Combinatorial Supply Chain Optimisation
Jun 20, 2026Rifny Rachman, Bahrul Ilmi Nasution, Josh Tingey +3Multi-Objective Reinforcement Learning Multi-Objective Evolutionary Optimization
The University of Manchester, United Kingdom · Peak AI, Ltd, United Kingdom
A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Apr 27, 2026Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2Multi-Objective Reinforcement Learning Reinforcement Learning
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology
PRISM: Parallel Reward Integration with Symmetry for MORL
Feb 20, 2026Finn van der Knaap, Kejiang Qian, Zheng Xu +1Reinforcement Learning Multi-Objective Reinforcement Learning
University of Edinburgh · Meta Superintelligence Labs.

Jun 20, 2026J/K move · Enter open · S save

Meta-Reinforcement Learning via Evolution for Multi-Objective Combinatorial Supply Chain Optimisation

Rifny Rachman, Bahrul Ilmi Nasution, Josh Tingey, Richard Allmendinger+2

The University of Manchester, United Kingdom · Peak AI, Ltd, United Kingdom

Multi-Objective Reinforcement Learning Multi-Objective Evolutionary Optimization Combinatorial Optimization Meta-Reinforcement Learning RL for Combinatorial Optimization Multi-Objective Optimization

Open paper AskPDF arXiv

Meta-reinforcement learning is a promising approach to multi-objective optimisation because it enables rapid policy adaptation across changing environments and preference settings. However, conventional few-shot methods usually fine-tune from a single shared meta-policy, which can reduce solution diversity and limit exploration of the Pareto front, especially in high-dimensional combinatorial problems such as supply chain optimisation. We propose a population-based Meta-reinforcement learning framework that combines decomposition with evolutionary search in scalarisation weight space. The framework maintains a population of weight vectors, each associated with a distinct meta-policy trained through gradient-based meta-learning, and iteratively refines this population through elitist selection, crossover, and mutation guided by hypervolume and entropy contributions. We evaluate the method in a multi-objective supply chain setting with conflicting economic, environmental, and social goals, and further test its generality on standard reinforcement learning problems. The results show that the proposed approach yields more diverse, better distributed Pareto front approximations, improves cross-task adaptation, increases hypervolume by up to 32% over Meta-multi-objective reinforcement learning in the complex case, and attains the lowest average Hausdorff distance among all compared methods.

Abstract

Explore similar work

Meta-Reinforcement Learning via Evolution for Multi-Objective Combinatorial Supply Chain Optimisation

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning

PRISM: Parallel Reward Integration with Symmetry for MORL