Multi-objective reinforcement learning (MORL) is effective for multi-echelon combinatorial supply chain optimisation, where tasks involve high dimensionality, uncertainty, and competing objectives. However, its deployment in dynamic environments is hindered by the need for task-specific retraining and substantial computational cost. We introduce MIRACL (Meta multI-objective Reinforcement leArning with Composite Learning), a hierarchical Meta-MORL framework that allows for a few-shot generalisation across diverse tasks. MIRACL decomposes each task into structured subproblems for efficient policy adaptation and meta-learns a global policy across tasks using a Pareto-based adaptation strategy to encourage diversity in meta-training and fine-tuning. To our knowledge, this is the first integration of Meta-MORL with such mechanisms in combinatorial optimisation. Although validated in the supply chain domain, MIRACL is theoretically domain-agnostic and applicable to broader dynamic multi-objective decision-making problems. Empirical evaluations show that MIRACL outperforms conventional MORL baselines in simple to moderate tasks, achieving up to 10% higher hypervolume and 5% better expected utility. These results underscore the potential of MIRACL for robust, efficient adaptation in multi-objective problems.
Figures & tables
Figure 1: Meta-learning and adaptation phases in MIRACL. Each task is decomposed using several weight vectors, then PSA is applied between meta-updates to improve the task diversity.
Figure 2: Fine-tuning phase requires only a few shots training since it utilises the meta-policy as a good initialisation.
Complexities
Action
Observation
Nodes
Edges
Simple
8
20
7
8
Moderate
21
49
13
21
Complex
59
131
24
59
Table 1: SC environments simulated in our experiment. Each environment complexity corresponds with the action and observation space dimension, number of nodes, and edges.
Figure 3: Normalised hypervolume comparison of MORL/D, MORL/D with SB and PSA, NSGA-II, Meta-MORL, and our proposed methods. Left panels show early time steps (proportional to NSGA-II generations), while right panels show later iterations. MIRACL consistently outperforms all baseline methods in simple ( 3(a) ) and moderate problems ( 3(b) ), but is exceeded by MORL/D in complex ones ( 3(c) ).
Algorithms
Simple
Moderate
Complex
MIRACL(MT)
8
12
14
MIRACL(FT)
14
47
77
MORL/D
84
95
111
NSGA-II
2
10
47
Table 2: Runtime comparison (in minutes) between MIRACL and traditional algorithms. Meta-training time (MT) is reported separately as a one-off overhead (using 4 parallel workers), while fine-tuning time (FT) reflects the per-task cost.
Problems
Aspects
Meta-MORL
MT
MT&FT
Δ MT
Δ MT&FT
Simple
Hypervolume
0.4333
0.4460
0.4600
2.92%
6.16% **
Sparsity
0.0056
0.0052
0.0057
-8.03%
1.36 %
EUM
0.7075
0.7398
0.7436
4.57%
5.10% **
Moderate
Hypervolume
0.6737
0.6853
0.7306
1.73%
8.45% ***
Sparsity
0.0015
0.0025
0.0013
71.75 %*
-7.78%
EUM
0.8387
0.8538
0.8394
2.75% *
1.03%
Table 3: Performance comparison between Meta-MORL and MIRACL. MIRACL is evaluated with PSA applied only in meta-training (MT) or in both meta-training and fine-tuning (MT&FT). We report hypervolume, sparsity, and EUM, with Δ indicating the percentage change relative to Meta-MORL. Statistical significance is assessed via a Kruskal–Wallis test followed by Dunn’s post hoc tests with Bonferroni correction, where *, **, and *** denote p<0.05 , p<0.01 , and p<0.001 , respectively.
Figure 4: PF approximation sets of NSGA-II, MORL/D, Meta-MORL, and MIRACL across simple, moderate, and complex SC problems. Meta-learning-based methods produce more diverse solutions than NSGA-II, yet, more concentrated than MORL/D. The concentration increases with the problem complexity.
Algorithm
Hypervolume
Sparsity
EUM
mo-hopper-v4
MIRACL
0.1108*
0.0234*
0.4003*
Meta-MORL
0.0756
0.0047
0.3718
mo-halfcheetah-v4
MIRACL
0.7002
0.0143
0.6910
Meta-MORL
0.6556
0.0114
0.6655
Table 4: Cross-domain benchmarks (mean over 10 runs). The asterisk (*) denote statistically significantly different results using the Wilcoxon test on p<0.05 , respectively against Meta-MORL.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Notations
Definition
qtijτ
Transport quantity at period t , from node i to j
qtkψ
Manufacturing quantity at period t , at manufacturer k
Prof
Total profit of all periods
E
Total emission of all periods
F
Total SL inequality measure of all periods
SLy
SL at node y of all periods
Appendix
Table 1: The nodes i , j , and k symbolise the SC network facilities, with period t as the daily interval. The variables ς , ψ , and τ signify inventory, production, and transport activities.
Node
I0j
cjς
ejς
ckψ
vkψ
ekψ
3
380
0.1100
0.0002
2.0000
1.0000
5.0126
4
350
0.1300
0.0002
2.2000
1.0000
4.5754
5
400
0.1200
0.0002
NA
NA
NA
6
80
0.1500
0.0002
NA
NA
NA
Appendix
Table 2: Simple SC. Node parameter values that correspond to the facilities in Simple SC environment.
From Node
To Node
cijτ
eijτ
1
2
0.2200
0.1258
3
0.6900
0.3947
2
4
1.0550
0.6035
5
0.4300
0.2460
3
4
0.4850
0.2774
5
0.7500
0.4290
Appendix
Table 3: Transport parameter values that correspond to product delivery.
Node
I0j
cjς
ejς
ckψ
vkψ
ekψ
3
380
0.1100
0.0002
2.0000
1.0000
5.0126
4
350
0.1300
0.0002
2.2000
1.0000
4.5754
5
400
0.1200
0.0002
2.3000
1.0000
5.4491
6
80
0.1500
0.0002
NA
NA
NA
7
110
0.2000
0.0002
NA
NA
NA
8
100
0.2500
0.0002
NA
NA
NA
Appendix
Table 4: Moderate SC. Node parameter values that correspond to the facilities in Moderate SC environment.
From Node
To Node
cijτ
eijτ
1
3
0.2200
0.1258
4
0.6900
0.3947
5
0.5650
0.3232
2
3
1.0550
0.6035
4
0.6500
0.3718
5
0.6300
0.3604
Appendix
Table 5: Transport parameter values that correspond to product delivery.
Node
I0j
cjς
ejς
ckψ
vkψ
ekψ
4
155
0.2300
0.0002
2.0000
1.0000
5.0126
5
267
0.3500
0.0002
2.2000
1.0000
4.5754
6
342
0.2200
0.0002
2.1000
1.0000
5.4491
7
211
0.1100
0.0002
2.0000
1.0000
6.1232
8
162
0.2900
0.0002
2.3000
1.0000
5.5157
9
195
0.3700
0.0002
NA
NA
NA
Appendix
Table 6: Complex SC. Node parameter values that correspond to the facilities in Complex SC environment.
From Node
To Node
cijτ
eijτ
1
4
0.5350
0.3060
5
0.2650
0.1516
6
1.8450
1.0553
7
1.6000
0.9152
8
1.4400
0.8237
2
4
0.3600
0.2059
Appendix
Table 7: Transport parameter values that correspond to product delivery.
Hyperparameter
Value
Inner-loop learning rate α
0.003
Outer-loop learning rate β
0.001
PPO steps per meta-update
10
Adaptation steps
4–8
Network architecture
[64, 64]
Discount factor γ
0.99
Appendix
Table 8: Hyperparameters for MAML-based meta-training.
Hyperparameter
Value
Learning rate
3×10−4
Steps per update
2048
Minibatch size
64
Epochs per update
10
Discount factor γ
0.99
GAE weight λ
0.95
Appendix
Table 9: PPO hyperparameters used in fine-tuning.
Hyperparameters
Values
MORL/D
Discount factor
0.995
Population size
6
Exchange every
5×104
Total time steps
5×105
Neighbourhood size
1
Appendix
Table 10: Hyperparameters of MORL/D Algorithm with MOSAC Policy
Hyperparameters
Values
Population size
300
Offspring number
30
Cross-over operator
binary (SBX)
Cross-over probability
90%
Cross-over η
15
Mutation method
polynomial
Appendix
Table 11: Hyperparameters of NSGA-II to solve the SC problem across three network complexities: simple, moderate, and complex.
Figure 1: Sparsity and EUM of NSGA-II, MORL/D, MORL/D with SB and PSA, Meta-MORL, and our proposed algorithms. The left parts are zoomed in on the early timesteps with meta-learning, while the right parts show MORL/D timesteps and NSGA-II generations. The EUM reveal that meta-learning methods outperform MORL/D in simpler problems but increasingly struggle with more complex ones. Normalised sparsity consistently shows the lowest values for meta-learning-based methods across all complexities.
Setting
Hypervolume
Sparsity
EUM
Number of subproblems K
K=3
0.3832***
0.0088
0.7156
K=5
0.4003*
0.0064
0.7146
K=10
0.4387
0.0076
0.7141
PSA steps S
S=3
0.4371
0.0060
0.7055**
Appendix
Table 12: Ablation study for MIRACL on simple SC (mean over 10 runs), where *, **, *** denote statistically significant differences: p<0.05 , p<0.01 , and p<0.001 , respectively. The Kruskal–Wallis followed with Dunn’s post hoc (Bonferroni-corrected) is used for data sets >2 , and the Wilcoxon test for data sets =2 . Bold text represents the baseline value of each parameter.
Figure 2: PF approximation sets between algorithms in all problem complexities. In general, meta-learning-based algorithms reside in a narrower solution space. The solutions consistently spread in the range of moderate to high values of profit and GHG emissions, while low to moderate values of SL inequality in all problem complexities.
Figure 3: Manufacturing quantities of three algorithms. Our proposed method ( 3(c) , 3(f) , 3(i) ) exhibits the most stable daily production quantities and evenly distributed production amounts across facilities compared to MORL/D and NSGA-II.
Figure 4: The inventory levels yielded from the three algorithms in various complexities. Our proposed method ( 4(c) , 4(f) , 4(i) ) exhibits relatively lean and stable inventory levels daily compared to conventional RL. The average inventory levels tend to increase with the problem complexities for all algorithms.
Figure 5: Unmet demand resulting from the three algorithms across various problem complexities. Our proposed method mostly shows proximity values to MORL/D, which is lower than NSGA-II, especially in the simpler problems.
Meta-reinforcement learning is a promising approach to multi-objective optimisation because it enables rapid policy adaptation across changing environments and preference settings. However, conventional few-shot methods usually fine-tune from a single shared meta-policy, which can reduce solution diversity and limit exploration of the Pareto front, especially in high-dimensional combinatorial problems such as supply chain optimisation. We propose a population-based Meta-reinforcement learning framework that combines decomposition with evolutionary search in scalarisation weight space. The framework maintains a population of weight vectors, each associated with a distinct meta-policy trained through gradient-based meta-learning, and iteratively refines this population through elitist selection, crossover, and mutation guided by hypervolume and entropy contributions. We evaluate the method in a multi-objective supply chain setting with conflicting economic, environmental, and social goals, and further test its generality on standard reinforcement learning problems. The results show that the proposed approach yields more diverse, better distributed Pareto front approximations, improves cross-task adaptation, increases hypervolume by up to 32% over Meta-multi-objective reinforcement learning in the complex case, and attains the lowest average Hausdorff distance among all compared methods.
Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.
Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology
This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100% over the baseline and up to 32% over the oracle. The code is at https://github.com/EVIEHub/PRISM.
Finn van der Knaap, Kejiang Qian, Zheng Xu +1
University of Edinburgh · Meta Superintelligence Labs.