PRISM: Parallel Reward Integration with Symmetry for MORL

Finn van der Knaap; Kejiang Qian; Zheng Xu; Fengxiang He

cs.LGFeb 20, 2026

PRISM: Parallel Reward Integration with Symmetry for MORL

Authors: Finn van der Knaap, Kejiang Qian, Zheng Xu, Fengxiang He

Organizations: University of Edinburgh · Meta Superintelligence Labs.

Abstract

This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100% over the baseline and up to 32% over the oracle. The code is at https://github.com/EVIEHub/PRISM.

Figures & tables

Figure 1: Reflectional symmetry in a two-legged agent. The left panel shows a transition from state $s$ to $s^{\prime}$ under action $a$ , whereas the right panel shows the reflected transition, where states and actions are transformed by $L_{g}$ and $K_{g}$ , respectively.

Panel of Figure 3 — Figure 3: The obtained hypervolume for various levels of sparsity amongst various dimensions.

Panel of Figure 4 — Figure 4: The approximated Pareto front for dense rewards (blue dots) and sparse rewards (orange dots) for the first reward objective.

Environment	Metric	Oracle	Baseline	PRISM
Mo-hopper-v5	HV ( $\times 10^{7}$ )	1.30 $\pm$ 0.13	0.84 $\pm$ 0.05	1.58 $\pm$ 0.05
	EUM	129.04 $\pm$ 7.96	97.64 $\pm$ 4.18	147.43 $\pm$ 2.61
	VO	59.07 $\pm$ 3.45	43.36 $\pm$ 1.61	66.66 $\pm$ 1.40
Mo-walker2d-v5	HV ( $\times 10^{4}$ )	4.21 $\pm$ 0.11	3.34 $\pm$ 0.16	4.77 $\pm$ 0.07
	EUM	107.58 $\pm$ 2.86	82.13 $\pm$ 4.34	120.43 $\pm$ 1.64
	VO	53.22 $\pm$ 1.39	39.18 $\pm$ 2.49	59.35 $\pm$ 0.80

Table 1: Experimental results. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. The largest (best) values are in bold font.

Panel of Figure 5 — Figure 5: The dense (blue line) and shaped rewards (orange line) over time for mo-walker2d-v5 and the first reward objective.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\mathcal{S}$	State space
$\mathcal{A}$	Action space
$P(s^{\prime}\|s,a)$	Transition probability
$\bm{r}(s,a)\in\mathbb{R}^{L}$	Vector-valued reward with $L$ objectives
$\gamma\in[0,1)$	Discount factor
$\pi:\mathcal{S}\to\mathcal{A}$	Policy mapping

Appendix

Table 2: Notation.

	State Space	Action Space	Reward Space
Mo-hopper-v5	$\mathcal{S}\in\mathbb{R}^{11}$	$\mathcal{A}\in\mathbb{R}^{3}$	$\mathcal{R}\in\mathbb{R}^{3}$
Mo-walker2d-v5	$\mathcal{S}\in\mathbb{R}^{17}$	$\mathcal{A}\in\mathbb{R}^{6}$	$\mathcal{R}\in\mathbb{R}^{2}$
Mo-halfcheetah-v5	$\mathcal{S}\in\mathbb{R}^{17}$	$\mathcal{A}\in\mathbb{R}^{6}$	$\mathcal{R}\in\mathbb{R}^{2}$
Mo-swimmer-v5	$\mathcal{S}\in\mathbb{R}^{8}$	$\mathcal{A}\in\mathbb{R}^{2}$	$\mathcal{R}\in\mathbb{R}^{2}$

Appendix

Table 3: Considered MuJoCo environments.

Index	Observation Component	Type	Symmetry
0	z-coordinate of the torso	position	Asymmetric
1	angle of the torso	angle	Asymmetric
2	angle of the thigh joint	angle	Symmetric
3	angle of the leg joint	angle	Symmetric
4	angle of the foot joint	angle	Symmetric
5	velocity of the x-coordinate of the torso	velocity	Asymmetric

Appendix

Table 4: Reflectional symmetry partition for mo-hopper-v5 observation space.

Index	Observation Component	Type	Symmetry
0	z-coordinate of the torso	position	Asymmetric
1	angle of the torso	angle	Asymmetric
2	angle of the thigh joint	angle	Symmetric
3	angle of the leg joint	angle	Symmetric
4	angle of the foot joint	angle	Symmetric
5	angle of the left thigh joint	angle	Symmetric

Appendix

Table 5: Reflectional symmetry partition for mo-walker2d-v5 observation space.

Index	Observation Component	Type	Symmetry
0	z-coordinate of the front tip	position	Asymmetric
1	angle of the front tip	angle	Asymmetric
2	angle of the back thigh	angle	Symmetric
3	angle of the back shin	angle	Symmetric
4	angle of the back foot	angle	Symmetric
5	angle of the front thigh	angle	Symmetric

Appendix

Table 6: Reflectional symmetry partition for mo-halfcheetah-v5 observation space.

Index	Observation Component	Type	Symmetry
0	angle of the front tip	angle	Asymmetric
1	angle of the first rotor	angle	Symmetric
2	angle of the second rotor	angle	Symmetric
3	velocity of the tip along the x-axis	velocity	Asymmetric
4	velocity of the tip along the y-axis	velocity	Symmetric
5	angular velocity of the front tip	angular velocity	Asymmetric

Appendix

Table 7: Reflectional symmetry partition for mo-swimmer-v5 observation space.

	PRISM
Initial collection $N$	1000
Expert collection $E$	1000
Number of refinements $IR$	2
Timesteps per cycle $M$	100,000
Epochs	1000
Learning rate	0.005

Appendix

Table 8: Hyperparameters for ReSymNet.

	Mo-hopper-v5	Mo-walker2d-v5	Mo-halfcheetah-v5	Mo-swimmer-v5
$\lambda$	0.01	1	0.01	0.005

Appendix

Table 9: SymReg hyperparameter.

Panel of Figure 6 — Figure 6: The approximated Pareto front for dense rewards (blue dots), sparse rewards (orange dots), and shaped rewards (green dots). Sparsity is imposed on the first reward objective.

Environment	Metric	uniform	random
Mo-hopper-v5	HV ( $\times 10^{7}$ )	1.38 $\pm$ 0.08	0.49 $\pm$ 0.06
	EUM	135.19 $\pm$ 5.30	65.22 $\pm$ 6.63
	VO	63.90 $\pm$ 2.34	29.62 $\pm$ 3.68
Mo-walker2d-v5	HV ( $\times 10^{4}$ )	4.67 $\pm$ 0.07	1.18 $\pm$ 0.10
	EUM	116.72 $\pm$ 2.11	16.52 $\pm$ 4.98
	VO	56.22 $\pm$ 1.01	3.77 $\pm$ 2.46

Appendix

Table 11: ReSymNet ablation study results. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. w/o is the abbreviation of without.

Environment	Metric	Oracle	Baseline	PRISM
Mo-hopper-v5	HV ( $\times 10^{7}$ )	1.30 $\pm$ 0.13	1.19 $\pm$ 0.10	1.51 $\pm$ 0.11
	EUM	129.04 $\pm$ 7.96	124.82 $\pm$ 7.21	142.89 $\pm$ 7.38
	VO	59.07 $\pm$ 3.45	56.21 $\pm$ 3.20	67.58 $\pm$ 3.31
Mo-walker2d-v5	HV ( $\times 10^{4}$ )	4.21 $\pm$ 0.11	3.16 $\pm$ 0.13	4.59 $\pm$ 0.14
	EUM	107.58 $\pm$ 2.86	85.95 $\pm$ 3.27	114.62 $\pm$ 2.80
	VO	53.22 $\pm$ 1.39	41.29 $\pm$ 1.49	54.84 $\pm$ 1.25

Appendix

Table 12: Experimental results on the control cost objective. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. The largest (best) values are in bold font.

Panel of Figure 7 — Figure 7: The obtained hypervolume for various levels of sparsity for PRISM.

Environment	Metric	Oracle	Baseline	PRISM
Mo-hopper-v5	HV ( $\times 10^{7}$ )	1.65 $\pm$ 0.10	0.67 $\pm$ 0.04	1.65 $\pm$ 0.07
	EUM	151.45 $\pm$ 5.87	85.87 $\pm$ 3.17	148.19 $\pm$ 4.26
	VO	72.26 $\pm$ 2.90	41.21 $\pm$ 1.44	70.24 $\pm$ 2.51
Mo-walker2d-v5	HV ( $\times 10^{4}$ )	5.93 $\pm$ 0.10	3.20 $\pm$ 0.23	5.61 $\pm$ 0.10
	EUM	141.88 $\pm$ 2.38	76.41 $\pm$ 6.47	132.67 $\pm$ 2.26
	VO	67.63 $\pm$ 1.17	35.64 $\pm$ 3.91	63.19 $\pm$ 1.75

Appendix

Table 13: Experimental results of GPI-PD. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. The largest (best) values are in bold font.

Explore similar work

MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics
Mar 10, 2026Neil Janwani, Ellen Novoseller, Vernon J. Lawhern +1Multi-Objective Reinforcement Learning Reinforcement Learning
Georgia Institute of Technology, Atlanta, GA USA · DEVCOM Army Research Laboratory, MD, USA
A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Apr 27, 2026Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2Multi-Objective Reinforcement Learning Reinforcement Learning
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology
Momba: Network Modernization Improves Multi-Objective Reinforcement Learning
Aug 7, 2026Adam Štafa, Santeri Heiskanen, Petr Novotný +1Multi-Objective Reinforcement Learning
Faculty of Informatics, Masaryk University · Department of Electrical Engineering and Automation, Aalto University

Mar 10, 2026J/K move · Enter open · S save

MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics

Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker

Georgia Institute of Technology, Atlanta, GA USA · DEVCOM Army Research Laboratory, MD, USA

Multi-Objective Reinforcement Learning Reinforcement Learning RL for Robotics

Open paper AskPDF arXiv

Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL's application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging.

Abstract

Explore similar work

MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning

Momba: Network Modernization Improves Multi-Objective Reinforcement Learning