In this work, we present MOCHA, the first, to our knowledge, reinforcement learning based approach to computing a family of Pareto-optimal policies across the design space of a robot using a single network. Specifically, MOCHA leverages the hypernetwork architecture to learn a network that produces specialized network parameters optimized for a given objective and parameterized robot design; we term this a multi-objective design hypernetwork (MDH). We demonstrate the capabilities of MDHs to represent a complex family of design-dependent strategies on two distinct robot morphologies, each with six design dimensions and across 2-3 objectives. Moreover, we propose an approach for efficiently producing a Design Pareto set using evolutionary search of the learned policy network, generating the optimal design-policy combination for each objective prioritization. Lastly, we provide an efficient method for computing generalist robot designs which achieve the best cumulative performance across the entire set of objectives.
Figures & tables
Fig. 1 : By learning a control policy over multiple objectives and designs, our algorithm is capable of finding the optimal design for a given objective prioritization. For example, (a) is optimized for jumping, (b) is optimized for energy consumption, and (c) is optimized for running speed.
Fig. 2 : The MO-UP Hypernetwork is trained by first sampling M designs and K trade-offs. The rewards obtained from rolling out these designs and trade-offs are used to update the value and policy networks using Multi-Objective PPO.
Fig. 3 : Some examples of procedurally generated Cheetah and Walker robots.
Fig. 4 : The Design Pareto frontier of the Cheetah sliced along the Run-Energy, Run-Height, and Energy-Height axes.
Network
Env.
Hypervolume
Spacing
Train Time
MDH
Cheetah
6.838e9
5.195e-2
2h 6m
MDMLP
Cheetah
5.353e9
0.1224
6h 20m
MDH
Walker
2.715e6
5.656e-2
1h 48m
MDMLP
Walker
2.576e6
9.566e-2
5h 55m
TABLE I : Performance of the hypernetwork algorithm on Walker and Cheetah Environments compared to the MLP baseline
Fig. 5 : Design Pareto frontier for the 2-objective Walker, and some rollouts of Pareto-optimal designs.
Fig. 6 : Comparison of the Design Pareto Frontier metrics attained by a Hypernetwork and MLP policy for the Cheetah (top) and Walker (bottom) environments. The Hypernetwork achieves a higher hypervolume and lower spacing than the MLP in both environments.
Fig. 7 : Fixed-Design Pareto frontiers for the generalist Cheetah, and a few selected specialists. Note that the run-optimized design far outperforms the others at running and energy savings, and the generalist design performs approximately similar to the height-optimized design.
Fig. 8 : Procedurally-generated generalist and specialist Cheetah robots obtained using MOCHA. The generalist robot maximizes the hypervolume of its associated Pareto frontier, while each task-optimized design maximizes the scalar reward according to its objective prioritization.
Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, thus facing vastly increased computation time. Ultimately, this has limited the application of MORL towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 26-271x speed-ups compared to legacy CPU-based approaches and up to 19x speed-ups over prior GPU-based approaches whilst learning superior Pareto front hypervolumes. We demonstrate MO-Playground's versatility by implementing a custom BRUCE humanoid robot environment and learning Pareto-optimal locomotion policies across 6 practical objectives in simulation, such as smoothness, efficiency and arm swinging. Our toolbox is available at https://dynamicmobility.github.io/moplayground/.
Neil Janwani, Ellen Novoseller, Vernon J. Lawhern +1
Georgia Institute of Technology, Atlanta, GA USA · DEVCOM Army Research Laboratory, MD, USA
We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs). PQLRM combines Pareto Q-Learning (PQL), which maintains sets of vector-valued Q-estimates to approximate the Pareto front, with enhancements from Q-Learning with Reward Machines (QRM), which exploits the factored automaton structure of the reward signal. This yields a multi-policy algorithm that remains sample-efficient under non-Markovian, RM-encoded rewards. Experimental trials show that PQLRM converges faster than a naive PQL baseline applied to the cross-product MDP and can synthesize Pareto-optimal policies that QRM cannot.
Linköping University, Sweden · Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStal, F-59000 Lille, France · Univ. Toulouse, INRAE-MIAT, Toulouse, France
In this paper, we introduce a model of evolution and learning in robots that co-optimizes a distribution of latent design vectors (genotypes) and a mixture of control experts (neural modules), which are gated by the latent coordinates of each decoded design (phenotype). This provides a scalable alternative to co-design algorithms that either train an individual policy for every robot, which is inefficient, or a monolithic universal controller for all robots, which results in overly conservative structures and behaviors. Our approach lies somewhere between these two extremes, preserving ancestral knowledge in a unified yet modular framework in which different body plans activate and deactivate different combinations of learned sensorimotor circuits for goal-directed behavior. This allows one part of the controller to be overhauled to better suit new species of designs as they emerge without disrupting the hard-earned knowledge contained within other expert modules. It also allows pretrained expert policies to be directly plugged into the mixture, which can steer evolution into otherwise unexplored areas of latent space containing desired morphological traits. We refer to this process as "evo by demo" and explore how it may be used to guide freeform evolution toward canonical structures defined by the pretrained model. Videos and code can be found at: https://eco-moe.github.io.