Social Learning Strategies for Evolved Virtual Soft Robots
Authors: K. Ege de Bruin, Kyrre Glette, Kai Olav Ellefsen, Giorgia Nadizar, Eric Medvet
Organizations: Department of Informatics, University of Oslo, Oslo, Norway · Also with RITMO, University of Oslo. · University Toulouse Capitole, IRIT - CNRS UMR5505, Toulouse, France · University of Trieste, Trieste, Italy
Optimizing the body and brain of a robot is a coupled challenge: the morphology determines what control strategies are effective, while the control parameters influence how well the morphology performs. This joint optimization can be done through nested loops of evolutionary and learning processes, where the control parameters of each robot are learned independently. However, the control parameters learned by one robot may contain valuable information for others. Thus, we introduce a social learning approach in which robots can exploit optimized parameters from their peers to accelerate their own brain optimization. Within this framework, we systematically investigate how the selection of teachers, deciding which and how many robots to learn from, affects performance, experimenting with virtual soft robots in four tasks and environments. In particular, we study the effect of inheriting experience from morphologically similar robots due to the tightly coupled body and brain in robot optimization. Our results confirm the effectiveness of building on others' experience, as social learning clearly outperforms learning from scratch under equivalent computational budgets. In addition, while the optimal teacher selection strategy remains open, our findings suggest that incorporating knowledge from multiple teachers can yield more consistent and robust improvements.
Figures & tables
Figure 1. Overview of the SL approach. When a new VSR is generated through mutation, we need to optimize its brain. We use Bayesian optimization , which works by using initial samples ( i.e. , initial evaluated brains) to predict new candidates ( i.e. , new brains). The best sample is then used as the VSR brain to determine its fitness. With SL , initial candidates are obtained by inheriting samples from other VSRs in the population. This differs from a standard case, IL , where initial candidates are independent from other robots. Several strategies exist for selecting which samples to inherit; in this figure we show inheritance of one sample from the most similar robots in the previous generation (we call this “Similar-many”, see Section 4.2 ).
Figure 2. A snapshot of four episodes where a VSR performs the four tasks.
Figure 3. Distribution of the quality q⋆ of the best-performing VSR in the four tasks for the eight approaches. Stars above the boxes show significant differences: for example, a red star on top of a gray box indicates that IL is significantly different from Best-One.
Figure 4. The evolution of the population diversity, averaged over runs and environments. The line corresponds to the mean value, the shaded area to the first-third quartile range.
Figure 5. Body features for all the optimized VSRs .
Figure 7. A selection of the optimized VSR bodies, one per each approach (column) and task (row).
Figure 8. Distribution of the quality q⋆ of the best-performing VSR subjected to the re-learning of the brain with a large budget for BO .
Figure 9. Distribution of the quality q⋆ of the best-performing VSR optimized on a source task (row of plots) subjected to the re-learning of the brain with a large budget for BO and for another destination task (groups of boxes).
Figure 10. Performance over learning iterations for three brain learning approaches evaluated on eight fixed robot bodies. The vertical red dotted line corresponds to the learning budget of 50 in our main experiment.
Shanghai Jiao Tong University, Shanghai 200240, China · Intelligent Game and Decision Laboratory, Beijing 100048, China · Renmin University of China, Beijing 100872, China +1