Distilling a Modular Reservoir Through a Genomic Bottleneck
Authors: Mani Hamidi, Sina Khajehabdollahi, Charley M. Wu, Emmanouil Giannakakis
Organizations: Department of Computer Science, University of Tübingen, Tübingen, Germany · Flowers Team Inria, Bordeaux, France · Max Planck Institute for Biological Cybernetics, Tübingen, Germany · Human and Machine Cognition Lab, Center for Cognitive Science, TU Darmstadt, Darmstadt, Germany · Hessian.AI, Darmstadt, Germany · Department of Bioengineering, Imperial College London, London, UK
The intricate structures of biological neural networks largely emerge during development, guided by a comparatively compressed blueprint encoded in the genome. The connectivity that emerges from this decoding process is rich in structure, and already equips the organism with functional modules upon birth. This initial structure serves as a scaffold that can be gradually refined and fine-tuned through lifelong experience, via a variety of plasticity mechanisms. Drawing inspiration from this interaction between evolutionary and developmental modes of learning, we use hypernetworks to learn a compressed generative process that generates the connectivity of a modular reservoir. We show that this marriage between curriculum-based meta-learning and modular reservoir computing can generate sparse recurrent networks that solve difficult temporal tasks with minimal training and without concessions to robustness.