Humans can learn rapidly from limited experience, while neural networks typically require extensive training. In this work, we investigate what properties of a neural network can enable it to learn rapidly. We study neural networks that have been optimized via meta-learning, a technique that exposes a network to many related functions to facilitate its subsequent learning of similar functions. Prior work has shown that these meta-learned networks can learn new functions rapidly, but how they do so remains unclear. To investigate their internal structure, we take inspiration from classic work demonstrating additive relationships among the learned vectors that neural networks use to represent words (e.g., "king" minus "man" plus "woman" approximates "queen"). Analogously, we look for additive relationships among a meta-learned network's "representations" of different functions; when a meta-learned network rapidly learns to implement a specific function, all of its adapted parameter values can be concatenated into a single vector that we treat as representing the function. We find that such parameter vectors can be added and subtracted according to symbolic relationships among the functions they represent, yielding new parameter vectors that successfully represent the functions predicted by the corresponding symbolic manipulations: the networks specified by these artificially recombined vectors actually implement the intended functions without any direct training on them. Our results suggest that meta-learning can tune a network's starting point to organize subsequent learning along systematic, functionally meaningful trajectories through the space of possible parameter vectors, providing a geometric basis for realizing inductive biases in neural networks.
Figures & tables
Figure 1: Outline of the full analogy pipeline. A concrete example illustrates how meta-learning a parameter initialization \uptheta for rapid adaptation within a structured domain of functions [a,b] induces corresponding geometric structure in the space of parameter vectors θ[a,b] adapted from \uptheta to implement specific functions. (A) We specify a domain of sequence-generating functions [a,b] , characterized by the sequence distributions D( [a,b] ) they generate. (B) The network meta-learns \uptheta by iteratively adapting it to implement specific [a,b] functions ( [5,42] in the step shown), then updating \uptheta based on how well the resulting parameter vectors ( θ[5,42] in this step) implement the intended functions. (C) After meta-learning, we rapidly adapt \uptheta to several functions (adaptation shown with thin arrows), verifying for each that the resulting network generates a distribution of sequences that matches the distribution generated by the corresponding function — that is, that the adapted parameter vector successfully implements the intended function. (D) We construct a new parameter vector via analogy over these adapted parameter vectors ( W = θ[1,5] + θ[2,6] - θ[2,5] ) and show that it implements the function predicted by the corresponding symbolic analogy ( [1,6] = [1,5] + [2,6] - [2,5] ) without any direct adaptation to that function.
[A=16,B=49]+[A=99,B=97]−[A=99,B=49]=[A=16,B=97]
Figure 2: Examples of randomly generated analogies used in our experiments. The intended function is shown on the right-hand side of the equation. As written, each analogy varies B while holding A fixed, but swapping the first two terms yields an equivalent analogy that can be interpreted as varying A while holding B fixed.
Figure 3: Model and analogy performance across architectures and function domains. Each panel shows F-scores across ten independent MAML-tuned initializations for one (architecture, domain) pair. “Adaptation” shows the performance of networks directly adapted to implement novel functions in the domain. “Analogy: single type” shows how well networks constructed using analogies over the initially specified component pair implement the intended novel functions, as measured by the mean F-score, while “Analogy: mean-max” reports, for each initialization, the highest mean F-score obtained across analogy types corresponding to alternative syntactic representations (see Section 3.4 ). Colored bars show means across initializations, black points show individual initializations, and black error bars show ±1 standard deviation. For ABgeom , only one analogy type is available, so the two analogy measures are identical.
Figure 4: Examples of the different ABlength analogy types used in our experiments. Each row shows an analogy drawn from one of our experimental analogy sets, with functions specified in terms of all three equivalent syntactic components, L.A , L.B , and L.total . Checkmarks indicate the syntactic representation under which the analogy is well-formed; crosses indicate alternative representations under which the same analogy is not well-formed. Thus, each analogy type tests a particular hypothesis about which pair of syntactic components might be reflected in the geometry of accessible parameter space.
Figure 5: Analogy performance across initialization and training conditions. Each panel shows mean-max F-scores across ten independent initializations for one (architecture, domain) pair. “Random, converge” and “Pre-train, converge” show analogy performance for parameter vectors trained to convergence from random and conventionally pre-trained initializations, respectively. “MAML, converge” shows analogy performance for parameter vectors trained to convergence from MAML-tuned initializations, while “MAML, rapid adapt” shows analogy performance for parameter vectors rapidly adapted from MAML-tuned initializations, as in the main experiment. Colored bars show means across initializations, black points show individual initializations, and black error bars show ±1 standard deviation.
Figure 6: Schematic making explicit the compositionality of adaptation updates. We recast the Figure 1 D analogy over adaptation endpoints (labeled there with θ ’s) as an analogy over the corresponding adaptation updates (thin arrows, labeled here with v ’s). The single-step update vector vf is computed from the shared initialization based on 100 example sequences generated by the target function f . These analogies are equivalent in our experiments because each adaptation endpoint can be decomposed into the shared initialization plus its single-step update vector, θf=\uptheta+vf , causing the shared initialization terms to cancel in the original endpoint analogy.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Schematic of one outer-loop step of MAML. A random function f is drawn from the domain and the network rapidly adapts from \uptheta to task-specific parameters θf using one training step and limited data. The performance of θf on unseen data is then used to compute the meta-update from \uptheta to \uptheta′ . The process repeats from \uptheta′ with a new function.
The structure of human visual representations underpins our capacity for adaptive behaviour. While pretrained neural networks model human visual representations with unprecedented success, a large discrepancy remains. We propose one reason: these networks optimise a single fixed objective, whereas human representations must support open-ended tasks. We hypothesise this flexibility arises from meta-learning (learning to learn), a pressure shaping representations to acquire new tasks from few observations. To test this, we train a sequence model, without any supervision from human data, across thousands of semantically rich tasks mapping images to high-level concepts. Compared to their pretrained base encoders, meta-learned representations better predict human similarity judgements, semantic rule learning, and high-level visual cortex. Behavioural gains depend on disentangled, high-level task distributions, while brain alignment is driven primarily by the learning-to-learn pressure. Our results suggest the flexibility of human visual representations reflects the functional demand to learn new semantic relationships on the fly.
Can Demircan, Marcel Binz, Alireza Modirshanechi +1
Helmholtz Munich, Munich, Germany · Max Planck Institute for Biological Cybernetics, Tübingen, Germany
How does compositional reasoning emerge during learning? We address this question by mathematically analyzing the learning dynamics of deep linear networks. We train these networks in structured synthetic environments and derive a theory linking the structure of experience to compositional learning. Our theory predicts when compositional inferences emerge, whether they are identifiable from the available evidence, and how new linking evidence can rapidly unlock previously unavailable inferences. These results provide a qualitative explanation for several phenomena observed in human cognition. They account for why a composition can fail despite knowing its premises, why similar compositions can emerge at different times, and how a single linking fact can suddenly enable many new inferences. Taken together, these findings establish a mathematical link between the statistical structure of experience and the development of compositional reasoning.
Neural networks are typically adapted by computing gradients and updating model parameters. We investigate whether task-specific adaptation can instead emerge from a meta-learned self-organising process that requires no gradients at adaptation time. We instantiate this idea with a Neural Cellular Automaton in which locally interacting recurrent cells maintain both a recurrent state and a fast associative memory. During meta-training, backpropagation is used to learn the recurrent dynamics together with how the memory is read and written. Once training is complete, the slow model parameters remain fixed, and online adaptation occurs only through cellwise memory updates driven by local prediction errors and a delta rule. We evaluate whether the learned mechanism can adapt to semantically distinct held-out classification tasks. A single pass over the support data produces substantial improvements in held-out performance without gradient computation or parameter updates during adaptation, and the mechanism remains effective across large changes in the number of examples processed jointly. These results show that task-specific adaptation can be achieved through explicit fast-memory updates while keeping the slow model parameters fixed.