Overview

Neural Processes study how to predict a function from a small set of observed input-output pairs, by learning across many related tasks and returning a distribution of possible outputs. Conditional, attentive, latent-variable, convolutional, and transformer variants differ in how they aggregate context and represent uncertainty; extensions use spectral features or neural operators for periodic data and physical fields. These methods are useful for sparse or irregularly sampled settings such as spatial interpolation, time-series forecasting, and partial-observation PDE prediction. A central tension is balancing expressive context interactions and calibrated uncertainty with consistency, computational efficiency, and the limits imposed by finite representations and amortized inference.

Latest papers

62 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Feb 26, 2024

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks

Behavior can be described as a temporal sequence of actions driven by neural activity. To learn complex sequential patterns in neural networks, memories of past activities need to persist on significantly longer timescales than the relaxation times of single-neuron activity. While recurrent networks can produce such long transients, training these networks is a challenge. Learning via error propagation confers models such as FORCE, RTRL or BPTT a significant functional advantage, but at the expense of biological plausibility. While reservoir computing circumvents this issue by learning only the readout weights, it does not scale well with problem complexity. We propose that two prominent structural features of cortical networks can alleviate these issues: the presence of a certain network scaffold at the onset of learning and the existence of dendritic compartments for enhancing neuronal information storage and computation. Our resulting model for Efficient Learning of Sequences (ELiSe) builds on these features to acquire and replay complex non-Markovian spatio-temporal patterns using only local, always-on and phase-free synaptic plasticity. We showcase the capabilities of ELiSe in a mock-up of birdsong learning, and demonstrate its flexibility with respect to parametrization, as well as its robustness to external disturbances.
Sep 14, 2022

Sketch of a novel approach to a neural model

We present an account of neuroplasticity with respect to cell-internal processing pathways in relation to membrane and synaptic plasticity. We think traditional synapse-centric, weight-based models of memorization are not sufficient or adequate to capture the complexity of neuroplasticity. In these accounts, the model is a network of neurons connected by adaptive transmission links. The adaptation of the transmission links relies on weight changes according to use of the transmission link (short-term and long-term potentiation/depression). In contrast, we propose a paradigm switch from a synapse-centric model (each synapse learns independently, based on its history of use) to a neuron-centric model (each neuron uses signal selection for intracellular pathways to express plasticity at the membrane). A neural model consists of (a) expression of parameters at the membrane, in particular dendritic synapses or spines, and axonal boutons (b) internal parameters in the sub-membrane zone and the cytoplasm with its protein signaling network and (c) core parameters in the nucleus for genetic and epigenetic information. In a neuron-centric model, each node (=neuron) in the network has its own internal memory. Neural transmission and information storage are separated, not automatically combined by coupling strength. There is filtering and selection of signals for storage. Not every transmission event leaves a trace. This represents an important conceptual advance over synaptic weight models. We present the neuron as a self-programming device, rather than as passively determined by ongoing input. We believe a new approach to neural modeling is necessary, because the experimental evidence is not well captured by traditional synapse-centric models. Ultimately, we are interested in the possibilities of a flexible memory system that processes external signals according to its inherent structure.