Characterizing the Representational Capacity of Neural Processes

Robin Young

cs.LGMay 22, 2026

Characterizing the Representational Capacity of Neural Processes

Authors: Robin Young

Organizations: University of Cambridge

Abstract

What functions can Neural Processes represent? We analyze the representational capacity of popular NP architectures: Conditional Neural Processes (CNPs), Attentive Neural Processes (ANPs), Transformer Neural Processes (TNPs), and their latent variants. We prove these architectures form a strict hierarchy. CNP-representable functions are exactly those depending on finitely many expected features of the context distribution. ANPs strictly generalize CNPs via query-dependent reweighting, enabling kernel smoothers. ConvCNPs and ANPs are incomparable; each contains functions outside the other, separated by stationarity versus translation equivariance. TNPs with $L$ self-attention layers capture $L$ -hop context interactions. For latent NPs, we show finite-dimensional latents provide coherent sampling but do not circumvent encoder limitations; matching GP posterior distributions requires latent dimension scaling with context size. These results provide a theoretical foundation for architecture selection based on task structure.

Explore similar work

Revisiting Neural Processes via Fourier Transform and Volterra Series
May 31, 2026Peiman Mohseni, Nick Duffield, Raymond K. W. WongFourier Convolutional
Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models
Jun 1, 2026Alexander GuhaRepresentational Capacity Representation Learning
Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders
Sep 21, 2026Rithin Nagaraj, Rupa Laalasa Oruganti, Prerna Subhashchandra Kunder +1Semantic Representations Bidirectional Mamba

May 31, 2026 · cs.LGJ/K move · Enter open · S save

Revisiting Neural Processes via Fourier Transform and Volterra Series

Peiman Mohseni, Nick Duffield, Raymond K. W. Wong

Texas A&M University, College Station, Texas, USA.

Modeling unknown latent functions from finite, irregularly sampled measurements is a recurring challenge across science and engineering. Neural processes (NPs), a family of probabilistic functional models, are promising solutions -- especially when endowed with domain-specific symmetries like translation equivariance, which improve sample efficiency and generalization. Yet existing translation-equivariant NPs face two limitations: (i) they stack generic components with non-linearities, obscuring the induced function class and limiting interpretability; and (ii) convolutional designs are limited by local receptive fields and the need to embed inputs onto a dense uniform grid, while attention-based alternatives lift these restrictions at quadratic cost in the number of observations. We address both with two contributions. First, using the Volterra expansion, we approximate continuous translation-equivariant operators by sums of higher-order convolutions, yielding analytical transparency while admitting efficient evaluation via first-order convolutions. Second, we introduce set Fourier convolutions (SFConvs), a frequency-domain parameterization that operates directly on irregularly sampled points, achieves approximately global receptive fields, and scales linearly in the number of observations. Building on these ideas, we propose two conditional NPs (CNPs): SFConvCNPs, which stack SFConv blocks with non-linearities, and SFVConvCNPs, which integrate the Volterra formulation. Experiments on synthetic and real-world datasets demonstrate our methods' efficacy against state-of-the-art baselines.

Fourier Convolutional Equivariant Neural Networks

Open paper Ask about it PDF

Abstract

Explore similar work

Revisiting Neural Processes via Fourier Transform and Volterra Series

Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders