Structured Priors: Towards an algebra of component priors for prior-data fitted networks
Abstract
Prior-data fitted networks are typically trained using synthetic data generated by a single, manually constructed generator. In cases where such a generator combines multiple components, e.g., in TabPFN, Mitra, or TabICLv2, this is achieved through a mixture using manually specified weights, the impact of which on the network only becomes apparent after pre-training. Instead of such a hand-built generator, this work proposes a structured prior: a small set of named component priors, linked by operators whose algebraic properties are formally proven and whose influence on the trained network has been validated. Within a measure-theoretic framework, the operators of mixture, convolution, amplitude scaling, local transform, and projection are defined. Their governing laws are proven. These operators are implemented in the macl2l software package. Experiments demonstrate that the implemented generators reproduce the identities within the resolution of a permutation test. Furthermore, it is shown that trained networks obey the distributive law; that a mixed prior composed of small components performs as well as or better than any individual component in seven out of eight open regression benchmarks; and that on two of these tables a structured prior reaches, with a ninth of the data budget, an error that a generic random-function prior does not reach even with the full budget. A theorem on model averaging demonstrates that networks trained on individual components constitute the Bayes predictor for the mixture, provided that the component networks are themselves the Bayes predictors for their respective priors. However, the pre-registered test of this averaged predictor with trained component networks failed, because the networks identify the generating component too rarely. The paper proposes a research program focused on the algebra of structured priors for prior-data fitted networks.
Figures & tables
| Operator | Map | Result | Law proved | Purpose |
|---|---|---|---|---|
| mixture | dataset density ; predictive the Bayesian model average; cost at most per component; addition of the semiring | hedge between alternatives; component inference made explicit | ||
| convolution and amplitude scaling | ; ; | in | commutative semiring with the as endomorphisms; labels on shared rows; kernels add for Gaussian processes | add independent sources of structure; set the size of each |
| local transform and composition | monoid under composition; additive and positively homogeneous, so it distributes over mixtures; | fit a data-driven teacher to the draws of a prior; reaches priors outside the semiring | ||
| projection onto a basis | unique minimiser of the maximum mean discrepancy over ; fixes every mixture of the basis; non-expansive | explain a target by named parts; weights found by computation |
| Symbol | Meaning | Type |
|---|---|---|
| amplitude scales | scalar | |
| vector of amplitude scales of a weighted convolution | vector | |
| dataset density of | density | |
| vectors of clean labels on a row block | vector | |
| mixture weights, weight vector in , projection weights | scalar, vector | |
| number of component priors | integer |
Prior-Data Fitted Networks
Prior-Data Fitted Networks (PFNs) learn in-context prediction by training on synthetic datasets sampled from designed priors.
See all 55 papers