All roads lead to Markov kernel networks: A representation theorem for discretization-stable learning
Organizations: Rice University · University of Basel · University of Helsinki
Abstract
Operator learning seeks to approximate function-to-function maps in a way that remains stable when the input discretization changes, a property sometimes called "discretization invariance". We consider a quantitative refinement of this idea, called , which builds on a common representation of input functions and their discretizations as probability measures and provides a Hölder bound relating output changes to the Wasserstein distance between these measures. Our main result is a representation theorem showing all uniformly discretization stable operators can be written exactly as a ---an abstract architecture that is a joint generalization of transformers, Fourier neural operators (FNO), Flowers, and other successful learned operators. Our representation theorem characterizes a design space for discretization-stable operator learning; designing new architectures becomes a question of which parts of a Markov kernel network to parameterize, and how. We further show that the read kernels of these networks can be realized as solutions of regularized variational problems, with particular choices of costs and regularizers recovering the kernels of transformers, FNO, Flowers, their hybrids, and beyond. This variational formulation provides a systematic approach to architecture design, together with sufficient conditions for uniform discretization stability.
Figures & tables
| Divergence | Generator | Response | Attention mechanism |
|---|---|---|---|
| KL | Softmax ( Vaswani et al., 2017 ) | ||
| Pearson | Sparsemax ( Martins and Astudillo, 2016 ) | ||
| Tsallis / power, | Entmax ( Martins et al., 2022 ) | ||
| Shifted entropy, | — | ||
| KL plus Pearson, | — | ||
| Centered power, | Sparsemax for ( Martins and Astudillo, 2016 ) |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Notation | Description |
|---|---|
| Spaces, measures, and norms | |
| Compact spatial domain (metric space , metric ). | |
| Nonempty compact convex input/output feature range. | |
| , | Graph space and its product metric. |
| Compact embedding spaces; indexes layers. | |
| , | Radon probability measures; those with finite first moment. |
| Read / architecture | Regularizer | Cost |
|---|---|---|
| SelfWarp | , | |
| Attention | ||
| FNO uniform read | ||
| Representation theorem |
Operator Learning
Operator learning studies models that map functions or fields to functions, enabling fast prediction across scientific and dynamical systems.
See all 111 papers