Stability of Measure-to-Measure Transformers on Sub-Gaussian Data

Frank Cole; Nicholas H. Nelsen; Takashi Furuya

stat.MLOct 6, 2026

Stability of Measure-to-Measure Transformers on Sub-Gaussian Data

Authors: Frank Cole, Nicholas H. Nelsen, Takashi Furuya

Organizations: UCLA · UT Austin · Doshisha University, RIKEN AIP

Abstract

Transformers have exhibited impressive empirical success across various domains, but their theoretical foundations remain less developed. This work constitutes a mathematical study of the measure-to-measure operators defined by transformers. We show that transformers map sub-Gaussian inputs to sub-Gaussian outputs; this ensures that taking arbitrary-length compositions of the softmax operator is well-defined. We then show that transformers are Hölder continuous with respect to the 1-Wasserstein distance on appropriate spaces of sub-Gaussian inputs. This allows us to establish estimates on the error propagation along a transformer between a sub-Gaussian input and its empirical approximation. We also study a mean-field analog of the cross-attention mechanism, which is an operator from a pair of probability measures to a single probability measure. We show that cross-attention exhibits different Hölder regularity and sample-complexity in its two input arguments. Last, we apply our results to deduce approximation guarantees for measure-to-measure transformers. Together, these results provide a firm stability and finite-sample theory for transformers on sub-Gaussian data.

Explore similar work

Universal Approximation of Measure-to-Measure Operators by Pushforwards
Sep 28, 2026Takashi Furuya, Nicholas H. Nelsen, Frank ColeProbability Measures Approximation
Doshisha University, RIKEN AIP · UT Austin · UCLA
Universality and Generalization of Causal Transformers Across Context Lengths
Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréTransformer Architectures Regularity
Doshisha University, RIKEN AIP · Rice University · CNRS, ENS, PSL Université
Approximation Error Upper and Lower Bounds for Hölder Class with Transformers
May 8, 2026Xin He, Yuling Jiao, Xiliang Lu +1Transformer Architectures Approximation
School of Mathematics and Statistics, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

Universal Approximation of Measure-to-Measure Operators by Pushforwards

Takashi Furuya, Nicholas H. Nelsen, Frank Cole

Doshisha University, RIKEN AIP · UT Austin · UCLA

Probability Measures Approximation

Open paper AskPDF arXiv

Many learning tasks map an input distribution to an output distribution. A natural way to model such an operator is to transform each input sample using a continuous function that may depend on the entire input distribution, and then take the distribution of the transformed samples. This defines a measure-dependent pushforward model and includes measure-theoretic formulations of transformers. We ask when such models can approximate arbitrary continuous operators between spaces of probability measures. We first show that universal approximation fails when atomic inputs are allowed: some continuous measure-to-measure operators that split or redistribute atomic mass cannot be approximated arbitrarily well by deterministic pushforward models. We then introduce the uniform level set condition, which requires a continuous measure-dependent scalarization whose shrinking level set neighborhoods carry uniformly vanishing mass over the input family. This condition is satisfied, in particular, by compact families of absolutely continuous measures. On every compact family satisfying this condition, we prove that any continuous measure-to-measure operator with outputs of finite

p

-th moment can be uniformly approximated, in the

p

-Wasserstein distance, by continuous measure-dependent pushforwards. Combining our theorem with existing approximation results for measure-dependent in-context maps yields universal approximation by measure-theoretic transformers. We also extend the framework to continuously-varying source measures, yielding a corresponding universality result for a class of pushforward models that are closely aligned with cross-attention architectures.

Abstract

Explore similar work

Universal Approximation of Measure-to-Measure Operators by Pushforwards

Universality and Generalization of Causal Transformers Across Context Lengths

Approximation Error Upper and Lower Bounds for Hölder Class with Transformers