Learning Conditional Expectation Operators via Functional Newton Updates

Thiago Ramos; Alek Fröhlich; Daniel Perazzo; Massimiliano Pontil

stat.MLSep 28, 2026

Learning Conditional Expectation Operators via Functional Newton Updates

Authors: Thiago Ramos, Alek Fröhlich, Daniel Perazzo, Massimiliano Pontil

Organizations: Federal University of São Carlos · CSML, Istituto Italiano di Tecnologia and University of Genoa · CSML, Istituto Italiano di Tecnologia and University College London

Abstract

We introduce the Functional Spectral-Newton Method (FSNM) for learning the leading singular structure of a conditional expectation operator without fixing a basis or reproducing kernel Hilbert space. FSNM fits a low-rank representation of the centered joint-to-product density ratio kernel by alternating functional Newton updates. Each update reduces to a preconditioned regression, which we approximate with vector-valued regression trees in a stagewise boosting procedure. At the population level, we establish descent and an $O(1/T)$ best-iterate block-stationarity rate under a relative weak-learner accuracy condition, and show that every nondegenerate local minimum over the full centered $L^2$ spaces is a globally optimal rank- $d$ approximation. Synthetic experiments show that FSNM recovers a low-rank density ratio and its leading spectral structure, and that the same learned kernel can answer multiple conditional queries without refitting.

Figures & tables

Figure 1: Rank-three synthetic experiment with close singular values. From left to right: the exact density ratio $\kappa$ , the FSNM estimate $\widehat{\kappa}$ , and the signed error $\widehat{\kappa}-\kappa$ .

Figure 2: Empirical training and validation losses over $40$ boosting iterations. The vertical line marks the iteration selected by validation.

Method	Kernel RMSE	Spectrum error	Subspace error
FSNM	$\mathbf{0.1261\pm 0.0075}$	$\mathbf{0.0286\pm 0.0058}$	$\mathbf{0.3579\pm 0.0242}$
ACE	$0.1489\pm 0.0046$	$\mathbf{0.0238\pm 0.0048}$	$\mathbf{0.4010\pm 0.0322}$
uLSIF	$\mathbf{0.1149\pm 0.0044}$	–	–
Kernel CCA	$0.1756\pm 0.0198$	$0.0720\pm 0.0196$	$\mathbf{0.3613\pm 0.0663}$

Table 1: Baseline comparison on the rank-three synthetic experiment, over

10

independent data draws (mean

\pm

95\%

CI). Bold entries are statistically tied for best in that column, i.e., their interval overlaps the best mean’s interval. uLSIF does not factorize

\widehat{\kappa}

, so it has no associated spectrum or subspace error.

Figure 3: Discontinuous regional experiment. From left to right: the exact piecewise-constant density ratio, the FSNM estimate using tree weak learners, and their signed difference.

Method	Kernel RMSE	Spectrum error	Subspace error
FSNM	$\mathbf{0.1393\pm 0.0198}$	$\mathbf{0.0301\pm 0.0113}$	$\mathbf{0.2678\pm 0.0846}$
ACE	$\mathbf{0.1583\pm 0.0152}$	$\mathbf{0.0353\pm 0.0066}$	$\mathbf{0.2978\pm 0.0742}$
uLSIF	$0.2312\pm 0.0065$	–	–
Kernel CCA	$0.2908\pm 0.0148$	$0.0716\pm 0.0287$	$0.4878\pm 0.0317$

Table 2: Baseline comparison on the discontinuous regional experiment (mean

\pm

95\%

CI, bold = tied for best, as in Table 1 ).

Figure 4: Twenty-dimensional tabular experiment. Left: exact and estimated density ratios on held-out product pairs. Center: exact and estimated spectra. Right: post-fit permutation sensitivity; red bars are the three relevant inputs and gray bars are the $17$ irrelevant inputs.

Method	Kernel RMSE	Spectrum error	Subspace error
FSNM	$\mathbf{0.1880\pm 0.0227}$	$\mathbf{0.0363\pm 0.0114}$	$\mathbf{0.4689\pm 0.1062}$
ACE	$\mathbf{0.1818\pm 0.0152}$	$\mathbf{0.0375\pm 0.0065}$	$\mathbf{0.3754\pm 0.0689}$
uLSIF	$0.7102\pm 0.0042$	–	–
Kernel CCA	$0.6651\pm 0.0180$	$0.3618\pm 0.0117$	$0.6994\pm 0.0088$

Table 3: Baseline comparison on the twenty-dimensional tabular experiment with seventeen irrelevant input coordinates (mean

\pm

95\%

CI, bold = tied for best, as in Table 1 ).

Figure 5: Conditional queries obtained from the learned rank-three kernel. From left to right: conditional mean, conditional variance, and conditional tail probability. Solid black curves are exact and dashed red curves are computed from $\widehat{\kappa}$ .

Figure 6: Projected direct conditional CDF estimates and pointwise bootstrap bands at three values of $x$ using tree weak learners. Solid black curves are exact, dashed blue curves are the fitted estimates, and shaded regions are pointwise $95\%$ bands.

Figure 7: Independent case. Left: distribution of $\widehat{\kappa}$ over independent reference pairs, with the dashed line marking the population value one. Right: estimated singular values, on the same vertical scale as Figure 8 .

Figure 8: Nonlinear dependent case $Y=X^{2}+\epsilon$ . From left to right: test observations, learned centered density ratio kernel, and estimated singular values. Pearson’s correlation is near zero, whereas the learned kernel and the held-out permutation test detect dependence.

Group	Oxides	Motivation
Network formers	SiO 2 , B 2 O 3 , P 2 O 5	Backbone-forming oxides
Alkali modifiers	Li 2 O, Na 2 O, K 2 O	Network modification and fluxing
Optical-property components	TiO 2 , Nb 2 O 5 , Ta 2 O 5 , La 2 O 3	High-index optical-glass systems

Table 4: Ten-oxide family retained for the real-data experiment.

Figure 9: Fit of the restricted glass composition–RI. Left: training and validation losses for the selected rank- $20$ trajectory; early stopping chooses iteration $39$ . Right: fitted singular values and cumulative share of $\sum_{j}\widehat{\sigma}_{j}^{2}$ .

Figure 10: Prediction of RI using $g(y)=y$ . Left: the dashed line is perfect prediction and the dotted line is the training mean. Right: mean observed and predicted within prediction octiles, with pointwise $95\%$ normal intervals for the observed means.

Figure 11: Conditional queries. Left: for a random subset of $150$ held-out compositions, the shaded bands are validation-calibrated $70\%$ and $90\%$ intervals. Right: reliability of the direct indicator query $\widehat{\mathbb{P}}(\mathrm{RI}>1.8\mid x)$ in eight equal-count bins, with pointwise $95\%$ normal intervals for observed frequencies.

Explore similar work

Verifiable Regularity Criterion for Conditional Expectation Operators and Conditional Mean Embeddings with Applications to Nonparametric Regression, Bayesian Inverse Problems, and Koopman Operators
Aug 6, 2026Maximiliano Hertel, Ilja Klebanov, Manuel Schaller +1Kernel Hilbert Spaces Regularity
Optimization-based Control Group, Institute of Mathematics, Technische Universität Ilmenau, Germany · Department of Mathematics and Computer Science, Freie Universität Berlin, Germany · Faculty of Mathematics, Chemnitz University of Technology, Germany
One Operator for Many Densities: Amortized Approximation of Conditioning by Neural Operators
May 7, 2026Panos Tsimpos, Edoardo Calvello, Ayoub Belhadji +1Conditional Distribution Probabilistic Operator Learning
Operations Research Center MASSachusetts Institute of Technology Cambridge, MA 02139 · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125 · Laboratory for Information and Decision Systems Center for Computational Science and Engineering MASSachusetts Institute of Technology Cambridge, MA 02139 +2
A Kernel-based Stochastic Approximation Framework for Nonlinear Operator Learning
Sep 14, 2025Jia-Qi Yang, Lei ShiProbabilistic Operator Learning Nonlinear Operators
School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

Aug 6, 2026, math.DSJ/K move · Enter open · S save

Verifiable Regularity Criterion for Conditional Expectation Operators and Conditional Mean Embeddings with Applications to Nonparametric Regression, Bayesian Inverse Problems, and Koopman Operators

Maximiliano Hertel, Ilja Klebanov, Manuel Schaller, Karl Worthmann

Optimization-based Control Group, Institute of Mathematics, Technische Universität Ilmenau, Germany · Department of Mathematics and Computer Science, Freie Universität Berlin, Germany · Faculty of Mathematics, Chemnitz University of Technology, Germany

Kernel Hilbert Spaces Regularity Koopman Operator Exponential Family Bayesian Inverse Problems

Open paper AskPDF arXiv

Conditional expectation operators (CEOs) and their associated conditional mean embeddings (CMEs) play a central role across applied mathematics and machine learning, appearing in nonparametric regression, Bayesian inverse problems, and Koopman operator theory. A fundamental question is when a CEO maps a function space on

\mathcal{Y}

into a prescribed function space on

\mathcal{X}

, particularly a reproducing kernel Hilbert space (RKHS). We show that such mapping properties are characterized by the regularity of the Radon--Nikodym density of the conditional law, and establish a simple, verifiable sufficient condition under which the CEO is bounded and Hilbert--Schmidt. For RKHSs norm-equivalent to Sobolev spaces, this condition reduces to Sobolev regularity of the conditional density. The result yields a direct route to validate CME representations and error bounds for Galerkin-type and CME-based estimators. We verify the regularity condition in three settings: nonparametric regression, Bayesian inverse problems, and Koopman operator theory for stochastic dynamical systems. We show in each case that classical regularity results on the underlying probabilistic model imply the required mapping properties. The resulting framework offers a unified perspective on conditional expectation operators across probability, operator theory, kernel methods, and stochastic dynamics.

Abstract

Explore similar work

Verifiable Regularity Criterion for Conditional Expectation Operators and Conditional Mean Embeddings with Applications to Nonparametric Regression, Bayesian Inverse Problems, and Koopman Operators

One Operator for Many Densities: Amortized Approximation of Conditioning by Neural Operators

A Kernel-based Stochastic Approximation Framework for Nonlinear Operator Learning