stat.MLSep 5, 2026

Functional Attentive Interpretable Regression

Authors: Haixu Wang, Tianyu Guan, Jiguo Cao

Organizations: Department of Mathematics and Statistics, University of Calgary · Department of Mathematics and Statistics, York University · Department of Statistics and Actuarial Science, Simon Fraser University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In function-on-function regression, the coefficient surface β(s,t)β(s,t) may exhibit complex support structure---from localized patches to global patterns such as disconnected regions, bands, or rings---where effect similarity does not align with Euclidean proximity. Projection-based methods that rely on fixed basis expansions can obscure such structure, while direct smoothing approaches risk oversmoothing the surface and its boundaries. We propose Functional Attentive Interpretable Regression (FAIR), which represents β(s,t)β(s,t) directly through coordinate features and uses self-attention to learn effect-adaptive neighborhoods, enabling information sharing at both local and global scales. A scalar compression network maps these learned representations to the coefficient surface. Sparsity and smoothness penalties applied over these neighborhoods promote localized support with coherent boundaries. We establish a sieve equivalence to tensor-product spline spaces and derive convergence rates. Simulations and applications to oceanographic and hydrological data demonstrate that FAIR recovers support geometry more accurately than existing methods while achieving superior prediction, particularly under sparse sampling.

Figures & tables

Figure 1: Motivation for an attention-based representation in function-on-function regression. (a): the true coefficient surface β(s,t)\beta(s,t) with annular support. (b): evaluation points in the input domain (s,t)(s,t) colored by β(s,t)\beta(s,t) . (c): the corresponding learned representations z(s,t)\bm{z}(s,t) , projected onto the first two principal components for visualization and colored by the same β(s,t)\beta(s,t) values.
Figure 2: FAIR architecture. Top: function-on-function regression with coefficient surface β(s,t)\beta(s,t) linking the predictor X(s)X(s) to the response Y(t)Y(t) . Bottom: coordinate-based parameterization of the surface coefficient: a coordinate (s,t)(s,t) is featurized by uθu_{\theta} , transformed by an attention module ϕθ\phi_{\theta} into an embedding zθ(s,t)\bm{z}_{\theta}(s,t) (via interactions with a set of key locations), and mapped by a scalar readout gθg_{\theta} to βθ(s,t)\beta_{\theta}(s,t) . The lower panels illustrate (left) a schematic attention neighborhood for a query location, (middle) an example embedding vector, and (right) an example learned coefficient surface.
Figure 3: Heatmaps show the attention weights assigned to reference locations (s′,t′)∈Gref(s^{\prime},t^{\prime})\in G_{\mathrm{ref}} for two query coordinates (top row: (s,t)=(0.3,0.3)(s,t)=(0.3,0.3) ; bottom row: (s,t)=(0.5,0.5)(s,t)=(0.5,0.5) ) across H=4H=4 heads (columns). In each panel, color intensity represents the normalized scaled dot-product attention weight ah((s,t),(s′,t′))a_{h}\big((s,t),(s^{\prime},t^{\prime})\big) (higher = more attended) used to form the head embedding zh(s,t)=∑(s′,t′)∈Grefah((s,t),(s′,t′))vh(s′,t′)\bm{z}_{h}(s,t)=\sum_{(s^{\prime},t^{\prime})\in G_{\mathrm{ref}}}a_{h}\big((s,t),(s^{\prime},t^{\prime})\big)\bm{v}_{h}(s^{\prime},t^{\prime}) . The star marks the query location; dashed contours are shown for visual reference.
GroupSymbolRoleCandidates
ArchitectureMMNumber of stacked attention blocks{1,2,3,...}\{1,2,3,...\}
 HHNumber of attention heads per block{22,23,24,...}\{2^{2},2^{3},2^{4},...\}
 DDFeature dimension of u(⋅)u(\cdot){24,25,...,28}\{2^{4},2^{5},...,2^{8}\}
 LuL_{u}Depth (number of layers) of u(⋅)u(\cdot){24,25,26}\{2^{4},2^{5},2^{6}\}
 LgL_{g}Depth (number of layers) of g(⋅)g(\cdot){23,24,25}\{2^{3},2^{4},2^{5}\}
 ActivationNonlinearity used in MLP blocksfixed (ReLU)
Table 1: Model specification and hyperparameters for FAIR.
Figure 4: Ground-truth coefficient surfaces used in the simulation study. Panels (A)–(C) show β(s,t)\beta(s,t) for the Annulus, Rectangle, and Two-disk settings on [0,1]2[0,1]^{2} , respectively. The three settings are designed to represent localized, axis-aligned, and irregular nonzero regions.
TypeMethodMSEISE all{}_{\text{all}}ISE zero{}_{\text{zero}}ISE nonzero{}_{\text{nonzero}}Concentration
AnnulusFAIR9.96 ⁣× ⁣10−4 (3.57 ⁣× ⁣10−5)\mathbf{9.96\!\times\!10^{-4}\ (3.57\!\times\!10^{-5})}3.68 ⁣× ⁣10−2 (3.16 ⁣× ⁣10−3)\mathbf{3.68\!\times\!10^{-2}\ (3.16\!\times\!10^{-3})}9.56 ⁣× ⁣10−39.56\!\times\!10^{-3} ( 1.08 ⁣× ⁣10−31.08\!\times\!10^{-3} )1.21 ⁣× ⁣10−1 (1.41 ⁣× ⁣10−2)\mathbf{1.21\!\times\!10^{-1}\ (1.41\!\times\!10^{-2})}0.761 (0.013)
TP4.46 ⁣× ⁣10−34.46\!\times\!10^{-3} ( 1.34 ⁣× ⁣10−41.34\!\times\!10^{-4} )1.22 ⁣× ⁣10−11.22\!\times\!10^{-1} ( 3.45 ⁣× ⁣10−43.45\!\times\!10^{-4} )1.62 ⁣× ⁣10−4 (1.80 ⁣× ⁣10−5)\mathbf{1.62\!\times\!10^{-4}\ (1.80\!\times\!10^{-5})}5.12 ⁣× ⁣10−15.12\!\times\!10^{-1} ( 9.28 ⁣× ⁣10−39.28\!\times\!10^{-3} )0.473 (0.008)
Bern4.66 ⁣× ⁣10−34.66\!\times\!10^{-3} ( 1.42 ⁣× ⁣10−41.42\!\times\!10^{-4} )1.27 ⁣× ⁣10−11.27\!\times\!10^{-1} ( 2.04 ⁣× ⁣10−42.04\!\times\!10^{-4} )3.12 ⁣× ⁣10−43.12\!\times\!10^{-4} ( 3.68 ⁣× ⁣10−53.68\!\times\!10^{-5} )5.33 ⁣× ⁣10−15.33\!\times\!10^{-1} ( 9.57 ⁣× ⁣10−39.57\!\times\!10^{-3} )0.203 (0.002)
FPCA2.20 ⁣× ⁣10−32.20\!\times\!10^{-3} ( 3.98 ⁣× ⁣10−43.98\!\times\!10^{-4} )6.58 ⁣× ⁣10−26.58\!\times\!10^{-2} ( 9.26 ⁣× ⁣10−39.26\!\times\!10^{-3} )1.63 ⁣× ⁣10−21.63\!\times\!10^{-2} ( 2.27 ⁣× ⁣10−32.27\!\times\!10^{-3} )1.91 ⁣× ⁣10−11.91\!\times\!10^{-1} ( 3.71 ⁣× ⁣10−23.71\!\times\!10^{-2} )0.534 (0.041)
RectangleFAIR6.27 ⁣× ⁣10−4 (4.73 ⁣× ⁣10−5)\mathbf{6.27\!\times\!10^{-4}\ (4.73\!\times\!10^{-5})}1.15 ⁣× ⁣10−2 (3.98 ⁣× ⁣10−3)\mathbf{1.15\!\times\!10^{-2}\ (3.98\!\times\!10^{-3})}1.35 ⁣× ⁣10−31.35\!\times\!10^{-3} ( 7.54 ⁣× ⁣10−47.54\!\times\!10^{-4} )1.48 ⁣× ⁣10−2 (5.66 ⁣× ⁣10−3)\mathbf{1.48\!\times\!10^{-2}\ (5.66\!\times\!10^{-3})}0.981 (0.004)
TP1.79 ⁣× ⁣10−21.79\!\times\!10^{-2} ( 1.04 ⁣× ⁣10−31.04\!\times\!10^{-3} )4.31 ⁣× ⁣10−14.31\!\times\!10^{-1} ( 4.18 ⁣× ⁣10−34.18\!\times\!10^{-3} )2.84 ⁣× ⁣10−52.84\!\times\!10^{-5} ( 1.45 ⁣× ⁣10−51.45\!\times\!10^{-5} )6.57 ⁣× ⁣10−16.57\!\times\!10^{-1} ( 1.70 ⁣× ⁣10−21.70\!\times\!10^{-2} )0.969 (0.004)
Table 2: Simulation results under dense and uniform grid. Mean (SD) over 100100 Monte Carlo replications for prediction and surface estimation accuracy across the three coefficient geometries (Annulus/Rectangle/Two-disk). Lower values are better for all metrics except Concentration, where higher is better. Best performance in each column within each geometry is shown in bold.
TypeMethodMSEISE all{}_{\text{all}}ISE zero{}_{\text{zero}}ISE nonzero{}_{\text{nonzero}}Concentration
AnnulusFAIR1.28 ⁣× ⁣10−3 (1.14 ⁣× ⁣10−4)\mathbf{1.28\!\times\!10^{-3}\ (1.14\!\times\!10^{-4})}4.52 ⁣× ⁣10−2 (2.05 ⁣× ⁣10−3)\mathbf{4.52\!\times\!10^{-2}\ (2.05\!\times\!10^{-3})}2.02 ⁣× ⁣10−22.02\!\times\!10^{-2} ( 1.67 ⁣× ⁣10−31.67\!\times\!10^{-3} )1.20 ⁣× ⁣10−1 (8.55 ⁣× ⁣10−3)\mathbf{1.20\!\times\!10^{-1}\ (8.55\!\times\!10^{-3})}0.640 (0.011)
TP1.04 ⁣× ⁣10−21.04\!\times\!10^{-2} ( 2.10 ⁣× ⁣10−32.10\!\times\!10^{-3} )1.02 ⁣× ⁣10−11.02\!\times\!10^{-1} ( 5.67 ⁣× ⁣10−45.67\!\times\!10^{-4} )9.82 ⁣× ⁣10−39.82\!\times\!10^{-3} ( 4.87 ⁣× ⁣10−44.87\!\times\!10^{-4} )3.88 ⁣× ⁣10−13.88\!\times\!10^{-1} ( 8.66 ⁣× ⁣10−38.66\!\times\!10^{-3} )0.275 (0.003)
Bern1.61 ⁣× ⁣10−21.61\!\times\!10^{-2} ( 2.61 ⁣× ⁣10−32.61\!\times\!10^{-3} )1.13 ⁣× ⁣10−11.13\!\times\!10^{-1} ( 6.41 ⁣× ⁣10−46.41\!\times\!10^{-4} )5.40 ⁣× ⁣10−3 (5.21 ⁣× ⁣10−4)\mathbf{5.40\!\times\!10^{-3}\ (5.21\!\times\!10^{-4})}4.56 ⁣× ⁣10−14.56\!\times\!10^{-1} ( 9.74 ⁣× ⁣10−39.74\!\times\!10^{-3} )0.240 (0.004)
FPCA7.18 ⁣× ⁣10−37.18\!\times\!10^{-3} ( 3.29 ⁣× ⁣10−33.29\!\times\!10^{-3} )8.95 ⁣× ⁣10−28.95\!\times\!10^{-2} ( 9.17 ⁣× ⁣10−49.17\!\times\!10^{-4} )3.57 ⁣× ⁣10−23.57\!\times\!10^{-2} ( 1.45 ⁣× ⁣10−31.45\!\times\!10^{-3} )2.48 ⁣× ⁣10−12.48\!\times\!10^{-1} ( 6.48 ⁣× ⁣10−36.48\!\times\!10^{-3} )0.309 (0.003)
RectangleFAIR4.60 ⁣× ⁣10−3 (8.18 ⁣× ⁣10−4)\mathbf{4.60\!\times\!10^{-3}\ (8.18\!\times\!10^{-4})}2.61 ⁣× ⁣10−2 (1.18 ⁣× ⁣10−2)\mathbf{2.61\!\times\!10^{-2}\ (1.18\!\times\!10^{-2})}3.66 ⁣× ⁣10−3 (2.57 ⁣× ⁣10−3)\mathbf{3.66\!\times\!10^{-3}\ (2.57\!\times\!10^{-3})}3.17 ⁣× ⁣10−2 (1.63 ⁣× ⁣10−2)\mathbf{3.17\!\times\!10^{-2}\ (1.63\!\times\!10^{-2})}0.972 (0.013)
TP5.48 ⁣× ⁣10−25.48\!\times\!10^{-2} ( 2.78 ⁣× ⁣10−22.78\!\times\!10^{-2} )1.87 ⁣× ⁣10−11.87\!\times\!10^{-1} ( 5.89 ⁣× ⁣10−35.89\!\times\!10^{-3} )3.84 ⁣× ⁣10−33.84\!\times\!10^{-3} ( 1.32 ⁣× ⁣10−31.32\!\times\!10^{-3} )2.88 ⁣× ⁣10−12.88\!\times\!10^{-1} ( 1.04 ⁣× ⁣10−21.04\!\times\!10^{-2} )0.924 (0.005)
Table 3: Simulation results under sparse and non-uniform sampling. Mean (SD) over 100100 Monte Carlo replications for prediction and surface estimation accuracy across the three coefficient geometries (Annulus/Rectangle/Two-disk). Lower values are better for all metrics except Concentration, where higher is better. Best performance in each column within each geometry is shown in bold.
Figure 5: Estimated coefficient surfaces β^(s,t)\hat{\beta}(s,t) for the hydrological data under four methods (columns: FAIR, tensor product, Bernstein polynomial, FPCA) and two sampling designs (rows: dense, sparse). The horizontal and vertical axes represent normalized precipitation time ( ss ) and streamflow time ( tt ), respectively.
 DenseSparse
MethodMSESup-NormMSESup-Norm
FAIR0.069 (0.006)0.631 (0.038)0.073 (0.005)0.655 (0.026)
FPCA0.071 (0.006)0.646 (0.028)0.083 (0.008)0.695 (0.055)
TP0.080 (0.006)0.663 (0.035)0.082 (0.005)0.677 (0.037)
Bern0.082 (0.005)0.675 (0.035)0.082 (0.005)0.677 (0.035)
Table 4: Mean squared error (MSE) and sup-norm prediction error for dense and sparse sampling designs (mean with standard deviation in parentheses over 100 random train–test splits). Best performance in each column is shown in bold.
Figure 6: Estimated coefficient surfaces β^(s,t)\hat{\beta}(s,t) for the Hawaii ocean data under four methods (columns: FAIR, tensor product, Bernstein polynomial, FPCA) and two sampling designs (rows: dense, sparse).
 DenseSparse
MethodMSESup-NormMSESup-Norm
FAIR0.482 (0.149)1.246 (0.492)0.575 (0.084)1.243 (0.585)
FPCA0.578 (0.119)1.374 (0.528)0.693 (0.107)1.704 (1.052)
TP0.658 (0.128)1.455 (0.541)0.721 (0.115)1.513 (0.960)
Bern0.672 (0.126)1.462 (0.578)0.720 (0.108)1.520 (0.980)
Table 5: Mean squared error (MSE) and sup-norm prediction error for dense and sparse sampling designs (mean with standard deviation in parentheses over 100 random train–test splits).

Explore similar work

CardsList

  1. Functional linear regression from sparse to dense designs: a pooling-ridge method and minimax optimality

    Aug 26, 2026Shunxing Yan, Fang YaoLinear RegressionMinimax Estimation

    Department of Statistics and Data Science, Tsinghua University · School of Mathematical Sciences, Center for Statistical Science, Peking University

  2. Functional Attention: From Pairwise Affinities to Functional Correspondences

    May 29, 2026Jiefang Xiao, Maolin Gao, Simon Weber +2PDE Operator LearningAttention Mechanisms

    Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Germany · PIXL, Department of Computer Science, University of Oxford, United Kingdom +1

  3. Sparse Bayesian Deep Functional Learning with Structured Region Selection

    Feb 24, 2026Xiaoxian Zhu, Yingmeng Li, Shuangge Ma +1Bayesian Neural NetworksFeature Selection

    School of Statistics and Data Science, Shanghai University of Finance and Economics · Department of Biostatistics, Yale School of Public Health