Organizations: School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, 611731, Sichuan, China. · School of Mathematical and Statistics, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China. · Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, Taiyuan, 030006, Shanxi, China.
Multivariate function approximation is a fundamental problem in machine learning. Classic multivariate function approximations rely on hand-crafted basis functions (e.g., polynomial basis and Fourier basis), which limits their approximation ability and data adaptation ability, resulting in unsatisfactory performance. To address these challenges, we introduce the neural basis function by leveraging an untrained neural network as the basis function. Equipped with the proposed neural basis function, we suggest the neural approximation (NeuApprox) paradigm for multivariate function approximation. Specifically, the underlying multivariate function behind the multi-dimensional data is decomposed into a sum of block terms. The clear physically-interpreted block term is the product of expressive neural basis functions and their corresponding learnable coefficients, which allows us to faithfully capture distinct components of the underlying data and also flexibly adapt to new data by readily fine-tuning the neural basis functions. Attributed to the elaborately designed block terms, the suggested NeuApprox enjoys strong approximation ability and flexible data adaptation ability over the hand-crafted basis function-based methods. We also theoretically prove that NeuApprox can approximate any multivariate continuous function to arbitrary accuracy. Extensive experiments on diverse multi-dimensional datasets (including multispectral images, light field data, videos, traffic data, and point cloud data) demonstrate the promising performance of NeuApprox in terms of both approximation capability and adaptability.
Figures & tables
Figure 1: Comparison of multivariate function approximation behind multi-dimensional data (i.e., MSI Painting and MSI Toy with SR = 0.1) based on different basis function. For different data, the hand-crafted basis functions (e.g., polynomial, Fourier, and Gaussian basis functions) are predefined, while the proposed neural basis functions are data-adaptive. The enlarged patch and corresponding residuals are displayed under the recovered results.
Figure 2: Different block terms of the proposed NeuApprox and their corresponding Fourier spectrum images (bottom) on MSI Toy with SR = 0.1. (a)-(c) first, second, and third block terms of the proposed NeuApprox with corresponding Fourier spectrum images.
Figure 3: Illustration of the proposed NeuApprox for a three-variable function f(x,y,z) . Leveraging multiple block terms to approximate the target function behind the multi-dimensional data. Each block term, which can inherently capture different components of the target function, is formed by the product of a neural basis function and its corresponding coefficient. From top to bottom: multi-dimensional data, cross-section of the three-variable functions behind the multi-dimensional data, and the flowchart of our proposed NeuApprox. Here, (x,y,z) is the coordinate; fθ1j(⋅) , fθ2j(⋅) , and fθ3j(⋅) are the suggested neural basis functions for block term index j=1,2,⋯,T .
Algorithm 1 NeuApprox-based Multi-Dimensional Data Completion
Sampling rate
0.1
0.15
0.2
0.25
0.3
Data
Method
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
MSIs Toys Painting (256 × 256 × 31)
Observed
12.30
0.315
0.247
12.54
0.349
0.240
12.81
0.382
0.232
13.09
0.413
0.225
13.38
0.444
0.218
TRLRF
30.08
0.919
0.031
31.58
0.939
0.026
32.79
0.953
0.023
33.73
0.961
0.021
34.54
0.967
0.019
TNN
29.20
0.928
0.035
31.64
0.956
0.026
33.73
0.972
0.021
35.71
0.981
0.016
37.62
0.988
0.013
FCTN
32.08
0.940
0.025
33.74
0.956
0.021
34.25
0.962
0.019
35.63
0.970
0.017
36.12
0.973
0.016
FSA
21.54
0.664
0.084
21.55
0.669
0.084
21.75
0.675
0.082
21.84
0.680
0.081
21.78
0.687
0.082
Table 1: The average quantitative results by different methods for multi-dimensional image inpainting. The best and second-best values are highlighted. (PSNR ↑ , SSIM ↑ , and NRMSE ↓ )
Figure 4: The results of multi-dimensional image inpainting with SR=0.15 by different methods on MSIs (i.e., Toys and Painting ), videos (i.e., Foreman and Carphone ), and light field data (i.e., Greek and Origami ).
Missing rate
0.1
0.15
0.2
0.25
0.3
Data
Method
RMSE
MAPE
RMSE
MAPE
RMSE
MAPE
RMSE
MAPE
RMSE
MAPE
Guangzhou
HaLRTC
40.09
1.000
40.97
1.000
39.70
1.000
39.55
1.000
39.23
1.000
LRTC-TNN
40.09
1.000
40.97
1.000
39.70
1.000
39.55
1.000
39.23
1.000
LATC
40.09
1.000
40.97
1.000
39.70
1.000
39.55
1.000
39.23
1.000
FSA
10.25
0.241
9.71
0.227
9.48
0.220
9.45
0.220
10.2111
0.237
LRTFR
9.55
0.215
9.44
0.213
9.31
0.215
9.69
0.217
9.8618
0.225
Table 2: The quantitative results by different methods for traffic data completion. The best and second-best values are highlighted. (RMSE ↓ and MAPE ↓ )
Figure 5: The results of traffic data completion by different methods. From top to bottom: the 58-th slice of PEMS07 , 40-th slice of Seattle , and 148-th slice of Seattle .
Data
Doll
Duck
Frog
Mario
Rabbit
Average
Methods
NRMSE
R-Square
NRMSE
R-Square
NRMSE
R-Square
NRMSE
R-Square
NRMSE
R-Square
NRMSE
R-Square
DT
0.2032
0.9074
0.1793
0.6991
0.1510
0.9140
0.2658
0.7573
0.2145
0.5996
0.2028
0.7747
KNR
0.1426
0.9544
0.1355
0.8282
0.1287
0.9375
0.2112
0.8469
0.1621
0.7715
0.1561
0.8677
RF
0.1698
0.9353
0.1375
0.8231
0.1252
0.9199
0.2022
0.8596
0.1649
0.7635
0.1651
0.7461
FSA
0.2061
0.9048
0.2085
0.5932
0.2291
0.8021
0.3296
0.6270
0.2689
0.3710
0.2485
0.7849
LRTFR
0.1433
0.9588
0.1224
0.8387
0.1252
0.9214
0.2139
0.8515
0.1636
0.7729
0.1536
0.8687
Table 3: The quantitative results by different methods for point cloud data completion. The best and second-best values are highlighted. (NRMSE ↓ and SR-Square ↓ )
Figure 6: The results of point cloud data completion by different methods. From top to bottom: the visual results of Doll , Duck , Frog , Mario , and Rabbit by different methods.
Sampling rate
0.1
0.15
0.2
0.25
0.3
Basis function
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
PSNR
SSIM
NRMSE
Poly
22.15
0.510
0.078
22.29
0.520
0.077
23.87
0.520
0.064
25.86
0.610
0.051
29.99
0.875
0.032
Fourier
26.27
0.696
0.049
27.77
0.769
0.041
28.43
0.804
0.038
29.66
0.825
0.033
30.30
0.827
0.031
Gaussian
30.07
0.827
0.031
30.43
0.836
0.030
31.30
0.867
0.027
31.66
0.876
0.026
32.22
0.892
0.024
Neural
33.28
0.864
0.022
35.82
0.919
0.016
37.45
0.942
0.013
38.66
0.949
0.012
39.49
0.968
0.011
Table 4: The average quantitative results of the proposed NeuApprox with different neural basis functions on MSI Toy and Painting with different SRs.
Data
Sampling rate
0.1
0.15
0.2
0.25
0.3
Method
PSNR
SSIM
Time
PSNR
SSIM
Time
PSNR
SSIM
Time
PSNR
SSIM
Time
PSNR
SSIM
Time
Painting
Pretraining
29.25
0.759
2.35
30.97
0.813
3.67
33.52
0.891
2.98
34.46
0.910
3.75
35.78
0.930
3.48
Fine-Tuning
30.92
0.802
4.07
32.65
0.855
7.15
35.19
0.915
8.23
36.69
0.939
9.74
37.88
0.952
7.14
Training from Scratch
31.32
0.839
28.55
33.96
0.903
30.62
35.35
0.928
31.72
36.83
0.946
41.09
37.78
0.980
40.53
Toys
Pretraining
29.59
0.758
2.14
30.83
0.799
2.55
33.32
0.889
2.77
34.82
0.913
3.49
35.70
0.932
3.74
Fine-Tuning
32.04
0.820
4.18
35.19
0.905
6.47
38.05
0.953
7.62
39.50
0.967
9.56
40.44
0.974
10.53
Table 5: Inpainting results of different training strategies on MSI Toys and Painting with different SRs. The neural basis functions are pretrained on out-of-distribution datasets.
Figure 7: The PSNR curves of the recovered MSI Toy and Painting by NeuApprox with different numbers of layers. From left to right: The PSNR curves with respect to number of layers of neural networks used in basis functions on MSI Toy and Painting with SR=0.1, 0.15, 0.2, 0.25, and 0.3.
Figure 8: The PSNR values of the recovered by the proposed NeuApprox with single block term representation and multiple block terms on MSI Toy with respect to the number of parameters. From left to right: The PSNR curves with respect to number of parameters in NeuApprox on MSI Toy and Painting with SR=0.1, 0.15, 0.2, 0.25, and 0.3.
Figure 9: Different block terms (top) of the proposed NeuApprox and their Fourier spectrum images (bottom) on MSI Toy with SR=0.1. (a)-(c) first, second, and third block terms of the proposed NeuApprox with corresponding Fourier spectrum images.
Figure 10: The PSNR curves of the recovered MSI Toy and Painting by NeuApprox with different coefficient sizes. From top to bottom: The PSNR curves with respect to R1 values and the PSNR curves with respect to R2 values. From left to right: The PSNR curves of the recovered MSI Toy and Painting with SR=0.1, 0.15, 0.2, 0.25, and 0.3.
Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom. However, practical computation operates under finite precision: parameters must be encoded using a finite number of bits. Therefore, approximation efficiency should be evaluated in terms of computational bit complexity, which is intrinsically connected to the metric entropy of the underlying function class. In this work, we develop a unified approximation framework based on binary encoding and metric entropy. We analyze classical methods (including polynomial approximation, sparse grids, and finite elements) as well as shallow and deep neural networks, and compare their approximation rates for function classes with comparable metric entropy. We observe that, when evaluated in terms of bits, most classical methods are in general suboptimal relative to the intrinsic limits dictated by metric entropy, while neural network methods may exhibit different behaviors. We show that when complexity is measured in bits rather than parameters, no method fundamentally exceeds the approximation order achieved by classical approaches. Our results also indicate that many seeming advantages of neural networks, including dimension-independent rates and superconvergence phenomena, stem from differences in function class complexity rather than intrinsic architectural superiority. In this sense, the traditional curse of dimensionality can be misleading; the fundamental limitation is instead a curse of bit complexity, governed by metric entropy.
Tong Mao, Jinchao Xu
King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia
Classical representation systems such as Fourier series, wavelets, and fixed dictionaries provide analytically tractable basis expansions, but they are not intrinsically adapted to the empirical structure of modern high-dimensional data. Neural networks overcome this limitation by learning features from data, yet they do so through layered nonlinear parameterizations that often sacrifice interpretability, explicit control over basis structure, and mathematical transparency. In this manuscript we develop a non-neural alternative that learns basis functions directly from data through variational optimization. The proposed framework, termed Data Driven Variational Basis Learning (DVBL), treats basis atoms as primary optimization variables and learns them jointly with sample-specific coefficients and, when appropriate, a latent linear evolution operator. This yields a data-adaptive basis expansion that remains explicit, interpretable, and amenable to rigorous analysis. We formulate the model, establish existence of minimizers, prove blockwise descent properties for an alternating minimization algorithm, give conditions for coefficient recovery and basis identifiability, and show how manifold and dynamical regularization can be integrated without invoking neural architectures. We also discuss the conceptual novelty of the framework relative to classical dictionary learning, spectral methods, Koopman operator methods, and deep representation learning.
This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and Lp functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general Lp functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.
ZeYu Li, FengLei Fan, TieYong Zeng
Chinese University of Hong Kong, Department of Mathematics, Hong Kong, 999077, NT, Hong Kong SAR, China · City University of Hong Kong, Department of Data Science, Hong Kong, 999077, Kowloon, Hong Kong SAR, China · Guangzhou Nanfang College, School of Mathematics and Statistics, Guangzhou, 510970, Guangdong, China +1