cs.LGFeb 20, 2026

Hardware-Friendly Input Expansion for Accelerating Function Approximation

Authors: Hu Lou, Yin-Jun Gao, Dong-Xiao Zhang, Tai-Jiao Du, Jun-Jie Zhang, Jia-Rui Zhang

Organizations: Northwest Institute of Nuclear Technology, Xi’an, 710000, Shaanxi, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

One-dimensional function approximation is a fundamental problem in scientific computing and engineering applications. While neural networks possess powerful universal approximation capabilities, their optimization process is often hindered by flat loss landscapes induced by parameter-space symmetries, leading to slow convergence and poor generalization, particularly for high-frequency components. Inspired by the principle of symmetry breaking in physics, this paper proposes a hardware-friendly approach for function approximation through input-space expansion. The core idea involves augmenting the original one-dimensional input (e.g., xx) with constant values (e.g., ππ) to form a higher-dimensional vector (e.g., [π,π,x,π,π][π, π, x, π, π]), effectively breaking parameter symmetries without increasing the network's parameter count. We evaluate the method on ten representative one-dimensional functions, including smooth, discontinuous, high-frequency, and non-differentiable functions. Experimental results demonstrate that input-space expansion significantly accelerates training convergence (reducing LBFGS iterations by 12% on average) and enhances approximation accuracy (reducing final MSE by 66.3% for the optimal 5D expansion). Ablation studies further reveal the effects of different expansion dimensions and constant selections, with ππ consistently outperforming other constants. Our work proposes a low-cost, efficient, and hardware-friendly technique for algorithm design.

Figures & tables

ModelParametersIterations to TargetFinal MSE ( ×10−2\times 10^{-2} )Convergence Rate
Standard17951473 ±\pm 827.60 ±\pm 11.131.00 ±\pm 0.17
Expanded-318151461 ±\pm 1214.73 ±\pm 10.111.03 ±\pm 0.27
Expanded-518351416 ±\pm 1312.56 ±\pm 4.711.14 ±\pm 0.36
Expanded-718551464 ±\pm 1082.38 ±\pm 4.091.02 ±\pm 0.24
Adjusted18875496 ±\pm 104.99 ±\pm 6.660.95 ±\pm 0.02
Table 1: Convergence performance comparison (average across 10 functions, target MSE = 10−510^{-5} )
Figure 1: Comprehensive ablation study of symmetry breaking with expanded input dimensionality across 10 benchmark functions. Functions 1-5: Multi-frequency sine, Square wave, Sawtooth wave, Triangle wave, and Modulated sine.
Figure 2: Comprehensive ablation study of symmetry breaking with expanded input dimensionality across 10 benchmark functions. Functions 6-10: Frequency chirp, Duty-cycle square wave, Van der Pol oscillator, Weierstrass function, and Comb function.
CategoryRepresentative FunctionsMSE ImprovementIteration Reduction
SmoothF1, F5, F689.3%21.0%
DiscontinuousF2, F3, F758.6%0.0%
Non-differentiableF8, F938.4%0.0%
Complex SpectrumF4, F1097.2%31.1%
Overall AverageAll 10 functions66.3%12.0%
Table 2: Performance improvement by function category (Expanded-5 vs. Standard)
ModelIterations to 10−510^{-5}Final MSEConvergence Pattern
Standard5007.89×10−37.89\times 10^{-3}Slow, plateaued convergence
Expanded-35002.81×10−52.81\times 10^{-5}Improved final accuracy
Expanded-53871.40×10−61.40\times 10^{-6}Accelerated + accurate
Expanded-75002.72×10−42.72\times 10^{-4}Improved accuracy only
Adjusted5002.36×10−32.36\times 10^{-3}Similar to Standard
Table 3: Detailed performance on F1: Multi-frequency Sine
Figure 3: Experiments on ablation using different constant values in 5-dimensional extension. Functions 1-5: Multi-frequency sine, Square wave, Sawtooth wave, Triangle wave, and Modulated sine.
Figure 4: Experiments on ablation using different constant values in 5-dimensional extension. Functions 6-10: Frequency chirp, Duty-cycle square wave, Van der Pol oscillator, Weierstrass function, and Comb function.
ConfigurationPerformance RankRelative MSEConvergence Factor
All π\pi11.001.00
Mixed21.080.95
All ee31.150.92
All 141.320.87
All 051.470.81
Table 4: Performance ranking of different constant configurations (average across 10 functions)
More in this topic

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 236 papers

Explore similar work

CardsList
Mar 11, 2026

Efficient Approximation to Analytic and LpL^p functions by Height-Augmented ReLU Networks

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and LpL^p functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general LpL^p functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.

ZeYu Li, FengLei Fan, TieYong Zeng

Chinese University of Hong Kong, Department of Mathematics, Hong Kong, 999077, NT, Hong Kong SAR, China · City University of Hong Kong, Department of Data Science, Hong Kong, 999077, Kowloon, Hong Kong SAR, China · Guangzhou Nanfang College, School of Mathematics and Statistics, Guangzhou, 510970, Guangdong, China +1

Neural Network Approximation TheoryReLU Neural Networks
AskPDF
Feb 26, 2025

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

The architecture of a neural network and the choice of its activation function are both fundamental to its performance. Equally important is ensuring that these two elements are well matched, as their alignment is key to effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a model that combines sine-type activations with the multi-component and multi-layer structure of MMNNs. In an FMMNN, each component is represented as a trainable linear combination of fixed random sine-type basis functions, while multi-layer composition generates more complex and adaptive high-frequency features. We establish that FMMNNs retain exponential expressive power for function approximation even under a low-rank architectural structure. We also analyze the optimization landscape of FMMNNs and find it to be substantially more favorable than that of standard fully connected neural networks, especially for high-frequency targets. In addition, we propose a scaled random initialization method for the first-layer weights in FMMNNs, which accelerates training and improves final performance when sufficient samples are available. Extensive numerical experiments support our theoretical insights, showing that FMMNNs achieve strong accuracy and favorable convergence behavior on oscillatory function-approximation benchmarks.

Shijun Zhang, Hongkai Zhao, Yimin Zhong +1

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1

Fourier Feature EmbeddingsNeural Network OptimizationNeural Network Activation FunctionsNeural Network Approximation TheoryNeural Network ExpressivityModular Neural Networks
AskPDF
Jun 15, 2026

Functional Gradient Descent with Adaptive Representations

Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis. An interesting alternative is functional gradient descent (FGD), that is, gradient descent directly in function space, which benefits from strong convergence results and admits a clean theory. However, FGD is difficult to implement in practice because functional gradients are infinite-dimensional, and thus cannot be fully computed nor stored in memory. Existing implementations therefore rely on fixed approximations, which introduce approximation error. We propose a new, theoretically-grounded FGD algorithm that adapts the representation of the functional gradients over the course of optimization. By explicitly incorporating this approximation into the analysis, we establish convergence to a stationary point (for smooth losses) and to a global minimizer (under smoothness + a Polyak-Lojasiewicz-type condition) regardless of our approximations. To the best of our knowledge, this is the first implementable FGD method with such guarantees in a general setting. We demonstrate the effectiveness of our method on regression, numerical solution of PDEs, and modern computer vision. Across settings, our method consistently outperforms both FGD with fixed approximations and neural network baselines in efficiency and accuracy.

Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia +3

FGV EMAp · IMPA · Stanford University

Representation LearningGradient Descent
AskPDF