Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

David Orlando Salazar Torres; Diyar Altinses; Andreas Schwung

cs.LGMar 17, 2026

Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

Authors: David Orlando Salazar Torres, Diyar Altinses, Andreas Schwung

Organizations: Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

Abstract

Neural network architectures designed for function parameterization, such as the Bag-of-Functions (BoF) framework, bridge the gap between the expressivity of deep learning and the interpretability of classical signal processing. However, these models are inherently sensitive to parameter initialization, as traditional data-agnostic schemes fail to capture the structural properties of the target signals, often leading to suboptimal convergence. In this work, we propose a prior-informed design strategy that leverages the intrinsic spectral and temporal structure of the data to guide both network initialization and architectural configuration. A principled methodology is introduced that uses the Fast Fourier Transform to extract dominant seasonal priors, informing model depth and initial states, and a residual-based regression approach to parameterize trend components. Crucially, this structural alignment enables a substantial reduction in encoder dimensionality without compromising reconstruction fidelity. A supporting theoretical analysis provides guidance on trend estimation under finite-sample regimes. Extensive experiments on synthetic and real-world benchmarks demonstrate that embedding data-driven priors significantly accelerates convergence, reduces performance variability across trials, and improves computational efficiency. Overall, the proposed framework enables more compact and interpretable architectures while outperforming standard initialization baselines, without altering the core training procedure.

Figures & tables

Category	Function	Parameters
Seasonality	$a_{1}\sin(a_{2}t+a_{3})$	$a_{1},a_{2},a_{3}$
	$a_{1}\cos(a_{2}t+a_{3})$	$a_{1},a_{2},a_{3}$
	$a_{1}\mathrm{sinc}(a_{2}t+a_{3})$	$a_{1},a_{2},a_{3}$
Trend	$a_{1}$	$a_{1}$
	$a_{1}t$	$a_{1}$
	$a_{1}t^{2}+a_{2}t$	$a_{1},a_{2}$

Table I : Representative basis functions categorized by component type [ 4 ] . The symbolic parameters

a_{j}

correspond to the learned parameters

\mathbf{z}_{i,a}

inferred by the BoF encoders.

Figure 1: Overview of the proposed data-informed framework. Extracted seasonal and trend statistics serve as priors that jointly determine the architectural topology, including stage depth and input dimensions, and initialize the Bag-of-Functions encoder weights to align the model with the data structure before training.

Figure 2: Spectral analysis of the synthetic dataset used for prior extraction. The average periodogram identifies dominant modes at $3.48$ Hz, $7.56$ Hz, and $12.29$ Hz, complemented by the spectral heatmap across all $N=2,000$ samples. The cumulative energy ratio reaches $\rho_{\text{spec}}=0.873$ by combining these three identified modes.

Panel of Figure 3 — Figure 3: Visual validation of the trend estimation on two dataset samples. The plots display the input signal (gray), the deseasonalized input (blue), and the ground truth trend (green dotted). The red markers indicate the specific observations selected for the OLS regression, yielding the estimated trend (red dashed).

Figure 4: Spectral analysis of the PJM Hourly dataset used for prior extraction. The average periodogram identifies dominant modes at $6.97$ and $13.99$ cycles/week, while the accompanying spectral heatmap displays the energy distribution across all 156 samples. The cumulative energy ratio reaches $\rho_{\text{spec}}=0.960$ by combining the two identified dominant modes.

Figure 5: Spectral analysis of the Thermal Power Plant dataset used for prior extraction. The average periodogram identifies distributed dominant modes at $1.42$ , $6.95$ , $14.00$ , and $21.20$ cycles/week, complemented by the spectral heatmap across all 1000 samples. The cumulative energy ratio reaches $\rho_{\text{spec}}=0.991$ by combining the four identified modes.

Panel of Figure 6 — Figure 6: Trend input configuration based on deseasonalized residuals. The blue lines represent the signal proxy obtained by removing the detected harmonics $\mathcal{K}$ . The red markers indicate the $n_{opt}$ equidistant observations selected to maximize stability, dimensioned according to the estimated residual volatility.

Dataset	Win. ( $W$ )	Frequency Analysis (Seasonal Priors)			Trend Analysis (Trend Priors)
Dataset	Win. ( $W$ )	Ratio ( $\rho$ )	Depth	Mode Initialization ( $\mathcal{K}$ )	Noise ( $\hat{\sigma}$ )	Input ( $n_{opt}$ )
Synthetic	100	0.873	3	$[3.48,7.56,12.29]$ Hz	0.850	11
PJM Hourly	168	0.960	2	$[6.97,13.99]$ cycles/win	0.048	3
Thermal Plant	168	0.991	4	$[14.00,6.95,1.42,21.20]$ cycles/win	0.486	13

Table II : Data-Driven Configuration and Initialization Priors. Consolidated architectural parameters derived from the two-stage diagnostic process. The table segregates the hyperparameters obtained from spectral decomposition (Seasonal Priors) and residual analysis (Trend Priors).

Figure 7: Distribution of the final MSE loss on the training (left) and testing (right) sets. The results summarize 10 independent trials for each of the four evaluated architectures. Reported values represent the average loss at the converged state. Within each box, the solid orange and dashed green lines denote the median and the mean, respectively.

Figure 8: Loss trajectories of the best-performing trial for each architecture. The solid lines represent the moving average, while the lighter shaded regions indicate batch volatility.

Figure 9: Evolution of the parameter displacement during training for the best-performing trial of each architecture. Solid lines represent moving averages, while shaded regions indicate batch-wise variability.

Training MSE
Dataset	BoF	H-BoF	I-BoF	IT-BoF
PJM	$0.0116\scriptstyle{\pm 0.0005}$	$0.0066\scriptstyle{\pm 0.0026}$	$0.0060\scriptstyle{\pm 0.0007}$	$\mathbf{0.0046\scriptstyle{\pm 0.0008}}$
TPP	$0.4423\scriptstyle{\pm 0.0178}$	$0.1957\scriptstyle{\pm 0.0431}$	$\mathbf{0.1715\scriptstyle{\pm 0.0025}}$	$0.1755\scriptstyle{\pm 0.0043}$
Testing MSE
PJM	$0.0155\scriptstyle{\pm 0.0008}$	$0.0130\scriptstyle{\pm 0.0131}$	$0.0099\scriptstyle{\pm 0.0024}$	$\mathbf{0.0074\scriptstyle{\pm 0.0011}}$
TPP	$0.4621\scriptstyle{\pm 0.0192}$	$0.2309\scriptstyle{\pm 0.0481}$	$\mathbf{0.1958\scriptstyle{\pm 0.0027}}$	$0.2035\scriptstyle{\pm 0.0077}$

Table III: Reconstruction performance on real-world benchmarks. The table reports the Mean Squared Error for the training and testing partitions across the PJM and Thermal Power Plant (TPP) datasets. Results are averaged over ten independent trials per architecture, and the best performance per dataset is highlighted in bold.

Panel of Figure 10 — Figure 10: Evolution of parameter displacement during training on real-world datasets for the best-performing trial of each architecture. Results are shown for the PJM (left) and Thermal Power Plant (right) datasets. Parameter displacement is measured relative to the initialization and averaged across network parameters. Solid lines denote moving averages, while shaded regions indicate batch-wise variability. Colors correspond to BoF (red), H-BoF (blue), I-BoF (green), and IT-BoF (pink).

Dataset	Model	Params ( $\times 10^{3}$ )	FLOPs ( $\times 10^{3}$ )	Latency (ms)	Throughput (samples/s)
Synthetic	BoF	62.25	61.54	4.484	3895.1
	H-BoF			4.417	3879.2
	I-BoF			4.401	3388.5
	IT-BoF	47.38	46.81	4.329	4012.0
PJM	BoF	63.23	62.75	2.993	5749.1
	H-BoF			2.993	5886.0

Table IV : Computational efficiency comparison (Parameters, FLOPs, Latency, and Throughput) for Synthetic and PJM datasets.

Dataset	Metric	Original	R-GAN [ 25 ]	LSTM-GAN [ 26 ]	ITF-GAN [ 27 ]	CR-VAE [ 28 ]	Wave-GAN [ 29 ]	CRNN-GAN [ 30 ]	Time-VAE [ 31 ]	ITF-VAE [ 15 ]	ITF-VAE-I [ 15 ]	ITF-VAE-Init [ 4 ]	ITF-VAE-Informed Init
Sine	Discriminative	-	0.1275	0.3255	0.2274	0.1756	0.4970	0.2580	0.2705	0.0740	0.1910	0.0660	0.0550
Sine	Predictive	0.0320	0.1000	0.1162	0.1061	0.1008	0.2566	0.0988	0.1616	0.0612	0.0707	0.0103	0.0087
Stocks	Discriminative	-	0.2143	0.4922	0.2325	0.4627	0.4582	0.1254	0.3632	0.1002	0.2709	0.0099	0.0050
Stocks	Predictive	0.0730	0.1020	0.1145	0.0967	0.1857	0.1180	0.1042	0.1200	0.0881	0.2395	0.0180	0.0170
Air	Discriminative	-	0.3809	0.4935	0.4521	0.4556	0.4878	0.3387	0.3015	0.3955	0.4376	0.3995	0.3565
Air	Predictive	0.2766	0.3806	0.5954	0.4787	0.5073	0.6428	0.4137	0.3850	0.5112	0.6136	0.4246	0.4145

Table V : Empirical Test Data Results. The discriminative and predictive tasks are performed across four different time series datasets and ten synthetic data generators. The predictive scores are compared to the original train and test data domain.

Neural Network Initialization

Neural network initialization studies how starting weights shape signal propagation, optimization, generalization, and reproducibility.

See all 51 papers

Mar 30, 2026

The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

The approximation and convergence properties of implicit neural representations (INRs) are known to be highly sensitive to parameter initialization strategies. While several data-driven initialization methods demonstrate significant improvements over standard random sampling, the reasons for their success -- specifically, whether they encode classical statistical signal priors or more complex features -- remain poorly understood. In this study, we explore this phenomenon through a series of experimental analyses leveraging noise pretraining. We pretrain INRs on diverse noise classes (e.g., Gaussian, Dead Leaves, Spectral) and measure their ability to both fit unseen signals and encode priors for an inverse imaging task (denoising). Our analyses on image and video data reveal a surprising finding: simply pretraining on unstructured noise (Uniform, Gaussian) dramatically improves signal fitting capacity compared to all other baselines. However, unstructured noise also yields poor deep image priors for denoising. In contrast, we also find that noise with the classic

1/|f^α|

spectral structure of natural images achieves an excellent balance of signal fitting and inverse imaging capabilities, performing on par with the best data-driven initialization methods. This finding enables more efficient INR training in applications lacking sufficient prior domain-specific data. For more details, visit project page at https://kushalvyas.github.io/noisepretraining.html

Kushal Vyas, Alper Kayabasi, Daniel Kim +3

Rice University · University of California, Riverside

Implicit Neural Representations Neural Network Initialization Synthetic Data Pretraining Self-Supervised Pre-Training

Ask PDF

Jun 15, 2026

RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization

Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNN, a reparameterized neural network model with ReLU or tanh activations designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one-, two-, and four-dimensional function approximations, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning for an earthquake problem using real data, demonstrate that RepNN improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features. These results indicate that RepNN provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.

Yong Wang, Tao Zhou, Xuhui Meng

Institute of Interdisciplinary Research for Mathematics and Applied Science, School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan 430074, China · Institute of Computational Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China

PDE Solving Spectral Bias Neural Network Approximation Theory

Ask PDF

Jun 4, 2026

Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation

Existing implicit neural representation (INR) approaches suffer from stochastic initialization that does not guarantee consistent or high-quality performance across runs, with variations reaching more than 2.5 dB (~78%) in image regression. This variation is problematic for scientific computing and simulation, where result reproducibility is crucial. To address this problem, we present Jacobi-Anger Sinusoidal Representation Network (JA-SIREN), a deterministic initialization scheme for sinusoidal networks grounded in classical spectral analysis. By computing the Discrete Sine Transform (DST) of the target signal and leveraging the Jacobi-Anger expansion, we derive closed-form weights for a two-layer sinusoidal MLP that analytically match the network's initial spectral response to the target signal, requiring no random seed or additional hyperparameter tuning. On the Kodak dataset, JA-SIREN achieves a mean PSNR of 67.18 dB, a 21.30 dB improvement over the best baseline. This is achieved with zero run-to-run variance, confirming that spectrally-informed initialization is a more effective and reproducible alternative to stochastic initialization for sinusoidal INRs.

Mohammed Alsakabi, Kejia Hu, John M. Dolan +1

Department of Electrical and Computer Engineering, College of Engineering · The Robotics Institute, School of Computer Science Carnegie Mellon University, Pittsburgh, PA, USA

Implicit Neural Representations Neural Network Initialization ML Reproducibility Sinusoidal Representation Networks

Ask PDF

Abstract

Neural Network Initialization

Explore similar work

The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization

Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation