Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

David Orlando Salazar Torres; Diyar Altinses; Andreas Schwung

cs.LGMar 17, 2026

Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

Authors: David Orlando Salazar Torres, Diyar Altinses, Andreas Schwung

Organizations: Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

Abstract

Neural network architectures designed for function parameterization, such as the Bag-of-Functions (BoF) framework, bridge the gap between the expressivity of deep learning and the interpretability of classical signal processing. However, these models are inherently sensitive to parameter initialization, as traditional data-agnostic schemes fail to capture the structural properties of the target signals, often leading to suboptimal convergence. In this work, we propose a prior-informed design strategy that leverages the intrinsic spectral and temporal structure of the data to guide both network initialization and architectural configuration. A principled methodology is introduced that uses the Fast Fourier Transform to extract dominant seasonal priors, informing model depth and initial states, and a residual-based regression approach to parameterize trend components. Crucially, this structural alignment enables a substantial reduction in encoder dimensionality without compromising reconstruction fidelity. A supporting theoretical analysis provides guidance on trend estimation under finite-sample regimes. Extensive experiments on synthetic and real-world benchmarks demonstrate that embedding data-driven priors significantly accelerates convergence, reduces performance variability across trials, and improves computational efficiency. Overall, the proposed framework enables more compact and interpretable architectures while outperforming standard initialization baselines, without altering the core training procedure.

Figures & tables

Category	Function	Parameters
Seasonality	$a_{1}\sin(a_{2}t+a_{3})$	$a_{1},a_{2},a_{3}$
	$a_{1}\cos(a_{2}t+a_{3})$	$a_{1},a_{2},a_{3}$
	$a_{1}\mathrm{sinc}(a_{2}t+a_{3})$	$a_{1},a_{2},a_{3}$
Trend	$a_{1}$	$a_{1}$
	$a_{1}t$	$a_{1}$
	$a_{1}t^{2}+a_{2}t$	$a_{1},a_{2}$

Table I : Representative basis functions categorized by component type [ 4 ] . The symbolic parameters

a_{j}

correspond to the learned parameters

\mathbf{z}_{i,a}

inferred by the BoF encoders.

Figure 1: Overview of the proposed data-informed framework. Extracted seasonal and trend statistics serve as priors that jointly determine the architectural topology, including stage depth and input dimensions, and initialize the Bag-of-Functions encoder weights to align the model with the data structure before training.

Figure 2: Spectral analysis of the synthetic dataset used for prior extraction. The average periodogram identifies dominant modes at $3.48$ Hz, $7.56$ Hz, and $12.29$ Hz, complemented by the spectral heatmap across all $N=2,000$ samples. The cumulative energy ratio reaches $\rho_{\text{spec}}=0.873$ by combining these three identified modes.

Panel of Figure 3 — Figure 3: Visual validation of the trend estimation on two dataset samples. The plots display the input signal (gray), the deseasonalized input (blue), and the ground truth trend (green dotted). The red markers indicate the specific observations selected for the OLS regression, yielding the estimated trend (red dashed).

Figure 4: Spectral analysis of the PJM Hourly dataset used for prior extraction. The average periodogram identifies dominant modes at $6.97$ and $13.99$ cycles/week, while the accompanying spectral heatmap displays the energy distribution across all 156 samples. The cumulative energy ratio reaches $\rho_{\text{spec}}=0.960$ by combining the two identified dominant modes.

Figure 5: Spectral analysis of the Thermal Power Plant dataset used for prior extraction. The average periodogram identifies distributed dominant modes at $1.42$ , $6.95$ , $14.00$ , and $21.20$ cycles/week, complemented by the spectral heatmap across all 1000 samples. The cumulative energy ratio reaches $\rho_{\text{spec}}=0.991$ by combining the four identified modes.

Panel of Figure 6 — Figure 6: Trend input configuration based on deseasonalized residuals. The blue lines represent the signal proxy obtained by removing the detected harmonics $\mathcal{K}$ . The red markers indicate the $n_{opt}$ equidistant observations selected to maximize stability, dimensioned according to the estimated residual volatility.

Dataset	Win. ( $W$ )	Frequency Analysis (Seasonal Priors)			Trend Analysis (Trend Priors)
Dataset	Win. ( $W$ )	Ratio ( $\rho$ )	Depth	Mode Initialization ( $\mathcal{K}$ )	Noise ( $\hat{\sigma}$ )	Input ( $n_{opt}$ )
Synthetic	100	0.873	3	$[3.48,7.56,12.29]$ Hz	0.850	11
PJM Hourly	168	0.960	2	$[6.97,13.99]$ cycles/win	0.048	3
Thermal Plant	168	0.991	4	$[14.00,6.95,1.42,21.20]$ cycles/win	0.486	13

Table II : Data-Driven Configuration and Initialization Priors. Consolidated architectural parameters derived from the two-stage diagnostic process. The table segregates the hyperparameters obtained from spectral decomposition (Seasonal Priors) and residual analysis (Trend Priors).

Figure 7: Distribution of the final MSE loss on the training (left) and testing (right) sets. The results summarize 10 independent trials for each of the four evaluated architectures. Reported values represent the average loss at the converged state. Within each box, the solid orange and dashed green lines denote the median and the mean, respectively.

Figure 8: Loss trajectories of the best-performing trial for each architecture. The solid lines represent the moving average, while the lighter shaded regions indicate batch volatility.

Figure 9: Evolution of the parameter displacement during training for the best-performing trial of each architecture. Solid lines represent moving averages, while shaded regions indicate batch-wise variability.

Training MSE
Dataset	BoF	H-BoF	I-BoF	IT-BoF
PJM	$0.0116\scriptstyle{\pm 0.0005}$	$0.0066\scriptstyle{\pm 0.0026}$	$0.0060\scriptstyle{\pm 0.0007}$	$\mathbf{0.0046\scriptstyle{\pm 0.0008}}$
TPP	$0.4423\scriptstyle{\pm 0.0178}$	$0.1957\scriptstyle{\pm 0.0431}$	$\mathbf{0.1715\scriptstyle{\pm 0.0025}}$	$0.1755\scriptstyle{\pm 0.0043}$
Testing MSE
PJM	$0.0155\scriptstyle{\pm 0.0008}$	$0.0130\scriptstyle{\pm 0.0131}$	$0.0099\scriptstyle{\pm 0.0024}$	$\mathbf{0.0074\scriptstyle{\pm 0.0011}}$
TPP	$0.4621\scriptstyle{\pm 0.0192}$	$0.2309\scriptstyle{\pm 0.0481}$	$\mathbf{0.1958\scriptstyle{\pm 0.0027}}$	$0.2035\scriptstyle{\pm 0.0077}$

Table III: Reconstruction performance on real-world benchmarks. The table reports the Mean Squared Error for the training and testing partitions across the PJM and Thermal Power Plant (TPP) datasets. Results are averaged over ten independent trials per architecture, and the best performance per dataset is highlighted in bold.

Panel of Figure 10 — Figure 10: Evolution of parameter displacement during training on real-world datasets for the best-performing trial of each architecture. Results are shown for the PJM (left) and Thermal Power Plant (right) datasets. Parameter displacement is measured relative to the initialization and averaged across network parameters. Solid lines denote moving averages, while shaded regions indicate batch-wise variability. Colors correspond to BoF (red), H-BoF (blue), I-BoF (green), and IT-BoF (pink).

Dataset	Model	Params ( $\times 10^{3}$ )	FLOPs ( $\times 10^{3}$ )	Latency (ms)	Throughput (samples/s)
Synthetic	BoF	62.25	61.54	4.484	3895.1
	H-BoF			4.417	3879.2
	I-BoF			4.401	3388.5
	IT-BoF	47.38	46.81	4.329	4012.0
PJM	BoF	63.23	62.75	2.993	5749.1
	H-BoF			2.993	5886.0

Table IV : Computational efficiency comparison (Parameters, FLOPs, Latency, and Throughput) for Synthetic and PJM datasets.

Dataset	Metric	Original	R-GAN [ 25 ]	LSTM-GAN [ 26 ]	ITF-GAN [ 27 ]	CR-VAE [ 28 ]	Wave-GAN [ 29 ]	CRNN-GAN [ 30 ]	Time-VAE [ 31 ]	ITF-VAE [ 15 ]	ITF-VAE-I [ 15 ]	ITF-VAE-Init [ 4 ]	ITF-VAE-Informed Init
Sine	Discriminative	-	0.1275	0.3255	0.2274	0.1756	0.4970	0.2580	0.2705	0.0740	0.1910	0.0660	0.0550
Sine	Predictive	0.0320	0.1000	0.1162	0.1061	0.1008	0.2566	0.0988	0.1616	0.0612	0.0707	0.0103	0.0087
Stocks	Discriminative	-	0.2143	0.4922	0.2325	0.4627	0.4582	0.1254	0.3632	0.1002	0.2709	0.0099	0.0050
Stocks	Predictive	0.0730	0.1020	0.1145	0.0967	0.1857	0.1180	0.1042	0.1200	0.0881	0.2395	0.0180	0.0170
Air	Discriminative	-	0.3809	0.4935	0.4521	0.4556	0.4878	0.3387	0.3015	0.3955	0.4376	0.3995	0.3565
Air	Predictive	0.2766	0.3806	0.5954	0.4787	0.5073	0.6428	0.4137	0.3850	0.5112	0.6136	0.4246	0.4145

Table V : Empirical Test Data Results. The discriminative and predictive tasks are performed across four different time series datasets and ten synthetic data generators. The predictive scores are compared to the original train and test data domain.

Neural Network Initialization

Neural network initialization studies how starting weights shape signal propagation, optimization, generalization, and reproducibility.

See all 51 papers

The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations
Mar 30, 2026Kushal Vyas, Alper Kayabasi, Daniel Kim +3Implicit Neural Representations Neural Network Initialization
Rice University · University of California, Riverside
RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization
Jun 15, 2026Yong Wang, Tao Zhou, Xuhui MengPDE Solving Spectral Bias
Institute of Interdisciplinary Research for Mathematics and Applied Science, School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan 430074, China · Institute of Computational Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China
Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation
Jun 4, 2026Mohammed Alsakabi, Kejia Hu, John M. Dolan +1Implicit Neural Representations Neural Network Initialization
Department of Electrical and Computer Engineering, College of Engineering · The Robotics Institute, School of Computer Science Carnegie Mellon University, Pittsburgh, PA, USA

Mar 30, 2026J/K move · Enter open · S save

The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

Kushal Vyas, Alper Kayabasi, Daniel Kim, Vishwanath Saragadam+2

Rice University · University of California, Riverside

Implicit Neural Representations Neural Network Initialization Synthetic Data Pretraining Self-Supervised Pre-Training

Open paper AskPDF arXiv

The approximation and convergence properties of implicit neural representations (INRs) are known to be highly sensitive to parameter initialization strategies. While several data-driven initialization methods demonstrate significant improvements over standard random sampling, the reasons for their success -- specifically, whether they encode classical statistical signal priors or more complex features -- remain poorly understood. In this study, we explore this phenomenon through a series of experimental analyses leveraging noise pretraining. We pretrain INRs on diverse noise classes (e.g., Gaussian, Dead Leaves, Spectral) and measure their ability to both fit unseen signals and encode priors for an inverse imaging task (denoising). Our analyses on image and video data reveal a surprising finding: simply pretraining on unstructured noise (Uniform, Gaussian) dramatically improves signal fitting capacity compared to all other baselines. However, unstructured noise also yields poor deep image priors for denoising. In contrast, we also find that noise with the classic

1/|f^α|

spectral structure of natural images achieves an excellent balance of signal fitting and inverse imaging capabilities, performing on par with the best data-driven initialization methods. This finding enables more efficient INR training in applications lacking sufficient prior domain-specific data. For more details, visit project page at https://kushalvyas.github.io/noisepretraining.html

Abstract

Neural Network Initialization

Explore similar work

The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization

Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation