A Novel Fourier Feature Network for Solving Partial Differential Equations

Qihong Yang; Zhijie Su; Yangtao Deng; Qiaolin He

cs.LGAug 13, 2026

A Novel Fourier Feature Network for Solving Partial Differential Equations

Authors: Qihong Yang, Zhijie Su, Yangtao Deng, Qiaolin He

Organizations: School of Mathematics, Sichuan University, city=Chengdu, postcode=610065, country=China

Abstract

Building on the foundation of single-hidden-layer neural networks, Fourier Feature Networks (FENs) are proposed, which incorporate Fourier features using $\cos$ , $\sin$ , or a combination of both. Similar to Extreme Learning Machines (ELMs), FENs employ a single-hidden-layer architecture to generate a set of basis functions. The target function is then approximated as a linear combination of these basis functions, with the coefficients determined using the least squares method. However, unlike ELMs, which often rely on affine transformations to improve representational power, FENs can achieve high-precision solutions without requiring such transformations on the input variables. To evaluate the representational capacity of these networks, we search for an optimal scaling factor within a predefined range for the randomly initialized and fixed weights and biases. By adjusting this scaling factor, we ensure a fair comparison between FENs and ELMs using various activation functions, such as $\text{sigmoid}$ , $\tanh$ , and $\text{swish}$ . Our numerical experiments demonstrate that FENs consistently achieve higher accuracy than ELMs.

Figures & tables

Figure 1: Error variation with the scaling factor $\rho$ for different numbers of basis functions.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,10]$	0.01	1.52	2.19	2.94	3.81
$\tanh$	$(0,10]$	0.01	0.76	1.47	1.55	1.92
swish	$(0,10]$	0.01	1.49	2.61	3.06	3.76
$\cos$	$(0,50]$	0.1	7.4	9.5	9.7	22.8
$\sin$	$(0,50]$	0.1	7.7	9.1	10.3	22.7

Table 1: Function approximation: Parameters used in approximating the two-dimensional function defined by Equation ( 25 ).

Figure 2: Function approximation: $L_{\infty}$ errors of neural networks in approximating the two-dimensional function defined by Equation ( 25 ).

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	3.2425E-05	9.2209E-06	6.3796E-08	6.2611E-09	4.0454E-09	2.7714E-10	1.7599E-10	1.8013E-11
$\tanh$	2.7418E-05	6.1370E-06	1.2666E-07	1.0780E-08	3.1869E-09	2.4560E-10	1.9782E-10	1.8775E-11
swish	3.3855E-05	1.0940E-05	3.5856E-08	4.5318E-09	9.6770E-10	1.6892E-10	3.7835E-10	2.0718E-11
$\cos$	1.8362E-07	7.4465E-08	1.2727E-14	2.3325E-15	9.6481E-15	1.6508E-15	3.3154E-14	3.5549E-15
$\sin$	2.2316E-07	9.5567E-08	1.0184E-14	3.9475E-15	1.5099E-14	2.0114E-15	1.7819E-14	3.4593E-15

Table 2: Function approximation: Performance comparison of FENs and ELMs with sigmoid ,

\tanh

, and swish activations in approximating the two-dimensional function defined by Equation ( 25 ). The

L_{\infty}

and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,10]$	0.01	1.38	1.66	2.48	2.98
$\tanh$	$(0,10]$	0.01	0.6	0.94	1.1	1.5
swish	$(0,50]$	0.01	1.4	2.0	2.2	2.4
$\cos$	$(0,100]$	0.1	5.6	9.3	9.0	13.1
$\sin$	$(0,100]$	0.1	5.8	9.9	9.9	12.0

Table 3: Helmholtz equation: Parameters when solving the Helmholtz equation ( 26 ) with solution in Equation ( 27 ).

Figure 3: Helmholtz equation: $L_{\infty}$ errors of neural networks when solving Helmholtz equation ( 26 ) with solution in Equation ( 27 ).

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	3.3875E-03	8.3152E-04	6.5565E-07	5.7295E-07	8.3819E-09	2.8206E-09	6.0754E-10	3.9546E-10
$\tanh$	2.7466E-03	6.7760E-04	4.4703E-07	2.0869E-07	1.1059E-08	6.0855E-09	4.9477E-10	2.7595E-10
swish	6.3477E-03	1.5769E-03	7.8976E-07	2.2362E-07	6.6357E-09	3.0735E-09	1.0987E-09	3.9585E-10
$\cos$	6.3797E-05	1.7419E-05	5.8978E-13	1.5736E-13	1.7333E-13	7.0401E-14	1.7977E-13	7.1273E-14
$\sin$	1.2699E-04	1.8981E-05	9.0587E-13	2.7940E-13	1.8736E-13	7.4318E-14	5.3300E-14	2.7200E-14

Table 4: Helmholtz equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the Helmholtz equation ( 26 ) with solution in Equation ( 27 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Method	$h$	DoF	$e_{L_{\infty}}$	$e_{L_{2}}$
FEM	0.1	1969	1.5408E-02	7.4130E-03
	0.05	7577	1.5028E-03	8.9865E-04
	0.02	46905	1.4795E-04	4.1033E-05
	0.01	185913	1.8508E-05	5.0898E-06
	0.005	741209	1.3809E-06	9.4240E-07

Table 5: Helmholtz equation: Results of FEM when solving the Helmholtz equation ( 26 ) with solution in Equation ( 27 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,10]$	0.01	1.51	2.27	2.47	3.31
$\tanh$	$(0,10]$	0.01	0.73	1.06	1.53	1.59
swish	$(0,50]$	0.01	1.6	2.1	2.8	4.0
$\cos$	$(0,100]$	0.1	7.6	9.4	10.2	12.1
$\sin$	$(0,100]$	0.1	8.3	9.2	10.3	15.9

Table 6: Helmholtz equation: Parameters when solving the Helmholtz equation ( 26 ) with solution in Equation ( 29 ).

Methods	M=400		M=900		M=1600		M=2500
Methods	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
ELMs ( $\text{sigmoid})$	2.9184E-08	1.5916E-08	2.5498E-11	1.5287E-11	2.4958E-13	1.5588E-13	2.0040E-14	2.1436E-14
ELMs ( $\text{tanh})$	2.3647E-08	1.2001E-08	1.6025E-11	1.0172E-11	1.8674E-13	1.5324E-13	1.1768E-14	9.1806E-15
ELMs ( $\text{swish})$	2.5343E-08	1.0115E-08	1.7768E-11	1.8566E-11	3.4794E-13	3.9641E-13	3.8497E-14	5.8373E-14
ELMs ( $\sin$ )	1.0436E-09	4.3592E-10	1.6043E-14	8.7617E-15	4.1078E-15	3.8454E-15	4.2188E-15	3.9039E-15
FENs ( $\cos$ $\&$ $\sin$ )	4.0931E-10	1.6929E-10	1.2101E-14	9.6805E-15	3.2613E-15	2.4689E-15	3.7192E-15	2.2473E-15

Table 7: Helmholtz equation: Performance comparison of FENs and ELMs activated by different functions, when solving the Helmholtz equation ( 26 ) with solution ( 29 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,10]$	0.01	1.55	1.86	2.61	3.35
$\tanh$	$(0,10]$	0.01	0.9	1.01	1.38	2.03
swish	$(0,50]$	0.01	1.55	2.5	2.53	3.96
$\cos$	$(0,100]$	0.1	5.5	5.0	5.4	6.4
$\sin$	$(0,100]$	0.1	4.0	5.5	5.3	6.6

Table 8: Diffusion equation: Parameters when solving the diffusion equation ( 31 ) with

t_{f}=1

Figure 4: Diffusion equation: the $L_{\infty}$ errors of neural networks when solving diffusion equation ( 31 ) with $t_{f}=1$ .

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	1.0276E-01	7.3530E-03	2.6306E-05	2.4873E-06	2.4815E-07	1.8386E-08	5.7246E-09	3.8524E-10
$\tanh$	1.4518E-01	7.2468E-03	1.9460E-05	1.1581E-06	2.4414E-07	1.7353E-08	1.0454E-08	4.3533E-10
swish	6.3047E-02	4.5548E-03	1.8239E-05	1.5684E-06	1.1645E-07	6.2690E-09	1.1023E-08	9.1794E-10
$\cos$	7.4218E-09	5.5655E-10	1.3012E-13	5.9433E-15	1.1724E-13	9.1437E-15	1.8852E-13	9.1456E-15
$\sin$	1.4405E-08	8.5286E-10	9.2371E-14	5.9186E-15	1.0303E-13	8.3162E-15	2.1139E-13	6.9327E-15

Table 9: Diffusion equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the diffusion equation ( 31 ) with

t_{f}=1

. The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,5]$	0.01	0.15	0.3	0.46	0.56
$\tanh$	$(0,5]$	0.01	0.07	0.14	0.19	0.22
swish	$(0,5]$	0.01	0.12	0.21	0.31	0.48
$\cos$	$(0,50]$	0.1	1.1	1.4	2.2	1.7
$\sin$	$(0,50]$	0.1	1.2	1.3	1.6	2.4

Table 10: Heat equation: Parameters when solving the heat equation ( 33 ).

Figure 5: Heat equation: $L_{\infty}$ errors of neural networks when solving the heat equation ( 33 ).

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	1.5832E-07	7.3817E-08	2.4738E-10	2.3643E-10	3.0013E-11	2.2705E-11	5.9936E-12	3.5015E-12
$\tanh$	1.2910E-07	5.3362E-08	1.1386E-10	5.7319E-11	1.5007E-11	7.7614E-12	4.1442E-12	3.0037E-12
swish	1.4388E-07	7.5731E-08	1.4778E-10	7.2960E-11	2.0669E-11	8.1287E-12	2.4443E-12	1.4761E-12
$\cos$	1.5015E-07	4.9863E-08	2.4385E-12	1.2761E-12	2.9932E-13	1.2883E-13	4.5741E-14	2.8907E-14
$\sin$	6.6993E-08	3.8847E-08	2.8706E-12	1.1786E-12	1.2257E-13	9.7663E-14	2.6090E-14	1.7562E-14

Table 11: Heat equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the heat equation ( 33 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,5]$	0.01	0.17	0.27	0.5	0.54
$\tanh$	$(0,5]$	0.01	0.06	0.13	0.18	0.24
swish	$(0,5]$	0.01	0.13	0.24	0.35	0.49
$\cos$	$(0,10]$	0.1	1.28	1.34	1.46	1.7
$\sin$	$(0,10]$	0.1	1.3	1.38	1.47	1.39

Table 12: Wave equation: Parameters when solving the wave equation ( 34 ).

Figure 6: Wave equation: $L_{\infty}$ errors of neural networks when solving the wave equation ( 34 ).

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	2.4436E-07	6.4600E-08	1.9004E-10	1.3386E-10	2.0691E-11	7.1025E-12	2.1174E-12	1.0359E-12
$\tanh$	1.7975E-07	3.5759E-08	8.8139E-11	4.3830E-11	8.2991E-12	3.5998E-12	1.6200E-12	5.1755E-13
swish	1.0241E-07	3.8084E-08	8.5493E-11	3.3717E-11	9.0949E-12	4.1869E-12	1.4412E-12	6.2451E-13
$\cos$	3.7392E-08	1.6164E-08	2.1672E-13	1.2717E-13	5.2736E-15	2.3601E-15	2.4425E-15	1.7659E-15
$\sin$	6.0666E-08	2.4112E-08	4.3354E-13	2.1115E-13	6.9944E-15	3.3327E-15	5.6621E-15	3.9232E-15

Table 13: Wave equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the wave equation ( 34 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,10]$	0.01	9.65	9.99	9.95	9.72
$\tanh$	$(0,10]$	0.01	9.65	9.71	9.9	9.98
swish	$(0,10]$	0.01	10.0	9.94	9.94	9.84
$\cos$	$(0,100]$	1	16	33	34	64
$\sin$	$(0,100]$	1	20	34	24	74

Table 14: Nonlinear Helmholtz equation: Parameters when solving the nonlinear Helmholtz equation ( 35 ).

Figure 7: Nonlinear Helmholtz equation: $L_{\infty}$ errors of neural networks when solving the nonlinear Helmholtz equation ( 35 ).

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	3.4384E-04	1.6943E-05	5.2436E-06	2.4732E-07	1.5165E-06	7.1510E-08	1.4590E-06	7.7030E-08
$\tanh$	2.0442E-06	1.0492E-07	1.0126E-08	4.7025E-10	3.7986E-11	4.4912E-12	5.6331E-10	3.1179E-11
swish	3.1724E-04	1.9411E-05	9.2887E-06	5.5102E-07	1.4125E-06	6.6785E-08	1.4330E-06	8.1166E-08
$\cos$	2.9874E-12	1.5546E-13	2.2427E-13	1.3705E-14	9.2371E-13	4.8289E-14	7.0832E-13	3.5773E-14
$\sin$	8.6642E-13	4.2264E-14	8.4621E-13	4.6449E-14	1.4064E-12	6.4684E-14	7.4474E-13	3.7450E-14

Table 15: Nonlinear Helmholtz equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the nonlinear Helmholtz equation ( 35 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=400$	$M=900$	$M=1600$	$M=2500$
sigmoid	$(0,20]$	0.01	18.79	18.12	18.46	19.19
$\tanh$	$(0,10]$	0.01	9.15	9.83	9.93	9.70
swish	$(0,20]$	0.01	19.90	19.33	19.39	18.95
$\cos$	$(0,1000]$	1	176	130	205	564
$\sin$	$(0,1000]$	1	147	150	234	890

Table 16: Poisson equation with an oscillating solution: Parameters when solving the Poisson equation ( 36 ).

Figure 8: Poisson equation with an oscillating solution: comparison between the exact solution and the approximate solution obtained using FEN with a $\cos$ activation, where $M=900$ and $\rho_{opt}=130$ .

Figure 9: Poisson equation with an oscillating solution: $L_{\infty}$ errors of neural networks when solving the Poisson equation ( 36 ).

Activations	M=400		M=900		M=1600		M=2500
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	3.6288E-05	3.4848E-05	2.9000E-07	6.0888E-07	1.6587E-07	1.8036E-07	1.0117E-06	1.0275E-06
$\tanh$	3.0062E-05	3.8163E-07	2.6801E-07	2.8331E-07	6.1572E-08	8.0396E-08	2.0924E-07	1.7937E-07
swish	1.7748E-04	2.4055E-04	1.4265E-06	1.5708E-06	5.9915E-07	8.1065E-07	1.9759E-06	2.1461E-06
$\cos$	3.0724E-11	9.7238E-11	1.3878E-11	3.0395E-11	7.2102E-11	9.4554E-11	1.7749E-11	3.2850E-11
$\sin$	3.2458E-11	5.7302E-11	2.9600E-11	4.9456E-11	2.7607E-10	4.0441E-10	4.6233E-11	9.1841E-11

Table 17: Poisson equation with an oscillating solution: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the Poisson equation ( 36 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$M=900$	$M=1600$	$M=2500$	$M=5000$
sigmoid	$(0,10]$	0.1	6.7	8.1	5.0	9.7
$\tanh$	$(0,10]$	0.1	6.3	5.7	3.5	5.9
swish	$(0,10]$	0.1	9.9	5.7	9.0	9.6
$\cos$	$(0,150]$	1	45	62	79	111
$\sin$	$(0,150]$	1	44	49	78	112

Table 18: Nonlinear Burgers’ equation: Parameters when solving the nonlinear Burgers’ equation ( 38 ).

Figure 10: Nonlinear Burgers’ equation: the heat maps of the exact solution and the approximate solution of FEN with a $\sin$ activation, where $M=5000$ and $\rho_{opt}=112$ .

Figure 11: Nonlinear Burgers’ equation: $L_{\infty}$ errors of neural networks when solving the nonlinear Burgers’ equation ( 38 ).

Activations	M=900		M=1600		M=2500		M=5000
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	8.8672E-01	2.0842E-01	3.6381E-01	1.3008E-01	1.1966E-01	1.6866E-02	1.1641E-02	1.1815E-03
$\tanh$	9.1862E-01	2.0184E-01	3.3211E-01	1.1212E-01	8.8108E-02	2.0741E-02	1.4296E-02	1.9656E-03
swish	1.1182E+00	6.4755E-01	4.7318E-01	9.8497E-02	2.0387E-01	2.9728E-02	1.2937E-02	2.9260E-03
$\cos$	5.8175E-01	1.4085E-01	2.3469E-01	5.9826E-02	4.3138E-02	5.1853E-03	9.9221E-04	4.8742E-05
$\sin$	1.2913E+00	3.7190E-01	4.6918E-01	1.3188E-01	4.1016E-02	5.6931E-03	6.2835E-04	5.4839E-05

Table 19: Nonlinear Burgers’ equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the nonlinear Burgers’ equation ( 38 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$\rho_{opt}$
Activations	$(\rho_{min},\rho_{max}]$	$\rho_{s}$	$d=5$	$d=7$	$d=10$	$d=15$
sigmoid	$(0,1]$	0.001	0.143	0.049	0.027	0.016
$\tanh$	$(0,1]$	0.001	0.046	0.015	0.014	0.008
swish	$(0,1]$	0.001	0.12	0.033	0.035	0.024
$\cos$	$(0,1]$	0.001	0.25	0.111	0.091	0.043
$\sin$	$(0,1]$	0.001	0.262	0.102	0.052	0.026

Table 20: High-dimensional Poisson equation: Parameters when solving the high dimensional Poisson equation ( 40 ).

Figure 12: High-dimensional Poisson equation: $L_{\infty}$ errors of neural networks when solving the high dimensional Poisson equation ( 40 ).

Activations	d=5		d=7		d=10		d=15
Activations	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$	$e_{L_{\infty}}$	$e_{L_{2}}$
sigmoid	1.8921E-11	8.7116E-12	2.7337E-09	3.8450E-10	1.0511E-06	3.1487E-07	6.2374E-05	4.0279E-05
$\tanh$	9.5997E-10	1.0016E-10	1.5291E-08	1.9115E-09	2.1064E-05	8.1063E-06	4.9823E-04	3.1662E-04
swish	1.5994E-12	4.5378E-13	5.2239E-09	1.0120E-09	5.5396E-06	4.2761E-07	3.3137E-04	5.9395E-05
$\cos$	1.2396E-13	3.9641E-14	5.6010E-09	7.7473E-10	5.7360E-06	3.5996E-07	3.3140E-04	5.9459E-05
$\sin$	3.6643E-13	7.9493E-14	1.5148E-09	3.1943E-10	1.1275E-06	4.4159E-07	7.9356E-05	5.2070E-05

Table 21: High-dimensional Poisson equation: Performance comparison of FENs and ELMs activated by sigmoid ,

\tanh

and swish when solving the high dimensional Poisson equation ( 35 ). The

L_{\infty}

errors and

L_{2}

errors for each model configuration are presented.

Activations	$d=5$	$d=7$	$d=10$	$d=15$
sigmoid	2.3847	2.6247	3.1373	3.7696
$\tanh$	2.2354	2.4611	2.8802	3.5063
swish	2.6865	3.1198	3.8061	4.9203
$\cos$	2.0713	2.1767	2.5219	3.0101
$\sin$	2.0117	2.2124	2.6537	3.1934
$\cos$ $\&$ $\sin$	1.5786	1.7345	2.0190	2.4217

Table 22: High-dimensional Poisson equation: Computational time for solving the high dimensional Poisson equation ( 40 ).

Shallow Neural Networks

Shallow neural networks use one or a few layers to learn functions, offering a setting to study efficient approximation, training dynamics, and limits of depth.

See all 56 papers

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential
Feb 26, 2025Shijun Zhang, Hongkai Zhao, Yimin Zhong +1Fourier Feature Embeddings Neural Network Optimization
Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1
Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs
May 29, 2026Qihong Yang, Qiaolin HeNeural PDE Solvers PDE Solving
School of Mathematics, Sichuan University, city=Chengdu, postcode=610065, country=China
FEKAN: Feature-Enriched Kolmogorov-Arnold Networks
Feb 18, 2026Sidharth S. Menon, Ameya D. JagtapFunction Approximation Kolmogorov-Arnold Networks
Aerospace Engineering Department, Worcester Polytechnic Institute, Worcester, MA 01609, USA.

Feb 26, 2025J/K move · Enter open · S save

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

Shijun Zhang, Hongkai Zhao, Yimin Zhong, Haomin Zhou

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University+1

Fourier Feature Embeddings Neural Network Optimization Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity Modular Neural Networks

Open paper AskPDF arXiv

The architecture of a neural network and the choice of its activation function are both fundamental to its performance. Equally important is ensuring that these two elements are well matched, as their alignment is key to effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a model that combines sine-type activations with the multi-component and multi-layer structure of MMNNs. In an FMMNN, each component is represented as a trainable linear combination of fixed random sine-type basis functions, while multi-layer composition generates more complex and adaptive high-frequency features. We establish that FMMNNs retain exponential expressive power for function approximation even under a low-rank architectural structure. We also analyze the optimization landscape of FMMNNs and find it to be substantially more favorable than that of standard fully connected neural networks, especially for high-frequency targets. In addition, we propose a scaled random initialization method for the first-layer weights in FMMNNs, which accelerates training and improves final performance when sufficient samples are available. Extensive numerical experiments support our theoretical insights, showing that FMMNNs achieve strong accuracy and favorable convergence behavior on oscillatory function-approximation benchmarks.

Abstract

Shallow Neural Networks

Explore similar work

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

FEKAN: Feature-Enriched Kolmogorov-Arnold Networks