Extrapolation of Periodic Functions Using Binary Encoding of Continuous Numerical Values
Authors: Brian P. Powell, Jordan A. Caraballo-Vega, Mark L. Carroll, Thomas Maxwell, Andrew Ptak, Greg Olmschenk, Jorge Martinez-Palomera
Organizations: NASA Goddard Space Flight Center, Greenbelt, MD 20771, USA · ARSC Federal Holding Company, Reston, VA 20190, USA · University of Maryland, College Park, MD 20190, USA · University of Maryland Baltimore County, Baltimore, MD 20190, USA
We introduce Normalized Base-2 Encoding (NB2E) as a method for encoding continuous numerical values and demonstrate that, using this input encoding, vanilla multi-layer perceptrons (MLP) successfully extrapolate four periodic waveforms without prior knowledge of their functional form. We contrast this finding with fixed Fourier encoding, which extrapolates well when the number of signal cycles over the normalized domain is an even integer and loses coherence at most other values, whereas NB2E degrades gradually away from those values and, except near odd cycle counts, remains substantially more accurate. These findings are limited to a single MLP architecture trained on one synthetic, exactly periodic, one-dimensional signal at a time, with extrapolation extending only to the end of the normalized domain. We do not propose a mechanism for this phenomenon. Rather, we present empirical observations which merit further study.
Figures & tables
Figure 1: 16-element NB2E of the number 0.9876. The given value is encoded into a vector of ones and zeros in the manner described by Equation 3 to be used as input to a neural network.
Figure 2: For FFE (top panels) and NB2E (bottom panels) waveforms of the first three bits or octaves in the domain [0,1) are shown on the left and the frequency content of each waveform is shown on the right in corresponding colors. Note that the sine and cosine components of FFE have the same frequency.
Bit
Frequency
NB2E
FFE
1
20=1
2−1
sin(2πx),cos(2πx)
2
21=2
2−2
sin(4πx),cos(4πx)
3
22=4
2−3
sin(8πx),cos(8πx)
4
23=8
2−4
sin(16πx),cos(16πx)
⋮
⋮
⋮
⋮
i
2i−1
2−i
sin(2iπx),cos(2iπx)
Table 1: NB2E bits and the equivalent FFE. Each NB2E bit i with positional value 2−i corresponds to FFE at frequency 2i−1 . The two encodings share a fundamental frequency at every index but differ in waveform as the square wave carries additional odd harmonics (see Figure 2 ).
Figure 3: MLP structure. The NB2E is not part of the neural network, just noted as the encoding of the input. Replace NB2E with FFE for the appropriate examples, whereas for continuous numerical input there is no encoding. There are five dense layers of 512 neurons each with ELU activation and L2 regularization with a regularization factor of 10−4 , followed by a dense layer with a single neuron and linear activation.
Figure 4: A schematic of our process. Raw coordinates, x , are normalized to x′ by the chosen normalization value, z . x′ is then encoded into NB2E (top blue square). The MLP is then trained with the NB2E of x′ as the input and f(x) as the output. With t as the boundary between the training region and extrapolation region, we train the MLP on all x′<t , with 10% of the training data reserved as validation data for selection of the best training epoch. The best weights are then used to predict the value of the function in the extrapolation region, (t,1) .
Figure 5: Results of training the MLP from Figure 3 with encoded input data for f(x′)=sin(2π⋅24x′) . Predictions for the training data (blue) and the extrapolation region (red) for ( top ) NB2E, ( middle ) FFE, and ( bottom ) continuous, with residuals plotted on the same scale for visual comparison. The true function is in black. Fit and extrapolation nMAE for each case are given in the text boxes in each panel.
Figure 6: Extrapolation results for the sine, triangle, sawtooth, and square functions at c=24 (top two rows), c=24.5 (middle two rows), and c=17 (bottom two rows) at N=16 , for NB2E and FFE. Each panel shows the seed with the median extrapolation error. All c=24 examples extrapolate well for both FFE and NB2E, only NB2E extrapolation remains coherent when c=24.5 , and both encodings are degraded at c=17 . The NB2E triangle at c=24.5 varies substantially across seeds (see Table 2 ).
Figure 7: Extrapolation nMAE as a function of distance in c from a power of two, for c0={4,8,16,32,64} . NB2E is shown in shades of blue, while FFE is shown in shades of red. Points are the median nMAE and error bars show the interquartile range over 15 runs (5 seeds × 3 phase offsets).
Figure 8: Extrapolation nMAE for FFE (red) and NB2E (blue) from c=4 to c=32 in increments of 0.1. Error bars show the interquartile range over 15 runs (5 seeds × 3 phase offsets).
Training
Function
NB2E
FFE
Continuous
sine, c=24
0.004 (0.003–0.005)
0.001 (0.001–0.001)
1.003 (1.003–1.004)
triangle, c=24
0.006 (0.005–0.006)
0.003 (0.002–0.003)
0.995 (0.995–0.997)
sawtooth, c=24
0.012 (0.011–0.015)
0.007 (0.007–0.008)
1.007 (1.006–1.008)
square, c=24
0.007 (0.006–0.008)
0.004 (0.004–0.005)
0.987 (0.986–0.999)
sine, c=24.5
0.006 (0.005–0.006)
0.002 (0.002–0.002)
1.005 (0.998–1.007)
Table 2: Training and extrapolation error (nMAE) at encoding width N=16 (16 bits for NB2E, 16 octaves for FFE) for the three cycle counts of Section 4.1 . The weights retained are those of the epoch with the lowest MAE on the validation subset. The median over five seeds is provided with the (min - max) range for each value. The best result for each row is in bold. All tests share the same MLP, optimizer, starting seeds, and epoch budget.
Training
Function
NB2E
FFE
Continuous
sine, c=24
0.006 (0.006–0.007)
0.002 (0.002–0.002)
1.003 (1.003–1.004)
triangle, c=24
0.009 (0.008–0.009)
0.004 (0.004–0.004)
0.995 (0.995–0.997)
sawtooth, c=24
0.016 (0.014–0.017)
0.010 (0.009–0.011)
1.007 (1.006–1.008)
square, c=24
0.008 (0.007–0.011)
0.006 (0.005–0.006)
0.987 (0.986–0.999)
sine, c=24.5
0.011 (0.010–0.012)
0.004 (0.003–0.005)
1.005 (0.998–1.007)
Table 3: Same as Table 2 for N=32 .
Training
Function
NB2E
FFE
Continuous
sine, c=24
0.008 (0.007–0.009)
0.003 (0.003–0.003)
1.003 (1.003–1.004)
triangle, c=24
0.011 (0.010–0.011)
0.006 (0.006–0.007)
0.995 (0.995–0.997)
sawtooth, c=24
0.023 (0.022–0.026)
0.018 (0.015–0.020)
1.007 (1.006–1.008)
square, c=24
0.012 (0.011–0.015)
0.008 (0.007–0.008)
0.987 (0.986–0.999)
sine, c=24.5
0.016 (0.015–0.018)
0.010 (0.008–0.012)
1.005 (0.998–1.007)
Table 4: Same as Table 2 for N=64 .
Figure 9: For Xtrain′<t<Xextrap′ , from top to bottom, predictions and residuals for the training data (blue) and the extrapolation region (red) for t=0.3 , t=0.4 , t=0.5 , t=0.6 , and t=0.7 .
Figure 10: ( upper panels ) Predictions of the sine curve at c=24.5 for the training data (blue) and the extrapolation region (red) shown with the true function (black). ( lower panels ) Residuals of the predictions. The number of data points in each data set are labeled to the left of each panel. The dashed gray vertical line is shown at t=0.7 .
Figure 11: ( upper panels ) Predictions of the sine curve at c=24.5 for the training data (blue) and the extrapolation region (red) shown with the true function (black). ( lower panels ) Residuals of the predictions. Standard deviation of the Gaussian noise is shown to the left of each panel.
Deutsche Telekom Chair of Communication Networks, Dresden University of Technology, Germany · Centre for Tactile Internet with Human-in-the-Loop (CeTI), Dresden, Germany
Department of Electrical and Photonics Engineering Technical University of Denmark Denmark · Department of Computational Science and Technology KTH Royal Institute of Technology Sweden