Extrapolation of Periodic Functions Using Binary Encoding of Continuous Numerical Values
Authors: Brian P. Powell, Jordan A. Caraballo-Vega, Mark L. Carroll, Thomas Maxwell, Andrew Ptak, Greg Olmschenk, Jorge Martinez-Palomera
Organizations: NASA Goddard Space Flight Center, Greenbelt, MD 20771, USA · ARSC Federal Holding Company, Reston, VA 20190, USA · University of Maryland, College Park, MD 20190, USA · University of Maryland Baltimore County, Baltimore, MD 20190, USA
We introduce Normalized Base-2 Encoding (NB2E) as a method for encoding continuous numerical values and demonstrate that, using this input encoding, vanilla multi-layer perceptrons (MLP) successfully extrapolate four periodic waveforms without prior knowledge of their functional form. We contrast this finding with fixed Fourier encoding, which extrapolates well when the number of signal cycles over the normalized domain is an even integer and loses coherence at most other values, whereas NB2E degrades gradually away from those values and, except near odd cycle counts, remains substantially more accurate. These findings are limited to a single MLP architecture trained on one synthetic, exactly periodic, one-dimensional signal at a time, with extrapolation extending only to the end of the normalized domain. We do not propose a mechanism for this phenomenon. Rather, we present empirical observations which merit further study.
Figures & tables
Figure 1: 16-element NB2E of the number 0.9876. The given value is encoded into a vector of ones and zeros in the manner described by Equation 3 to be used as input to a neural network.
Figure 2: For FFE (top panels) and NB2E (bottom panels) waveforms of the first three bits or octaves in the domain [0,1) are shown on the left and the frequency content of each waveform is shown on the right in corresponding colors. Note that the sine and cosine components of FFE have the same frequency.
Bit
Frequency
NB2E
FFE
1
20=1
2−1
sin(2πx),cos(2πx)
2
21=2
2−2
sin(4πx),cos(4πx)
3
22=4
2−3
sin(8πx),cos(8πx)
4
23=8
2−4
sin(16πx),cos(16πx)
⋮
⋮
⋮
⋮
i
2i−1
2−i
sin(2iπx),cos(2iπx)
Table 1: NB2E bits and the equivalent FFE. Each NB2E bit i with positional value 2−i corresponds to FFE at frequency 2i−1 . The two encodings share a fundamental frequency at every index but differ in waveform as the square wave carries additional odd harmonics (see Figure 2 ).
Figure 3: MLP structure. The NB2E is not part of the neural network, just noted as the encoding of the input. Replace NB2E with FFE for the appropriate examples, whereas for continuous numerical input there is no encoding. There are five dense layers of 512 neurons each with ELU activation and L2 regularization with a regularization factor of 10−4 , followed by a dense layer with a single neuron and linear activation.
Figure 4: A schematic of our process. Raw coordinates, x , are normalized to x′ by the chosen normalization value, z . x′ is then encoded into NB2E (top blue square). The MLP is then trained with the NB2E of x′ as the input and f(x) as the output. With t as the boundary between the training region and extrapolation region, we train the MLP on all x′<t , with 10% of the training data reserved as validation data for selection of the best training epoch. The best weights are then used to predict the value of the function in the extrapolation region, (t,1) .
Figure 5: Results of training the MLP from Figure 3 with encoded input data for f(x′)=sin(2π⋅24x′) . Predictions for the training data (blue) and the extrapolation region (red) for ( top ) NB2E, ( middle ) FFE, and ( bottom ) continuous, with residuals plotted on the same scale for visual comparison. The true function is in black. Fit and extrapolation nMAE for each case are given in the text boxes in each panel.
Figure 6: Extrapolation results for the sine, triangle, sawtooth, and square functions at c=24 (top two rows), c=24.5 (middle two rows), and c=17 (bottom two rows) at N=16 , for NB2E and FFE. Each panel shows the seed with the median extrapolation error. All c=24 examples extrapolate well for both FFE and NB2E, only NB2E extrapolation remains coherent when c=24.5 , and both encodings are degraded at c=17 . The NB2E triangle at c=24.5 varies substantially across seeds (see Table 2 ).
Figure 7: Extrapolation nMAE as a function of distance in c from a power of two, for c0={4,8,16,32,64} . NB2E is shown in shades of blue, while FFE is shown in shades of red. Points are the median nMAE and error bars show the interquartile range over 15 runs (5 seeds × 3 phase offsets).
Figure 8: Extrapolation nMAE for FFE (red) and NB2E (blue) from c=4 to c=32 in increments of 0.1. Error bars show the interquartile range over 15 runs (5 seeds × 3 phase offsets).
Training
Function
NB2E
FFE
Continuous
sine, c=24
0.004 (0.003–0.005)
0.001 (0.001–0.001)
1.003 (1.003–1.004)
triangle, c=24
0.006 (0.005–0.006)
0.003 (0.002–0.003)
0.995 (0.995–0.997)
sawtooth, c=24
0.012 (0.011–0.015)
0.007 (0.007–0.008)
1.007 (1.006–1.008)
square, c=24
0.007 (0.006–0.008)
0.004 (0.004–0.005)
0.987 (0.986–0.999)
sine, c=24.5
0.006 (0.005–0.006)
0.002 (0.002–0.002)
1.005 (0.998–1.007)
Table 2: Training and extrapolation error (nMAE) at encoding width N=16 (16 bits for NB2E, 16 octaves for FFE) for the three cycle counts of Section 4.1 . The weights retained are those of the epoch with the lowest MAE on the validation subset. The median over five seeds is provided with the (min - max) range for each value. The best result for each row is in bold. All tests share the same MLP, optimizer, starting seeds, and epoch budget.
Training
Function
NB2E
FFE
Continuous
sine, c=24
0.006 (0.006–0.007)
0.002 (0.002–0.002)
1.003 (1.003–1.004)
triangle, c=24
0.009 (0.008–0.009)
0.004 (0.004–0.004)
0.995 (0.995–0.997)
sawtooth, c=24
0.016 (0.014–0.017)
0.010 (0.009–0.011)
1.007 (1.006–1.008)
square, c=24
0.008 (0.007–0.011)
0.006 (0.005–0.006)
0.987 (0.986–0.999)
sine, c=24.5
0.011 (0.010–0.012)
0.004 (0.003–0.005)
1.005 (0.998–1.007)
Table 3: Same as Table 2 for N=32 .
Training
Function
NB2E
FFE
Continuous
sine, c=24
0.008 (0.007–0.009)
0.003 (0.003–0.003)
1.003 (1.003–1.004)
triangle, c=24
0.011 (0.010–0.011)
0.006 (0.006–0.007)
0.995 (0.995–0.997)
sawtooth, c=24
0.023 (0.022–0.026)
0.018 (0.015–0.020)
1.007 (1.006–1.008)
square, c=24
0.012 (0.011–0.015)
0.008 (0.007–0.008)
0.987 (0.986–0.999)
sine, c=24.5
0.016 (0.015–0.018)
0.010 (0.008–0.012)
1.005 (0.998–1.007)
Table 4: Same as Table 2 for N=64 .
Figure 9: For Xtrain′<t<Xextrap′ , from top to bottom, predictions and residuals for the training data (blue) and the extrapolation region (red) for t=0.3 , t=0.4 , t=0.5 , t=0.6 , and t=0.7 .
Figure 10: ( upper panels ) Predictions of the sine curve at c=24.5 for the training data (blue) and the extrapolation region (red) shown with the true function (black). ( lower panels ) Residuals of the predictions. The number of data points in each data set are labeled to the left of each panel. The dashed gray vertical line is shown at t=0.7 .
Figure 11: ( upper panels ) Predictions of the sine curve at c=24.5 for the training data (blue) and the extrapolation region (red) shown with the true function (black). ( lower panels ) Residuals of the predictions. Standard deviation of the Gaussian noise is shown to the left of each panel.
Interfacing with Biological Neural Networks (BNNs) requires encoding information into stimulation patterns that can be effectively processed and that enable the underlying system to adapt. Nevertheless, the role of stimulation encoding remains poorly understood. In this work, we compare multiple encoding strategies, including rate-based, phase-based, burst-based, and time-to-first-spike temporal encodings, in a closed-loop neural classification task using cultured BNNs. We encode visual inputs as spatiotemporal stimulation patterns delivered via a Multi-Electrode Array (MEA) and evaluate classification performance for each encoding scheme. We find that burst-based temporal encoding yields the highest observed performance, achieving up to 95.6 % accuracy in a binary classification task, compared to substantially lower performance from rate- and phase-based approaches. We further show that performance is highly sensitive to the spatial distribution of stimulation, with suboptimal electrode selection significantly degrading accuracy. These findings indicate that effective interfacing with biological neural systems requires the joint optimization of temporal and spatial encoding strategies, and highlight temporal encoding as a key design dimension for bio-digital computing.
Martin Schottlender, Veronika Volkova, Pengjie Zhou +3
Deutsche Telekom Chair of Communication Networks, Dresden University of Technology, Germany · Centre for Tactile Internet with Human-in-the-Loop (CeTI), Dresden, Germany
Spike-based encodings are sparse and energy-efficient, but have largely been formulated probabilistically, disconnected from most signal processing literature. We recast spike encoders as time-causal wavelet frames with quantitative bandwidths and reconstruction error bounds. The proposed wavelets preserve the sparsity and locality of spiking representations, with reconstruction up to spike quantization and time discretization. We demonstrate reconstruction on ECG and audio datasets, achieving a normalized RMSE comparable to continuous wavelet transforms. The spiking wavelets map directly to neuromorphic hardware.
Jens Egholm Pedersen, Tony Lindeberg, Peter Gerstoft
Department of Electrical and Photonics Engineering Technical University of Denmark Denmark · Department of Computational Science and Technology KTH Royal Institute of Technology Sweden
We show that MLP layers in transformer language models perform binary routing of continuous signals: the decision of whether a token needs nonlinear processing is well-captured by binary neuron activations, even though the signals being routed are continuous. In GPT-2 Small (124M parameters), we find that specific neurons implement a consensus architecture -- seven "default-ON" neurons and one exception handler (N2123 in Layer 11) that are 93-98% mutually exclusive -- creating a binary routing switch. A cross-layer analysis reveals a developmental arc: early layers (L1-3) use single gateway neurons to route exceptions without consensus quorums; middle layers (L4-6) show diffuse processing with neither gateway nor consensus; and late layers (L7-11) crystallize full consensus/exception architectures with increasing quorum size (1 to 3 to 7 consensus neurons). Causal validation confirms the routing is functional: removing the MLP at consensus breakdown costs 43.3% perplexity, while at full consensus removing it costs only 10.1% -- exceeding a 4x difference. Comparing binary vs. continuous features for the routing decision confirms that binarization loses essentially no information (79.2% vs. 78.8% accuracy), while continuous activations carry additional magnitude information (R^2 = 0.36 vs. 0.22). This binary routing structure explains why smooth polynomial approximation fails: cross-validated polynomial fits (degrees 2-7) never exceed R^2 = 0.06 for highly nonlinear layers. We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization: along the natural data manifold, the piecewise boundaries implement binary decisions about which tokens need nonlinear processing, routing continuous signals through qualitatively different computational paths.