Dynamical systems describe how a physical system evolves over time. Physical processes can evolve faster or slower in different environmental conditions. We use time-warping as rescaling the time in a model of a physical system. This thesis proposes a new method of transfer learning for Recurrent Neural Networks (RNNs) based on time-warping. We prove that for a class of linear, first-order differential equations known as time lag models, an LSTM can approximate these systems with any desired accuracy, and the model can be time-warped while maintaining the approximation accuracy. The Time-Warping method of transfer learning is then evaluated in an applied problem on predicting fuel moisture content (FMC), an important concept in wildfire modeling. An RNN with LSTM recurrent layers is pretrained on fuels with a characteristic time scale of 10 hours, where there are large quantities of data available for training. The RNN is then modified with transfer learning to generate predictions for fuels with characteristic time scales of 1 hour, 100 hours, and 1000 hours. The Time-Warping method is evaluated against several known methods of transfer learning. The Time-Warping method produces predictions with an accuracy level comparable to the established methods, despite modifying only a small fraction of the parameters that the other methods modify.
Figures & tables
Figure 2.1 : Two complementary views of an RNN. (a) Left: a single RNN cell, following the formulation in [ 12 , p. 539] . (b) Right: the RNN unrolled in time.
Figure 2.2 : A single LSTM unit. This is an original graphic, developed following the formulation in [ 12 , p. 569] . The computations are described in Equations ( 2.8 ).
Time-Lag (T)
Fuel Diameter
mt+1≈
1
Less than 1/4 in.
0.63⋅E+0.37⋅mt
10
1/4 - 1 in.
0.095⋅E+0.905⋅mt
100
1-3 in.
0.01⋅E+0.99⋅mt
1000
3-8 in.
0.001⋅E+0.999⋅mt
Table 2.1 : Characteristic Time Scales for FMC Fuel Classes.
Setup: Pretrained LSTM with bias vectors (bf,bi) ; training set Dtrain ; test set Dtest ; fixed temporal scaling γ ; specified ranges of scalar candidate values for αf and αi .
1. Construct a grid of candidate values for αf and a grid of candidate values for αi .
2. Form all possible pairs (αf,αi) by combining each value of αf with each value of αi .
Table 3.1 : Algorithm 1: Grid search over time-warping parameters (forget and input gate bias shifts) for a pretrained LSTM under fixed temporal scaling, with model selection based on training performance and final evaluation on a test set.
Setup: Pretrained LSTM with bias vectors (bf,bi) ; training set Dtrain ; validation set Dval ; test set Dtest ; fixed temporal scaling γ ; specified ranges of scalar candidate values for αf and αi .
1. Construct a grid of candidate values for αf and a grid of candidate values for αi .
2. Form all possible pairs (αf,αi) by combining each value of αf with each value of αi .
Table 3.2 : Algorithm 2: Grid search over time-warping parameters (forget and input gate bias shifts) for a pretrained LSTM under fixed temporal scaling, with fine-tuning on a training set, model selection based on validation performance, and final evaluation on a test set.
Figure 3.1 : Deterministic trajectories from Newton’s Law of Cooling for varying cooling constants. Varying values of k change the characteristic time scale. The initial temperature and the ambient temperature of the environment are kept constant in each case.
Figure 3.2 : RNN model architecture, pre-trained on FM10 from RAWS. An RNN with one LSTM layer and three subsequent dense layers.
Variable
Units
Description
Mean
(Low, High)
Drying Equilibrium
%
Derived from RH and temperature.
18.02
(1.29, 60.56)
Wetting Equilibrium
%
Derived from RH and temperature.
16.53
(0.61, 56.10)
Solar Radiation
Wm−2
Downward shortwave radiative flux.
208.41
(0, 1177)
Wind Speed
ms−1
Wind speed at 10m.
2.52
(0.40, 8.61)
Rain
mmh−1
Rain accumulation over the hour.
0.08
(0.00, 42.17)
Hour of Day
hours
Hour of day, UTC time.
11.50
(0, 23)
Table 3.3 : Oklahoma Field Study Weather Data Summary. Weather variables and geographic features are from the Slapout Mesonet station. The time period is from March 26, 1996 through the end of 1997.
Fuel Class
N. Observations
Mean
(Low, High)
FM1
1,233
15.33
(0.0, 109.2)
FM10
1,232
15.02
(1.6, 64.3)
FM100
871
13.44
(5.4, 35.7)
FM1000
874
11.37
(4.7, 27.9)
Table 3.4 : Oklahoma Field Study FMC Data Summary. Observations are made with gravimetric measurements on arrays of wooden dowels. The time period is from March 26, 1996, through the end of 1997.
Training Set
Validation Set
Test Set
Start Date (UTC)
1996-03-26, 23:00
1997-03-27, 00:00
1997-08-13, 12:00
End Date (UTC)
1997-03-26, 23:00
1997-08-13, 11:00
1997-12-30, 22:00
Table 3.5 : Temporal definition of the training, validation, and test splits for the Oklahoma field data. All dates are reported in UTC.
N. Observations
Training Set
Validation Set
Test Set
Weather
8,761
3,348
3,347
FM1
704
258
271
FM10
704
258
270
FM100
481
184
206
FM1000
482
183
209
Table 3.6 : Number of observations in the training, validation, and test splits for each variable in the Oklahoma field data. Note that not all analyses utilize the validation set.
Method Name
Details
Literature Status
No Transfer
Random initialization and direct training on the target learning task.
Standard supervised training baseline.
Full Fine-Tuning
Model pretrained on a source task and all parameters updated on the target task.
Standard transfer learning baseline.
Freeze Recurrent Layer
Recurrent weights fixed while dense/output layers are retrained on the target task.
Common transfer learning strategy, widely used for CNNs.
Freeze Dense/Output Layers
Dense/output layers fixed while input and recurrent layers are retrained.
Reverse-freezing baseline; rarely explored in literature.
Time-Warping
Parameters for shifting the forget and input gate biases are fit, without additional training.
Novel to this thesis.
Time-Warping + Fine-Tuning
Time-warp initialization followed by full fine-tuning on the target task.
Novel to this thesis.
Table 3.7 : Transfer learning methods comparison for FMC prediction.
All observed FM10
Observed FM10 ≤ 30%
Metric
Nelson (hourly)
RNN
Nelson (hourly)
RNN
R2
0.79
0.56±0.02
0.61
0.65±0.06
Bias (%)
−0.9
0.91±0.52
−1.1
−0.74±0.53
RMSE (%)
-
6.79±0.16
-
3.21±0.25
Table 4.1 : Accuracy metrics for FM10 Zero-Shot RNN Predictions and Nelson model [ 6 , Table 4] . Comparisons of the models are presented for all FM10 values ( n=1,232 ) and for FM10 ≤30% ( n=1,134 ). RNN metrics are reported as mean ± standard deviation across 100 replications.
Figure 4.1 : Observed vs Predicted FM10 for Zero-Shot Transfer RNN. The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications. Note the different x-axis ranges in the plots. (a) Top: All FM10 observations (n=1,232). The zero-shot RNN predictions substantially underestimate the FM10 for wet fuels. (b) Bottom: FM10 observations filtered to less than or equal to 30% (n=1,134). The zero-shot RNN predictions are much more accurate, though the model is overestimating the FM10 for very dry fuels and underestimating the FM10 for wetter fuels.
Figure 4.2 : One week of FM10 sensor measurements and corresponding HRRR weather at BAWC2, southwest of Denver. The wetting and drying equilibria are derived from the forecasted air temperature and RH. The sensor appears to have a maximal response to rain, with the FM10 reaching a maximum value of around 27 % .
Figure 4.3 : One week of FM10 field measurements in Oklahoma, the corresponding ground-based weather observations, and the RNN zero-shot predictions. The RNN predictions are from the model replication with the median RMSE in the test set. The RNN predictions reach a maximum of about 27 % , similar to the RAWS training data.
Metric
Values
RMSE (Mean ± SD)
2.81 ± 0.49
RMSE Range (Low, High)
(2.63, 2.91)
N (Mean ± SD)
65,691,366 ± 589,543
Table 4.2 : Fitting accuracy of the FM10 RNN across 100 replications.
Metric
Forget Gate Bias
Input Gate Bias
Difference
Mean of per-model means ( ± SD across reps)
1.14 ± 0.04
0.11 ± 0.08
1.03 ± 0.10
Range of per-model means (low, high)
(1.08, 1.32)
(-0.45, 0.24)
(0.90, 1.56)
Mean of per-model SDs ( ± SD across reps)
0.23 ± 0.15
0.39 ± 0.17
0.47 ± 0.22
Range of per-model SDs (low, high)
(0.14, 1.37)
(0.21, 1.42)
(0.25, 1.91)
Table 4.3 : Summary Statistics for Empirical Distributions of Gate Biases.
Figure 4.4 : Distribution of the mean LSTM forget-gate ( bf ) and input-gate ( bi ) bias parameters across 100 training replications. In each replication, the mean bias across the 64 units in the LSTM layer is computed, producing one value per run. The histograms therefore summarize the distribution of these mean biases over the 100 replications.
Method
R2
Bias (%)
RMSE (%)
No Transfer
0.78±0.03
0.84±0.71
6.31±0.40
Full Fine-Tuning
0.84±0.02
2.24±0.53
5.31±0.38
Freeze Recurrent Layer
0.79±0.02
1.23±0.76
6.06±0.36
Freeze Dense/Output Layers
0.84±0.03
2.19±0.46
5.39±0.43
Time-Warping
0.41±0.03
2.46±0.42
10.29±0.23
Time-Warping + Fine-Tuning
0.84±0.05
1.11±1.04
5.26±0.71
Table 4.4 : Accuracy metrics for FM1 RNN predictions compared to field observations in Oklahoma for different transfer learning methods. Metrics are reported as mean ± standard deviation across 100 replications (n=271).
Figure 4.5 : FM1 RMSE comparison for different transfer learning methods. Bars represent the mean RMSE when predicting FM1 on the test set over 100 replications, and brackets show ± 1 standard deviation of the RMSE.
Method
R2
Bias (%)
RMSE (%)
No Transfer
0.66±0.06
−0.30±0.62
3.60±0.29
Full Fine-Tuning
0.67±0.05
1.43±0.49
3.59±0.29
Freeze Recurrent Layer
0.43±0.11
0.30±0.74
4.69±0.45
Freeze Dense/Output Layers
0.72±0.05
1.40±0.40
3.30±0.27
Time-Warping
0.71±0.04
−0.28±0.43
3.35±0.25
Time-Warping + Fine-Tuning
0.64±0.11
0.62±0.92
3.69±0.53
Table 4.5 : Accuracy metrics for FM1 RNN predictions compared to field observations in Oklahoma for different transfer learning methods. Metrics are reported as mean ± standard deviation across 100 replications. FM1 values are filtered ≤30% (n=247).
Figure 4.6 : FM1 RMSE comparison for different transfer learning methods with FM1 filtered to ( ≤30% ). Bars represent the mean RMSE when predicting FM1 on the test set over 100 replications, and brackets show ± 1 standard deviation of the RMSE.
Figure 4.7 : Observed vs Predicted FM1 for Time-Warped RNN. The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications. (a) Top: All FM1 observations in the test set (n=271). The RNN predictions substantially underestimate the FM1 for wet fuels. (b) Bottom: FM1 observations filtered to less than or equal to 30% (n=247). The RNN predictions are much more accurate, though the model is overestimating the FM1 for very dry fuels and underestimating the FM1 for wetter fuels.
Method
R2
Bias (%)
RMSE (%)
No Transfer
0.64±0.13
−0.19±0.33
2.60±0.44
Full Fine-Tuning
0.84±0.06
−0.41±0.41
1.75±0.29
Freeze Recurrent Layer
0.73±0.08
−0.34±0.33
2.25±0.34
Freeze Dense/Output Layers
0.75±0.09
0.12±0.44
2.15±0.38
Time-Warping
0.70±0.08
−0.33±0.72
2.38±0.28
Time-Warping + Fine-Tuning
0.78±0.10
−0.64±0.56
2.00±0.40
Table 4.6 : Accuracy metrics for FM100 RNN predictions compared to field observations in Oklahoma for different transfer learning methods (n=206). Metrics are reported as mean ± standard deviation across 100 replications.
Figure 4.8 : FM100 RMSE comparison for different transfer learning methods. Bars represent the mean RMSE when predicting FM100 on the test set over 100 replications, and brackets show ± 1 standard deviation of the RMSE.
Figure 4.9 : Observed vs Predicted FM100 for Time-Warped RNN. The plot includes all FM100 observations in the test set (n=206). The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications.
Method
R2
Bias (%)
RMSE (%)
No Transfer
0.57±0.12
−0.50±0.29
2.16±0.31
Full Fine-Tuning
0.59±0.05
−0.92±0.21
2.13±0.14
Freeze Recurrent Layer
0.50±0.09
−0.42±0.55
2.33±0.20
Freeze Dense/Output Layers
0.50±0.09
0.65±0.70
2.33±0.21
Time-Warping
0.54±0.18
−0.81±0.71
2.22±0.43
Time-Warping + Fine-Tuning
0.65±0.10
−0.75±0.36
1.94±0.27
Table 4.7 : Accuracy metrics for FM1000 RNN predictions compared to field observations in Oklahoma for different transfer learning methods (n=209). Metrics are reported as mean ± standard deviation across 100 replications.
Figure 4.10 : FM1000 RMSE comparison for different transfer learning methods. Bars represent the mean RMSE when predicting FM1000 on the test set over 100 replications, and brackets show ± 1 standard deviation of the RMSE.
Figure 4.11 : Observed vs Predicted FM1000 for Time-Warped RNN. The plot includes all FM1000 observations in the test set (n=209). The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications.
Figure 4.12 : Example Time Series of Predictions for FM1, FM100, and FM1000 from the Time-Warped RNN, with no fine-tuning. The time period plotting is 72 hours during the first week of the test period in August 1997. The predictions are from the models that had the median RMSE across the 100 replications.
Time-Warping Parameter
Mean
Std
Median
Low
High
bf shift
−1.31
0.50
−1.25
−3.33
−0.42
bi shift
1.82
2.19
1.67
−1.67
5.00
Table 4.8 : Summary statistics for the time-warp parameters across 100 replications for FM1 with no fine-tuning.
Time-Warping Parameter
Mean
Std
Median
Low
High
bf shift
1.89
0.61
2.08
0.42
3.33
bi shift
−1.01
2.06
−1.25
−4.17
4.17
Table 4.9 : Summary statistics for the time-warp parameters across 100 replications for FM100 with no fine-tuning.
Time-Warping Parameter
Mean
Std
Median
Low
High
bf shift
1.15
1.46
1.25
−4.17
3.75
bi shift
−1.62
3.81
−4.17
−5.00
5.00
Table 4.10 : Summary statistics for the time-warp parameters across 100 replications for FM1000 with no fine-tuning..
Figure 4.13 : Autocorrelation functions for FM1 (top), FM100 (middle), and FM1000 (bottom) from predictions on the test set. The plots represent the ACF for the single models that had the median RMSE on the test set. The ACF for FM1 decays relatively quickly and has a visible diurnal periodicity, while FM100 and FM1000 show much slower decay consistent with stronger persistence in the system.
Figure 4.14 : Partial autocorrelation functions for FM1 (top), FM100 (middle), and FM1000 (bottom) from predictions on the test set. The plots represent the ACF for the single models that had the median RMSE on the test set. FM100 and FM1000 show a dominant lag-1 effect consistent with strong persistence, whereas FM1 shows additional short-lag structure and weaker persistence beyond the first lag.
Figure 7.1 : Forget Gate Bias Distribution across Replications. The distributions are visualized using Gaussian kernel density estimates to provide a smooth representation of the empirical bias distributions for comparison, and the replications are ordered by the sample variance of each distribution.
Figure 7.2 : Input Gate Bias Distribution across Replications. The distributions are visualized using Gaussian kernel density estimates to provide a smooth representation of the empirical bias distributions for comparison, and the replications are ordered by the sample variance of each distribution.
Figure 7.3 : Per-Unit Difference of Forget and Input Gate Biases across Replications. The distributions are visualized using Gaussian kernel density estimates to provide a smooth representation of the empirical bias distributions for comparison, and the replications are ordered by the sample variance of each distribution.
FM1: Static Baselines
All observed FM1
Observed FM1 ≤ 30%
Metric
XGBoost
LM
XGBoost
LM
R2
0.63
0.56
0.35
0.42
Bias (%)
0.71
−0.05
−0.92
−2.21
RMSE (%)
8.15
8.81
5.03
4.73
Table 7.1 : Accuracy metrics for FM1 predictions during the test period from XGBoost and linear regression model (LM). Comparisons of the models are presented for Oklahoma field observations of all FM1 values and for FM1 ≤30% .
FM10: Static Baselines
All observed FM10
Observed FM10 ≤ 30%
Metric
XGBoost
LM
XGBoost
LM
R2
0.56
0.59
0.49
0.59
Bias (%)
1.31
0.77
0.31
−0.36
RMSE (%)
5.80
5.62
3.88
3.47
Table 7.2 : Accuracy metrics for FM10 predictions during the test period from XGBoost and linear regression model (LM). Comparisons of the models are presented for Oklahoma field observations of all FM1 values and for FM1 ≤30% .
FM100: Static Baselines
All observed FM100
Metric
XGBoost
LM
R2
0.57
0.53
Bias (%)
−0.58
−0.51
RMSE (%)
2.86
3.00
Table 7.3 : Accuracy metrics for FM100 predictions during the test period from XGBoost and linear regression model (LM) compared against field observations. Results are shown for all observed FM100 values.
FM1000: Static Baselines
All observed FM1000
Metric
XGBoost
LM
R2
0.38
0.44
Bias (%)
−0.71
−0.64
RMSE (%)
2.61
2.47
Table 7.4 : Accuracy metrics for FM1000 predictions from XGBoost and linear regression model (LM) during the test period compared against field observations. Results are shown for all observed FM1000 values.
This paper proposes a time-warping transfer learning method, a technique for temporally rescaling the learned dynamics of a recurrent neural network (RNN) with a Long Short-Term Memory (LSTM) layer to enable task transfer across fuel moisture classes. Fuel moisture content (FMC) is divided into idealized classes based on characteristic lag time. Large quantities of real-time data are available for 10h fuels from sensors on weather stations, but observations of other fuel classes are sparse in space and time. We use transfer learning to adapt an RNN pretrained on 10h FMC to predict FMC for 1h, 100h, and 1000h fuels. We validate this method using data from a landmark field study conducted in Oklahoma that was used to calibrate the state-of-the-art Nelson fuel moisture model.
Jonathon Hirschi, Jan Mandel, Adam Kochanski
Department of Mathematical and Statistical Sciences, University of Colorado Denver, Colorado · Department of Meteorology and Climate Science, Wildfire Interdisciplinary Research Center, San Jose State University, San Jose, California
Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels (mt,xt+1)→mt+1. SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective--retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable O(1) length gradient path between any two tokens--without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.
Time-warping estimation is a fundamental problem in signal processing with applications in bioacoustics, radar, and biomedical analysis. This paper introduces a Time-Warping Estimation Trainable (TWET) model for estimating timewarping functions from a single observation. The proposed approach formulates time-warping estimation as a stationarization problem in the wavelet domain and leverages a hierarchical dilated convolutional architecture to estimate the time-warping functions. A differentiable stationarity criterion is introduced for end-to-end optimization. TWET is compared with existing approaches. Experimental results show improved deformation reconstruction accuracy together with significantly reduced computation time, making the framework compatible with low-latency applications.