Time-Warping Recurrent Neural Networks for Transfer Learning

Jonathon Hirschi

Figure 2.1 : Two complementary views of an RNN. (a) Left: a single RNN cell, following the formulation in [ 12 , p. 539] . (b) Right: the RNN unrolled in time.

Figure 2.2 : A single LSTM unit. This is an original graphic, developed following the formulation in [ 12 , p. 569] . The computations are described in Equations ( 2.8 ).

Time-Lag (T)	Fuel Diameter	$\boldsymbol{m_{t+1}\approx}$
1	Less than 1/4 in.	$0.63\cdot E+0.37\cdot m_{t}$
10	1/4 - 1 in.	$0.095\cdot E+0.905\cdot m_{t}$
100	1-3 in.	$0.01\cdot E+0.99\cdot m_{t}$
1000	3-8 in.	$0.001\cdot E+0.999\cdot m_{t}$

Table 2.1 : Characteristic Time Scales for FMC Fuel Classes.

Setup: Pretrained LSTM with bias vectors

(b_{f},b_{i})

; training set

\mathcal{D}_{\text{train}}

; test set

\mathcal{D}_{\text{test}}

; fixed temporal scaling

\gamma

; specified ranges of scalar candidate values for

\alpha_{f}

and

\alpha_{i}

.

1. Construct a grid of candidate values for

\alpha_{f}

and a grid of candidate values for

\alpha_{i}

.

2. Form all possible pairs

(\alpha_{f},\alpha_{i})

by combining each value of

\alpha_{f}

with each value of

\alpha_{i}

.

3. For each pair

(\alpha_{f},\alpha_{i})

:

3.1 Shift LSTM biases:

\widetilde{b}_{f}=b_{f}+\alpha_{f}

,

\widetilde{b}_{i}=b_{i}+\alpha_{i}

.

Table 3.1 : Algorithm 1: Grid search over time-warping parameters (forget and input gate bias shifts) for a pretrained LSTM under fixed temporal scaling, with model selection based on training performance and final evaluation on a test set.

Setup: Pretrained LSTM with bias vectors

(b_{f},b_{i})

; training set

\mathcal{D}_{\text{train}}

; validation set

\mathcal{D}_{\text{val}}

; test set

\mathcal{D}_{\text{test}}

; fixed temporal scaling

\gamma

; specified ranges of scalar candidate values for

\alpha_{f}

and

\alpha_{i}

.

1. Construct a grid of candidate values for

\alpha_{f}

and a grid of candidate values for

\alpha_{i}

.

2. Form all possible pairs

(\alpha_{f},\alpha_{i})

by combining each value of

\alpha_{f}

with each value of

\alpha_{i}

.

3. For each pair

(\alpha_{f},\alpha_{i})

:

3.1 Shift LSTM biases:

\widetilde{b}_{f}=b_{f}+\alpha_{f}

,

\widetilde{b}_{i}=b_{i}+\alpha_{i}

.

Table 3.2 : Algorithm 2: Grid search over time-warping parameters (forget and input gate bias shifts) for a pretrained LSTM under fixed temporal scaling, with fine-tuning on a training set, model selection based on validation performance, and final evaluation on a test set.

Figure 3.1 : Deterministic trajectories from Newton’s Law of Cooling for varying cooling constants. Varying values of

k

change the characteristic time scale. The initial temperature and the ambient temperature of the environment are kept constant in each case.

Figure 3.2 : RNN model architecture, pre-trained on FM10 from RAWS. An RNN with one LSTM layer and three subsequent dense layers.

Variable	Units	Description	Mean	(Low, High)
Drying Equilibrium	$\%$	Derived from RH and temperature.	18.02	(1.29, 60.56)
Wetting Equilibrium	$\%$	Derived from RH and temperature.	16.53	(0.61, 56.10)
Solar Radiation	$\text{W\,m}^{-2}$	Downward shortwave radiative flux.	208.41	(0, 1177)
Wind Speed	$\text{m\,s}^{-1}$	Wind speed at 10m.	2.52	(0.40, 8.61)
Rain	$\text{mm\,h}^{-1}$	Rain accumulation over the hour.	0.08	(0.00, 42.17)
Hour of Day	hours	Hour of day, UTC time.	11.50	(0, 23)

Table 3.3 : Oklahoma Field Study Weather Data Summary. Weather variables and geographic features are from the Slapout Mesonet station. The time period is from March 26, 1996 through the end of 1997.

Fuel Class	N. Observations	Mean	(Low, High)
FM1	1,233	15.33	(0.0, 109.2)
FM10	1,232	15.02	(1.6, 64.3)
FM100	871	13.44	(5.4, 35.7)
FM1000	874	11.37	(4.7, 27.9)

Table 3.4 : Oklahoma Field Study FMC Data Summary. Observations are made with gravimetric measurements on arrays of wooden dowels. The time period is from March 26, 1996, through the end of 1997.

	Training Set	Validation Set	Test Set
Start Date (UTC)	1996-03-26, 23:00	1997-03-27, 00:00	1997-08-13, 12:00
End Date (UTC)	1997-03-26, 23:00	1997-08-13, 11:00	1997-12-30, 22:00

Table 3.5 : Temporal definition of the training, validation, and test splits for the Oklahoma field data. All dates are reported in UTC.

N. Observations	Training Set	Validation Set	Test Set
Weather	8,761	3,348	3,347
FM1	704	258	271
FM10	704	258	270
FM100	481	184	206
FM1000	482	183	209

Table 3.6 : Number of observations in the training, validation, and test splits for each variable in the Oklahoma field data. Note that not all analyses utilize the validation set.

Method Name	Details	Literature Status
No Transfer	Random initialization and direct training on the target learning task.	Standard supervised training baseline.
Full Fine-Tuning	Model pretrained on a source task and all parameters updated on the target task.	Standard transfer learning baseline.
Freeze Recurrent Layer	Recurrent weights fixed while dense/output layers are retrained on the target task.	Common transfer learning strategy, widely used for CNNs.
Freeze Dense/Output Layers	Dense/output layers fixed while input and recurrent layers are retrained.	Reverse-freezing baseline; rarely explored in literature.
Time-Warping	Parameters for shifting the forget and input gate biases are fit, without additional training.	Novel to this thesis.
Time-Warping + Fine-Tuning	Time-warp initialization followed by full fine-tuning on the target task.	Novel to this thesis.

Table 3.7 : Transfer learning methods comparison for FMC prediction.

	All observed FM10		Observed FM10 $\leq$ 30%
Metric	Nelson (hourly)	RNN	Nelson (hourly)	RNN
$R^{2}$	$0.79$	$0.56\pm 0.02$	$0.61$	$\;\;\,0.65\pm 0.06$
Bias (%)	$-0.9$	$0.91\pm 0.52$	$-1.1$	$-0.74\pm 0.53$
RMSE (%)	-	$6.79\pm 0.16$	-	$\;\;\,3.21\pm 0.25$

Table 4.1 : Accuracy metrics for FM10 Zero-Shot RNN Predictions and Nelson model [ 6 , Table 4] . Comparisons of the models are presented for all FM10 values (

n=1,232

) and for FM10

\leq 30\%

(

n=1,134

). RNN metrics are reported as mean

\pm

standard deviation across 100 replications.

Figure 4.1 : Observed vs Predicted FM10 for Zero-Shot Transfer RNN. The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications. Note the different x-axis ranges in the plots. (a) Top: All FM10 observations (n=1,232). The zero-shot RNN predictions substantially underestimate the FM10 for wet fuels. (b) Bottom: FM10 observations filtered to less than or equal to

30\%

(n=1,134). The zero-shot RNN predictions are much more accurate, though the model is overestimating the FM10 for very dry fuels and underestimating the FM10 for wetter fuels.

Figure 4.2 : One week of FM10 sensor measurements and corresponding HRRR weather at BAWC2, southwest of Denver. The wetting and drying equilibria are derived from the forecasted air temperature and RH. The sensor appears to have a maximal response to rain, with the FM10 reaching a maximum value of around 27

\%

.

Figure 4.3 : One week of FM10 field measurements in Oklahoma, the corresponding ground-based weather observations, and the RNN zero-shot predictions. The RNN predictions are from the model replication with the median RMSE in the test set. The RNN predictions reach a maximum of about 27

\%

, similar to the RAWS training data.

Metric	Values
RMSE (Mean $\pm$ SD)	2.81 $\pm$ 0.49
RMSE Range (Low, High)	(2.63, 2.91)
N (Mean $\pm$ SD)	65,691,366 $\pm$ 589,543

Table 4.2 : Fitting accuracy of the FM10 RNN across 100 replications.

Metric	Forget Gate Bias	Input Gate Bias	Difference
Mean of per-model means ( $\pm$ SD across reps)	1.14 $\pm$ 0.04	0.11 $\pm$ 0.08	1.03 $\pm$ 0.10
Range of per-model means (low, high)	(1.08, 1.32)	(-0.45, 0.24)	(0.90, 1.56)
Mean of per-model SDs ( $\pm$ SD across reps)	0.23 $\pm$ 0.15	0.39 $\pm$ 0.17	0.47 $\pm$ 0.22
Range of per-model SDs (low, high)	(0.14, 1.37)	(0.21, 1.42)	(0.25, 1.91)

Table 4.3 : Summary Statistics for Empirical Distributions of Gate Biases.

Figure 4.4 : Distribution of the mean LSTM forget-gate (

b_{f}

) and input-gate (

b_{i}

) bias parameters across 100 training replications. In each replication, the mean bias across the 64 units in the LSTM layer is computed, producing one value per run. The histograms therefore summarize the distribution of these mean biases over the 100 replications.

Method	$R^{2}$	Bias (%)	RMSE (%)
No Transfer	$0.78\pm 0.03$	$0.84\pm 0.71$	$6.31\pm 0.40$
Full Fine-Tuning	$0.84\pm 0.02$	$2.24\pm 0.53$	$5.31\pm 0.38$
Freeze Recurrent Layer	$0.79\pm 0.02$	$1.23\pm 0.76$	$6.06\pm 0.36$
Freeze Dense/Output Layers	$0.84\pm 0.03$	$2.19\pm 0.46$	$5.39\pm 0.43$
Time-Warping	$0.41\pm 0.03$	$2.46\pm 0.42$	$10.29\pm 0.23$
Time-Warping + Fine-Tuning	$0.84\pm 0.05$	$1.11\pm 1.04$	$5.26\pm 0.71$

Table 4.4 : Accuracy metrics for FM1 RNN predictions compared to field observations in Oklahoma for different transfer learning methods. Metrics are reported as mean

\pm

standard deviation across 100 replications (n=271).

Figure 4.5 : FM1 RMSE comparison for different transfer learning methods. Bars represent the mean RMSE when predicting FM1 on the test set over 100 replications, and brackets show

\pm

1 standard deviation of the RMSE.

Method	$R^{2}$	Bias (%)	RMSE (%)
No Transfer	$0.66\pm 0.06$	$-0.30\pm 0.62$	$3.60\pm 0.29$
Full Fine-Tuning	$0.67\pm 0.05$	$1.43\pm 0.49$	$3.59\pm 0.29$
Freeze Recurrent Layer	$0.43\pm 0.11$	$0.30\pm 0.74$	$4.69\pm 0.45$
Freeze Dense/Output Layers	$0.72\pm 0.05$	$1.40\pm 0.40$	$3.30\pm 0.27$
Time-Warping	$0.71\pm 0.04$	$-0.28\pm 0.43$	$3.35\pm 0.25$
Time-Warping + Fine-Tuning	$0.64\pm 0.11$	$0.62\pm 0.92$	$3.69\pm 0.53$

Table 4.5 : Accuracy metrics for FM1 RNN predictions compared to field observations in Oklahoma for different transfer learning methods. Metrics are reported as mean

\pm

standard deviation across 100 replications. FM1 values are filtered

\leq 30\%

(n=247).

Figure 4.6 : FM1 RMSE comparison for different transfer learning methods with FM1 filtered to (

\leq 30\%

). Bars represent the mean RMSE when predicting FM1 on the test set over 100 replications, and brackets show

\pm

1 standard deviation of the RMSE.

Figure 4.7 : Observed vs Predicted FM1 for Time-Warped RNN. The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications. (a) Top: All FM1 observations in the test set (n=271). The RNN predictions substantially underestimate the FM1 for wet fuels. (b) Bottom: FM1 observations filtered to less than or equal to

30\%

(n=247). The RNN predictions are much more accurate, though the model is overestimating the FM1 for very dry fuels and underestimating the FM1 for wetter fuels.

Method	$R^{2}$	Bias (%)	RMSE (%)
No Transfer	$0.64\pm 0.13$	$-0.19\pm 0.33$	$2.60\pm 0.44$
Full Fine-Tuning	$0.84\pm 0.06$	$-0.41\pm 0.41$	$1.75\pm 0.29$
Freeze Recurrent Layer	$0.73\pm 0.08$	$-0.34\pm 0.33$	$2.25\pm 0.34$
Freeze Dense/Output Layers	$0.75\pm 0.09$	$0.12\pm 0.44$	$2.15\pm 0.38$
Time-Warping	$0.70\pm 0.08$	$-0.33\pm 0.72$	$2.38\pm 0.28$
Time-Warping + Fine-Tuning	$0.78\pm 0.10$	$-0.64\pm 0.56$	$2.00\pm 0.40$

Table 4.6 : Accuracy metrics for FM100 RNN predictions compared to field observations in Oklahoma for different transfer learning methods (n=206). Metrics are reported as mean

\pm

standard deviation across 100 replications.

Figure 4.8 : FM100 RMSE comparison for different transfer learning methods. Bars represent the mean RMSE when predicting FM100 on the test set over 100 replications, and brackets show

\pm

1 standard deviation of the RMSE.

Figure 4.9 : Observed vs Predicted FM100 for Time-Warped RNN. The plot includes all FM100 observations in the test set (n=206). The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications.

Method	$R^{2}$	Bias (%)	RMSE (%)
No Transfer	$0.57\pm 0.12$	$-0.50\pm 0.29$	$2.16\pm 0.31$
Full Fine-Tuning	$0.59\pm 0.05$	$-0.92\pm 0.21$	$2.13\pm 0.14$
Freeze Recurrent Layer	$0.50\pm 0.09$	$-0.42\pm 0.55$	$2.33\pm 0.20$
Freeze Dense/Output Layers	$0.50\pm 0.09$	$0.65\pm 0.70$	$2.33\pm 0.21$
Time-Warping	$0.54\pm 0.18$	$-0.81\pm 0.71$	$2.22\pm 0.43$
Time-Warping + Fine-Tuning	$0.65\pm 0.10$	$-0.75\pm 0.36$	$1.94\pm 0.27$

Table 4.7 : Accuracy metrics for FM1000 RNN predictions compared to field observations in Oklahoma for different transfer learning methods (n=209). Metrics are reported as mean

\pm

standard deviation across 100 replications.

Figure 4.10 : FM1000 RMSE comparison for different transfer learning methods. Bars represent the mean RMSE when predicting FM1000 on the test set over 100 replications, and brackets show

\pm

1 standard deviation of the RMSE.

Figure 4.11 : Observed vs Predicted FM1000 for Time-Warped RNN. The plot includes all FM1000 observations in the test set (n=209). The plotted RNN model is the set of weights that corresponds to the median RMSE on the test set out of 100 replications.

Figure 4.12 : Example Time Series of Predictions for FM1, FM100, and FM1000 from the Time-Warped RNN, with no fine-tuning. The time period plotting is 72 hours during the first week of the test period in August 1997. The predictions are from the models that had the median RMSE across the 100 replications.

Time-Warping Parameter	Mean	Std	Median	Low	High
$b_{f}$ shift	$-1.31$	$0.50$	$-1.25$	$-3.33$	$-0.42$
$b_{i}$ shift	$1.82$	$2.19$	$1.67$	$-1.67$	$5.00$

Table 4.8 : Summary statistics for the time-warp parameters across 100 replications for FM1 with no fine-tuning.

Time-Warping Parameter	Mean	Std	Median	Low	High
$b_{f}$ shift	$1.89$	$0.61$	$2.08$	$0.42$	$3.33$
$b_{i}$ shift	$-1.01$	$2.06$	$-1.25$	$-4.17$	$4.17$

Table 4.9 : Summary statistics for the time-warp parameters across 100 replications for FM100 with no fine-tuning.

Time-Warping Parameter	Mean	Std	Median	Low	High
$b_{f}$ shift	$1.15$	$1.46$	$1.25$	$-4.17$	$3.75$
$b_{i}$ shift	$-1.62$	$3.81$	$-4.17$	$-5.00$	$5.00$

Table 4.10 : Summary statistics for the time-warp parameters across 100 replications for FM1000 with no fine-tuning..

Figure 4.13 : Autocorrelation functions for FM1 (top), FM100 (middle), and FM1000 (bottom) from predictions on the test set. The plots represent the ACF for the single models that had the median RMSE on the test set. The ACF for FM1 decays relatively quickly and has a visible diurnal periodicity, while FM100 and FM1000 show much slower decay consistent with stronger persistence in the system.

Figure 4.14 : Partial autocorrelation functions for FM1 (top), FM100 (middle), and FM1000 (bottom) from predictions on the test set. The plots represent the ACF for the single models that had the median RMSE on the test set. FM100 and FM1000 show a dominant lag-1 effect consistent with strong persistence, whereas FM1 shows additional short-lag structure and weaker persistence beyond the first lag.

Figure 7.1 : Forget Gate Bias Distribution across Replications. The distributions are visualized using Gaussian kernel density estimates to provide a smooth representation of the empirical bias distributions for comparison, and the replications are ordered by the sample variance of each distribution.

Figure 7.2 : Input Gate Bias Distribution across Replications. The distributions are visualized using Gaussian kernel density estimates to provide a smooth representation of the empirical bias distributions for comparison, and the replications are ordered by the sample variance of each distribution.

Figure 7.3 : Per-Unit Difference of Forget and Input Gate Biases across Replications. The distributions are visualized using Gaussian kernel density estimates to provide a smooth representation of the empirical bias distributions for comparison, and the replications are ordered by the sample variance of each distribution.

FM1: Static Baselines
	All observed FM1		Observed FM1 $\leq$ 30%
Metric	XGBoost	LM	XGBoost	LM
$R^{2}$	$0.63$	$0.56$	$0.35$	$0.42$
Bias (%)	$0.71$	$-0.05$	$-0.92$	$-2.21$
RMSE (%)	$8.15$	$8.81$	$5.03$	$4.73$

Table 7.1 : Accuracy metrics for FM1 predictions during the test period from XGBoost and linear regression model (LM). Comparisons of the models are presented for Oklahoma field observations of all FM1 values and for FM1

\leq 30\%

.

FM10: Static Baselines
	All observed FM10		Observed FM10 $\leq$ 30%
Metric	XGBoost	LM	XGBoost	LM
$R^{2}$	$0.56$	$0.59$	$0.49$	$0.59$
Bias (%)	$1.31$	$0.77$	$0.31$	$-0.36$
RMSE (%)	$5.80$	$5.62$	$3.88$	$3.47$

Table 7.2 : Accuracy metrics for FM10 predictions during the test period from XGBoost and linear regression model (LM). Comparisons of the models are presented for Oklahoma field observations of all FM1 values and for FM1

\leq 30\%

.

FM100: Static Baselines
	All observed FM100
Metric	XGBoost	LM
$R^{2}$	$0.57$	$0.53$
Bias (%)	$-0.58$	$-0.51$
RMSE (%)	$2.86$	$3.00$

Table 7.3 : Accuracy metrics for FM100 predictions during the test period from XGBoost and linear regression model (LM) compared against field observations. Results are shown for all observed FM100 values.

FM1000: Static Baselines
	All observed FM1000
Metric	XGBoost	LM
$R^{2}$	$0.38$	$0.44$
Bias (%)	$-0.71$	$-0.64$
RMSE (%)	$2.61$	$2.47$

Table 7.4 : Accuracy metrics for FM1000 predictions from XGBoost and linear regression model (LM) during the test period compared against field observations. Results are shown for all observed FM1000 values.

Time-Warping Recurrent Neural Networks for Transfer Learning

Abstract

Figures & tables

Transfer Learning

Transfer Learning for Dead Fuel Moisture Prediction Using Time-Warping Recurrent Neural Networks

Pretraining Recurrent Networks without Recurrence

Time-warping estimation via stationarity-based learning of the de-warped signal

Abstract

Transfer Learning

Explore similar work

Transfer Learning for Dead Fuel Moisture Prediction Using Time-Warping Recurrent Neural Networks

Pretraining Recurrent Networks without Recurrence

Time-warping estimation via stationarity-based learning of the de-warped signal