Variability Aware Recursive Neural Network (VARNN): A Residual-Memory Model for Capturing Temporal Deviation in Sequence Regression Modeling

Haroon Gharwi; Yue Dai; Kai Shu

cs.LGOct 10, 2025

Variability Aware Recursive Neural Network (VARNN): A Residual-Memory Model for Capturing Temporal Deviation in Sequence Regression Modeling

Authors: Haroon Gharwi, Yue Dai, Kai Shu

Organizations: Department of Computer Science, Illinois Institute of Technology, Chicago, IL, USA · Department of Computer Science, Emory University, Atlanta, GA, USA

Abstract

Real-world time-series regression often involves non-stationarity, heteroscedasticity, and regime changes, under which recent prediction errors may contain structured information about local temporal mismatch between model predictions and observations. Learning how to represent and reuse these errors can therefore provide useful information for subsequent prediction. We introduce the Variability-Aware Recursive Neural Network (VARNN), a residual-aware architecture for supervised time-series regression that learns an explicit residual-memory state from recent prediction errors and uses it to condition subsequent predictions. Specifically, VARNN maps scalar prediction innovations into a learned nonlinear, vector-valued residual representation over a short context. Across nine datasets spanning energy, healthcare, and environmental domains, VARNN achieves lower test MSE than the compared static, lag-based, and sequence-model baselines. Targeted ablations further show that learned projected residual memory improves predictive accuracy over direct scalar residual feedback, supporting the benefit of a learned nonlinear representation of prediction deviations.

Figures & tables

Figure 1: VARNN architecture: the Residual Memory Block (green) converts the innovation $e_{\tau}$ into a residual-memory state $\mathbf{h}_{\tau}$ ; the Predictor Block (blue) fuses $\mathbf{x}_{\tau}$ with $\mathbf{h}_{\tau-1}$ and, when enabled, $\mathbf{u}_{\tau-1}$ , to produce $\hat{y}_{\tau}$ .

Domain	Dataset	Size	Missing	#Feat. $(d)$	Target
Energy	Appliances	19,735	No	27	Appliance energy use (Wh)
	ETTh1	17,420	No	6	Oil temperature
	ETTh2	17,420	No	6	Oil temperature
Healthcare	BIDMC–HR	25,436	No	375	Heart rate (bpm)
	BIDMC–RR	25,436	No	375	Respiratory rate (breaths/min)
	BIDMC–SPO2	25,436	No	375	Oxygen saturation (SpO 2 )

Table 1: Dataset summary (post-cleaning): size, missingness, feature count

d

, and scalar target.

Static (no lags)
Model	ENERGY						Healthcare						Environmental
	Appliances		ETTh1		ETTh2		BIDMC HR		BIDMC RR		BIDMC SPO2		Beijing PM2.5		Beijing PM10		WEATHER
	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test
LR	0.00799	0.00657	0.02792	0.03335	0.02243	0.04112	0.01830	0.02256	0.00984	0.00822	0.03659	0.03103	0.00212	0.00171	0.00375	0.00242	0.01516	0.01483
RF	0.00057	0.04815	0.00234	0.02888	0.00174	0.03274	0.00031	0.01842	0.00032	0.00705	0.00057	0.03925	0.00012	0.00162	0.00025	0.00227	0.00229	0.01235
MLP	0.00833	0.00778	0.00833	0.00778	0.02622	0.02049	0.00216	0.00168	0.01276	0.00933	0.04683	0.03452	0.00216	0.00168	0.00392	0.00224	0.01413	1.21301

Table 2: Train and test mean squared error (MSE; ↓) on all datasets. Static models use only contemporaneous covariates

\mathbf{x}_{t}

; Dynamic models additionally incorporate lagged outputs/inputs or recurrent states. Best test scores are in bold .

Panel of Figure 2 — Figure 2: Learning curves for VARNN-RM and VARNN-RM+AM across Appliances and Beijing

Panel of Figure 3 — Figure 3: Learning curves for VARNN-RM and RM+AM across Appliances and Beijing datasets.

Panel of Figure 4 — Figure 4: Residual memory size vs scalar across three datasets.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

$w$	Appliances	BIDMC-HR	Beijing PM2.5
2	0.00338	0.00016	0.00028
3	0.00358	0.00018	0.00029
5	0.00328	0.00015	0.00026
7	0.00340	0.00016	0.00026
9	0.00340	0.00020	0.00027

Appendix

Table 3: Sensitivity of VARNN-RM to the context window length

w

. Test MSE is reported; lower is better. Best results for each dataset are shown in bold.

Panel of Figure 5 — Figure 5: Train vs. validation loss on Beijing PM2.5 using VARNN-RM with ReLU vs. Tanh activations.

Figure 6: Beijing PM2.5: learning curves for VARNN variants. Accumulating residuals (ARM) improves stability and validation accuracy, while combining accumulation with activation memory (ARM+AM) yields similar performance with slightly smoother training dynamics.

	Power				HR				PM2.5
Model	$H{=}1$	$H{=}3$	$H{=}5$	$H{=}7$	$H{=}1$	$H{=}3$	$H{=}5$	$H{=}7$	$H{=}1$	$H{=}3$	$H{=}5$	$H{=}7$
RNN	0.007057	0.007459	0.007310	0.007337	0.010834	0.019918	0.017077	0.016677	0.001407	0.001773	0.001995	0.002268
GRU	0.006663	0.006711	0.006870	0.006772	0.016132	0.017971	0.017825	0.015146	0.001412	0.001765	0.002015	0.002303
LSTM-H	0.006522	0.006510	0.006793	0.006731	0.017026	0.018671	0.016860	0.017115	0.001365	0.001747	0.001984	0.002257
LSTM-H+C	0.006484	0.007736	0.007053	0.006923	0.017054	0.017482	0.018643	0.016395	0.001408	0.001757	0.002021	0.002260
VARNN-RM	0.003430	0.005029	0.005817	0.006147	0.000153	0.000335	0.000397	0.000483	0.000230	0.000656	0.001082	0.001493

Appendix

Table 4: Direct multi-step forecasting performance in MSE for prediction horizons

H\in\{1,3,5,7\}

. Lower is better. All models use the same historical window length (

w{=}5

) and direct multi-step prediction protocol. The best result for each dataset and horizon is shown in bold.

Model	ENERGY						Healthcare						Environmental
	Appliances		ETTh1		ETTh2		BIDMC HR		BIDMC RR		BIDMC SPO2		Beijing PM2.5		Beijing PM10		WEATHER
	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test
VARNN–RM	0.00412	0.00328	0.00037	0.00020	0.00041	0.00048	0.00016	0.00015	0.00017	0.00031	0.00015	0.00024	0.00024	0.00026	0.00085	0.00058	0.00028	0.00045
VARNN–RM+AM	0.00378	0.00329	0.00035	0.00018	0.00044	0.00049	0.00016	0.00015	0.00017	0.00029	0.00015	0.00022	0.00022	0.00025	0.00077	0.00057	0.00037	0.00042
VARNN–ARM	0.00382	0.00330	0.00035	0.00017	0.00040	0.00046	0.00015	0.00015	0.00017	0.00029	0.00016	0.00024	0.00022	0.00025	0.00077	0.00058	0.00026	0.00031
VARNN–RM+ARM	0.00374	0.00329	0.00037	0.00017	0.00041	0.00051	0.00016	0.00015	0.00017	0.00029	0.00015	0.00024	0.00021	0.00024	0.00078	0.00055	0.00024	0.00032

Appendix

Table 5: Train and test mean squared error (MSE; ↓) on all datasets. Static models use only contemporaneous covariates

\mathbf{x}_{t}

; Dynamic models additionally incorporate lagged outputs/inputs or recurrent states. Best test scores are in bold .

Explore similar work

StateFlow: Dual-State Recurrent Modeling for Long-Horizon Time Series Forecasting
Jun 30, 2026Haroon Gharwi, Yue Dai, Kai ShuRetrieval-Augmented Forecasting Time Series Forecasting
Department of Computer Science, Illinois Institute of Technology, Chicago, IL, USA · Department of Computer Science, Emory University, Atlanta, GA, USA
Improving Spatio-Temporal Residual Error Propagation by Mitigating Over-Squashing
May 18, 2026Seyed Mohamad Moghadas, Esther Rodrigo Bonet, Bruno Cornelis +1Autoregressive Model Recurrent Model
ETRO Department, Vrije Universiteit Brussel, B-1050 Brussels, Belgium · imec Kapeldreef 75, B-3001 Leuven, Belgium
RATL: Learning from Retrieved Residuals for Robust Multivariate Time-Series Forecasting
Sep 3, 2026Yuchen He, Yueyang Cang, Zhiyuan Ning +2Residual Learning Multivariate Time-Series
Department of Automation, Tsinghua University · State Key Laboratory of Hydroscience and Engineering, Tsinghua University

Jun 30, 2026, cs.LGJ/K move · Enter open · S save

StateFlow: Dual-State Recurrent Modeling for Long-Horizon Time Series Forecasting

Haroon Gharwi, Yue Dai, Kai Shu

Department of Computer Science, Illinois Institute of Technology, Chicago, IL, USA · Department of Computer Science, Emory University, Atlanta, GA, USA

Retrieval-Augmented Forecasting Time Series Forecasting Recurrent State Conditional Variational Autoencoder

Open paper AskPDF arXiv

Long-horizon multivariate time series forecasting (LTSF) remains challenging due to non-stationarity, regime shifts, and error accumulation. The Variability-Aware Recursive Neural Network (VARNN) is designed to track such variability by maintaining a residual-memory state driven by one-step prediction errors. However, its original formulation is limited to one-step sequence regression and does not directly support multi-step forecasting. In this work, we extend VARNN to long-horizon forecasting and introduce StateFlow, a recurrent forecasting framework that uses VARNN as a dual-state recurrent backbone to capture two complementary signals from the lookback sequence: a hidden-state trajectory representing primary temporal dynamics, including trend, seasonality, level changes, and recurring patterns, and a residual-memory trajectory representing structured local prediction deviations, driven from a nonlinear recurrent transformation of errors between one-step base predictions and observed values. A chunk-based decoder separately summarizes these trajectories and maps them to the future horizon for direct multi-step forecasting. We further employ a two-stage optimization strategy that first trains the VARNN encoder through a one-step base prediction objective to optimize the internal representations over the lookback sequence, and then trains a horizon-specific decoder for direct multi-step forecasting. Experiments on standard LTSF benchmarks show that StateFlow achieves competitive performance against strong linear, recurrent, convolutional, and Transformer-based baselines while preserving linear recurrent encoding and a compact model design.

Abstract

Explore similar work

StateFlow: Dual-State Recurrent Modeling for Long-Horizon Time Series Forecasting

Improving Spatio-Temporal Residual Error Propagation by Mitigating Over-Squashing

RATL: Learning from Retrieved Residuals for Robust Multivariate Time-Series Forecasting