Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization

Jianing Chen; Vajiheh Farhadi; Yan Li; Thomas La Porta

cs.LGAug 28, 2026

Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization

Authors: Jianing Chen, Vajiheh Farhadi, Yan Li, Thomas La Porta

Organizations: School of Electrical Engineering and Computer Science, The Pennsylvania State University, University Park, PA 16802, USA · Department of Electrical and Computer Engineering, Bucknell University, Lewisburg, PA 17837, USA

Abstract

Short-term load forecasting (STLF) provides essential information for numerous applications in modern power systems. However, accurate STLF often relies on fine-grained smart-meter data from distributed users, raising increasing concerns about data privacy. Federated learning (FL) has therefore emerged as a promising privacy-preserving paradigm for STLF. Nevertheless, this paper reveals structured heterogeneity in clients' load data. Specifically, clients exhibit different responses to exogenous factors and distinct temporal load profiles, which can degrade forecasting performance in FL. To mitigate these issues, this paper studies the role of model initialization in federated STLF, and proposes two initialization strategies from global and local perspectives. For global model initialization, when auxiliary public load data are available, a pretrained initialization strategy is developed to initialize the global model before federated training, thereby reducing client drift during the training process. For local model initialization, we propose SLIAvg, a sequential local initialization strategy that promotes a more consistent training process by allowing participating clients to start from progressively adapted models within each communication round. Since the proposed strategies only modify the initialization process, they are compatible with most existing FL frameworks and privacy-enhancing techniques. Experiments on real smart-meter data with two representative forecasting architectures demonstrate that the proposed strategies effectively improve forecasting performance, as evidenced by reduced client drift, improved convergence behavior, and lower forecasting errors.

Figures & tables

Figure 1: Overview of load heterogeneity analysis and the proposed initialization strategies.

Figure 2: Illustration of client drift in FedAvg compared with centralized optimization. Two clients perform three local update steps (

K=2

E=3

). Local updates (gray arrows) move toward client-specific optima

\mathbf{w}_{1}^{*}

and

\mathbf{w}_{2}^{*}

(blue and green squares). Under heterogeneous local objectives, the aggregated server trajectory (red arrows) may deviate from the centralized optimization trajectory and move toward an averaged local solution

\mathbf{w}_{\text{server}}^{(\text{final})}

(black square), rather than the centralized optimum

\mathbf{w}^{*}

(gray square).

Figure 3: Illustration of the proposed SLIAvg strategy with randomized client ordering, sequential local initialization, and server-side aggregation.

Figure 4: Visualization of client heterogeneity in exogenous weather responses and temporal load-profile distributions. (a) Distributions of client-specific response coefficients to continuous and categorical weather factors. Temp., Dew pt., App. temp., Humid., Wind spd., Press., Vis., and Wind dir. denote temperature, dew point, apparent temperature, humidity, wind speed, pressure, visibility, and wind direction, respectively. PC day and PC night denote partly cloudy day and partly cloudy night. (b) Pairwise temporal heterogeneity matrix of client load profiles.

Pretraining	Transformer		LSTM
Epoch	MSE	MAE	MSE	MAE
No pretraining
0	0.1363	0.1993	0.0897	0.1599
Pretraining without FL training data
1	0.1034	0.1708	0.0856	0.1567
5	0.0928	0.1606	0.0839	0.1566

Table I: Effect of Pretraining Epochs on Test Loss (

K=15

)

Figure 5: The test loss of the Transformer-based models.

Figure 6: The test loss of the LSTM-based models.

Figure 7: Client drift of the Transformer model with and without pretraining.

Figure 8: Client drift of the LSTM model with and without pretraining.

House ID	FedAvg		SLI		SLIAvg		Centralized
	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
$1$	0.1605	0.2652	0.1611	0.2548	0.1556	0.2641	0.1294	0.2592
$2$	0.0899	0.1477	0.0943	0.1639	0.0939	0.1646	0.0926	0.1578
$3$	0.0106	0.0676	0.0102	0.0692	0.0090	0.0609	0.0076	0.0558
$4$	0.0334	0.1282	0.0368	0.1452	0.0322	0.1248	0.0319	0.1227
$5$	0.2343	0.2952	0.1951	0.2666	0.1896	0.2716	0.1633	0.2642

Table II: House-Level and Overall Forecasting Performance of Different Training Strategies

FL Strategy	K = 15		K = 30
	Transformer	LSTM	Transformer	LSTM
FedAvg	0.1363	0.0897	0.1016	0.0576
SLI	0.0908	0.0885	0.0749	0.0560
SLIAvg	0.0853	0.0764	0.0744	0.0537
Centralized	0.0749	0.0704	0.0573	0.0414

Table III: Test Loss Comparison of Different FL Strategies Under Different Numbers of Clients

Explore similar work

Global Federated Learning Strategies for Building Efficient Personalized Models
Aug 15, 2026Seongyoon KimFederated Learning Heterogeneous Federated Learning
한국과학기술원 Korea Advanced Institute of Science and Technology 산업및시스템공학과
Latent Information Sharing for Accelerating Federated Learning
Oct 1, 2026Seungjun Lee, Ensieh Khazaei, Dimitrios Hatzinakos +2Federated Learning Aggregation Federated Learning
Inha University · University of Toronto · University of Birmingham
FedPLT: Scalable, Resource-Efficient, and Heterogeneity-Aware Federated Learning via Partial Layer Training
May 4, 2026Ahmad Dabaja, Rachid El-AzouziCommunication-Efficient Distributed Training Heterogeneous Federated Learning
Laboratoire Informatique d’Avignon (LIA), Avignon University, France

Aug 15, 2026, cs.LGJ/K move · Enter open · S save

Global Federated Learning Strategies for Building Efficient Personalized Models

Seongyoon Kim

한국과학기술원 Korea Advanced Institute of Science and Technology 산업및시스템공학과

Federated Learning Heterogeneous Federated Learning Federated Fine-Tuning Reward Modeling

Open paper AskPDF arXiv

Federated learning (FL) is a practical framework that can train models on distributed user data while guaranteeing data privacy; however, due to heterogeneity in which each user has a different data distribution, problems frequently arise where both global and personalization performance deteriorate simultaneously. This dissertation presents methodologies for building efficient personalized models by identifying which strategies are effective in the global training stage and by showing how to preserve global knowledge while securing user-specific performance during local adaptation. First, we show that as data heterogeneity increases, the collapse of feature vectors is a more fundamental bottleneck than classifier weights, and propose a method that directly mitigates the discrepancy in representation magnitude between local and global models. Second, we analyze that a training approach that strengthens local alignment can induce forgetting of global knowledge (e.g., categories not observed locally), and propose a method that achieves both local alignment and global knowledge preservation by combining feature distillation based on the global model's feature vectors. Third, in federated personalized reward model learning with preference heterogeneity, we empirically verify the conventional belief that "increasing the number of global models yields better initialization," and we show that when sufficient local fine-tuning is allowed, a single global initialization can instead provide stronger personalization performance. This study redefines the role of global initialization under data and preference heterogeneity and provides practical training strategies that simultaneously satisfy global knowledge preservation and personalization.

Abstract

Explore similar work

Global Federated Learning Strategies for Building Efficient Personalized Models

Latent Information Sharing for Accelerating Federated Learning

FedPLT: Scalable, Resource-Efficient, and Heterogeneity-Aware Federated Learning via Partial Layer Training