Organizations: School of Artificial Intelligence, Hebei University of Technology, Tianjin 300401, China · University of Southern Queensland, Toowoomba 487-535, Australia · School of Information Engineering, Tianjin University of Commerce, Tianjin 300134, China · Hebei Province Key Laboratory of Big Data Calculation, Hebei University of Technology, Tianjin 300401, China · Trustworthy and General AI Lab, School of Engineering, Westlake University, Hangzhou, 310030, China · Discipline of Business Systems and Operations, School of Business and Law, Edith Cowan University, Joondalup, WA 6027, Australia
The inherent intermittency and high-frequency variability of solar irradiance, particularly during rapid cloud advection, present significant stability challenges to high-penetration photovoltaic grids. Although multimodal forecasting has emerged as a viable mitigation strategy, existing architectures predominantly rely on shallow feature concatenation and binary cloud segmentation, thereby failing to capture the fine-grained optical features of clouds and the complex spatiotemporal coupling between visual and meteorological modalities. To bridge this gap, this paper proposes M3S-Net, a novel multimodal feature fusion network based on multi-scale data for ultra-short-term PV power forecasting. First, a multi-scale partial channel selection network leverages partial convolutions to explicitly isolate the boundary features of optically thin clouds, effectively transcending the precision limitations of coarse-grained binary masking. Second, a multi-scale sequence to image analysis network employs Fast Fourier Transform (FFT)-based time-frequency representation to disentangle the complex periodicity of meteorological data across varying time horizons. Crucially, the model incorporates a cross-modal Mamba interaction module featuring a novel dynamic C-matrix swapping mechanism. By exchanging state-space parameters between visual and temporal streams, this design conditions the state evolution of one modality on the context of the other, enabling deep structural coupling with linear computational complexity, thus overcoming the limitations of shallow concatenation. Experimental validation on the newly constructed fine-grained PV power dataset demonstrates that M3S-Net achieves a mean absolute error reduction of 6.2% in 10-minute forecasts compared to state-of-the-art baselines. The dataset and source code will be available at https://github.com/she1110/FGPD.
Figures & tables
Figure 1: The overall diagram of the proposed M3S-Net. The framework accomplishes the prediction task through three core components: 1. Fine-grained visual extraction branch; 2. Multi-scale temporal imaging branch; 3. Cross-modal mamba fusion branch.
Figure 2: The structure of the proposed MPCS-Net, which is an Encoder-Decoder architecture. The spatial-channel selection mechanism is embedded within the encoder to form the MSPC and MSPA.
Figure 3: The structure of the proposed MPCS and MSPA, which is concluding the (a) SCSM. The core modules of the SCSM are (b) CSIA and (c) CE.
Figure 4: The flowchart of the multi-scale representation.
Figure 5: The flowchart of the CSA to capture nonlinear interdependencies among variables on the coarsest scale XM .
Figure 6: The flowchart of (a) FFT-based 2D transformation. (b) RA. (c) Amplitude-weighted fusion and prediction. The architecture of (b1) RA, (b2) D-MSA and S-MSA.
Figure 7: The architecture of the proposed MMIF.
Cloud class
Shape and characteristic
Color
White cloud
White transparent clouds, such as cirrus and cirrocumulus, are pure white and translucent or appear as white scaly flakes.
White
Gray cloud
Gray clouds, such as stratus and altostratus, are gray and foggy with a dark base.
Gray
Sun
Radiation source
Red
Background
There are no clouds in the sky
Blue
Table 1: Cloud classification standard for CSRC dataset.
Figure 8: Eight types of meteorological data under three types of typical weather.
Method
P( ↑ )
R( ↑ )
MIoU( ↑ )
U-Net
66.7
64.6
46.7
SegFormer
84.6
85.2
61.3
CloudFU-Net
86.9
86.5
62.1
CloudSwinNet
87.1
87.6
62.3
MPCM-Net
90.6
91.1
64.8
MPCS-Net
91.3
91.8
65.6
Table 2: Quantitative comparison with different methods on FGPD for GCI segmentation. ↓ (or ↑ ) indicates lower (or higher) is better. The best results are highlighted in bold.
Model structure
Method
MAE ( W/m2 ) ( ↓ )
10 MA
20 MA
30 MA
40 MA
50 MA
60 MA
Singlemodal
LSTM
39.76
42.28
45.53
48.97
54.64
61.42
DLinear
22.32
31.44
44.33
55.62
66.73
77.64
TimesNet
23.45
28.61
34.25
41.27
47.82
55.66
Multimodal
CNN-based
22.56
27.74
33.96
40.66
47.12
54.12
ViT-based
22.37
28.13
33.84
39.72
47.33
53.83
Table 3: Comparison results of GHI based on MAE ( W/m2 ) on the FGPD.
Model structure
Method
NRMSE ( % ) ( ↓ )
10 MA
20 MA
30 MA
40 MA
50 MA
60 MA
Singlemodal
LSTM
6.82
7.31
7.94
8.82
9.41
10.23
DLinear
5.14
5.72
7.16
8.73
10.26
11.78
TimesNet
5.42
5.96
6.78
7.64
8.86
9.35
Multimodal
CNN-based
5.23
5.88
6.52
7.36
8.29
9.14
ViT-based
5.09
5.63
6.38
7.14
7.93
8.89
Table 4: Comparison results of GHI based on NRMSE ( % ) on the FGPD.
Model structure
Method
R2 ( ↑ )
10 MA
20 MA
30 MA
40 MA
50 MA
60 MA
Single -modal
LSTM
0.912
0.903
0.894
0.882
0.864
0.845
DLinear
0.931
0.922
0.910
0.897
0.872
0.841
TimesNet
0.947
0.923
0.904
0.884
0.849
0.807
Multi -modal
CNN-based
0.954
0.931
0.916
0.902
0.874
0.851
ViT-based
0.953
0.934
0.915
0.894
0.871
0.848
Table 5: Comparison results of GHI based on R2 on the FGPD.
Figure 9: Visual results of average value of different error indicators.
Figure 10: Visual results under three typical weather categories of the proposed M3S-Net and other singlemodal methods.
Figure 11: Visual results under three typical weather categories of the proposed M3S-Net and other multimodal methods.
Version
MAE ( W/m2 ) ( ↓ )
NRMSE ( % ) ( ↓ )
R2 ( ↑ )
A
23.44
5.37
0.948
B
22.46
5.21
0.951
C
22.31
5.15
0.853
D
21.84
5.06
0.957
E
21.45
4.94
0.960
F
20.96
4.72
0.962
Table 6: Ablation experimental results on the FGPD.
Figure 12: Visualization of the impact of input lengths in SIFR-Net.
Model structure
Method
NRMSE ( % )
10 MA
20 MA
30 MA
40 MA
50 MA
60 MA
Single -modal
LSTM
7.73
7.83
8.12
8.64
9.37
10.01
DLinear
4.72
6.88
8.38
9.62
10.84
12.03
TimesNet
5.12
6.39
7.29
8.15
8.99
9.84
Multi -modal
CNN-based
5.03
6.19
7.08
7.88
8.71
9.33
ViT-based
5.04
6.16
7.02
7.92
8.66
9.36
Table 7: Comparison results of GHI with different methods on the SRRL dataset ( % ).
Figure 13: Visualization of the impact of image resolution in MPCS-Net.
Figure 14: Visualization results show different categories of the FGPD validation set using Grad-CAM. (a) Input image, (b) MPCM-Net, and (c) MPCS-Net.
Figure 15: Visualization of representation on FGPD across three different scales (length: 192, 96, 48) and three periods (96, 48, 36) with frequencies (2, 4, 5). Both high-frequency local fluctuations and low-frequency seasonal trends via spectral-temporal disentanglement are captured.
Figure 16: The heatmaps of visual and temporal weights of MMIF under different weather conditions. Highlight performance along the horizontal axis indicates the difference in the visual feature activation. Visual feature is activated to enhance temporal features to participate in GHI forecasting.
Ultra-short-term solar irradiance prediction is critical for photovoltaic system dispatch and power grid stability. Existing approaches suffer from three key shortcomings: single time-series models cannot capture the spatial dynamics of clouds under complex conditions, standard convolutions inadequately represent multi-scale cloud features, and fixed low-frequency compensation strategies fail to adapt to different prediction steps. To address these issues, this proposes a multi-source data fusion model for ultra-short-term irradiance prediction. The model first employs InceptionNeXt to extract multi-scale, multi-directional spatial features from ground-based cloud images. A step-adaptive low-frequency compensation unit is then introduced to dynamically modulate global low-frequency information based on the prediction step. Eventually, the enhanced image features are combined with meteorological time-series features, and a TempAttnLSTM network captures global temporal dependencies for multi-step prediction. Experiments on the public NREL dataset and practical photovoltaic stations in Shandong illustrate the effectiveness of the proposed method compared with several state-of-the-art approaches.
Jingxin Zhang Xiaoqin Wang
School of Automation, Southeast University, Nanjing, 210096, China
Photovoltaic (PV) power forecasting plays a critical role in power system dispatch and market participation. Because PV generation is highly sensitive to weather conditions and cloud motion, accurate forecasting requires effective modeling of complex spatiotemporal dependencies across multiple information sources. Although recent studies have advanced AI-based forecasting methods, most fail to fuse temporal observations, satellite imagery, and textual weather information in a unified framework. This paper proposes Solar-VLM, a large-language-model-driven framework for multimodal PV power forecasting. First, modality-specific encoders are developed to extract complementary features from heterogeneous inputs. The time-series encoder adopts a patch-based design to capture temporal patterns from multivariate observations at each site. The visual encoder, built upon a Qwen-based vision backbone, extracts cloud-cover information from satellite images. The text encoder distills historical weather characteristics from textual descriptions. Second, to capture spatial dependencies across geographically distributed PV stations, a cross-site feature fusion mechanism is introduced. Specifically, a Graph Learner models inter-station correlations through a graph attention network constructed over a K-nearest-neighbor (KNN) graph, while a cross-site attention module further facilitates adaptive information exchange among sites. Finally, experiments conducted on data from eight PV stations in a northern province of China demonstrate the effectiveness of the proposed framework. Our proposed model is publicly available at https://github.com/rhp413/Solar-VLM.
Hang Fan, Haoran Pei, Runze Liang +3
School of Economic and Management, North China Electric Power University, 102206, Beijing, China · School of Control and Computer Engineering, North China Electric Power University, 102206, Beijing, China · Department of Electrical Engineering, Tsinghua University, 100084, Beijing, China +1
Due to the rise in the use of renewable energies as an alternative to traditional ones, and especially solar energy, there is increasing interest in studying how to address photovoltaic forecasting in the face of the challenge of variability in photovoltaic energy production, using different methodologies. This work develops a hybrid approach for short and long-term forecasting based on two studies with the same purpose. A multimodal approach that combines images of the sky and photovoltaic energy history with meteorological data is proposed. The main goal is to improve the accuracy of ramp event prediction, increase the robustness of forecasts in cloudy conditions, and extend capabilities beyond nowcasting, to support more efficient operation of the power grid and better management of solar variability. Deep neural models are used for both nowcasting and forecasting solutions, incorporating individual and multiple meteorological variables, as well as an analytical solar position. The results demonstrate that the inclusion of meteorological data, particularly the surface long-wave, radiation downwards, and the combination of wind and solar position, significantly improves current predictions in both nowcasting and forecasting tasks, especially on cloudy days. This study highlights the importance of integrating diverse data sources to improve the reliability and interpretability of solar energy prediction models.
Ines Montoya-Espinagosa, Antonio Agudo
Universitat Pompeu Fabra, Barcelona, Spain. · Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain.