Solar-VLM: Multimodal Vision-Language Models for Augmented Solar Power Forecasting

Hang Fan; Haoran Pei; Runze Liang; Weican Liu; Long Cheng; Wei Wei

cs.AIApr 5, 2026

Solar-VLM: Multimodal Vision-Language Models for Augmented Solar Power Forecasting

Authors: Hang Fan, Haoran Pei, Runze Liang, Weican Liu, Long Cheng, Wei Wei

Organizations: School of Economic and Management, North China Electric Power University, 102206, Beijing, China · School of Control and Computer Engineering, North China Electric Power University, 102206, Beijing, China · Department of Electrical Engineering, Tsinghua University, 100084, Beijing, China · School of Electrical and Electronic Engineering, Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore

Abstract

Photovoltaic (PV) power forecasting plays a critical role in power system dispatch and market participation. Because PV generation is highly sensitive to weather conditions and cloud motion, accurate forecasting requires effective modeling of complex spatiotemporal dependencies across multiple information sources. Although recent studies have advanced AI-based forecasting methods, most fail to fuse temporal observations, satellite imagery, and textual weather information in a unified framework. This paper proposes Solar-VLM, a large-language-model-driven framework for multimodal PV power forecasting. First, modality-specific encoders are developed to extract complementary features from heterogeneous inputs. The time-series encoder adopts a patch-based design to capture temporal patterns from multivariate observations at each site. The visual encoder, built upon a Qwen-based vision backbone, extracts cloud-cover information from satellite images. The text encoder distills historical weather characteristics from textual descriptions. Second, to capture spatial dependencies across geographically distributed PV stations, a cross-site feature fusion mechanism is introduced. Specifically, a Graph Learner models inter-station correlations through a graph attention network constructed over a K-nearest-neighbor (KNN) graph, while a cross-site attention module further facilitates adaptive information exchange among sites. Finally, experiments conducted on data from eight PV stations in a northern province of China demonstrate the effectiveness of the proposed framework. Our proposed model is publicly available at https://github.com/rhp413/Solar-VLM.

Figures & tables

Figure 1: Overview of the proposed model.

Figure 2: Overview of the proposed two-stage cross-site joint modeling strategy.

Module	Hyperparameter	Symbol	Value
Time Series Encoder	Patch length	$l_{p}$	10
	Patch stride	$s_{p}$	8
	Embedding dimension	$d_{\mathrm{model}}$	128
	Memory bank size	$N_{m}$	100
	Number of retrieved memories	$k$	5
Text Encoder	Qwen-VLM output dimension	$d_{\text{text}}$	2048

Table 1: Model Hyperparameters

Model	$T=6$			$T=12$			$T=24$			$T=36$			$T=48$			$T=96$
Model	MSE	MAE	$R^{2}$	MSE	MAE	$R^{2}$	MSE	MAE	$R^{2}$	MSE	MAE	$R^{2}$	MSE	MAE	$R^{2}$	MSE	MAE	$R^{2}$
LSTM	0.175	0.231	0.932	0.279	0.299	0.891	0.703	0.508	0.727	0.714	0.501	0.723	0.564	0.461	0.781	0.595	0.449	0.770
Informer	0.937	0.609	0.635	0.594	0.494	0.769	0.427	0.361	0.834	0.494	0.425	0.808	0.452	0.381	0.825	0.495	0.392	0.808
FEDformer	0.383	0.413	0.851	0.427	0.474	0.834	0.458	0.481	0.822	0.450	0.484	0.825	0.485	0.495	0.812	0.534	0.567	0.793
TimesNet	0.228	0.298	0.911	0.295	0.377	0.885	0.407	0.412	0.842	0.428	0.425	0.834	0.443	0.452	0.828	0.681	0.534	0.736
TimeLLM	0.287	0.329	0.888	0.335	0.349	0.869	0.378	0.388	0.853	0.435	0.422	0.831	0.418	0.396	0.838	0.447	0.420	0.827

Table 2: Performance Comparison of Different Models

Figure 3: MAE boxplots for different models at different forecasting horizons.

Figure 4: PV power prediction curves of compared models against the ground truth on the test set at different forecasting horizons.

Setting	MSE	MAE	$R^{2}$
Full	0.207	0.254	0.888
Time-Series Encoder Only	0.256	0.296	0.862
Without Text Encoder	0.233	0.274	0.874
Without Visual Encoder	0.211	0.256	0.886
Without Graph Leaner	0.218	0.261	0.882
Without Cross-Site Attention	0.301	0.296	0.862

Table 3: Ablation Study on the Proposed Model

Figure 5: Forecasting results of different ablation variants on a representative cloudy day.

Figure 6: Forecasting results of different ablation variants on a representative sunny day.

Figure 7: Sensitivity analysis of key hyperparameters.

History Length $L$	MSE	MAE	$R^{2}$
12	0.681	0.476	0.632
24	0.495	0.416	0.733
48	0.348	0.342	0.812
96	0.311	0.312	0.832
192	0.249	0.282	0.865
288	0.207	0.254	0.888

Table 4: Sensitivity to Historical Time-Series Length

Patch Length $l_{p}$	MSE	MAE	$R^{2}$
4	0.250	0.289	0.865
8	0.228	0.262	0.877
10	0.207	0.254	0.888
12	0.256	0.283	0.862
16	0.225	0.265	0.878
18	0.218	0.262	0.882

Table 5: Sensitivity to Patch Length

l_{p}

Number of Images $k$	MSE	MAE	$R^{2}$
2	0.234	0.275	0.873
4	0.250	0.286	0.865
6	0.235	0.277	0.873
8	0.207	0.254	0.888
10	0.218	0.264	0.882
12	0.207	0.254	0.888

Table 6: Sensitivity to Number of Historical Satellite Images

KNN Neighbors $K$	MSE	MAE	$R^{2}$
1	0.217	0.259	0.883
2	0.231	0.269	0.875
3	0.228	0.266	0.877
4	0.221	0.259	0.881
5	0.207	0.254	0.888
6	0.217	0.260	0.883

Table 7: Sensitivity to Number of Neighbors

K

in KNN Graph

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Field	Description
Task Configuration	task	Fixed task identifier
	target	Target variable for forecasting
	output	Output format specification
	forecast_steps	Number of steps to forecast
	step_minutes	Temporal resolution in minutes
	context_short_steps	Short-term context window length

Appendix

Table 8: Prompt Field Definitions.

Explore similar work

Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models
Sep 10, 2026Ken Chen, Maneesha Perera, Wei Wang +3Multivariate Time Series Forecasting Multimodal Large Language Models
Department of Mechanical Engineering, The University of Melbourne, Melbourne, Australia · Department of Mechanical Engineering, Sri Lanka Institute of Information Technology, Sri Lanka
M3S-Net: Multimodal Feature Fusion Network Based on Multi-scale Data for Ultra-short-term PV Power Forecasting
Feb 23, 2026Penghui Niu, Taotao Cai, Suqi Zhang +4Solar Power Forecasting Multimodal Fusion
School of Artificial Intelligence, Hebei University of Technology, Tianjin 300401, China · University of Southern Queensland, Toowoomba 487-535, Australia · School of Information Engineering, Tianjin University of Commerce, Tianjin 300134, China +3
Meteorological data and Sky Images meets Neural Models for Photovoltaic Power Forecasting
Feb 17, 2026Ines Montoya-Espinagosa, Antonio AgudoSolar Power Forecasting Multivariate Time Series Forecasting
Universitat Pompeu Fabra, Barcelona, Spain. · Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain.

Sep 10, 2026J/K move · Enter open · S save

Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models

Ken Chen, Maneesha Perera, Wei Wang, Sachith Seneviratne+2

Department of Mechanical Engineering, The University of Melbourne, Melbourne, Australia · Department of Mechanical Engineering, Sri Lanka Institute of Information Technology, Sri Lanka

Multivariate Time Series Forecasting Multimodal Large Language Models LLM-Based Time Series Forecasting Solar Power Forecasting Multimodal Fusion

Open paper AskPDF arXiv

Short-term photovoltaic (PV) power and global horizontal irradiance (GHI) forecasts are essential for effective dispatch, reserve scheduling, and grid operations. At these forecasting horizons, errors are predominantly driven by cloud induced ramps: relying solely on historical numerical data may struggle to anticipate an incoming cloud, making ground-based sky images a crucial complementary physical signal. Furthermore, forecast performance is highly sensitive to location and local observing conditions, creating a strong need for site-specific data that are often scarce. Recently, large language models (LLMs) have demonstrated competitive performance and high data efficiency in time-series forecasting. Despite their success, existing LLM-based forecasting methods remain predominantly unimodal, relying primarily on historical numerical time-series data. Effectively incorporating sky imagery into an LLM-based forecasting framework remains under-explored and an open challenge. In this paper, we propose SolCloudLLM, an LLM-based multimodal forecasting framework. SolCloudLLM aligns sky-image patches with time-series patches and fuses their corresponding representations through bidirectional multimodal fusion, yielding a unified representation that is subsequently mapped into the embedding space of an LLM. Extensive experiments on the SIRTA and SKIPP'D datasets demonstrate that SolCloudLLM consistently outperforms the best baseline methods in MSE across all forecasting horizons, achieving a maximum relative MSE reduction of 25.4%. Stratified analysis further indicates that the benefits of multimodal fusion are concentrated primarily under cloudy conditions. Notably, SolCloudLLM achieves the best performance in nearly all few-shot settings, whereas other deep learning baselines experience substantial performance degradation and are frequently outperformed by the non-learning physical method.

Abstract

Explore similar work

Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models

M3S-Net: Multimodal Feature Fusion Network Based on Multi-scale Data for Ultra-short-term PV Power Forecasting

Meteorological data and Sky Images meets Neural Models for Photovoltaic Power Forecasting