Spectral-Aware Text-to-Time Series Generation with Billion-Scale Multimodal Meteorological Data

Shijie Zhang

cs.LGMar 28, 2026

Spectral-Aware Text-to-Time Series Generation with Billion-Scale Multimodal Meteorological Data

Authors: Shijie Zhang

Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China

Abstract

Text-to-time-series generation is particularly important in meteorology, where natural language offers intuitive control over complex, multi-scale atmospheric dynamics. Existing approaches are constrained by the lack of large-scale, physically grounded multimodal datasets and by architectures that overlook the spectral-temporal structure of weather signals. We address these challenges with a unified framework for text-guided meteorological time-series generation. First, we introduce MeteoCap-3B, a billion-scale weather dataset paired with expert-level captions constructed via a Multi-agent Collaborative Captioning (MACC) pipeline, yielding information-dense and physically consistent annotations. Building on this dataset, we propose MTransformer, a diffusion-based model that enables precise semantic control by mapping textual descriptions into multi-band spectral priors through a Spectral Prompt Generator, which guides generation via frequency-aware attention. Extensive experiments on real-world benchmarks demonstrate state-of-the-art generation quality, accurate cross-modal alignment, strong semantic controllability, and substantial gains in downstream forecasting under data-sparse and zero-shot settings. Additional results on general time-series benchmarks indicate that the proposed framework generalizes beyond meteorology.

Figures & tables

Fig. 1: Overview of our Multi-agent Collaborative Captioning (MACC) pipeline for MeteoCap-3B construction. Note that we used GPT-4o, Claude Sonnet 3.5, and Gemini-3-Flash in Phase 3 and used DeepSeek V3 [ 23 ] in Phase 4. Human experts contains eight PhD in meteorology.

Fig. 2: Representative samples across three subsets of MeteoCap-3B.

Category	Metric	Value
Global Scale	Total observation points	$3.15\times 10^{9}+$
	Number of stations	$31{,}400+$
	Time span	1991–2025
Series Modality	Variables per step	5 (Temp, Press, Humid, Wind, Precip)
	Sequence length ( $T$ )	168 (hourly) / 512 (sub-hourly)
	Steady / Transient / Volatile	60% / 30% / 10%

TABLE I: Statistics of MeteoCap-3B. Temp: temperature, Press: pressure, Humid: humidity, Wind: wind speed, Precip: precipitation.

Fig. 3: Our MTransformer for time series generation. Qwen-3-Embedding-7B [ 25 ] is used to achieve the caption embedding with 1024 dimensions.

			MTransformer (Ours)			T2S [ 22 ] (IJCAI 2025)			DiffusionTS [ 14 ] (ICLR 2024)			TimeVAE [ 29 ] (arXiv 2025)			GPT-4o-mini [ 30 ] (2024)			Llama3.1-8b [ 31 ] (2023)			Gemini-3 (2025)
Group	Datasets	Length	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$	WAPE $\downarrow$	MSE $\downarrow$	MRR@10 $\uparrow$
TSFragment-600K [ 22 ]	ETTh1	24	0.162	0.009	0.279	0.183	0.008	0.283	0.793	0.077	0.267	0.666	0.055	0.211	0.264	0.041	0.104	0.883	0.663	0.097	0.245	0.035	0.112
		48	0.216	0.014	0.300	0.234	0.013	0.289	1.207	0.120	0.298	0.647	0.055	0.286	0.414	0.080	0.100	0.923	1.260	0.086	0.385	0.072	0.108
		96	0.218	0.012	0.296	0.229	0.011	0.291	0.498	0.028	0.214	0.643	0.055	0.286	0.500	0.118	0.096	0.949	1.748	0.056	0.466	0.105	0.099
	ETTm1	24	0.376	0.032	0.287	0.426	0.033	0.286	0.604	0.040	0.251	0.666	0.048	0.219	0.244	0.031	0.101	1.134	0.798	0.099	0.238	0.030	0.108
		48	0.368	0.054	0.280	0.53	0.053	0.283	1.119	0.100	0.285	0.636	0.051	0.217	0.453	0.112	0.097	1.074	1.496	0.079	0.425	0.101	0.103

TABLE II: Performance comparison on TSFragment-600K datasets and our proposed MeteoCap-3B . Note that for Meteo datasets, we evaluate on longer sequences (up to 336) to test long-range consistency. Bold : the best.

Fig. 4: Qualitative analysis. Left: Qualitative time series reconstruction. Right: Power spectral density of real and generated time series.

Fig. 5: Scaling behavior of MTransformer with respect to model size (length of 96), data scale, and sequence length, evaluated on the Meteo-Volatile subset.

Training Data Source	Backbone: Transformer				Backbone: TimesNet
Training Data Source	96	192	336	720	96	192	336	720
Real Only (10% data)	0.342	0.385	0.441	0.512	0.285	0.324	0.388	0.442
+ TimeVAE [ 29 ]	0.335	0.372	0.430	0.498	0.278	0.315	0.375	0.430
+ Diffusion-TS [ 14 ]	0.318	0.355	0.412	0.475	0.262	0.301	0.358	0.412
+ T2S [ 22 ]	0.305	0.340	0.395	0.455	0.255	0.292	0.345	0.398
+ MTransformer (Ours)	0.272	0.305	0.352	0.408	0.238	0.274	0.322	0.375

TABLE III: Downstream forecasting performance (MSE

\downarrow

) on ISD-Test under the Few-shot (10% Real Data) setting with two distinct backbones: a standard Transformer and TimesNet.

Pre-training	Zero-Shot		Full FT		Avg. Gain
Pre-training	MSE	MAE	MSE	MAE	ZS	FT
Scratch (Random Init.)	0.612	0.525	0.285	0.318	-	-
Unimodal Pre-train (Series Only)	0.545	0.482	0.264	0.295	+10.1%	+7.3%
Multimodal Pre-train (Ours)	0.465	0.412	0.235	0.268	+22.8%	+16.4%

TABLE IV: Impact of Pre-training Modality on Downstream Performance under Zero-Shot Transfer and Full Fine-Tuning (FT).

Fig. 6: Controllability of MTransformer generated sequences.

Part A: Ablation	Model Variants	MSE $\downarrow$	WAPE $\downarrow$
	Full Model (Ours)	0.028	0.265
	w/o Spectral Scheduler (Fixed $\lambda=1$ )	0.030	0.278
	w/o SPAC (Standard Cross-Attn)	0.033	0.292
	w/o MSPG (Raw Text Embed.)	0.038	0.325
	w/o Text Cond. (Base DiT)	0.045	0.368
Part B: Hyperparameter	Parameter Settings	MSE $\downarrow$	WAPE $\downarrow$

TABLE V: Ablation study and hyperparameter sensitivity analysis on the Meteo-Volatile subset (

L=96

, MSE, WAPE report).

Method	Hallucination Rate ( $\downarrow$ )	Phys. Consistency ( $\uparrow$ )	Informativeness ( $\uparrow$ )
GPT-4o	24.5%	3.12 $\pm$ 0.45	3.45 $\pm$ 0.38
Claude-4.5	18.2%	3.35 $\pm$ 0.41	3.68 $\pm$ 0.42
MACC (Ours)	3.8%	4.82 $\pm$ 0.15	4.65 $\pm$ 0.22

TABLE VI: Human expert evaluation of captioning quality. Values are reported as mean

\pm

std.

Heuristic Baselines
Method	Overall Test Set (All Subsets)						Meteo-Volatile Subset (Extreme Events)
	Text-to-Series			Series-to-Text			Text-to-Series			Series-to-Text
	R@1	R@5	MRR	R@1	R@5	MRR	R@1	R@5	MRR	R@1	R@5	MRR
Random Guess	0.01	0.05	0.001	0.01	0.05	0.001	0.01	0.05	0.001	0.01	0.05	0.001
Stat-Feature Matching	8.5	22.1	0.145	8.2	21.5	0.142	4.2	12.8	0.085	3.8	11.5	0.080
DTW-Matching	12.4	26.8	0.185	11.5	25.2	0.178	6.8	15.5	0.110	6.2	14.2	0.105

TABLE VII: Bi-directional cross-modal retrieval performance on MeteoCap-3B test set. We compare against heuristic methods, trained baselines, and LLM-based adapters. Meteo-Volatile subset tests alignment under extreme conditions.

Explore similar work

Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusion
Mar 23, 2026Shixu LiuMulti-Agent LLM Systems Gradient Descent
School of Computer & Information Technology, Northeast Petroleum University, No. 99 Xuefu Street, Ranghulu District, Daqing, Heilongjiang, 163318, P.R. China
SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future
Apr 7, 2026Timothy B. Higgins, Antonios Mamalakis, Chirag AgarwalVLM Evaluation Forecasting Benchmarks
Department of Environmental Sciences, University of Virginia, Charlottesville, 22903, Virginia, United States. · School of Data Science, University of Virginia, Charlottesville, 22903, Virginia, United States.
Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting
Feb 2, 2026Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen +1Multivariate Time Series Forecasting Time Series Forecasting
Applied Artificial Intelligence Initiative Deakin University Geelong, Australia

Mar 23, 2026J/K move · Enter open · S save

Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusion

Shixu Liu

School of Computer & Information Technology, Northeast Petroleum University, No. 99 Xuefu Street, Ranghulu District, Daqing, Heilongjiang, 163318, P.R. China

Multi-Agent LLM Systems Gradient Descent

Open paper AskPDF arXiv

Generating interpretable natural language captions from weather time series data remains a significant challenge at the intersection of meteorological science and natural language processing. While recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in time series forecasting and analysis, existing approaches either produce numerical predictions without human-accessible explanations or generate generic descriptions lacking domain-specific depth. We introduce WeatherTGD, a training-free multi-agent framework that reinterprets collaborative caption refinement through the lens of Text Gradient Descent (TGD). Our system deploys three specialized LLM agents including a Statistical Analyst, a Physics Interpreter, and a Meteorology Expert that generate domain-specific textual gradients from weather time series observations. These gradients are aggregated through a novel Consensus-Aware Gradient Fusion mechanism that extracts common signals while preserving unique domain perspectives. The fused gradients then guide an iterative refinement process analogous to gradient descent, where each LLM-generated feedback signal updates the caption toward an optimal solution. Experiments on real-world meteorological datasets demonstrate that WeatherTGD achieves significant improvements in both LLM-based evaluation and human expert evaluation, substantially outperforming existing multi-agent baselines while maintaining computational efficiency through parallel agent execution.

Abstract

Explore similar work

Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusion

SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future

Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting