PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision

Zehua Han; Jing Xiao; Yiqi Duan; Mengyu Xiang; Yuheng Ji; Xiaolong Zheng; Chenghanyu Zhang; Zhendong She

cs.AIMar 30, 2026

PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision

Authors: Zehua Han, Jing Xiao, Yiqi Duan, Mengyu Xiang, Yuheng Ji, Xiaolong Zheng, Chenghanyu Zhang, Zhendong She,

Organizations: Beihang University · Civil Aviation University of China · Beijing Institute of Technology · University of Chinese Academy of Sciences ·

Abstract

Multimodal Large Language Models have demonstrated powerful cross-modal understanding and reasoning capabilities in general domains. However, in the electromagnetic (EM) domain, they still face challenges such as data scarcity and insufficient integration of domain knowledge. This paper proposes PReD, the first foundation model for the EM domain that covers the intelligent closed-loop of "perception, recognition, decision-making." We constructed a high-quality multitask EM dataset, PReD-1.3M, and an evaluation benchmark, PReD-Bench. The dataset encompasses multi-perspective representations such as raw time-domain waveform, frequency-domain spectrograms, and constellation diagrams, covering typical features of communication and radar signals. It supports a range of core tasks, including signal detection, modulation recognition, parameter estimation, protocol recognition, radio frequency fingerprint recognition, and anti-jamming decision-making. PReD adopts a multi-stage training strategy that unifies multiple tasks for EM signals. It achieves closed-loop optimization from end-to-end signal understanding to language-driven reasoning and decision-making, significantly enhancing EM domain expertise while maintaining general multimodal capabilities. Experimental results show that PReD achieves state-of-the-art performance on PReD-Bench constructed from both open-source and self-collected signal datasets. These results collectively validate the feasibility and potential of vision-aligned foundation models in advancing the understanding and reasoning of EM signals.

Figures & tables

Figure 1: Overview of PReD and its capabilities, instantiated across six core electromagnetic (EM) tasks that span the three tiers of perception, recognition, and decision-making. The figure demonstrates how PReD processes multi-view signal visualizations to address a diverse range of EM instructions, from quantitative parameter estimation to open-ended anti-jamming strategy generation.

Figure 2: Overall pipeline for the construction of the PReD-1.3M training set and the PReD-Bench. We first generate five types of signal visualizations from the raw IQ signals. We then create corresponding OpenQA and MCQA pairs for each task type to form a complete instruction set. Finally, a portion of this set is sampled to obtain the PReD-Bench, which is kept strictly separate from the training data.

Panel of Figure 3 — Figure 3: Scale and composition of the EM training dataset across six tasks and two QA formats.

Dimension	Task	OpenQA	MCQA
EM Perception	SPE	2250	750
EM Perception	SSD	1700	300
EM Recognition	MR	–	500
	PR	–	500
	EI	–	458
EM Decision-making	AJSD	2000	–

Table 1: PReD-Bench for perception, recognition, and decision-making. Abbreviations: SSD = Signal Segment Detection; SPE = Signal Parameter Estimation; MR = Modulation Recognition; PR = Protocol Recognition; EI = Emitter Identification; AJSD = Anti-Jamming Strategy Decision-making. OpenQA and MCQA denote open-ended and multiple-choice question–answer pairs, respectively.

Figure 4: Overall pipeline of PReD. (Bottom to top) Multi-view EM renderings are encoded and projected into the token space, where they are combined with tokenized instructions. A pre-trained LLM decoder is first aligned with visual features and then fine-tuned through a multi-stage curriculum to become the specialized PReD LLM Decoder . The model supports two output modes: open-ended generation (OpenQA) and structured multiple-choice question answering (MCQA).

Category	Hyperparameter	Stage-1	Stage-2	Stage-3	Stage-4
Vision	Resolution	$384$	Max $384\times(2\times 2)$	Max $384\times(6\times 6)$	Max $384\times(6\times 6)$
Vision	Tokens	$729$	Max $729\times 5$	Max $729\times 10$	Max $729\times 10$
Data	Dataset	LCS	General Single-Image	General Multi-Image*	PReD-1.3M *
Data	Samples	$558\mathrm{K}$	$858\mathrm{K}$	$832\mathrm{K}$	$1.8\mathrm{M}$
Model	Trainable Components	Projector	Full Model	Full Model	Full Model
Model	Tunable Parameters	$21.5\mathrm{M}$	$8.6\mathrm{B}$	$8.6\mathrm{B}$	$8.6\mathrm{B}$

Table 2: Training schedule for PReD across four stages on 16 NVIDIA A100 GPUs (80GB each). A star * indicates datasets mixed with general visual corpora from earlier stages.

Model	EM Perception				EM Recognition			EM Decision-Making
	SSD		SPE		MR	PR	EI	AJSD ( OpenQA # )
	MCQA	OpenQA ∗	MCQA	OpenQA ∗	MCQA	MCQA	MCQA	BLEU4	ROUGE	METEOR	CIDEr
qwen2.5-vl-7b † [ 38 ]	28.7	0.2	20.5	0.1	24.2	14.0	–	0.000	0.079	0.115	0.128
qwen3-vl-8b † [ 36 ]	21.3	1.3	18.8	2.0	11.6	2.8	–	0.000	0.094	0.121	0.228
qwen3-vl-235b-a22b † [ 36 ]	27.7	0.5	30.4	2.0	20.2	8.6	–	0.000	0.090	0.070	0.151
seed-1.6 vision [ 39 ]	41.3	1.5	40.4	3.1	23.4	21.4	–	0.000	0.095	0.039	0.169

Table 3: Benchmark results for EM perception, recognition, and decision-making tasks. A superscript † marks an Instruct (instruction-tuned) variant. OpenQA ∗ indicates answers with a specific tag whose results are directly matched; OpenQA # indicates answers without a specific tag, where matching is performed using BLEU4, ROUGE, METEOR, and CIDEr. Since EI task requires device-specific RF fingerprint learning, general MLLMs arenot comparable and their results are omitted.

Model	MMStar	AI2D	HallusionBench	SEEDBench IMG	POPE	ScienceQA TEST	Average
GPT - 4v [ 3 ]	49.7	75.9	46.5	71.6	75.4	82.1	66.9
PReD (Ours)	50.2	74.3	54.5	73.7	70.4	69.2	65.4
Qwen - VL - Max [ 43 ]	49.5	75.7	41.2	72.7	71.9	80.0	65.2
LLaVA - Next - Interleave - 7B [ 44 ]	44.5	73.8	34.8	72.0	85.5	73.2	64.0
DeepSeek - VL - 7B [ 45 ]	40.5	65.3	34.5	70.1	85.6	80.9	62.8
MiniCPM - V - 2 [ 46 ]	39.1	62.9	36.1	67.1	86.3	80.7	62.0

Table 4: Results on general multimodal benchmarks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Overview of our four-stage curriculum and data composition. The outer ring shows the training stages; the middle ring indicates data regimes (LCS, general single/multi-image); the inner ring lists EM tasks (SSD, SSE, MR, PR, EI, AJSD) emphasized in Stage 4. Percentages denote the proportion of samples used per stage.

Category	Hyperparameter	Stage-1	Stage-2	Stage-3	Stage-4
Vision	Resolution	$384$	Max $384\times(2\times 2)$	Max $384\times(6\times 6)$	Max $384\times(6\times 6)$
Vision	Tokens	$729$	Max $729\times 5$	Max $729\times 10$	Max $729\times 10$
Model	Trainable Components	Projector	Full Model	Full Model	Full Model
Model	Tunable Parameters	$21.5\mathrm{M}$	$8.6\mathrm{B}$	$8.6\mathrm{B}$	$8.6\mathrm{B}$
Training	Per-device Batch Size	$8$	$4$	$2$	$2$
	Gradient Accumulation	$1$	$1$	$1$	$1$

Appendix

Table 5: Training schedule across four curriculum stages.

Method	SSD	SPE	MR	PR	EI	AJSD	Average
PReD - (w/o General Corpora)	83.55	86.37	72.40	74.40	71.83	50.05	73.10
PReD (Ours)	83.40	85.70	72.00	72.00	69.43	48.59	71.85

Appendix

Table 6: Performance comparison on the specialized EM benchmark. The variant fine-tuned exclusively on EM data ( PReD - ) achieves a marginal performance gain on these domain-specific tasks. As shown in Table 7 , this slight advantage comes at the cost of a severe degradation in general multimodal capabilities.

Method	MMStar	AI2D	HallusionBench	SEEDBench IMG	POPE	ScienceQA TEST	Average
PReD - (w/o General Corpora)	39.47	54.44	8.10	56.75	0.74	56.02	35.92
PReD (Ours)	50.20	74.32	54.47	73.67	70.37	69.16	65.37

Appendix

Table 7: Ablation study on mixing general-domain data in the final training stage. We compare our full model ( PReD ) against a variant ( PReD - ) that was fine-tuned exclusively on the specialized EM dataset without the general visual corpora. Mixing general-domain data significantly prevents catastrophic forgetting and boosts performance across general benchmarks.

Constellation	FFT	STFT	IQ	Accuracy
$\checkmark$	$\times$	$\times$	$\times$	43.4
$\times$	$\checkmark$	$\times$	$\times$	35.0
$\times$	$\times$	$\checkmark$	$\times$	30.2
$\times$	$\times$	$\times$	$\checkmark$	50.8
$\checkmark$	$\checkmark$	$\times$	$\times$	54.8
$\checkmark$	$\times$	$\checkmark$	$\times$	58.4

Appendix

Table 8: MR accuracy (%) for different modality combinations. The optimal result is highlighted in bold.

Input Modality	BLEU4	ROUGE	METEOR	CIDEr	Average
Constellation	0.069	0.329	0.227	0.405	0.258
FFT	0.056	0.305	0.194	0.381	0.234
STFT	0.094	0.374	0.267	0.447	0.2955
IQ	0.149	0.438	0.337	0.530	0.363
Constellation+FFT	0.070	0.331	0.227	0.409	0.259
Constellation+STFT	0.104	0.384	0.279	0.465	0.308

Appendix

Table 9: Ablation on input modalities for the anti-jamming strategy decision-making task.

Figure 6: Zero-shot modulation recognition accuracy of PReD on the OOD RadioML2016.04c dataset across SNR levels. The overall accuracy is 66.0%.

Figure 7: Performance comparison on MR and PR task across varying SNR levels. Both tasks show a similar graceful degradation, highlighting the model’s consistent and robust behavior against noise.

Figure 8: Hierarchical composition of the PReD-Bench, showing the distribution across three core dimensions, six specific tasks, and two primary question types (OpenQA and MCQA).

Figure 9: Log-scale radar plot of averaged performance over five EM tasks. PReD demonstrates superior performance across all evaluated tasks.

Panel of Figure 10 — Figure 10: Qualitative examples of PReD’s performance across the six core tasks of PReD-Bench. Each panel showcases a different capability, from perception (a, b), to recognition (c, d, e), and finally to complex decision-making (f).

Wireless Foundation Models

Wireless foundation models learn reusable representations of radio channels, signals, network traces, and telemetry for transfer across wireless tasks and settings.

See all 33 papers

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals
Mar 9, 2026Junyu Shen, Zhendong She, Chenghanyu Zhang +13Multimodal Large Language Models Multimodal Pretraining
Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications +6
EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals
Aug 25, 2026Mingxu Zhang, Ying Sun, Yuhan Li +4LLM Evaluation Scientific Reasoning in Language Models
The Hong Kong University of Science and Technology (Guangzhou) · The 63rd Research Institute, National University of Defense Technology · Nanjing University of Aeronautics and Astronautics
Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization
Oct 8, 2026Lei Zhai, Zhihao Chang, Shuyuan Yang +1Multimodal Token Compression Multimodal Instruction Tuning
School of Artificial Intelligence, Xidian University, Xi’an, China

Mar 9, 2026J/K move · Enter open · S save

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals

Junyu Shen, Zhendong She, Chenghanyu Zhang, Yuchuang Sun+12

Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications+6

Multimodal Large Language Models Multimodal Pretraining Noise Robustness Multimodal Model Evaluation

Open paper AskPDF arXiv

The paradigm of Multimodal Large Language Models (MLLMs) offers a promising blueprint for advancing the electromagnetic (EM) domain. However, prevailing approaches often deviate from the native MLLM paradigm, instead using task-specific or pipelined architectures that lead to fundamental limitations in model performance and generalization. Fully realizing the MLLM potential in EM domain requires overcoming three main challenges: (1) Data. The scarcity of high-quality datasets with paired EM signals and descriptive text annotations used for MLLMs pre-training; (2) Benchmark. The absence of comprehensive benchmarks to systematically evaluate and compare the performance of models on EM signal-to-text tasks; (3) Model. A critical fragility in low Signal-to-Noise Ratio (SNR) environments, where critical signal features can be obscured, leading to significant performance degradation. To address these challenges, we introduce a tripartite contribution to establish a foundation for MLLMs in the EM domain. First, to overcome data scarcity, we construct and release EM-100k, a large-scale dataset comprising over 100,000 EM signal-text pairs. Second, to enable rigorous and standardized evaluation, we propose EM-Bench, the most comprehensive benchmark featuring diverse downstream tasks spanning from perception to reasoning. Finally, to tackle the core modeling challenge, we present MERLIN, a novel training framework designed not only to align low-level signal representations with high-level semantic text, but also to explicitly enhance model robustness and performance in challenging low-SNR environments. Comprehensive experiments validate our method, showing that MERLIN is state-of-the-art in the EM-Bench and exhibits remarkable robustness in low-SNR settings.

Abstract

Wireless Foundation Models

Explore similar work

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals

EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals

Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization