Organizations: Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications · Institute of Microelectronics of the Chinese Academy of Sciences · HKUST (Guangzhou) · Beihang University · Artificial Intelligence Institute of China Electronics Technology Group Corporation · Beijing Information Science and Technology University · National University of Defense Technology
The paradigm of Multimodal Large Language Models (MLLMs) offers a promising blueprint for advancing the electromagnetic (EM) domain. However, prevailing approaches often deviate from the native MLLM paradigm, instead using task-specific or pipelined architectures that lead to fundamental limitations in model performance and generalization. Fully realizing the MLLM potential in EM domain requires overcoming three main challenges: (1) Data. The scarcity of high-quality datasets with paired EM signals and descriptive text annotations used for MLLMs pre-training; (2) Benchmark. The absence of comprehensive benchmarks to systematically evaluate and compare the performance of models on EM signal-to-text tasks; (3) Model. A critical fragility in low Signal-to-Noise Ratio (SNR) environments, where critical signal features can be obscured, leading to significant performance degradation. To address these challenges, we introduce a tripartite contribution to establish a foundation for MLLMs in the EM domain. First, to overcome data scarcity, we construct and release EM-100k, a large-scale dataset comprising over 100,000 EM signal-text pairs. Second, to enable rigorous and standardized evaluation, we propose EM-Bench, the most comprehensive benchmark featuring diverse downstream tasks spanning from perception to reasoning. Finally, to tackle the core modeling challenge, we present MERLIN, a novel training framework designed not only to align low-level signal representations with high-level semantic text, but also to explicitly enhance model robustness and performance in challenging low-SNR environments. Comprehensive experiments validate our method, showing that MERLIN is state-of-the-art in the EM-Bench and exhibits remarkable robustness in low-SNR settings.
Figures & tables
Subset
Signal Type
Mission Type
Data Source
Number of Samples
MOD
–
14 types of modulation
Real-world
588,000
PE
LFM/Barker/Frank/Stepped/Rectangular
6 types of parameters
Simulation
7,310,600
PI
BPSK/QPSK/16-QAM/64-QAM/256-QAM
8 types of protocols
Simulation
911,895
SD
–
9 types of Jamming
Simulation
162,000
RJR
LFM/Barker/Noise/Frank/Sin/Rectangular
12 types of Jamming
Simulation
4,918,700
CJR
BPSK/QPSK/16-QAM/64-QAM/GFSK
9 types of Jamming
Simulation
4,984,875
Table 1: Composition of the large-scale data corpus used for creating EM-134K and EM-Bench. This table details the 10 subsets, totaling over 35 million signal samples, that form our foundational data pool.
Figure 1: The hierarchical evaluation framework of EM-Bench, which systematically assesses the perception and reasoning capabilities of MLLMs on electromagnetic IQ signals across 3 levels and 14 sub-tasks.
Figure 2: The two-stage construction pipeline for the EM-134K fine-tuning dataset and the EM-Bench evaluation benchmark. Stage 1 focuses on building a large-scale, expert-validated IQ data corpus. Stage 2 details the distinct generation processes: a direct, large-scale formatting for EM-134K, and a rigorously expert-validated pathway for the high-quality EM-Bench.
Figure 3: The architecture and training framework of MERLIN. (1) The baseline model architecture consists of a Signal Encoder, a Projector, and a LLM. (2) The knowledge distillation framework enhances low-SNR robustness by using a frozen high-SNR teacher model to guide a student model. (3) The Denoising Subspace Module (DSM) facilitates effective distillation by projecting noisy signal features into a clean, noise-invariant feature space.
Figure 4: Motivational analysis demonstrating that low-SNR degradation is a feature-collapse problem
Model
Perception Tasks
Reasoning Tasks
MOD
PE (Radar)
PI
CJR
RJR
SD
Avg.
Anti-CJ
Anti-RJ
CLS
RJS
BW
DC
NP
PRF
PW
Rouge-L/BLEU
Rouge/BLEU
Rouge/BLEU
Rouge/BLEU
GPT-5
28.00
12.67
0.00
52.67
14.00
6.67
30.00
19.67
26.00
42.33
23.20
0.01 / 0.00
0.00 / 0.00
0.03 / 0.00
0.01 / 0.00
Claude-4-Sonnet
30.00
24.00
38.00
52.00
25.17
31.00
29.67
25.67
30.67
44.33
32.35
0.11 / 0.00
0.10 / 0.00
0.10 / 0.00
0.12 / 0.01
DeepSeek-v3.2-exp
18.00
9.33
9.67
19.67
13.67
17.00
8.33
5.67
11.00
18.33
13.07
0.05 / 0.00
0.06 / 0.00
0.09 / 0.00
0.09 / 0.00
Qwen3-Next-80b-A3B
17.00
7.33
14.67
29.33
15.33
16.33
10.00
9.33
13.67
24.00
15.70
0.01 / 0.00
0.01 / 0.00
0.03 / 0.00
0.01 / 0.00
Table 2: Main results on EM-Bench. We compare MERLIN with leading proprietary and open-source LLMs. All baseline LLMs process EM signals in a textualized format. The best results are highlighted in bold . ”PE” denotes Parameter Estimation tasks. Its sub-tasks denote BW : Bandwidth, DC : Duty Cycle, NP : Num. Pulses, PRF : Pulse Rep. Freq, PW : Pulse Width. MERLIN achieves state-of-the-art performance across both perception and reasoning tasks.
Method
Components
Perf. (%)
LMLP
DSM
LLogits
Low-SNR
Overall
Stage-1
×
×
×
59.7
71.8
+ Stage-2
×
×
×
62.9
77.6
+ Feature KD
✓
×
×
64.2
77.9
+ DSM
✓
✓
×
64.4
78.0
MERLIN
✓
✓
✓
65.1
78.6
Table 3: Ablation of MERLIN’s distillation components. We show the impact of incrementally adding each component on model performance, starting from the pre-trained baseline.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Parameter Value
Dataset (Stage 1)
EM-100K (100K instruction pairs)
Dataset (Stage 2)
Synthetic low-SNR variants (SNR = − 20dB to 20dB)
Batch Size
256
LR: Signal Encoder (SIT)
5.0×10−5 (cosine decay)
LR: Projector, LLM
5.0×10−5
Optimizer
AdamW ( β1=0.9,β2=0.999,ϵ=1e-8 )
Appendix
Table 4: Training configuration details for MERLIN framework. Hyperparameters used during the two-stage training. SIT = Signal Integration Transformer encoder.
L1-Task
L2-Task
L3-Task
Abbr.
Annotation Format
Number of Samples
Answer Type
Perception
Signal Characterization
Modulation Classification
MOD
VQA
300
Single Choice(A/B/C/D/E)
Duty Cycle Estimation
PE.DC
VQA
300
Single Choice(A/B/C/D/E)
Pulse Repetition Frequency Estimation
PE.PRF
VQA
300
Single Choice(A/B/C/D/E)
Bandwidth Estimation
PE.BW
VQA
300
Single Choice(A/B/C/D/E)
Pulse Width Estimation
PE.PW
VQA
300
Single Choice(A/B/C/D/E)
Pulse Number Estimation
PE.NoP
VQA
300
Single Choice(A/B/C/D/E)
Appendix
Table 5: Overview of the EM-Bench evaluation framework. The benchmark features a three-level hierarchy spanning Perception and Reasoning, consisting of 14 distinct L3-Tasks. Each sub-task contains 300 samples, totaling 4,200 VQA pairs. The table lists the specific abbreviations, data quantities, and answer types (Single Choice vs. Open-ended Strategy) for each task.
Figure 5: Detailed performance comparison across 10 sub-tasks. The red line denotes MERLIN and the gray line denotes the Stage-1 baseline. MERLIN demonstrates significant improvements in complex parameter estimation tasks (e.g., Duty Cycle, Pulse Width) across all SNR levels, while maintaining robust performance in classification tasks.
Multimodal Large Language Models have demonstrated powerful cross-modal understanding and reasoning capabilities in general domains. However, in the electromagnetic (EM) domain, they still face challenges such as data scarcity and insufficient integration of domain knowledge. This paper proposes PReD, the first foundation model for the EM domain that covers the intelligent closed-loop of "perception, recognition, decision-making." We constructed a high-quality multitask EM dataset, PReD-1.3M, and an evaluation benchmark, PReD-Bench. The dataset encompasses multi-perspective representations such as raw time-domain waveform, frequency-domain spectrograms, and constellation diagrams, covering typical features of communication and radar signals. It supports a range of core tasks, including signal detection, modulation recognition, parameter estimation, protocol recognition, radio frequency fingerprint recognition, and anti-jamming decision-making. PReD adopts a multi-stage training strategy that unifies multiple tasks for EM signals. It achieves closed-loop optimization from end-to-end signal understanding to language-driven reasoning and decision-making, significantly enhancing EM domain expertise while maintaining general multimodal capabilities. Experimental results show that PReD achieves state-of-the-art performance on PReD-Bench constructed from both open-source and self-collected signal datasets. These results collectively validate the feasibility and potential of vision-aligned foundation models in advancing the understanding and reasoning of EM signals.
Zehua Han, Jing Xiao, Yiqi Duan +13
Beihang University · Civil Aviation University of China · Beijing Institute of Technology +10
Large language models (LLMs) are increasingly used as code agents for scientific and engineering analysis, but their ability to analyze raw physical-layer measurements remains untested. We introduce EMRB (Electromagnetic Reasoning Benchmark), which evaluates whether LLMs can analyze raw I/Q data by writing and running code. EMRB contains 200 problems across five difficulty levels and 27 question types, from signal detection to OFDM design, generated from 11 signal types with verified ground truth. Unlike benchmarks built on preprocessed features or structured tables, EMRB provides only the raw capture; the quantities each question refers to must first be discovered through code. We evaluate 14 LLMs spanning proprietary, open-weight, and reasoning-oriented families. Scores range from 24.1% to 78.9%, with the mean dropping from 84.9% on basic measurement to 21.2% on system design. We also propose ReconPilot, a structured method that separates signal reconnaissance, targeted analysis, and self-verification. Across three backbones, ReconPilot raises the overall score by 3.8 to 17.6 points and improves 13 of 15 backbone-level combinations tested. All data and code are publicly released in blue{our GitHub repository}.
Mingxu Zhang, Ying Sun, Yuhan Li +4
The Hong Kong University of Science and Technology (Guangzhou) · The 63rd Research Institute, National University of Defense Technology · Nanjing University of Aeronautics and Astronautics
Multimodal Emotion Recognition (MER) is critical for interpreting real-world interactions. While Multimodal Large Language Models (MLLM) have shown promise in MER, their internal decision-making mechanisms under modality conflict and missingness remain largely underexplored. In this paper, to systematically investigate these behaviors, we introduce EmoMM, a comprehensive benchmark featuring modality-aligned, conflict, and missing subsets. Through extensive evaluation, we uncover a Video Contribution Collapse (VCC) phenomenon, where MLLM marginalize video evidence due to high token redundancy and modality preferences. To address this, we propose Conflict-aware Head-level Attention Steering (CHASE), a lightweight mechanism that detects modality conflicts and performs inference-time attention steering, effectively mitigating decision bias without retraining the backbone. Experimental results demonstrate that CHASE consistently improves performance across various settings, significantly enhancing the reliability of MLLM in complex affective scenarios.
Yueru Sun, Yimeng Zhang, Haoyu Gu +5
School of Future Technology, South China University of Technology, Guangzhou, China