EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals

Mingxu Zhang; Ying Sun; Yuhan Li; Yang Ji; Dazhong Shen; Ke Zhang; Shan Huang

cs.AIAug 25, 2026

EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals

Authors: Mingxu Zhang, Ying Sun, Yuhan Li, Yang Ji, Dazhong Shen, Ke Zhang, Shan Huang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The 63rd Research Institute, National University of Defense Technology · Nanjing University of Aeronautics and Astronautics

Abstract

Large language models (LLMs) are increasingly used as code agents for scientific and engineering analysis, but their ability to analyze raw physical-layer measurements remains untested. We introduce EMRB (Electromagnetic Reasoning Benchmark), which evaluates whether LLMs can analyze raw I/Q data by writing and running code. EMRB contains 200 problems across five difficulty levels and 27 question types, from signal detection to OFDM design, generated from 11 signal types with verified ground truth. Unlike benchmarks built on preprocessed features or structured tables, EMRB provides only the raw capture; the quantities each question refers to must first be discovered through code. We evaluate 14 LLMs spanning proprietary, open-weight, and reasoning-oriented families. Scores range from 24.1% to 78.9%, with the mean dropping from 84.9% on basic measurement to 21.2% on system design. We also propose ReconPilot, a structured method that separates signal reconnaissance, targeted analysis, and self-verification. Across three backbones, ReconPilot raises the overall score by 3.8 to 17.6 points and improves 13 of 15 backbone-level combinations tested. All data and code are publicly released in blue{our GitHub repository}.

Figures & tables

Figure 1. Overview of EMRB. (A) Benchmark input: raw I/Q capture paired with an engineering question (200 problems, 11 signal types, verified ground truth). (B) Five difficulty levels from basic measurement to system design. (C) Code-execution evaluation: the model writes and runs Python over up to 15 turns, scored by deterministic verifiers.

Figure 2. ReconPilot three-stage architecture. Stage 1 turns raw I/Q into a bounded but deliberately unresolved spectral region map; Stage 2 guides targeted LLM analysis; Stage 3 verifies consistency before submission.

Figure 3. Overall EMRB results across 14 LLMs. (a) Average scores over 200 problems. (b) Code calls per problem versus accuracy. Colors and icons are shared between panels.

Figure 4. EMRB scores for all 27 question types across 14 models, expressed as a percentage of the points available for that type. Models are ordered by overall score; the bottom row gives the mean over models for each type.

Level	Oracle	Union	Best	Reference
	replay	of 14	model	solver
L1	100.0	99.8	95.8	–
L2	100.0	99.4	88.7	–
L3	100.0	99.6	93.2	–
L4	100.0	98.8	84.8	–
L5	100.0	53.2	39.8	75.7

Table 1. EMRB headroom is reachable. Oracle replay: ground truth scored by the verifier. Union of 14: best per-criterion result across models. Reference solver: classical DSP (L5 only).

Backbone	Method	L1	L2	L3	L4	L5	Avg.
DeepSeek V4 Flash	Free form	91.2	65.1	67.5	41.8	16.5	56.4
	ReconPilot	88.2	86.3	84.9	67.8	24.8	70.4
	Improvement	$-$ 3.0	+21.2	+17.4	+26.0	+8.3	+14.0
Gemini 3.5 Flash	Free form	91.9	46.3	72.6	47.2	28.3	57.2
	ReconPilot	94.9	89.2	84.3	72.2	33.1	74.8
	Improvement	+3.0	+42.9	+11.7	+25.0	+4.8	+17.6

Table 2. Controlled comparison between free form code execution and ReconPilot across three backbones. Bold marks the higher score in each free form/ReconPilot pair.

Stage 1	Stage 3	L1	L2	L3	L4	L5	Avg.
–	–	91.9	46.3	72.6	47.2	28.3	57.2
✓	–	94.8	87.8	72.6	77.9	11.0	68.8
–	✓	94.7	83.1	76.4	61.3	35.2	70.2
✓	✓	94.9	89.2	84.3	72.2	33.1	74.8

Table 3. Stage 1

\times

Stage 3 grid on Gemini 3.5 Flash, with Stage 2 held fixed as the unmodified free form loop. Bold marks the best in each column.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Lvl	Theme	Question Types	Emitters	Qs
L1	Basic Meas.	Detection, Power, Sampling, Noise Floor, Classification	2–3	5
L2	Signal Proc.	FFT/Window, Bandwidth, Autocorrelation, STFT, Energy/Power	3–4	5
L3	Comm. Theory	Bitrate, Eb/N0, PAPR, ADC, DDC	3–4	5
L4	Multi-Signal	Symbol Rate/Mod. Order, FM, AM, Chirp/Radar, Burst, Spectral Gap, Interference/SIR, OFDM, Capacity	3–5	5
L5	System Design	Spectrum Survey, Radar/Coexist, OFDM Design	6–7	3

Appendix

Table 4. EMRB difficulty levels with question types. Each level tests distinct skills with zero question-type overlap across levels. Each L1–L3 and L5 problem instantiates all of its level’s question types; each L4 problem draws its five sub-questions from the level’s nine types.

Panel of Figure 5 — Figure 5. Complete L1 problem prompt as delivered to the model, reproduced verbatim from EMRB_L1_5000.json with no abridgement, including the hint formulas the prompt supplies. The model receives this text together with an execute_python tool and the raw I/Q file.

Ground Truth	Accepted Aliases
QPSK	4-QAM, 4QAM, Quadrature PSK, 4-PSK
16QAM	16-QAM, QAM-16, QAM16
FM	Frequency Modulation, FM modulation
AM-DSB	AM, DSB-AM, Double-Sideband AM
OFDM	Orthogonal FDM
Chirp	LFM, Linear FM, Swept frequency

Appendix

Table 10

Model	L1	L2	L3	L4	L5	Avg.
GPT-5.5	95.8	86.0	93.2	84.8	34.5	78.9
Grok-4.5	93.4	86.4	92.3	77.2	39.8	77.8
Claude Opus 4.7	84.0	88.7	88.3	70.0	20.5	70.3
GLM-5.2	92.9	83.8	73.7	58.6	20.0	65.8
Claude Sonnet 4.6	91.5	82.6	80.2	58.1	15.8	65.6
Gemini 3.1 Pro	91.7	79.6	57.8	63.4	34.8	65.5

Appendix

Table 5. Complete EMRB results for all 14 evaluated LLMs. The best score in each column is bold. Averages are computed from unrounded per-level scores, so they can differ in the last digit from the mean of the rounded values shown.

Figure 6. Score distributions across the five EMRB levels. Each gray point represents one model, the black line is the median, and the orange diamond is the mean.

Panel of Figure 7 — Figure 7. L1 response comparison on EMRB_L1_5000 (Q1–Q2 excerpts). GPT-5.5 executes multiple rounds of calibrated signal-processing code; Llama 3.3 70B produces superficially plausible answers from the prompt hint without code-verified measurements.

Panel of Figure 8 — Figure 8. L5 system-design response excerpt. EMRB L5 requires chained reasoning where the signal inventory feeds interference analysis, which feeds filter and system design. The verifier checks each step against ground truth and physical constraints.

Panel of Figure 9 — Figure 9. Gemini 3.5 Flash on EMRB_L3_4037. Free form execution guesses a root symbol rate after two code calls that returned no measurement, and the $2.9\times$ error propagates through Q1’s rate quantities, Q2’s noise and margin chain, and all of Q5. ReconPilot derives the root quantity from the reconnaissance region map and reaches full credit on Q1, Q2 and Q5.

Panel of Figure 10 — Figure 10. GPT-5.5 on EMRB_L5_2005. Under prerequisite-gated L5 scoring, a four-source inventory zeroes 43 points of otherwise competent downstream work. ReconPilot recovers the fifth source by following the region map’s audit instruction and the sixth through Stage 3’s dependency check, which is the division of labor the stage ablation in Section 6.6 predicts.

Benchmark	Input Format	Task Description	Output Format
EMRB	Raw full-length I/Q files	Open-ended EM signal analysis	Text answers
EM-Bench	Encoded signal-text pairs	Signal question answering	Text answers
RadioML	Short synthetic I/Q snippets	Modulation classification	Modulation labels
WiSig	WiFi I/Q packet captures	RF fingerprinting	Device IDs
WASD	Wireless measurements or spectrograms	Spectrum anomaly detection	Anomaly classes or regions
DeepSense 6G	Multimodal wireless data	Beam prediction and sensing tasks	Beams or locations

Appendix

Table 6. Comparison with representative EM/wireless and data-analysis benchmarks.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals
Mar 9, 2026Junyu Shen, Zhendong She, Chenghanyu Zhang +13Multimodal Large Language Models Multimodal Pretraining
Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications +6
PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision
Mar 30, 2026Zehua Han, Jing Xiao, Yiqi Duan +13Wireless Foundation Models Multimodal Foundation Models
Beihang University · Civil Aviation University of China · Beijing Institute of Technology +10
CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
Oct 16, 2026Jin Gene Wong, Anjney Midha, Joseph Mathew Tennyson +3LLM Evaluation Scientific Reasoning in Language Models
Physics Department and Stanford Laboratory for Intelligent Matter, Stanford University, Stanford, CA 94305 · Applied Physics Department, Stanford University, Stanford, CA 94305 · Electrical Engineering & Computer Sciences, University of California, Berkeley, Berkeley, CA 94720 +1

Mar 9, 2026J/K move · Enter open · S save

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals

Junyu Shen, Zhendong She, Chenghanyu Zhang, Yuchuang Sun+12

Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications+6

Multimodal Large Language Models Multimodal Pretraining Noise Robustness Multimodal Model Evaluation

Open paper AskPDF arXiv

The paradigm of Multimodal Large Language Models (MLLMs) offers a promising blueprint for advancing the electromagnetic (EM) domain. However, prevailing approaches often deviate from the native MLLM paradigm, instead using task-specific or pipelined architectures that lead to fundamental limitations in model performance and generalization. Fully realizing the MLLM potential in EM domain requires overcoming three main challenges: (1) Data. The scarcity of high-quality datasets with paired EM signals and descriptive text annotations used for MLLMs pre-training; (2) Benchmark. The absence of comprehensive benchmarks to systematically evaluate and compare the performance of models on EM signal-to-text tasks; (3) Model. A critical fragility in low Signal-to-Noise Ratio (SNR) environments, where critical signal features can be obscured, leading to significant performance degradation. To address these challenges, we introduce a tripartite contribution to establish a foundation for MLLMs in the EM domain. First, to overcome data scarcity, we construct and release EM-100k, a large-scale dataset comprising over 100,000 EM signal-text pairs. Second, to enable rigorous and standardized evaluation, we propose EM-Bench, the most comprehensive benchmark featuring diverse downstream tasks spanning from perception to reasoning. Finally, to tackle the core modeling challenge, we present MERLIN, a novel training framework designed not only to align low-level signal representations with high-level semantic text, but also to explicitly enhance model robustness and performance in challenging low-SNR environments. Comprehensive experiments validate our method, showing that MERLIN is state-of-the-art in the EM-Bench and exhibits remarkable robustness in low-SNR settings.

Abstract

LLM Evaluation

Explore similar work

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals

PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision

CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?