Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization

Lei Zhai; Zhihao Chang; Shuyuan Yang; Zhixi Feng

cs.AIOct 8, 2026

Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization

Authors: Lei Zhai, Zhihao Chang, Shuyuan Yang, Zhixi Feng

Organizations: School of Artificial Intelligence, Xidian University, Xi’an, China

Abstract

Electromagnetic spectrum monitoring increasingly requires flexible analysis beyond task-specific recognition and detection. Multimodal large language models offer a unified interface, but extending vision-language models (VLMs) to raw I/Q signals requires tokenization that balances fidelity against a strict budget. For signals, dense encoding causes token costs to grow with observation length, whereas fixed-resolution compression may discard short-duration or localized signal evidence. Thus, we propose \textbf{BATok}, a budget-adaptive signal tokenizer that adjusts token capacity to the input length while allocating that capacity according to the signal content. BATok constructs candidate representations from signal-derived features using lightweight multi-resolution branches, then combines a local energy prior with learnable queries to resample these representations into compact signal tokens. The number of tokens adapts to the input length while remaining strictly bounded. The resulting tokens are projected into the language embedding space of VLMs. We further introduce \textbf{EMSpec-Instruct}, a multimodal instruction dataset aligning raw I/Q signals, waterfall images, and language supervision for modulation recognition, structured detection, and language-conditioned signal grounding. Experiments show that BATok learns effective signal representations and achieves competitive performance across all tasks.

Figures & tables

Figure 1: Motivation for unified instruction-conditioned spectrum understanding. Task-specific models require manual redesign, while detector–LLM cascades inherit upstream errors. Our framework couples signal tokenization with a VLM through a unified interface.

Figure 2: Overview of BATok and its integration with a VLM. Variable-length I/Q signals are augmented with signal-derived features, encoded at multiple temporal resolutions, assigned a bounded length-adaptive token budget, and resampled using learned relevance and an energy prior. The resulting multi-resolution signal tokens are aggregated, projected into the language embedding space, and jointly processed with image and text tokens for unified spectrum understanding.

Figure 3: Construction and statistics of EMSpec-Instruct. Structured observation- and instance-level metadata are converted into metadata-grounded instructions. The bottom panels summarize the task composition of the training and test sets and the distribution of signal lengths.

		Modulation Recognition			Structured Detection			Signal Grounding
Backbone	BATok	OA (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Parse (%) $\uparrow$	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$	Parse (%) $\uparrow$	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$	Parse (%) $\uparrow$	Empty Acc. (%) $\uparrow$
Qwen3-VL-2B		33.90	35.38	100.00	62.25	94.66	98.08	82.52	94.75	99.99	93.19
Qwen3-VL-2B	✓	40.91	41.26	86.02	64.27	95.20	98.29	83.00	95.24	99.96	92.43
LLaVA-1.5-7B		25.58	26.26	100.00	45.99	88.72	99.43	66.94	88.81	99.95	91.92
LLaVA-1.5-7B	✓	44.14	43.20	94.95	47.80	89.40	97.17	68.27	89.39	99.87	91.23

Table 1: Unified spectrum-understanding results on EMSpec-Instruct. The original VLMs receive image–text inputs, whereas their BATok-enhanced counterparts receive aligned image–signal–text inputs. Parse is the proportion of outputs satisfying the required task-specific format, and Empty Acc. is the accuracy of returning an empty set for unmatched grounding queries. Within each backbone, the better value between the original VLM and its BATok-enhanced counterpart is bolded.

Panel of Figure 4 — Figure 4: Visualization of detection and grounding results. Detection question: “Enumerate every transmission using aligned box and class arrays.” Ground truth: $[141,424,846,924]$ (WiFi 20 MHz 64QAM) and $[145,443,842,943]$ (WiFi 20 MHz QPSK). Grounding question: “Return boxes of the longest-duration transmissions.” Ground truth: $[2,956,887,969]$ .

Method	Acc. (%) $\uparrow$	Macro-F1 (%) $\uparrow$
MAMC ( Zhang et al. 2024 )	43.20	42.31
SiT ( Zhai et al. 2025 )	53.56	53.36
DNCNet ( Du et al. 2022 )	58.18	56.89
BATok	58.64	58.54

Table 2: Standalone raw-I/Q modulation-recognition results. Each average is the unweighted mean across the two primary benchmarks, HisarMod2019 and EMSpec-Sim . Best and second-best averages are bolded and underlined, respectively. More results are provided in Supplementary Section S4.1.

Group	Variant	Acc. (%) $\uparrow$	Macro-F1 (%) $\uparrow$
Token budget	Fixed $K=32$	57.68	57.53
	Fixed $K=48$	57.97	57.60
	Linear-high	58.70	58.48
	Piecewise $K(T)^{\dagger}$	58.64	58.54
Input features	I/Q only	56.37	56.30
Input features	I/Q + $A$ + $P$ + $\Delta\phi$ $\dagger$	58.64	58.54

Table 3: Ablation study of BATok in standalone modulation recognition. Accuracy and Macro-F1 are percentages averaged over HisarMod2019 and EMSpec-Sim ;

\dagger

denotes the full BATok configuration within each ablation group.

Branch	Kernel (samples)	Stride (samples)	Approximate length (samples)
Small	7	4	$\lceil T/4\rceil$
Mid	31	16	$\lceil T/16\rceil$
Large	127	64	$\lceil T/64\rceil$

Table S1: Temporal-branch configuration used by BATok. The output length is approximate because convolution padding and the valid-length mask are handled separately.

Valid input length $T$ (samples)	Total budget $K(T)$ (tokens)
$T\leq 2^{12}$	32
$2^{12}<T\leq 2^{15}$	48
$2^{15}<T\leq 2^{16}$	64
$2^{16}<T\leq 2^{18}$	96
$2^{18}<T\leq 2^{20}$	128
$2^{20}<T\leq 2^{22}$	192

Table S2: BATok token schedule.

Setting	Qwen3-VL-2B	LLaVA-1.5-7B
LLM adaptation	LoRA ( $r=32$ )	LoRA ( $r=8$ )
Vision tower	Trainable	Trainable
Native multimodal projector	Trainable	Trainable
BATok	Trainable	Trainable
Signal projector	Trainable	Trainable
Signal position handling	Extended M-RoPE	Learned 1D embedding

Table S3: Backbone-specific VLM integration settings. The language models are adapted with LoRA, while BATok and its projector are trained and saved as complete modules.

Source	Task stream	Category coverage	Train (observations)	Test (observations)
HisarMod-Sub	Recognition	26 classes: PSK 6 , QAM 7 , FSK 4 , PAM 3 , and six analog AM/FM/PM variants	44,000	22,000
EMSpec-Sim-Sub	Recognition	34 classes: generic digital/analog modulations, WiFi 6 , BLE LE1M/LE2M, LoRa 125/250/500 kHz, Zigbee, and SRRC-shaped QPSK/16QAM	22,000	11,000
SpaceNet	Det./ground.	11 wireless classes: WiFi 6 , BLE LE1M/LE2M, LoRa 250 kHz, Zigbee, and SRRC-shaped QPSK	7,500	2,500
Multi-signal simulation	Det./ground.	14 classes: the SpaceNet vocabulary augmented with AM, FM, and SRRC-shaped 16QAM	20,000	7,500
Total			93,500	43,000

Table S4: Source observations and category coverage used to construct EMSpec-Instruct. Counts refer to electromagnetic observations before instruction expansion. WiFi 6 denotes the six combinations of 20/40-MHz bandwidth and QPSK/16QAM/64QAM modulation.

Task	Example instruction	Target
Recognition	Classify the modulation using both modalities.	<c>["8ASK"]</c>
Detection	Detect every signal and return all boxes and classes.	<d><b>[[339,166,982,833]]</b> <c>["16QAM"]</c></d>
Grounding	Find regions that overlap 2419.589–2448.466 MHz.	<b>[[339,166,982,833],...]</b>
Negative grounding	Find all BLE LE2M regions in this scene.	<b>[]</b>

Table S5: Representative task realizations and target formats. Scene context is abbreviated in the table but is present in the actual time/frequency-conditioned prompts.

Task	Train (records)	Train (%)	Test (records)	Test (%)
Modulation recognition	120,000	30.00	33,000	48.61
Structured detection	27,500	6.88	10,000	14.73
Signal grounding	252,500	63.12	24,883	36.66
Total	400,000	100.00	67,883	100.00

Table S6: Final task composition. The validation set is a balanced 3,000-example subset sampled from the held-out task files and is not included in the test total.

Grounding subtype	Train (records)	Test (records)
Class	27,500	1,631
Class fallback (no overlap)	–	313
Time	27,500	1,656
Frequency	27,500	1,640
Class–time–frequency	27,500	1,656
Longest duration	27,500	1,638

Table S7: Signal-grounding subtype counts. A dash denotes a subtype that is not selected for that partition.

		Mod. Rec.			Detection			Grounding
Backbone	Test Input	OA (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Parse (%) $\uparrow$	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$	Parse (%) $\uparrow$	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$	Parse (%) $\uparrow$	Empty Acc. (%) $\uparrow$
Qwen3-VL-2B + BATok	Image + Text	16.52	11.93	100.00	64.12	95.20	97.88	81.78	95.23	99.94	92.04
Qwen3-VL-2B + BATok	Signal + Text	40.36	40.33	88.96	3.64	72.48	99.44	13.36	74.37	99.90	89.39
Qwen3-VL-2B + BATok	Image + Signal + Text	40.91	41.26	86.02	64.27	95.20	98.29	83.00	95.24	99.96	92.43
LLaVA-1.5-7B + BATok	Image + Text	3.08	2.54	99.35	47.69	89.28	98.19	67.75	89.29	99.88	85.15
LLaVA-1.5-7B + BATok	Signal + Text	43.68	43.02	94.90	3.06	74.68	98.80	12.85	75.05	99.81	89.11

Table S8: BATok-enhanced VLM performance by input configuration. Values are percentages, and all input configurations for a backbone use the same trained checkpoint. Parse denotes successful task-format parsing, and Empty Acc. is reported for unmatched grounding queries.

Training tasks			Grounding
Mod. Rec.	Detection	Grounding	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$
		✓	81.97	95.15
✓		✓	81.99	95.24
	✓	✓	81.47	95.06
✓	✓	✓	83.00	95.24

Table S9: Grounding performance under different instruction-tuning task compositions. All variants use Qwen3-VL-2B+BATok and are evaluated on the same grounding test set.

Method	Input	Acc. (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Params (M) $\downarrow$	Latency (ms/sample) $\downarrow$
ResNet-50	Waterfall	32.43	34.33	23.610	4.105
ViT	Waterfall	31.06	33.13	85.837	3.424
MAMC ( Zhang et al. 2024 )	I/Q	29.62	27.64	0.009	1.466
SiT ( Zhai et al. 2025 )	I/Q	41.56	41.43	0.654	0.967
DNCNet ( Du et al. 2022 )	I/Q	44.17	42.80	3.870	1.860
BATok	I/Q	49.36	47.19	1.047	5.443

Table S10: Waterfall-image and raw-I/Q modulation recognition on the aligned EMSpec-Instruct subset, defined as the union of HisarMod-Sub and EMSpec-Sim-Sub . Latency is measured in milliseconds per sample with batch size 1 on one NVIDIA GeForce RTX 4090.

Method	mIoU (ratio) $\uparrow$	[email protected] (ratio) $\uparrow$	[email protected] (ratio) $\uparrow$	[email protected] (ratio) $\uparrow$
YOLO26n	0.9291	0.6466	0.6288	0.6654
YOLO26l	0.9475	0.6865	0.6514	0.7255
YOLO26x	0.9517	0.6925	0.6577	0.7311
DETR	0.8597	0.4127	0.2918	0.7044

Table S11: Task-specific detection results on the EMSpec-Instruct detection test set. Best and second-best values are bolded and underlined, respectively. These systems use dedicated detection heads.

Backbone	Detection measure	Image + Text (%)	Image + Signal + Text (%)
Qwen3-VL-2B	Class-aware [email protected]	62.25	64.27
	Class-agnostic [email protected]	90.74	90.68
	Conditional class accuracy	68.57	70.72
	Class-agnostic matched-box mIoU	95.02	95.48
	Class-agnostic [email protected]	82.87	82.66
	Class-agnostic [email protected]:0.95	71.23	72.19

Table S12: Decomposition of structured-detection gains into class semantics and box geometry. Class-agnostic metrics ignore the predicted class during IoU matching; conditional class accuracy is computed only over boxes matched at IoU

\geq 0.5

	Qwen3-VL-2B + BATok				LLaVA-1.5-7B + BATok
Query Type	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$	Empty Acc. (%) $\uparrow$	Parse (%) $\uparrow$	[email protected] (%) $\uparrow$	mIoU (%) $\uparrow$	Empty Acc. (%) $\uparrow$	Parse (%) $\uparrow$
Modulation-conditioned	83.51	94.85	–	99.94	66.59	87.95	–	100.00
Time-conditioned	90.28	95.22	–	99.88	74.07	89.24	–	99.58
Frequency-conditioned	87.02	94.82	–	100.00	74.16	88.91	–	99.88
Compositional	89.46	94.50	–	99.94	79.49	88.80	–	99.88
Overlap / relational	78.00	95.58	–	99.92	66.13	90.01	–	99.87

Table S13: Language-conditioned grounding results by query type for the image–signal–text test configuration. Values are percentages; a dash marks an inapplicable metric, and Parse denotes successful required-format parsing. Overlap/relational pools duration, bandwidth, brightness, and overlap queries, while unmatched pools negative-class and class-fallback-no-overlap queries.

Method	Acc. (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Acc. (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Params (M) $\downarrow$	Latency (ms/sample) $\downarrow$
	HisarMod2019		EMSpec-Sim
MAMC ( Zhang et al. 2024 )	55.22	53.23	31.18	31.39	0.008	1.470
SiT ( Zhai et al. 2025 )	71.34	71.30	35.78	35.41	0.649	0.978
DNCNet ( Du et al. 2022 )	75.25	75.19	41.11	38.58	3.856	1.868
BATok	77.53	77.53	39.75	39.54	1.044	5.509

Table S14: Per-benchmark standalone raw-I/Q modulation-recognition results and classification efficiency. Best and second-best performance values within each benchmark are bolded and underlined, respectively. Latency is measured with batch size 1 on one NVIDIA GeForce RTX 4090.

Strategy	Avg. $K$ (tokens)	Max $K$ (tokens)	Avg. Acc. (%)	Avg. F1 (%)	Latency (ms/sample)
Fixed-16	16.00	16	57.91	57.62	5.358
Fixed-32	32.00	32	57.68	57.53	5.294
Fixed-48	48.00	48	57.97	57.60	5.300
Fixed-64	64.00	64	58.04	57.70	5.310
Fixed-128	128.00	128	58.08	57.90	5.469
Linear-low	32.50	48	58.60	58.34	5.469

Table S15: Unified performance–budget–latency ablation. Mean Accuracy and Macro-F1 are the unweighted means across HisarMod2019 and EMSpec-Sim . Latency is measured in milliseconds per sample on one NVIDIA GeForce RTX 4090.

\dagger

denotes the full BATok configuration. Best and second-best mean results are bolded and underlined, respectively.

Strategy	Avg. $K$ (tokens)	$T=1{,}024$ (samples) Acc./F1 (%)	$T=2{,}048$ (samples) Acc./F1 (%)	$T=4{,}096$ (samples) Acc./F1 (%)	$T=8{,}192$ (samples) Acc./F1 (%)	$T=16{,}384$ (samples) Acc./F1 (%)	$T=32{,}768$ (samples) Acc./F1 (%)
Fixed-32	32.00	31.99 /30.56	34.69/33.34	38.72 /36.74	41.28/39.43	40.39 /38.67	40.76/39.24
Fixed-39	39.00	31.73/ 30.77	36.19/ 35.39	38.10/ 37.21	41.40/39.97	38.93/37.96	41.18/ 40.70
Fixed-48	48.00	29.23/28.76	34.27/34.04	35.05/34.77	40.62/ 40.35	39.54/ 39.25	39.86/39.23
Piecewise $K(T)^{\dagger}$	39.23	31.84/29.70	36.73 /34.54	37.35/35.44	42.11 /38.93	40.39 /37.97	41.42 /38.99

Table S16: Controlled within-dataset token-budget comparison. EMSpec-Sim-VarLen is a variable-length subset of EMSpec-Sim . Each length column reports Accuracy/Macro-F1 in percent; Avg.

K

is weighted by the number of test samples, and bold marks the best result for each metric at each length.

Group	Variant	HisarMod2019 Acc./F1 (%)	EMSpec-Sim Acc./F1 (%)	Avg. Acc. (%)	Avg. F1 (%)
Input features	I/Q only	74.07/73.97	38.66/38.62	56.37	56.30
	I/Q+ $A$ + $P$ + $\Delta\phi$ $\dagger$	77.53/77.53	39.75/39.54	58.64	58.54
Temporal scales	Small	68.89/68.88	40.51/40.18	54.70	54.53
	Small+mid	72.64/72.53	39.48/39.63	56.06	56.08
	Small+mid+large $\dagger$	77.53/77.53	39.75/39.54	58.64	58.54
Resampling bias	Energy prior only	76.99/77.07	36.95/37.36	56.97	57.22

Table S17: Per-dataset component ablations for standalone BATok classification. Dataset columns report Accuracy/Macro-F1 in percent; Avg. columns are arithmetic means across HisarMod2019 and EMSpec-Sim .

\dagger

denotes the full BATok configuration, and bold marks the best cross-dataset mean within each ablation group.

GT	LLaVA	LLaVA+BATok	Qwen3-VL	Qwen3-VL+BATok

Figure S1: Detection of a broadband transmission and a narrow-band target. Question: “Detect all occupied signal regions and pair each bounding box with the matching class label.” Ground truth:

[285,500,904,1000]

(WiFi 20 MHz 64QAM) and

[346,843,958,849]

(LoRa 250 kHz). Coordinates are normalized to

[0,1000]

; the GT panel uses red boxes.

GT	LLaVA	LLaVA+BATok	Qwen3-VL	Qwen3-VL+BATok

Figure S2: Detection in a crowded heterogeneous scene. Question: “Perform exhaustive detection and respond only with the requested tags.” Ground truth:

[289,25,888,691]

(WiFi 20 MHz 64QAM),

[304,929,897,962]

(BLE LE1M),

[308,392,863,459]

(SRRC QPSK), and

[310,261,828,928]

(WiFi 20 MHz QPSK). Coordinates are normalized to

[0,1000]

; the GT panel uses red boxes.

GT	LLaVA	LLaVA+BATok	Qwen3-VL	Qwen3-VL+BATok

(a) Class-conditioned query

(b) Frequency-conditioned query

Figure S3: Grounding under class and frequency constraints. (a) Question: “Which regions correspond to the LoRa 250 kHz class?” GT:

[46,123,741,129]

. (b) Question: “Find signal regions that overlap the frequency band 2455.015597–2457.837200 MHz.” GT:

[77,0,1000,800]

and

[96,230,994,270]

. Coordinates are normalized to

[0,1000]

; GT panels use red boxes.

GT	LLaVA	LLaVA+BATok	Qwen3-VL	Qwen3-VL+BATok

(a) Time-conditioned query

(b) Overlap-conditioned query

Figure S4: Grounding under time and overlap constraints. (a) Question: “Find signal regions that overlap the time interval 1.092–10.124 ms.” GT:

[46,891,483,916]

and

[96,886,463,911]

. (b) Question: “Find the signal regions involved in time–frequency overlap.” GT:

[152,0,886,400]

and

[788,0,1000,800]

. The overlap case is deliberately included because all compared variants succeed on it. Coordinates are normalized to

[0,1000]

; GT panels use red boxes.

Explore similar work

Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals
May 30, 2026Shenghao DingWavelet Methods Multimodal Token Compression
Yet Another AI
M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models
Sep 16, 2026Chunpu Xu, Zhixuan Liang, Yuhao Zhang +6Vector Quantization Vision-Language-Action Models
The Hong Kong Polytechnic University, HongKong SAR, China · Shanghai AI Laboratory, Shanghai, China · The University of Hong Kong, HongKong SAR, China +2
LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1

May 30, 2026, eess.ASJ/K move · Enter open · S save

Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals

Shenghao Ding

Yet Another AI

Wavelet Methods Multimodal Token Compression Visual Token Compression

Open paper AskPDF arXiv

This paper studies whether audio, images, and video can share a common wavelet token schema rather than relying on separate modality-specific latent grids. It introduces a preliminary continuous-token model built around a one-level Haar DWT/IDWT frontend, a shared coefficient-token layout, optional structural metadata, lightweight modality value adapters, and a shared token-wise encoder-decoder trunk. On Speech Commands, EuroSAT RGB, and DAVIS 2017 data, a dense shared model reaches 39.92 dB audio, 29.37 dB image, and 23.93 dB video PSNR. A matched-rate sweep under continuous latent scalar budgets indicates that the visual gains are not explained solely by latent capacity, while also showing that additive metadata embeddings are not a universal source of improvement. Finally, fixed-rate energy selection provides a strong non-parametric baseline: energy_global improves average PSNR over uniform selection by 16.73 dB for audio, 16.90 dB for images, and 15.86 dB for video under compressed keep ratios. Masked sparse training reaches 34.45 dB video PSNR with 50% of dense tokens. The results support a unified wavelet token schema and sparse token interface, while stopping short of establishing a universal discrete vocabulary.

Abstract

Explore similar work

Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals

M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models