We study a problem of signal separation: estimating a signal of interest (SOI) contaminated by an unknown non-Gaussian background/interference. Given the training data consisting of examples of SOI and interference, we show how to build a fully data-driven signal separator. To that end we learn a good discrete tokenizer for SOI and then train an end-to-end transformer on a cross-entropy loss. Training with a cross-entropy shows substantial improvements over the conventional mean-squared error (MSE). Our tokenizer is a modification of Google's SoundStream, which incorporates additional transformer layers and switches from VQVAE to finite-scalar quantization (FSQ). Across real and synthetic mixtures from the MIT RF Challenge dataset, our method achieves competitive performance, including a 122x reduction in bit-error rate (BER) over prior state-of-the-art techniques for separating a QPSK signal from 5G interference. The learned representation adapts to the interference type without side information and shows zero-shot generalization to unseen mixtures at inference time, underscoring its potential beyond RF. Although we instantiate our approach on radio-frequency mixtures, we expect the same architecture to apply to gravitational-wave data (e.g., LIGO strain) and other scientific sensing problems that require data-driven modeling of background and noise.
Figures & tables
Figure 1: The schematic overview of the proposed architecture
Figure 2: Overview of the SOI Tokenizer architecture. The main differences from the SoundStream architecture are: (i) additional Transformer blocks after downsampling and before upsampling; (ii) the use of FSQ instead of RVQ for discretization; and (iii) the omission of the discriminator network.
Figure 3: Source separation performance for separating mixtures with CommSignal5G and EMISignal interference using different methods. In both cases our proposed architecture is highly competitive and surpasses most baselines across a wide range of SIRs.
MSE (dB)
BER ( log10 )
Method or team
CS2
CS3
CS5G
EMI
CS2
CS3
CS5G
EMI
RF transformer ( ours )
-27.22
-6.18
-46.32
-33.01
-2.92
-0.83
- 4.91
-3.52
RF transformer multi ( ours )
-28.71
-6.22
-5.54
-27.72
-3.07
-0.92
-0.86
-3.05
Wavenet
-24.14
-
-39.43
-28.92
-3.05
-
-4.23
-3.33
KU-TII
-
-6.04
-30.17
-29.07
-
-1.10
-3.41
-3.33
OneInAMillion
-23.54
-4.41
-37.11
-28.92
-3.03
-0.86
-3.94
-2.97
Table 1: Performance of source separation methods on different interference types
Figure 4: The BER comparison of matched filter and Multi-type RF transformer model on mixture dataset
Figure 5: Ablations studies evaluating key design choices for the SOI tokenizer. In (a), we show that combining FSQ with four transformer blocks yields the lowest validation loss among all configurations. In (b), we observe that tokenizer performance improves with longer input signal lengths.
Figure 6: Denoising performance of a continuous transformer on synchronized QPSK signals corrupted by mixtures of CommSignal2 and Gaussian noise. Although trained without any Gaussian corruption, the model generalizes well and outperforms matched filtering and LMMSE at high SINRs.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Mixture y=s+b )
Tokenization of s
CE Target
RF communications
SOI + co-channel intf.
Constellation / codebook (FSQ)
Next-token
Gravitational waves (LIGO)
Chirp hθ + noise
Quantized TF atoms / phase-increment
Template / token
LHC pileup mitigation
Leading-vertex + pileup
Per-particle/track (keep / PU / vertex)
Per-token labels
Seismology (phase picking)
P/S phases + ambient
{P,S,none} on time grid
Framewise
Neural spike sorting
Spikes + overlap/noise
(unit_id, binned t ) tokens
Event sequence
Radio astronomy (FRBs)
Dispersed transient + RFI/sky
Dedispersed path / track tokens
Path / track
Appendix
Table 2: Possible domains for our method. We list the field name; how its data fits our y=s+b setup; the tokenization of the SOI s ; and the target for cross-entropy training of an auto-regressive model.
Parameter
CommSignal2
CommSignal3
CommSignal5G
EMISignal
Train signal length
40960
2560
Encoder layers
14
Decoder layers
14
Embedding dimension
768
Attention heads
12
Window size
16
Appendix
Table 3: Training setup for reproducing results.
Figure 7: Source separation performance for separating mixtures with CommSignal2 and CommSignal3 interference using different methods.
Figure 8: Effect of window stride on downstream source separation with 5G interference.
Interference
Dataset Type
Description
# Recordings
Recording Length
CommSignal2
Recorded
Unknown
100
43560
CommSignal3
Recorded
Unknown
139
260000
CommSignal5G
Synthetic
5G OFDM signal
149
230000
EMISignal
-
Microwave Emission
530
230000
Appendix
Table 4: Summary of the interference datasets used in our experiments.
Figure 9: Real waveform and spectrogram of a sample from QPSK dataset
Figure 10: Real waveform and spectrogram of a sample from interference datasets
MSE (dB)
log10(BER)
INR
∞
10
7
3
−∞
∞
10
7
3
−∞
EMI
Single-type RF transformer
-33.01
-23.17
-20.58
-16.54
-11.66
-3.52
-2.54
-2.35
-1.93
-1.43
Multi-type RF transformer
-27.72
-17.79
-15.90
-13.80
-10.20
-3.05
-2.47
-2.13
-1.95
-1.44
Matched Filter
—
—
—
—
—
—
—
—
—
-1.43
CS2
Appendix
Table 5: Zero-shot Gaussian generalization across datasets. Columns are INR values; entries are averages over SINR.
Figure 11: LMMSE constellation plots at 0 dB, -3 dB, and -15 dB SINR. Despite large variance at low SINR, the decoded symbols tend to remain within the correct QPSK quadrant.
Figure 12: Transformer constellation plots at 0 dB, -3 dB, and -15 dB SINR. Outputs remain tightly clustered near ideal QPSK symbols even in highly noisy settings.
Figure 13: LMMSE real-valued waveform outputs at 0 dB, -3 dB, and -15 dB SINR, overlaid against ground truth. Note the increasing distortion and phase error at lower SINRs.
Figure 14: Raw transformer waveform outputs at 0 dB, -3 dB, and -15 dB SINR. Unlike the LMMSE results, the transformer is able to preserve signal structure even under high interference.
Figure 15: Real parts of remodulated LMMSE output vs. ground truth at 0 dB, -3 dB, and -15 dB. Despite correct bit decisions, amplitudes show mismatch at low SINRs.
Figure 16: Real parts of remodulated transformer output vs. ground truth at 0 dB, -3 dB, and -15 dB. Transformer waveform closely matches ground truth at all SINRs.
Attribute
RF Transformer
WaveNet
Parameters
240M
4M
Signal length
2560
40960
Batch size
130
8
Batch latency
0.29s
0.07s
Step count
256,000
375,000
Training time
20.5 h
6.5 h
Appendix
Table 6: Comparison of RF Transformer and WaveNet training characteristics.
Figure 17: Comparison of tokenizer performance for the signal length of N=2560 . To achieve comparable performance in the causal setting, the number of Transformer blocks was increased from 4 to 8 .
Figure 18: Performance of the original (non-causal) RF transformer, the proposed causal variant and (non-causal) WaveNet in separating QPSK and CommSignal5G interference. The proposed causal variant, though preliminary, shows competitive performance especially at high SIR.
Single-channel radio-frequency (RF) signal separation is important for interference mitigation when a signal of interest overlaps with a co-channel interferer in time and frequency. Classical matched filtering and LMMSE receivers are limited under structured non-Gaussian interference, while diffusion and score-based methods often require iterative sampling or optimization. We propose FlowSep-RF, a rectified flow matching framework that learns a clean-endpoint transport from the received mixture to the signal of interest. The model combines time-domain IQ samples with frequency-domain features and recovers the separated waveform using one endpoint velocity evaluation. We evaluate FlowSep-RF on QPSK mixtures with OFDM-BPSK, OFDM-QPSK, and RFChallenge CommSignal2 interference against classical, discriminative, and generative baselines. FlowSep-RF achieves the lowest average BER and MSE among evaluated baselines across all three settings with one-step inference.
Mohamed Kamel, Tamer Nadeem
Department of Computer Science, Virginia Commonwealth University, Richmond, VA, USA
AI-enhanced interference rejection in radio frequency (RF) transmissions has recently attracted interest because deep learning approaches trained on both the signal of interest (SOI) and the signal mixture (SOI plus interference) can outperform traditional approaches which only consider the SOI. The goal is to detect, demodulate, and decode signals over a range of signal-to-interference-plus-noise (SINR) levels without having a detailed, design-level knowledge of the interfering signal or the propagation conditions. Our present AI interference suppression results are based on Autoregressive Transformer Decoder models which exhibit orders of magnitude faster throughput at inference time than WaveNet models developed in earlier work. As a specific example, we investigate an analog FM "Walkie Talkie" radio signal of interest in the presence of an Orthogonal Frequency-Division Multiplexing (OFDM) interferer. This type of interferer is near-ubiquitous in the current RF landscape. Our results clearly show the benefits of transformer-based interference mitigation in tactical settings. We show that unintelligible transmissions become intelligible via metrics such as Perceptual Evaluation of Speech Quality (PESQ), while overall latency is kept to a minimum using readily available lightweight GPUs such as a Jetson AGX Orin. We believe these same techniques can also be applied to a broader set of national security scenarios, as well as having commercial applications.
AI-based structured interference rejection has grown more popular because deep learning approaches can outperform traditional methods by jointly considering the signal of interest (SOI) and the signal mixture (SOI plus interference). This work builds on a previous AI-enabled approach utilizing autoregressive transformer-based models by adding a Finite Scalar Quantization (FSQ) tokenizer layer which aims to improve the interference rejection performance while keeping overall latency to a minimum. Additionally, we experiment with other inference optimization techniques with the goal of speeding up inference without much accuracy loss. We explore this space with an experiment where the SOI is a digitally modulated radio frequency (RF) signal and the structured interference is a digital television signal, an extremely common type of Orthogonal Frequency-Division Multiplexing (OFDM) transmission. Our results achieve low latency and increased interference rejection over traditional techniques and prior work with other AI-enabled methods. We demonstrate the benefits of the AI-enabled approaches via audio metrics such as Perceptual Evaluation of Speech Quality (PESQ). Additionally, we explore a variety of applications and detail how our interference rejection algorithm may be used in operationally-relevant scenarios.