A Universal Raw-IQ Radar Encoder for Cross-Sensor Human Sensing

Jonas Leo Mueller; Dario Zanca; Sebastian Hoefler; Jan Petermann; Thomas Altstidl; Naga Venkata Sai Jitin Jami; Emmanuelle Salin; Bjoern M. Eskofier

cs.CVNov 3, 2026

A Universal Raw-IQ Radar Encoder for Cross-Sensor Human Sensing

Authors: Jonas Leo Mueller, Dario Zanca, Sebastian Hoefler, Jan Petermann, Thomas Altstidl, Naga Venkata Sai Jitin Jami, Emmanuelle Salin, Bjoern M. Eskofier

Organizations: Institute for AI in Medicine, LMU Hospital, Munich, Germany · Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Chair of AI-supported Therapy Decisions, Ludwig-Maximilians-Universität München, Munich, Germany ·

Abstract

Radar-based human sensing typically relies on end-to-end task-specific models trained on hand-chosen signal projections, which generalize poorly across unseen users, tasks, and hardware. We argue that the dominant signal processing choice, the micro-Doppler spectrogram, is a poor starting point for transfer because it keeps the Doppler axis, which carrier frequency rescales, and discards the angular axis, which carrier frequency leaves unchanged. We instead pretrain a convolutional beta-VAE without labels on clutter-removed raw radar measurements from four platforms spanning 60 and 77 GHz and 4 to 12 virtual antennas, aligned by a unified spectral interpolation grid. For every downstream benchmark the encoder stays frozen, and only a small supervised temporal classifier is trained on its embeddings. We compare against supervised models trained end to end on each benchmark and a pretrained radar foundation model across ten settings covering activity, gesture, sign-language, and gait recognition. When the test users and the radar are seen during training, our encoder is best on all five benchmarks. For users never seen in training, it leads on three of four benchmarks, by 4.8 to 23.2 points. On subject-disjoint gait recognition, it is best for each of the four held-out subjects and 8.3 points above the strongest baseline on average, even though its pretraining excluded every recording of this radar. Excluding the test radar from pretraining costs at most 5.3 points on any benchmark. Code is available at https://github.com/jonasmueler/UniversalRadarEncoder_WACV.

Figures & tables

Figure 1: One frozen encoder across four radar platforms. A $\beta$ -VAE is pretrained without labels on raw IQ from four sensors spanning 60 and 77 GHz and 4 to 12 virtual antennas, aligned on a common index grid by spectral interpolation. The encoder is then frozen and reused unchanged, with only a small temporal head trained per benchmark, including under simultaneous shifts in subject, sensor, and task.

Dataset	Platform	Carrier	$\boldsymbol{N_{v}}$	Benchmark	Split and evaluation role
mmRadPose	IWR6843AOPEVM	60 GHz	12	12 rehabilitation exercises	participant-disjoint, unseen-subject test
CubeLearn	IWR6843ISK	60 GHz	12	HAR, HGR, AGR	in-set comparison, out-of-set unseen-user test
CI4R-ASL	AWR1642BOOST	77 GHz	8	18 daily-activity and ASL classes	subject-mixed, published-protocol comparison
CI4R-Gait	IWR1443BOOST	77 GHz	4	11 activities and gaits	subject-mixed comparison, leave-one-subject-out test

Table 1: Sensors and downstream benchmarks. HAR, HGR, AGR, and ASL denote human activity recognition, hand gesture recognition, arm gesture recognition, and American Sign Language, respectively.

N_{v}

is the number of virtual antenna channels embedded in the common

4{\times}4

grid.

(a) Accuracy
	Cube:HAR		Cube:HGR		Cube:AGR
Model	in-set	out	in-set	out	in-set	out
RDAT (DFT) [ 36 ]	$0.992_{\pm 0.003}$	$0.721_{\pm 0.080}$	$0.973_{\pm 0.001}$	$0.491_{\pm 0.030}$	$0.986_{\pm 0.011}$	$0.784_{\pm 0.013}$
RDAT (CubeLearn) [ 36 ]	$0.992_{\pm 0.006}$	$0.765_{\pm 0.103}$	$0.969_{\pm 0.005}$	$0.491_{\pm 0.043}$	$0.991_{\pm 0.008}$	$0.847_{\pm 0.016}$
RadMamba [ 34 ]	$0.930_{\pm 0.050}$	$0.865_{\pm 0.097}$	$0.744_{\pm 0.042}$	$0.629_{\pm 0.053}$	$0.929_{\pm 0.009}$	$0.810_{\pm 0.011}$
GRT [ 6 ]	$0.994_{\pm 0.006}$	$0.468_{\pm 0.028}$	$0.969_{\pm 0.005}$	$0.696_{\pm 0.009}$	$0.990_{\pm 0.003}$	$\mathbf{0.964}_{\pm 0.016}$

Table 2: CubeLearn under the official protocol of [ 36 ] , where in-set = seen users 0–5 and out = unseen users 6–7. Mean

{}_{\pm\text{std}}

over 3 seeds. Bold = best per column and panel.

Ours
Model	Accuracy	Macro-F1
RDAT (DFT) [ 36 ]	$0.088_{\pm 0.005}$	$0.013_{\pm 0.001}$
RDAT (CubeLearn) [ 36 ]	$0.091_{\pm 0.005}$	$0.014_{\pm 0.001}$
RadMamba [ 34 ]	$0.484_{\pm 0.039}$	$0.473_{\pm 0.045}$
GRT [ 6 ]	$0.648_{\pm 0.088}$	$0.629_{\pm 0.101}$
Spectrogram CNN	$0.525_{\pm 0.005}$	$0.515_{\pm 0.009}$

Table 3: mmRadPose, 12-class participant-disjoint HAR, chance

{=}0.083

. Mean

{}_{\pm\text{std}}

over 3 seeds. Bold = best per column.

	CI4R-ASL		CI4R-Gait
Model	Acc.	Macro-F1	Acc.	Macro-F1
GRT [ 6 ]	$0.938_{\pm 0.010}$	$0.899_{\pm 0.015}$	$0.915_{\pm 0.023}$	$0.927_{\pm 0.020}$
RDAT (DFT) [ 36 ]	$0.737_{\pm 0.036}$	$0.535_{\pm 0.064}$	$0.675_{\pm 0.051}$	$0.696_{\pm 0.060}$
RDAT (CubeLearn) [ 36 ]	$0.679_{\pm 0.019}$	$0.444_{\pm 0.031}$	$0.633_{\pm 0.025}$	$0.632_{\pm 0.041}$
RadMamba [ 34 ]	$0.934_{\pm 0.006}$	$0.890_{\pm 0.010}$	$0.810_{\pm 0.023}$	$0.815_{\pm 0.032}$
Spectrogram CNN	$0.951_{\pm 0.003}$	$0.918_{\pm 0.004}$	$0.723_{\pm 0.008}$	$0.716_{\pm 0.007}$

Table 4: The two 77 GHz CI4R benchmarks under their releases’ subject-mixed protocols, namely CI4R-ASL [ 9 ] (18 activity and sign classes, chance

{=}0.056

) and CI4R-Gait [ 3 ] (11 activities and gaits, chance

{=}0.091

). Mean

{}_{\pm\text{std}}

over 3 seeds. Every row is scored on one identical partition per release, so all rows are directly comparable. Bold = best per column.

	Per-fold accuracy				Mean over folds
Model	S1	S2	S3	S4	Accuracy	Macro-F1
Model	116	91	221	214	642
GRT [ 6 ]	$0.491_{\pm 0.026}$	$0.484_{\pm 0.094}$	$0.549_{\pm 0.019}$	$0.606_{\pm 0.010}$	$0.532_{\pm 0.027}$	$0.488_{\pm 0.020}$
RDAT (DFT) [ 36 ]	$0.474_{\pm 0.106}$	$0.509_{\pm 0.022}$	$0.439_{\pm 0.012}$	$0.540_{\pm 0.026}$	$0.490_{\pm 0.030}$	$0.447_{\pm 0.026}$
RDAT (CubeLearn) [ 36 ]	$0.368_{\pm 0.057}$	$0.381_{\pm 0.017}$	$0.395_{\pm 0.017}$	$0.422_{\pm 0.085}$	$0.392_{\pm 0.012}$	$0.358_{\pm 0.024}$
RadMamba [ 34 ]	$0.575_{\pm 0.088}$	$0.593_{\pm 0.038}$	$0.603_{\pm 0.025}$	$0.670_{\pm 0.035}$	$0.610_{\pm 0.028}$	$0.587_{\pm 0.026}$

Table 5: Leave-one-subject-out CI4R-Gait, where each fold holds one subject code out of training, validation, and model selection. Per-fold cells are accuracy with held-out recordings beneath, and the last two columns average over folds. All cells are mean

{}_{\pm\text{std}}

over 3 seeds. Bold = best per column.

Model	Params (M)	GFLOPs	Val./Par.
RDAT	$25.252$	$28.39$	$0.36$
RadMamba	$0.048$	$0.0035$	$1.04$
Spectrogram CNN	$0.393$	$0.25$	$0.09$
GRT	$25.886$	$5{,}967$	$1.40$
Raw-IQ VAE	$41.832$	$29{,}534$	$1.73$

Table 6: Parameters (encoder and head, or the end-to-end model), forward GFLOPs (model forward only) per recording on CI4R-ASL, the largest benchmark, with recordings of intermediate length (69 frames on average), and input values per parameter (Val./Par.).

mmWave Radar Sensing

Millimeter-wave radar sensing uses reflected radio waves to measure motion, distance, and structure for perception, health monitoring, and robotics.

See all 90 papers

Weather-Robust Scene Semantics with Vision-Aligned 4D Radar
May 8, 2026Kali Hamilton, Christoffer HeckmanCross-Modal Learning VLM Robustness
Autonomous Robotics and Perception Group at the University of Colorado Boulder, Boulder, CO 80309, USA.
mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction
Jul 25, 2026Cheng Guo, Qiming Cao, Shengkai Xu +4Synthetic Data Pretraining Sim-to-Real Transfer
University of North Carolina at Charlotte, Charlotte, NC, USA · Purdue University, West Lafayette, IN, USA
MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video
Apr 30, 2026Xijia Wei, Yuan Fang, Kevin Chetty +2Masked Autoencoders Self-Supervised Learning
University College London London, UK

May 8, 2026J/K move · Enter open · S save

Weather-Robust Scene Semantics with Vision-Aligned 4D Radar

Kali Hamilton, Christoffer Heckman

Autonomous Robotics and Perception Group at the University of Colorado Boulder, Boulder, CO 80309, USA.

Cross-Modal Learning VLM Robustness Image Captioning Millimeter-Wave Radar mmWave Radar Sensing Autonomous Driving Perception+1

Open paper AskPDF arXiv

Cameras and LiDAR degrade in rain, fog, and snow, while millimeter-wave radar remains largely unaffected. We align a radar encoder to frozen SigLIP vision embeddings and decode structured scene captions through a frozen vision-language model (VLM) with approximately 7M trainable parameters. On K-RADAR with held-out fog, light snow, and heavy snow sequences, all radar configurations outperform a camera baseline that collapses to over 90% hallucination. We identify a token-norm mismatch as the dominant failure mode when bridging radar to a frozen VLM and show that projector-output LayerNorm resolves it. Analysis of encoder complexity, caption format, and pooling strategy reveals tradeoffs that inform future radar-VLM pipeline design.

Abstract

mmWave Radar Sensing

Explore similar work

Weather-Robust Scene Semantics with Vision-Aligned 4D Radar

mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video