RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation

Anuvab Sen; Mir Sayeed Mohammad; Saibal Mukhopadhyay

eess.SPApr 6, 2026

RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation

Authors: Anuvab Sen, Mir Sayeed Mohammad, Saibal Mukhopadhyay

Organizations: Georgia Institute of Technology, Atlanta, Georgia, USA

Abstract

This paper presents RAVEN, a computationally efficient deep learning architecture for FMCW radar perception. The method processes raw ADC data in a chirp-wise streaming manner, preserves MIMO structure through independent receiver state-space encoders, and uses a learnable cross-antenna mixing module to recover compact virtual-array features. It also introduces an early-exit mechanism so the model can make decisions using only a subset of chirps when the latent state has stabilized. Across automotive radar benchmarks, the approach reports strong object detection and BEV free-space segmentation performance while substantially reducing computation and end-to-end latency compared with conventional frame-based radar pipelines.

Figures & tables

Figure 1: (a) Comparison of traditional radar processing paradigms: frame-wise CNN encoders, chirp-wise recurrent models, and sample-wise streaming SSM pipelines. (b) Our spatial-aware hybrid architecture preserves per-RX structure, performs cross-antenna attention and extracts chirp-wise virtual-array features for lightweight detection. (c) Runtime–performance characterization showing our method achieves higher accuracy at significantly lower latency and compute compared to existing radar perception models [ 25 , 5 , 29 , 28 ] .

Figure 2: MIMO radar virtual antenna formation and multiplexing. (a) $N_{tx}$ transmitters and $N_{rx}$ receivers form $N_{tx}\!\times\!N_{rx}$ virtual antennas. RX channels read simultaneously. (b) TDM: TX elements fire sequentially. (c) DDM: TX elements fire spectrally interleaved FMCW pulses; virtual-array information is mixed in frequency per receiver.

Figure 3: RAVEN Architecture: (1) Fast-time per-RX SSMs compress I/Q into compact 2-D tokens; (2) cross-antenna attention fuses RX channels and expands to virtual-MIMO features; (3) a chirp-wise SSM updates the state online across chirps; (4) a learned projection maps features to a $T\times H\times W$ grid; (5) lightweight decoders produce detection heatmaps/boxes and segmentation.

Figure 4: (a) Attention Mixer: Learnable transmitter queries are used to extract Doppler-division multiplexed information from the receiver signal in the time domain. These are fused together to form the virtual antenna array for retrieving the MIMO information. (b) Early Decision Supervision: During training, decoders take outputs from multiple chirp levels, and loss is computed simultaneously [ 13 ] , forcing the model to converge on earlier chirps.

Figure 5: Qualitative ablation of the adaptive decision module across four scenarios. Each example shows the RGB view, segmentation evolution over chirps (white - true positive, green - false positive, red - false negative), detection evolution (point-level RA predictions), and the chirp-state contribution signal. Sample (a) complex multi-vehicle scene where early-chirp assumptions about distant objects are refined into accurate detections. Sample (b) early-chirp false positives (“hallucinated” obstacles) are suppressed as more chirps arrive. Sample (c) early hallucinations fade but segmentation remains unreliable throughout. Sample (d) an object briefly emerges in clutter before vanishing, and a noisy chirp-similarity score depicts the irregularity of the data, resulting in poor segmentation and detection.

Figure 6: Design motivation for adaptive chirp selection. (Left) Minimum cosine-distance aggregate across all frames in train-set reveals a clear knee point beyond which new chirps add little novel information. (Middle) Validation-set scores show consistent gains from 32→64 chirps, with negligible improvements thereafter. (Right) Memory and latency scale strongly with chirp count, showing that reducing chirps provides substantial efficiency gains with minimal performance loss.

Metric / Model	ChirpNet	ChirpNetLite	ChirpNet-SSM	ChirpNet-Attn	T-FFTRadNet	FFT-RadNet	UNet	SSMRadNet	RAVEN
	[ 29 ]	[ 29 ]	[ 30 ]	[ 30 ]	[ 5 ]	[ 25 ]	[ 26 ]	[ 28 ]	(Ours)
Computational Complexity
GMACs $\downarrow$	1.480	0.320	0.340	0.350	15.990	41.740	15.140	0.108	0.053
Params (M) $\downarrow$	3.780	3.761	3.761	3.761	9.000	4.250	17.270	0.566	0.347
Accuracy Metrics
Dice Coefficient $\uparrow$	0.986	0.989	0.990	0.991	0.995	0.996	0.996	0.996	0.997

Table 1: Comparison with prior works on RaDICaL [ 16 ] . Best values per row are highlighted in bold .

Class	Model	mIoU $\uparrow$	F1 $\uparrow$	mAP $\uparrow$	mAR $\uparrow$	RE (m) $\downarrow$	AE ( ∘ ) $\downarrow$	GMACs $\downarrow$	Params (M) $\downarrow$	Lat. (ms) $\downarrow$
Convolution	Pixor (PC) [ 38 ]	—	—	0.96	0.32	0.17	0.25	—	—	—
	Pixor (RA) [ 38 ]	—	—	0.96	0.82	0.12	0.20	—	—	—
	PolarNet [ 20 ]	0.61	—	—	—	—	—	—	—	—
	Conv3D + FFT-RadNet [ 36 ]	0.75	0.47	0.58	0.39	0.19	0.33	—	—	—
	FFT-RadNet [ 25 ]	0.74	0.88	0.97	0.82	0.14	0.17	146.82	3.80	53.59
	RLSM [ 24 ]	0.71	0.86	0.91	0.82	—	—	—	—	—

Table 2: Overall segmentation and detection performance on RADIal [ 25 ] . Best values (global) in bold .

Figure 7: (a) RaDICaL [ 16 ] : label generation from RGB frames using a tiled RetinaNet detector (adapted from [ 29 ] ). (b) RADIal [ 25 ] : FFT of raw ADC data produces range–azimuth maps; CFAR yields radar point clouds; segmentation maps mark drivable (white) vs. non-drivable (black) areas; nearest and second-nearest vehicles are highlighted in red and green, respectively.

Figure 8: Per-block latency (ms) on a single GPU. The channel SSM is the main sequential bottleneck because it processes long fast-time sequences; the mixer and decoders are highly parallelizable.

Model Variant	Channel SSM	Antenna Mixer	mIoU	F1	mAP	GMACs
(A) Shared Fast-time SSM	✗	✗	0.79	0.77	0.81	1.67
(B) Cross-Antenna Attention + Shared Fast-time SSM	✗	✗	0.80	0.79	0.83	38.89
(C) Shared Fast-time SSM + Cross-Antenna Attention	✗	✓	0.79	0.80	0.83	1.62
(D) Cross-Antenna Attention + Channel SSM	✓	✓	0.84	0.88	0.88	34.56
(E) Channel SSM + Cross-Antenna Attention (RAVEN, full-frame)	✓	✓	0.90	0.93	0.95	1.02
(F) Channel SSM + Cross-Antenna Attention (RAVEN, sub-frame)	✓	✓	0.85	0.89	0.88	0.27

Table 3: Ablation of channel SSM and antenna mixer on RADIal. All variants share the same chirp-wise SSM backbone and decoders. Our physics-guided RAVEN encoders (blue rows), which apply per-RX channel SSMs before the antenna mixer, achieve the best trade-off between accuracy and compute; the sub-frame variant further improves efficiency for early-exit decisions.

Figure 9: Segmentation and detection maps across driving scenes with and without multi-chirp supervision. Without supervision across chirp levels, segmentation gradually approaches the ground truth, but detection remains unstable throughout the sequence, consistent with Figure 10 . In (a), the model forgets real objects mid-frame that only reappear at the end. In (b), it initially identifies one set of objects but later predicts an entirely different set. In (c), it begins to hallucinate obstacles near the final chirps. With multi-chirp supervision, these issues disappear: detection becomes consistent across the sequence, and both segmentation and detection remain accurate through the final frame.

Panel of Figure 10 — Figure 10: Design motivation for adaptive chirp selection. (a) Validation curves illustrate how detection and segmentation performance evolve with increasing chirp count. Without explicit early-convergence constraints, detection remains suboptimal until the full chirp frame is processed. (b) Introducing multi-chirp supervision during training encourages the model to saturate earlier and learn temporal continuity across chirps, yielding smoother convergence and higher overall detection–segmentation performance.

Component	Configuration
Antenna Mixer	Dim 64, 8 heads, expansion $4{\times}$ , init $\sim\!\mathcal{N}(0,1)$
SSM (Mamba)	State dim 16, conv kernel 4, expansion 2
Spatial Proj.	$1{\times}1$ Conv1D $\to$ 1792 ch. (grid $32{\times}56$ )

Table 4: RAVEN architectural hyperparameters.

Tokens per RX ( $K$ )	1 (RAVEN)	4	8	16
F1 (Detection)	0.934	0.936	0.936	0.937

Table 5: Fast-time token compression ablation on RADIal. Marginal gains from

K{=}1

K{=}16

confirm that a single token per RX channel sufficiently captures range and phase information.

Figure 11: Cosine distance vs. entropy as chirp-stopping signals. Cosine similarity (blue) produces a cleaner knee-point, enabling more consistent early-exit decisions than entropy (orange).

Stopping Rule	mAP	mAR	F1	mIoU
Cosine (Ours)	94.5	95.1	94.8	89.5
Entropy	93.6	94.0	93.8	88.8

Table 6: Early-exit decision rule comparison on RADIal (all metrics in %). Cosine similarity outperforms entropy on every metric.

Figure 12: Velocity distribution and adaptive chirp count. (Left) Velocity histogram of annotated objects in RADIal. (Right) Scatter plot of per-frame selected chirp count vs. object velocity. The absence of correlation confirms that adaptive stopping is stability-driven, not velocity-driven.

mmWave Radar Sensing

Millimeter-wave radar sensing uses reflected radio waves to measure motion, distance, and structure for perception, health monitoring, and robotics.

See all 85 papers

Real-Time Radar Semantic and Panoptic Scene Understanding
Oct 19, 2026Santosh Parit, Julia Hindel, Rohit Mohan +4Autonomous Driving Perception Panoptic Segmentation
University of Freiburg · University of Oxford
RADE-Net: Robust Attention Network for Radar-Only Object Detection in Adverse Weather
Feb 23, 2026Christof Leitgeb, Thomas Puchleitner, Max Peter Ronecker +13D Object Detection mmWave Radar Sensing
Infineon Technologies AG, Institute of Visual Computing, Graz University of Technology, Graz, Austria · Infineon Technologies AG, Graz, Austria · Virtual Vehicle Research GmbH, Institute of Visual Computing, Graz University of Technology, Graz, Austria +1
On Learning Spatial Structure from Pre-Beamforming Per-Antenna Range-Doppler Radar Measurements
Apr 2, 2026George Sebastian, Philipp Berthold, Bianca Forkel +2Representation Learning Millimeter-Wave Radar
Institute for Autonomous Driving, Department of Aerospace Engineering, University of the Bundeswehr Munich, Neubiberg, Germany

Oct 19, 2026J/K move · Enter open · S save

Real-Time Radar Semantic and Panoptic Scene Understanding

Santosh Parit, Julia Hindel, Rohit Mohan, Georgi Pramatarov+3

University of Freiburg · University of Oxford

Autonomous Driving Perception Panoptic Segmentation Semantic Segmentation mmWave Radar Sensing

Open paper AskPDF arXiv

Robust scene understanding under varying environmental conditions is essential for safe autonomous driving. While cameras and LiDAR degrade in adverse weather and poor lighting, Frequency-Modulated Continuous-Wave (FMCW) spinning radar provides dependable 360° perception at extended ranges with minimal compute cost. However, real-time FMCW radar scene understanding is challenging due to complex signal returns, limited temporal resolution, and scarce annotated datasets. We introduce StreamRPS, a novel framework for streaming FMCW radar panoptic segmentation, comprising an automated label-generation pipeline via LiDAR scene completion and a real-time streaming network for semantic and panoptic segmentation. Our encoder-decoder streaming architecture comprises a radar-specific hierarchical Transformer encoder and task-specific decoders for semantic or panoptic prediction. Our encoder introduces a stem to capture range-dependent signatures, anisotropic patch embeddings to preserve angular resolution, and axial polar attention to separately model the periodic azimuth and bounded range axes. We further introduce azimuth roll rotation augmentation to simulate ego-vehicle rotation during training. Comprehensive experiments on the SAX and BOREAS datasets validate both our auto-labeling methodology and streaming segmentation network against established baselines. We outperform the strongest baseline by at least 1.4pp while processing sectors at over 400 FPS in a 4-sector streaming setting. We additionally provide manual annotations for 100 SAX radar scans, enabling independent evaluation of the entire framework.

Abstract

mmWave Radar Sensing

Explore similar work

Real-Time Radar Semantic and Panoptic Scene Understanding

RADE-Net: Robust Attention Network for Radar-Only Object Detection in Adverse Weather

On Learning Spatial Structure from Pre-Beamforming Per-Antenna Range-Doppler Radar Measurements