Robust scene understanding under varying environmental conditions is essential for safe autonomous driving. While cameras and LiDAR degrade in adverse weather and poor lighting, Frequency-Modulated Continuous-Wave (FMCW) spinning radar provides dependable 360° perception at extended ranges with minimal compute cost. However, real-time FMCW radar scene understanding is challenging due to complex signal returns, limited temporal resolution, and scarce annotated datasets. We introduce StreamRPS, a novel framework for streaming FMCW radar panoptic segmentation, comprising an automated label-generation pipeline via LiDAR scene completion and a real-time streaming network for semantic and panoptic segmentation. Our encoder-decoder streaming architecture comprises a radar-specific hierarchical Transformer encoder and task-specific decoders for semantic or panoptic prediction. Our encoder introduces a stem to capture range-dependent signatures, anisotropic patch embeddings to preserve angular resolution, and axial polar attention to separately model the periodic azimuth and bounded range axes. We further introduce azimuth roll rotation augmentation to simulate ego-vehicle rotation during training. Comprehensive experiments on the SAX and BOREAS datasets validate both our auto-labeling methodology and streaming segmentation network against established baselines. We outperform the strongest baseline by at least 1.4pp while processing sectors at over 400 FPS in a 4-sector streaming setting. We additionally provide manual annotations for 100 SAX radar scans, enabling independent evaluation of the entire framework.
Figures & tables
Figure 1: StreamRPS yields dense semantic and panoptic radar labels using a pre-trained LiDAR scene completion network. The streaming FMCW radar segmentation architecture is trained on generated label sectors and operates in real time to deliver semantic or panoptic scene understanding.
Figure 2: StreamRPS employs PaSCo scene completion to create dense radar labels, which are converted to a polar representation. We then apply azimuth roll rotation to augment radar scans before processing radar sectors with our streaming network. Key novelties include our RadarStem and Axial Polar Attention. Semantic and panoptic decoders decode the resulting features. Cross-entropy, Dice, and L1 losses are applied per decoder, as appropriate.
Boreas
SAX
Method
1
obj L
obj S
bkg
2
4
8
16
32
1
obj L
obj S
bkg
2
4
8
16
32
TMVANet [ 10 ]
27.85
39.20
2.47
45.69
25.87
23.06
24.13
24.77
19.66
32.27
36.76
6.55
47.71
24.37
22.34
20.98
20.78
17.53
RSSNet [ 21 ]
23.56
27.45
1.83
42.70
22.26
19.30
18.63
17.42
11.98
31.02
35.43
5.35
47.16
27.59
23.05
25.55
26.33
20.14
TransRadar [ 8 ]
30.13
43.34
2.60
48.85
27.74
28.17
26.18
26.02
20.85
31.63
33.91
7.21
47.45
32.67
28.75
28.00
26.15
18.88
Mask2Former [ 28 ]
31.67
50.66
4.13
46.54
30.57
30.03
26.42
-
-
33.56
37.17
8.33
49.97
33.85
32.94
30.99
-
-
SegFormer [ 30 ]
29.53
39.60
2.85
49.49
29.53
29.39
28.55
-
-
30.88
36.81
5.76
46.29
29.83
30.24
30.17
-
-
Table I: Streaming semantic segmentation performance on Boreas and SAX for full-scan (1) and streaming settings with varying numbers of sectors (2, 4, 8, 16, 32), in IoU [%]. obj L : large object categories (car, other-vehicle / obstacle). obj S : small object categories (motorcycle, person, pole). bkg: background/static (road, pavement, terrain, vegetation, building, fence).
Boreas
SAX
Method
1
4
1
4
PQ
PQ st
PQ th
SQ
RQ
PQ
SQ
RQ
PQ
PQ st
PQ th
SQ
RQ
PQ
SQ
RQ
PanopticFPN [ 34 ]
18.94
22.57
12.88
59.59
27.94
20.23
55.29
29.30
13.09
19.22
4.51
55.47
19.48
13.44
57.44
19.67
Mask2Former [ 28 ]
18.94
27.17
5.22
47.35
27.63
21.53
48.69
30.57
15.45
22.58
5.48
55.94
22.69
14.40
62.06
20.97
OneFormer [ 37 ]
17.35
24.03
13.15
45.71
24.45
20.32
48.29
28.90
9.90
13.58
4.74
56.01
14.38
11.68
52.90
16.85
StreamRPS(Ours)
21.80
25.47
14.47
52.94
32.08
23.31
56.84
33.39
13.37
18.88
5.66
56.86
19.86
15.43
64.45
22.29
Table II: Streaming panoptic segmentation performance on Boreas and SAX datasets for full-scan (1) and streaming with 4 sectors. PQ: Panoptic Quality, PQ st : PQ for stuff classes, PQ th : PQ for thing classes, SQ: Segmentation Quality, RQ: Recognition Quality.
Figure 3: Qualitative results of StreamRPS on SAX and Boreas. (a, e, c, g): full-scan (1 sector); (b, f, d, h): streaming (4 sectors). (a, b, e, f): Semantic Segmentation in comparison to SegFormer; (c, d, g, h): Panoptic Segmentation in comparison to Mask2Former. Errors are highlighted in white-red.
Method
Sectors
IoU
FPS
Mem.
p50
p90
p99
RSSNet [ 2 ]
1
31.02
55.6
184
18.17
18.29
18.41
4
23.05
260.0
47
3.92
3.95
3.97
Mask2Former [ 28 ]
1
33.56
13.0
2080
77.15
78.02
78.30
4
32.94
46.4
2080
21.59
21.85
22.09
SegFormer [ 30 ]
1
30.88
145.8
299
7.06
7.10
7.14
4
30.24
782.3
38
1.34
1.35
1.36
Table III: Ablation study on the Inference performance on NVIDIA GeForce RTX 2080 Ti GPU, reporting throughput (FPS), memory footprint (Mem.[MiB]), and latency percentiles (p50-p95) on the SAX dataset full-scan and four sectors.
Training Label
Hand-Labeled Test
Dense Auto-Label Test
Generation
1
4
1
4
WaffleIron [ 38 ]
29.07
28.76
25.06
23.59
PaSCo (Ours)
31.47
30.22
34.35
34.33
Table IV: Performance comparison of different training data for StreamRPS.
Temporal
Radar
Azimuth Roll
Patch
Axial Polar
Frame
Stem
Aug.
Embed.
Att.
1
30.88
✓
30.84
✓
✓
31.61
✓
✓
✓
33.32
✓
✓
✓
✓
34.03
Table V: Ablation study on various components of StreamRPS on the SAX dataset full-scan.
{τ}
{τ,τ-1,τ-2}
{τ,…,τ-4}
{τ,τ-3,τ-5}
{τ,τ-4,τ-7}
32.05
34.35
33.04
32.63
32.17
Table VI: Ablation study on temporal frame aggregation of StreamRPS on SAX full-scan. τ denotes the timestamp of stacked radar frames.