Radar Panoptic Transformer: Semantic-Guided Mask Queries for Dynamic Scene Understanding
Organizations: Research and Development, Mercedes-Benz AG, Germany. · Department of Computer Science, University of Freiburg, Germany. · Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany.
Abstract
Automotive radar provides long-range sensing and direct Doppler velocity measurements, but its sparse and non-uniformly sampled point clouds pose significant challenges for panoptic segmentation. Dynamic objects often yield only a few detections amid a dominant static background, providing limiting the geometric cues for separating object instances. We propose Radar Panoptic Transformer (RPT), an end-to-end point-based network that incorporates radar-adaptive feature extraction with semantic-guided mask prediction. Its backbone aggregates pose-aligned temporal context through grouped vector attention and learns per-point neighborhood radii conditioned on range, azimuth, and radar cross section (RCS). Point-wise semantic predictions initialize queries at likely foreground detections and guide masked cross-attention through query-point class agreement, producing instance masks without geometric clustering. An auxiliary RCS-consistency loss matches instance-level statistics between predicted and annotated instances. RPT achieves state-of-the-art performance on RadarScenes with mIoU and PQ. We further evaluate RPT on MAN TruckScenes, where it outperforms the evaluated baselines in a long-range, full-surround radar setting with sensing ranges up to . RPT achieves real-time end-to-end inference at ,Hz on RadarScenes, exceeding the sensor frame rate.
Figures & tables
| Pts. | Stage | Encoder: Op./Ch./ / | Decoder: Op./Ch./ / |
|---|---|---|---|
| S1/D1 | KPConv /64/12/5 (shared) | KPConv /64/12/5 | |
| S2/D2 | KPConv /128/16/6 | KPConv /128/16/6 | |
| S3/D3 | GVA /256/24/8 | GVA /256/24/8 | |
| S4 | GVA /512/24/10 | – |
| Dataset | Radar setup | Coverage | Range | Supervision |
|---|---|---|---|---|
| RadarScenes [ 10 ] | 4 radar | front | 100 m | point cls. + track ID |
| NuScenes [ 27 ] | 5 radar | m | 3D boxes + track ID | |
| VoD [ 28 ] | 1 3+1D radar | front | m | 3D boxes + track ID |
| TruckScenes [ 15 ] | 6 4D radar | m | 3D boxes + track ID |
| Method | Input | PQ | mIoU | PQ | IoU | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| static | car | ped. | ped. grp. | bike | truck | static | car | ped. | ped. grp. | bike | truck | ||||
| RadarPNv1 [ 29 ] | multi-scan | – | 61.0 | – | – | – | – | – | – | 98.7 | 58.2 | 36.0 | 58.7 | 58.4 | 56.1 |
| RadarPNv2 [ 30 ] | – | 61.9 | – | – | – | – | – | – | 98.7 | 63.8 | 38.8 | 58.5 | 51.0 | 61.0 | |
| STA-Net [ 31 ] | – | 70.4 | – | – | – | – | – | – | 99.8 | 75.9 | 43.1 | 82.0 | 67.1 | 54.6 | |
| Mask3D [ 20 ] | single-scan | 56.9 | 56.1 | 98.4 | 70.6 | 18.4 | 31.9 | 57.7 | 64.3 | 98.6 | 68.7 | 11.8 | 35.9 | 57.7 | 63.9 |
| GRT [ 8 ] + DBSCAN [7] | 56.3 | 56.9 | 98.7 | 59.0 | 29.7 | 38.3 | 54.6 | 57.3 | 98.7 | 58.5 | 21.7 | 46.0 | 54.6 | 61.8 | |
| Method | Input | PQ | mIoU | PQ | IoU | ||||
|---|---|---|---|---|---|---|---|---|---|
| static | car | truck | static | car | truck | ||||
| STA-Net [ 31 ] | multi-scan | – | 72.9 | – | – | – | 96.8 | 58.1 | 63.9 |
| PTv3 [ 32 ] | 54.5 | 66.4 | 98.1 | 39.6 | 25.9 | 94.7 | 50.2 | 54.3 | |
| Mask3D [ 20 ] | single-scan | 56.3 | 68.8 | 98.4 | 42.8 | 27.6 | 95.2 | 53.5 | 57.7 |
| RPT (ours) | 62.8 | 73.3 | 98.4 | 50.6 | 39.4 | 96.9 | 58.5 | 64.6 | |
| Configuration | mIoU |
|---|---|
| No temporal module ( ) | 67.7 |
| Temporal + fixed-radius query | 70.9 |
| Temporal + learned-radius query | 71.6 |
| Configuration | PQ |
|---|---|
| Semantic Head + MeanShift Clustering [ 34 ] | 73.2 |
| Semantic Head + semantic-guided decoder | 81.6 |
| Semantic Head + semantic-guided decoder + RCS loss | 82.2 |
| Temporal input | mIoU | PQ |
|---|---|---|
| 69.4 | 80.0 | |
| (RPT) | 71.6 | 82.2 |
| 71.5 | 81.8 |
| Method | Params. [M] | Runtime [ms] | PQ |
|---|---|---|---|
| STA-Net [ 31 ] | 7.4 | 34.9 | – |
| RadarPNv1 [ 29 ] | 1.8 | 278.1 | – |
| Mask3D [ 20 ] | 39.6 | 85.2 | 56.9 |
| GRT [ 8 ] | 8.4 | 24.0 | 56.3 |
| RIT [ 12 ] | 3.8 | 31.7 | – |
| SemRafiner [ 11 ] | 4.5 | 42.1 | 81.4 |
Panoptic Segmentation
Panoptic segmentation assigns every pixel a semantic category and groups countable objects into distinct instances.
See all 40 papers