ReinSeg: Retrieval-Guided Bounded Multimodal Refinement for 3-D Road Anomaly Segmentation
Organizations: School of Electrical and Information Engineering, Tianjin University, Tianjin, China.
Abstract
3-D road anomaly segmentation is essential for risk-aware perception in autonomous driving. Its central challenge is to rank sparsely observed hazards ahead of abundant normal road structures while preserving point-wise metric localization. LiDAR provides accurate 3-D point support, whereas surround-view imagery offers complementary semantic context for separating unexpected hazards from challenging normal structures. We propose ReinSeg, a retrieval-guided bounded multimodal refinement framework for point-wise road anomaly segmentation. Occupied LiDAR bird's-eye-view cells retrieve scene-specific context from a shared memory of eight surround-view images without supplying calibration variables to the retrieval module. A relative-norm-bounded residual returns this context to the LiDAR point features, so the frozen LIDO anomaly-scoring interface is preserved, and the refinement is optimized with in-distribution semantic labels only. On the Spotting the Unexpected (STU) benchmark, ReinSeg improves validation average precision from 27.54% to 35.67% and reduces the false-positive rate at 95% anomaly recall from 34.85% to 10.61%, shrinking the high-recall screening set from 67.54 to 20.55 million points. Relative to REL, AP is 24.99 percentage points higher and FPR95 is 0.99 points lower. Matched interfaces and component isolation show that retrieved scene imagery and the relative-norm bound are both essential, and ReinSeg stays above LiDAR-only performance under missing, corrupted, or misaligned camera inputs while running at 21.8 frames per second. These results establish bounded retrieval of scene-specific visual evidence as an effective route to reliable 3-D road-hazard perception.
Figures & tables
| Method | Scene | Output | Inference input | Assoc. | Bounded update |
|---|---|---|---|---|---|
| M3DM [ 38 ] | Industrial | 3-D region | RGB + point cloud | Registered RGB–3-D | No |
| Unknown-road detection [ 24 ] | Road | 3-D object | RGB + LiDAR | Calibrated projection | No |
| DetSeg [ 20 ] | Road | 2-D object | RGB sequence | Temporal boxes | No |
| Feature Mixing [ 26 ] | Road | 3-D point | RGB + LiDAR | Calibrated point fusion | No |
| OccOoD [ 30 ] | Road | 3-D voxel | Multi-camera | Camera occupancy | No |
| LIDO [ 2 ] | Road | 3-D point | LiDAR | n/a | No |
| Method | Input | AP | AUROC | FPR95 |
|---|---|---|---|---|
| LiDAR only; no auxiliary-OOD supervision | ||||
| M4F3D + MC Dropout [ 52 ] | LiDAR | 0.17 | 65.76 | 79.82 |
| M4F3D + RbA [ 52 ] | LiDAR | 1.64 | 73.00 | 100.00 |
| M4F3D + Max Logit [ 52 ] | LiDAR | 2.02 | 87.27 | 68.76 |
| M4F3D + Void Classifier [ 52 ] | LiDAR | 2.62 | 89.77 | 79.50 |
| M4F3D + Deep Ensemble [ 52 ] | LiDAR | 6.94 | 90.93 | 37.34 |
| Interface | Calib. | AP | AUROC | FPR95 |
|---|---|---|---|---|
| LIDO (no images) | n/a | 27.54 | 95.05 | 34.85 |
| Projected local | Yes | 31.76 | 97.35 | 9.96 |
| Projected + global | Yes | 33.38 | 97.26 | 10.18 |
| ReinSeg retrieval | No | 35.67 | 97.24 | 10.61 |
| Variant | RGB | Ret. | Bound | AP | FPR95 |
|---|---|---|---|---|---|
| Frozen LiDAR reference | – | – | – | 30.02 | 12.87 |
| LiDAR adapter control | – | – | ✓ | 32.99 | 11.58 |
| Constant visual memory | – | ✓ | ✓ | 33.48 | 10.71 |
| Global pooled RGB | ✓ | – | ✓ | 32.76 | 11.43 |
| Retrieval, unbounded | ✓ | ✓ | – | 18.43 | 77.77 |
| Retrieval + bound + score gate | ✓ | ✓ | ✓ | 32.02 | 12.33 |
| Variant | AP | FPR95 | mIoU |
|---|---|---|---|
| only | 34.62 | 10.93 | 53.45 |
| 34.90 | 10.82 | 53.40 | |
| 33.88 | 10.68 | 53.45 | |
| All; no BEV fuser | 33.13 | 10.60 | 53.35 |
| All; LIDO unfrozen | 39.98 | 51.81 | 79.34 |
| All; LIDO frozen | 35.67 | 10.61 | 53.42 |
| Configuration | Cameras | Tokens/view | Trainable (M) | AP | FPR95 | Latency (ms) | FPS | Memory (GiB) |
|---|---|---|---|---|---|---|---|---|
| LIDO | 0 | – | – | 27.54 | 34.85 | 30.9 | 32.3 | 0.52 |
| ReinSeg, 4 cameras | 4 | 112 | 28.02 | 32.12 | 9.53 | 45.0 | 22.2 | 0.92 |
| ReinSeg, ResNet-18 | 8 | 112 | 1.07 | 34.20 | 9.23 | 41.0 | 24.4 | 0.83 |
| ReinSeg, 56 tokens | 8 | 56 | 28.02 | 35.34 | 10.70 | 47.1 | 21.2 | 0.93 |
| ReinSeg, 224 tokens | 8 | 224 | 28.02 | 34.08 | 10.48 | 46.1 | 21.7 | 0.93 |
| ReinSeg, 112 tokens | 8 | 112 | 28.02 | 35.67 | 10.61 | 45.9 | 21.8 | 0.93 |
Autonomous Driving Perception
Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.
See all 410 papers