RETR: A Structure-Preserving RGB-Event Transformer for Robust 3D Lane Detection
Organizations: Beijing Institute of Technology, Beijing, China
Abstract
Robust 3D lane detection requires accurate metric road geometry recovery from visual inputs, yet conventional RGB-based approaches suffer from fragile lane evidence under adverse illumination, motion blur, and long-range perspective compression. Event cameras offer complementary high-temporal-resolution, high dynamic-range structural cues that are robust to these challenges, but their sparse, motion-dependent responses cannot be directly fused into consistent 3D geometry. In this paper, we present the first attempt to introduce event cameras into 3D lane detection. To enable systematic research, we first build two multimodal benchmarks with metric 3D lane annotations: DSEC-3DLD (real-world sequences) and Ev-OpenLane (large-scale simulated sequences). We further propose RETR, a structure preserving transformer that converts complementary RGB and event observations into coherent 3D lane geometry. RETR first aligns modality consistent evidence through Reciprocal Context Flow Fusion, then preserves thin and uncertain lane structures with Uncertainty Aware Structural Consolidation, and finally decodes ordered lane hypotheses using a Geometry State Decoder with proposal conditioned initialization, reference conditioned query evolution, and diversified geometric inquiry. Extensive experiments show that RETR achieves state-of-the-art performance on both benchmarks, especially showing strong improvements in challenging lighting and complex road geometry scenarios. Code and dataset will be available at https://github.com/duskswell/RETR.
Figures & tables
| Method | Overall | Scenario-wise F1-Score (%) | ||||||||||
| F1 | P | R | CD | Str. | Cur. | Int. | M&S | Nor. | Ove. | Und. | U/D | |
| PersFormer [ 7 ] | 50.1 | 48.7 | 51.5 | 0.127 | 51.5 | 48.3 | 48.8 | 58.9 | 53.3 | 31.3 | 41.8 | 34.5 |
| SparsePers [ 62 ] | 50.2 | 48.1 | 52.3 | 0.134 | 51.6 | 48.8 | 46.1 | 59.2 | 52.2 | 35.2 | 45.4 | 45.6 |
| Anchor3DLane [ 19 ] | 54.7 | 57.9 | 51.8 | 0.127 | 56.6 | 53.8 | 51.2 | 50.7 | 57.3 | 28.3 | 49.0 | 45.7 |
| 3DLaneDT [ 56 ] | 53.7 | 55.2 | 52.2 | 0.130 | 54.1 | 54.2 | 51.2 | 53.3 | 56.2 | 24.9 | 48.8 | 43.5 |
| Anchor3DLanePP [ 20 ] | 61.8 | 67.1 | 57.4 | 0.121 | 63.8 | 59.9 | 59.3 | 65.6 | 64.4 | 39.7 | 56.2 | 48.6 |
| Component | RCF | UASC | GSD | F1-score (%) | |
| DSEC | Ev-Open | ||||
| RGB only | ✗ | ✗ | ✗ | 69.2 | 61.9 |
| RCF | ✓ | ✗ | ✗ | 72.9 | 64.5 |
| RCF + UASC | ✓ | ✓ | ✗ | 74.3 | 65.7 |
| Ours | ✓ | ✓ | ✓ | 75.6 | 66.7 |
| Method | All | Up & Down | Curve | Extreme Weather | Night | Intersection | Merge & Split |
| BEVLaneDet [ 55 ] | 58.4 | 48.7 | 63.1 | 53.4 | 53.4 | 50.3 | 53.7 |
| LATR [ 35 ] | 61.9 | 55.2 | 68.2 | 57.1 | 55.4 | 52.3 | 61.5 |
| HeightLane [ 45 ] | 62.7 | 53.6 | 69.3 | 55.4 | 54.6 | 54.1 | 61.1 |
| PVALane [ 63 ] | 62.7 | 54.1 | 67.3 | 62.0 | 57.2 | 53.4 | 60.0 |
| SC-Lane [ 44 ] | 64.3 | 54.6 | 70.7 | 58.1 | 56.5 | 56.1 | 63.0 |
| RETR (Ours) | 66.7 | 60.0 | 73.1 | 59.8 | 60.5 | 58.9 | 65.9 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Setting | Description |
| Original frame rate | 10 fps | Frame rate of the original OpenLane key frames. |
| Interpolated frame rate | 30 fps | Frame rate after temporal interpolation. |
| Interpolated frame interval | 33.33 ms | Temporal interval between adjacent interpolated frames. |
| Image resolution | Resolution used for ESIM simulation and event representation. | |
| Input timestamp interval | 33.33 ms | Timestamp interval for ESIM input frames. |
| Data source | Image folder | ESIM reads interpolated frames and timestamps from an image sequence. |
| Table 11: Dataset statistics and sequence-level splits of DSEC-3DLD. | Table 12: Default architectural settings of RETR, including UASC and GSD. |
| Split Sequences Samples Lane instances Training 33 19,048 52,422 Validation 6 4,749 12,503 Test 13 6,359 17,821 Total 52 30,156 82,746 | Symbol Description Value High-resolution structural stages in UASC 2 Initial lane proposals 12 Activation maps per proposal 4 Ordered anchors per proposal 20 Hidden dimension 256 Decoder layers 6 Inquiry branches per decoder layer 2 |
| Table 13: Expert assessment of DSEC-3DLD annotations, reporting mean scores (1–5) and quality-control pass rates. | Table 14: Comparison of RETR input variants on DSEC-3DLD, including RGB-only, event-only, RGB-event, and noisy-event configurations. | Table 15: Event window ablation on DSEC-3DLD with other settings fixed. |
| Criterion Score Pass (%) 2D annotation 4.62 96.4 3D lane 4.41 93.7 Scenario label 4.70 97.8 Overall 4.57 95.9 | Method Input F1-score (%) LATR [ 35 ] RGB 69.2 RETR-RGB RGB 72.5 RETR-Event Event 64.8 RETR-Noisy RGB + noisy event 74.6 RETR RGB + event 75.6 | Accum. window F1 (%) 25 ms 74.9 50 ms 75.6 100 ms 75.0 |
| Method | Params (M) | MACs (G) | Preproc. (ms) | Network (ms) | F1-score (%) |
| RETR-RGB | 83.56 | 117.6 | 69.9 | 65.03 | 72.5 |
| RETR-Event | 84.93 | 119.2 | 78.5 | 64.92 | 64.8 |
| RETR | 95.47 | 148.6 | 80.9 | 81.26 | 75.6 |