Deep learning-based online mapping has emerged as a cornerstone of autonomous driving, yet these models frequently fail to generalize beyond familiar environments. We propose a framework to identify and measure the underlying failure modes by disentangling two effects: Memorization of input features and overfitting to known map geometries. We propose measures based on evaluation subsets that control for geographical proximity and geometric similarity between training and validation scenes. We introduce Fréchet distance-based reconstruction statistics that capture per-element shape fidelity without threshold tuning, and define complementary failure-mode scores: a localization overfitting score quantifying the performance drop when geographic cues disappear, and a map geometry overfitting score measuring degradation as scenes become geometrically novel. Beyond models, we analyze dataset biases and contribute map geometry-aware diagnostics: A minimum-spanning-tree (MST) diversity measure for training sets and a symmetric coverage measure to quantify geometric similarity between splits. Leveraging these, we formulate an MST-based sparsification strategy that reduces redundancy and improves balancing and performance while shrinking training size. Experiments on nuScenes and Argoverse 2 across multiple state-of-the-art models yield more trustworthy assessment of generalization and show that map geometry-diverse and balanced training sets lead to improved performance. Our results motivate failure-mode-aware protocols and map geometry-centric dataset design for deployable online mapping.
Figures & tables
Figure 1: Validation set performance of different state-of-the-art online mapping models on the nuScenes dataset [ 2 ] with original and two different geographically disjoint [ 24 , 42 ] splits.
Figure 2: Correlation between d(v) and s(v) for the nuScenes original split (Pearson correlation coefficient r=0.724 ). Three representative pairs of s(v) are presented, showing each validation sample alongside its closest geometric match from the training set.
Figure 3: Visualization of two exemplary prediction and ground truth map element pairs. The Chamfer distance (red) and Fréchet distance (green) are shown as performance metrics for both cases. In example (a), both metrics produce meaningful results. In example (b), the Chamfer distance remains nearly unchanged compared to (a) because it ignores point ordering, whereas the Fréchet distance yields a much higher value, capturing the larger geometric deviation.
Figure 4: Sample-wise performance of MapTRv2 measured by M for the nuScenes original split, plotted over d(v) (left) and s(v) (right). In the left plot, an exemplary Tdist=5m is displayed, separating the validation set V into Vclose and Vfar . Performance is positively correlated with both d(v) and s(v) , with the correlation being stronger for s(v) (Pearson correlation coefficient r=0.568>0.379 ).
Dataset and split
geomdiv(T)
geomdiv(V)
geomsim(T,V)
d(v)<5m
mAP↑
M±IQR↓
Oloc↓
Ogeom↓
nuScenes
original
96.8 km
30.1 km
8.32 m
79.47 %
60.95
1.94±3.05
24.73
21.22
geo. [ 24 ]
80.6 km
22.5 km
14.66 m
0.95 %
24.96
4.07±6.14
n.a.
9.75
geo. [ 42 ]
90.2 km
22.4 km
13.85 m
0 %
28.53
3.24±5.50
n.a.
13.84
geometric
91.3 km
10.7 km
21.08 m
8.53 %
28.37
4.17±6.08
4.40
10.49
Argoverse 2
original*
91.0 km
23.7 km
8.98 m
44.89 %
63.97
1.77±2.99
7.29
11.17
geo. [ 24 ] *
87.3 km
19.8 km
11.37 m
0 %
49.53
2.40±4.18
n.a.
11.97
Table 1: Comparison of original, geographically disjoint [ 24 , 42 ] and geometric splits for nuScenes and Argoverse 2. Dataset split geometric diversities (geomdiv ) and similarities ( geomsim ) are listed for training set T and validation set V as well as their geographical overlap ( d(v)<5m ). For the dataset properties, the Argoverse 2 splits (*) were subsampled from 10 Hz to 2 Hz to enable comparison with nuScenes. Performance and overfitting measures are provided for MapTRv2 [ 23 ] .
Localization Overfitting Score Oloc (%) ↓
Geometric Overfitting Score Ogeom (%) ↓
nuScenes
Argoverse 2
nuScenes splits
Argoverse 2 splits
Model
original split
original split
original
geo. [ 24 ]
geo. [ 42 ]
original
geo. [ 24 ]
geo. [ 42 ]
MapTR [ 22 ]
24.42
22.06
18.66
13.55
18.66
10.69
22.43
13.86
MapTRv2 [ 23 ]
24.73
7.29
21.22
9.75
13.84
11.17
11.97
11.15
MapQR [ 27 ]
57.07
13.91
21.03
10.87
19.48
10.73
11.37
9.08
MGMap [ 25 ]
33.19
22.06
24.12
14.06
17.42
10.69
23.50
24.47
Table 2: Oloc and Ogeom for different state-of-the-art online mapping models on nuScenes and Argoverse 2 datasets with original and geographically disjoint splits [ 24 , 42 ] and Tdist=5m to separate between Vclose and Vfar .
Figure 5: Effect of MST-based training set sparsification on sample size and diversity (Top) and model performance for MapTRv2 on the validation set (Bottom). Besides examining the effect across all examined nuScenes and Argoverse 2 splits, we randomly sample from the training sets of the original splits for comparison.
Figure 7: Visualization of the topology-based similarity measure simtopo(v,t) and its computation steps. Candidate rigid transformations are derived via class and map element-wise Procrustes analysis and discrete Fréchet distance matching. For each transformed training sample, similarity is evaluated within the overlapping field-of-view and penalized for low overlap. The final similarity is obtained by selecting the minimum across all candidates.
Figure 8: Correlation between the topology-based similarity stopo(v) and the geographical distance d(v) for the nuScenes original split. Compared to the geometric similarity s(v) , the topology-based score exhibits a stronger correlation and a less cluttered distribution (Pearson correlation coefficient r=0.785>0.724 , cf. Fig. 2 ).
Figure 9: Comparison of geometric similarity s(v) and topology-based similarity stopo(v) for the nuScenes original split. Samples deviating from the diagonal most often satisfy s(v)>stopo(v) , demonstrating that stopo(v) identifies topologically similar scenes that differ due to translation or rotation. Two exemplary validation samples are displayed along their best matches from the training set for s(v) and stopo(v) .
Figure 10: Correlation between the topology-based similarity stopo(v) and the per-sample performance M(v) on the nuScenes original split. The slightly weaker correlation compared to s(v) (Pearson correlation coefficient r=0.552<0.568 , cf. Fig. 4 ) indicates that current models rely more strongly on geometric patterns than on topological structure.
Dataset and split
r(s(v),M(v))
r(stopo(v),M(v))
nuScenes
original
0.568
0.552
geo. [ 24 ]
0.226
0.211
geo. [ 42 ]
0.275
0.270
geometric
0.137
0.171
Argoverse 2
original
0.392
0.298
geo. [ 24 ]
0.433
0.366
Table 3: Pearson correlation between M(v) and geometric similarity s(v) versus topology-based similarity stopo(v) across all examined dataset splits. While s(v) correlates more strongly with performance for original and geographical splits, stopo(v) shows mildly higher correlation for the geometric splits, where geometric similarity is intentionally minimized.
Table 6: Geographically disjoint nuScenes split from [ 42 ]
Sparsification Threshold
Remaining Samples in T
geomdiv(T)
Performance on V (mAP)
–
22279
(100.00 %)
91003.65 m
(100.00 %)
63.97
( ± 0.00 %)
0.1
19892
(89.29 %)
90995.80 m
(99.99 %)
64.32
(+0.55 %)
0.2
19580
(87.89 %)
90991.00 m
(99.99 %)
64.97
(+1.56 %)
0.5
18633
(83.63 %)
90909.49 m
(99.90 %)
63.71
(-0.41 %)
1
17159
(77.02 %)
90555.71 m
(99.51 %)
64.22
(+0.39 %)
2
14440
(64.81 %)
88289.72 m
(97.02 %)
63.40
(-0.89 %)
Table 7: Original Argoverse 2 split, remaining samples in T and geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.
Sparsification Threshold
Remaining Samples in T
geomdiv(T)
Performance on V (mAP)
–
22223
(100.00 %)
87302.62 m
(100.00 %)
49.53
( ± 0.00 %)
0.1
19777
(88.99 %)
87294.12 m
(99.99 %)
50.47
(+1.90 %)
0.2
19454
(87.54 %)
87289.24 m
(99.98 %)
50.13
(+1.21 %)
0.5
18445
(83.00 %)
87188.20 m
(99.87 %)
49.50
(-0.06 %)
1
16731
(75.29 %)
86747.95 m
(99.36 %)
50.18
(+1.31 %)
2
13827
(62.22 %)
84389.69 m
(96.66 %)
49.78
(+0.50 %)
Table 8: Geographically disjoint Argoverse 2 split from [ 24 ] (Near Extrapolation Split), remaining samples in T and geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.
Sparsification Threshold
Remaining Samples in T
geomdiv(T)
Performance on V (mAP)
–
23954
(100.00 %)
97153.16 m
(100.00 %)
57.61
( ± 0.00 %)
0.1
21391
(89.30 %)
97144.47 m
(99.99 %)
57.55
(-0.10 %)
0.2
21077
(87.99 %)
97138.62 m
(99.99 %)
57.74
(+0.23 %)
0.5
20106
(83.94 %)
97050.71 m
(99.89 %)
58.36
(+1.30 %)
1
18403
(76.83 %)
96625.41 m
(99.46 %)
57.88
(+0.47 %)
2
15348
(64.07 %)
93994.73 m
(96.75 %)
56.75
(-1.49 %)
Table 9: Geographically disjoint Argoverse 2 split from [ 42 ] , remaining samples in T and geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.
Remaining Samples in T
geomdiv(T)
Performance on V (mAP)
28130
(100.00 %)
96828.00 m
(100.00 %)
60.95
( ± 0.00 %)
23084
(82.06 %)
87906.66 m
(90.79 %)
59.79
(-1.90 %)
22752
(80.88 %)
87715.78 m
(90.59 %)
59.50
(-2.38 %)
21417
(76.14 %)
85104.96 m
(87.89 %)
59.36
(-2.61 %)
18397
(65.40 %)
78757.05 m
(81.34 %)
58.48
(-4.05 %)
13969
(49.66 %)
68450.78 m
(70.69 %)
56.79
(-6.83 %)
Table 10: Random sampling for remaining sample amounts from Tab. 4 from the original nuScenes split
Remaining Samples in T
geomdiv(T)
Performance on V (mAP)
22279
(100.00 %)
91003.65 m
(100.00 %)
63.97
( ± 0.00 %)
19892
(89.29 %)
86989.97 m
(95.59 %)
63.79
(-0.28 %)
19580
(87.89 %)
86184.47 m
(94.70 %)
63.67
(-0.47 %)
18633
(83.63 %)
84021.93 m
(92.33 %)
63.06
(-1.42 %)
17159
(77.02 %)
80187.60 m
(88.11 %)
63.95
(-0.03 %)
14440
(64.81 %)
73394.62 m
(80.65 %)
62.34
(-2.55 %)
Table 11: Random sampling for remaining sample amounts from Tab. 7 from the original Argoverse 2 split, remaining samples in T and geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.
Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (SOTA) methods commonly predict map elements as ordered sequences of points that form polylines and polygons. The evaluation of these methods relies predominantly on mean average precision (mAP) based on thresholded Chamfer distance (CD). This framework lacks sensitivity to point ordering and provides limited granularity in assessing geometric quality, making it difficult to distinguish which methods truly excel over others. In this work, we address these limitations on two fronts. For the single-instance similarity measure, we introduce sequence optimal sub-pattern assignment (SOSPA), an order-aware metric that enables fine-grained evaluation of individual geometries while satisfying all metric axioms. For the multi-instance evaluation framework, we propose polyline localisation and detection (PLD), a soft metric that jointly captures detection quality and geometric accuracy, replacing the hard thresholding of mAP with a principled soft assignment. Through evaluations on nuScenes, we demonstrate that PLD effectively ranks SOTA online mapping methods (MapTRv2, StreamMapNet, MapTracker) while providing a decomposed error analysis. This analysis identifies detection capability as the dominant bottleneck in current methods, revealing a performance trend that mAP fails to capture. Code for evaluation using our metrics will be released.
Chouaib Bencheikh Lehocine, Adam Lilja, Junsheng Fu +1
1Zenseact AB · 2Chalmers University of Technology Gothenburg, Sweden
Runtime failure monitors can use a model's internal representations to anticipate failures. We audit this monitoring strategy across two autonomous-driving tasks: online vectorized map generation with LaneSegNet and end-to-end planning with VAD. We find that frame-level errors are predictable at inference in both tasks. For LaneSegNet, a supervised latent probe reaches Area Under the Receiver Operating Characteristic curve (AUROC) 0.780 for high Chamfer error; to our knowledge, this is the first post-hoc frame-level failure monitor for online vectorized map generation. For VAD, a supervised planning-latent probe reaches AUROC 0.868 for mean-ADE failure. Our audit shows that internal access is not necessary for strong failure prediction. A monitor using only LaneSegNet's prediction outputs reaches AUROC 0.825, while for VAD, ego state, driving command, and the planner's predicted trajectory reach 0.924 on the same mean-ADE endpoint. Adding latent features to either baseline yields no statistically resolved improvement. This observation persists across a broad suite of planning failure endpoints, including endpoints whose labels depend on geometry unavailable to the non-latent baseline. Thus, predicting failure from an internal representation does not establish that the representation provides useful information beyond observable inputs and outputs. We propose an evaluation protocol for testing the incremental value of latent access and release our per-frame failure endpoint labels.
Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .