Failure Modes for Deep Learning-Based Online Mapping: How to Measure and Address Them

Michael Hubbertz; Qi Han; Tobias Meisen

cs.CVMar 20, 2026

Failure Modes for Deep Learning-Based Online Mapping: How to Measure and Address Them

Authors: Michael Hubbertz, Qi Han, Tobias Meisen

Organizations: University of Wuppertal · Aptiv Services Deutschland GmbH

Abstract

Deep learning-based online mapping has emerged as a cornerstone of autonomous driving, yet these models frequently fail to generalize beyond familiar environments. We propose a framework to identify and measure the underlying failure modes by disentangling two effects: Memorization of input features and overfitting to known map geometries. We propose measures based on evaluation subsets that control for geographical proximity and geometric similarity between training and validation scenes. We introduce Fréchet distance-based reconstruction statistics that capture per-element shape fidelity without threshold tuning, and define complementary failure-mode scores: a localization overfitting score quantifying the performance drop when geographic cues disappear, and a map geometry overfitting score measuring degradation as scenes become geometrically novel. Beyond models, we analyze dataset biases and contribute map geometry-aware diagnostics: A minimum-spanning-tree (MST) diversity measure for training sets and a symmetric coverage measure to quantify geometric similarity between splits. Leveraging these, we formulate an MST-based sparsification strategy that reduces redundancy and improves balancing and performance while shrinking training size. Experiments on nuScenes and Argoverse 2 across multiple state-of-the-art models yield more trustworthy assessment of generalization and show that map geometry-diverse and balanced training sets lead to improved performance. Our results motivate failure-mode-aware protocols and map geometry-centric dataset design for deployable online mapping.

Figures & tables

Figure 1: Validation set performance of different state-of-the-art online mapping models on the nuScenes dataset [ 2 ] with original and two different geographically disjoint [ 24 , 42 ] splits.

Figure 2: Correlation between $d(v)$ and $s(v)$ for the nuScenes original split (Pearson correlation coefficient $r=0.724$ ). Three representative pairs of $s(v)$ are presented, showing each validation sample alongside its closest geometric match from the training set.

Figure 3: Visualization of two exemplary prediction and ground truth map element pairs. The Chamfer distance (red) and Fréchet distance (green) are shown as performance metrics for both cases. In example (a), both metrics produce meaningful results. In example (b), the Chamfer distance remains nearly unchanged compared to (a) because it ignores point ordering, whereas the Fréchet distance yields a much higher value, capturing the larger geometric deviation.

Figure 4: Sample-wise performance of MapTRv2 measured by $M$ for the nuScenes original split, plotted over $d(v)$ (left) and $s(v)$ (right). In the left plot, an exemplary $T_{\text{dist}}=5\,m$ is displayed, separating the validation set $V$ into $V_{\text{close}}$ and $V_{\text{far}}$ . Performance is positively correlated with both $d(v)$ and $s(v)$ , with the correlation being stronger for $s(v)$ (Pearson correlation coefficient $r=0.568>0.379$ ).

Dataset and split		$\mathrm{geomdiv}(T)$	$\mathrm{geomdiv}(V)$	$\mathrm{geomsim}(T,V)$	$d(v)<5\,m$	$\text{mAP}\uparrow$	$M^{\pm IQR}\downarrow$	$\mathcal{O}_{\text{loc}}\downarrow$	$\mathcal{O}_{\text{geom}}\downarrow$
nuScenes	original	96.8 km	30.1 km	8.32 m	79.47 %	60.95	$1.94^{\pm 3.05}$	24.73	21.22
	geo. [ 24 ]	80.6 km	22.5 km	14.66 m	0.95 %	24.96	$4.07^{\pm 6.14}$	n.a.	9.75
	geo. [ 42 ]	90.2 km	22.4 km	13.85 m	0 %	28.53	$3.24^{\pm 5.50}$	n.a.	13.84
	geometric	91.3 km	10.7 km	21.08 m	8.53 %	28.37	$4.17^{\pm 6.08}$	4.40	10.49
Argoverse 2	original*	91.0 km	23.7 km	8.98 m	44.89 %	63.97	$1.77^{\pm 2.99}$	7.29	11.17
	geo. [ 24 ] *	87.3 km	19.8 km	11.37 m	0 %	49.53	$2.40^{\pm 4.18}$	n.a.	11.97

Table 1: Comparison of original, geographically disjoint [ 24 , 42 ] and geometric splits for nuScenes and Argoverse 2. Dataset split geometric diversities

(\mathrm{geomdiv}

) and similarities (

\mathrm{geomsim}

) are listed for training set

T

and validation set

V

as well as their geographical overlap (

d(v)<5\,m

). For the dataset properties, the Argoverse 2 splits (*) were subsampled from 10 Hz to 2 Hz to enable comparison with nuScenes. Performance and overfitting measures are provided for MapTRv2 [ 23 ] .

	Localization Overfitting Score $\mathcal{O}_{\text{loc}}$ (%) $\downarrow$		Geometric Overfitting Score $\mathcal{O}_{\text{geom}}$ (%) $\downarrow$
	nuScenes	Argoverse 2	nuScenes splits			Argoverse 2 splits
Model	original split	original split	original	geo. [ 24 ]	geo. [ 42 ]	original	geo. [ 24 ]	geo. [ 42 ]
MapTR [ 22 ]	24.42	22.06	18.66	13.55	18.66	10.69	22.43	13.86
MapTRv2 [ 23 ]	24.73	7.29	21.22	9.75	13.84	11.17	11.97	11.15
MapQR [ 27 ]	57.07	13.91	21.03	10.87	19.48	10.73	11.37	9.08
MGMap [ 25 ]	33.19	22.06	24.12	14.06	17.42	10.69	23.50	24.47

Table 2:

\mathcal{O}_{\text{loc}}

and

\mathcal{O}_{\text{geom}}

for different state-of-the-art online mapping models on nuScenes and Argoverse 2 datasets with original and geographically disjoint splits [ 24 , 42 ] and

T_{\text{dist}}=5\;m

to separate between

V_{\text{close}}

and

V_{\text{far}}

Figure 5: Effect of MST-based training set sparsification on sample size and diversity (Top) and model performance for MapTRv2 on the validation set (Bottom). Besides examining the effect across all examined nuScenes and Argoverse 2 splits, we randomly sample from the training sets of the original splits for comparison.

Figure 7: Visualization of the topology-based similarity measure $\mathrm{sim}_{\mathrm{topo}}(v,t)$ and its computation steps. Candidate rigid transformations are derived via class and map element-wise Procrustes analysis and discrete Fréchet distance matching. For each transformed training sample, similarity is evaluated within the overlapping field-of-view and penalized for low overlap. The final similarity is obtained by selecting the minimum across all candidates.

Figure 8: Correlation between the topology-based similarity $s_{\mathrm{topo}}(v)$ and the geographical distance $d(v)$ for the nuScenes original split. Compared to the geometric similarity $s(v)$ , the topology-based score exhibits a stronger correlation and a less cluttered distribution (Pearson correlation coefficient $r=0.785>0.724$ , cf. Fig. 2 ).

Figure 9: Comparison of geometric similarity $s(v)$ and topology-based similarity $s_{\mathrm{topo}}(v)$ for the nuScenes original split. Samples deviating from the diagonal most often satisfy $s(v)>s_{\mathrm{topo}}(v)$ , demonstrating that $s_{\mathrm{topo}}(v)$ identifies topologically similar scenes that differ due to translation or rotation. Two exemplary validation samples are displayed along their best matches from the training set for $s(v)$ and $s_{\mathrm{topo}}(v)$ .

Figure 10: Correlation between the topology-based similarity $s_{\mathrm{topo}}(v)$ and the per-sample performance $M(v)$ on the nuScenes original split. The slightly weaker correlation compared to $s(v)$ (Pearson correlation coefficient $r=0.552<0.568$ , cf. Fig. 4 ) indicates that current models rely more strongly on geometric patterns than on topological structure.

Dataset and split		$r(s(v),M(v))$	$r(s_{\text{topo}}(v),M(v))$
nuScenes	original	0.568	0.552
	geo. [ 24 ]	0.226	0.211
	geo. [ 42 ]	0.275	0.270
	geometric	0.137	0.171
Argoverse 2	original	0.392	0.298
	geo. [ 24 ]	0.433	0.366

Table 3: Pearson correlation between

M(v)

and geometric similarity

s(v)

versus topology-based similarity

s_{\mathrm{topo}}(v)

across all examined dataset splits. While

s(v)

correlates more strongly with performance for original and geographical splits,

s_{\mathrm{topo}}(v)

shows mildly higher correlation for the geometric splits, where geometric similarity is intentionally minimized.

Sparsification Threshold	Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
–		28130	(100.00 %)		96828.00 m	(100.00 %)		60.95	( $\pm$ 0.00 %)
0.1		23084	(82.06 %)		96815.33 m	(99.99 %)		61.47	(+0.85 %)
0.2		22752	(80.88 %)		96805.15 m	(99.98 %)		61.56	(+1.00 %)
0.5		21417	(76.14 %)		96613.71 m	(99.78 %)		61.48	(+0.87 %)
1		18397	(65.40 %)		95585.00 m	(98.72 %)		59.71	(-2.03 %)
2		13969	(49.66 %)		92051.27 m	(95.07 %)		56.71	(-6.96 %)

Table 4: Original nuScenes split

Sparsification Threshold	Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
–		27840	(100.00 %)		80572.21 m	(100.00 %)		24.96	( $\pm$ 0.00 %)
0.1		22815	(81.95 %)		80559.35 m	(99.98 %)		25.73	(+3.08 %)
0.2		22427	(80.56 %)		80543.07 m	(99.96 %)		25.48	(+2.08 %)
0.5		20381	(73.21 %)		80197.95 m	(99.54 %)		25.64	(+2.72 %)
1		16515	(59.32 %)		78843.58 m	(97.85 %)		26.46	(+6.01 %)
2		11970	(43.00 %)		74909.77 m	(92.97 %)		25.07	(+0.44 %)

Table 5: Geographically disjoint nuScenes split from [ 24 ] (Near Extrapolation Split)

Sparsification Threshold	Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
–		28008	(100.00 %)		90172.57 m	(100.00 %)		28.53	( $\pm$ 0.00 %)
0.1		22801	(81.41 %)		90159.43 m	(99.99 %)		27.56	(-3.40 %)
0.2		22451	(80.16 %)		90148.35 m	(99.97 %)		28.44	(-0.32 %)
0.5		20840	(74.41 %)		89897.52 m	(99.69 %)		27.19	(-4.70 %)
1		17414	(62.18 %)		88718.11 m	(98.39 %)		27.95	(-2.03 %)
2		12825	(45.79 %)		85090.37 m	(94.36 %)		28.19	(-1.19 %)

Table 6: Geographically disjoint nuScenes split from [ 42 ]

Sparsification Threshold	Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
–		22279	(100.00 %)		91003.65 m	(100.00 %)		63.97	( $\pm$ 0.00 %)
0.1		19892	(89.29 %)		90995.80 m	(99.99 %)		64.32	(+0.55 %)
0.2		19580	(87.89 %)		90991.00 m	(99.99 %)		64.97	(+1.56 %)
0.5		18633	(83.63 %)		90909.49 m	(99.90 %)		63.71	(-0.41 %)
1		17159	(77.02 %)		90555.71 m	(99.51 %)		64.22	(+0.39 %)
2		14440	(64.81 %)		88289.72 m	(97.02 %)		63.40	(-0.89 %)

Table 7: Original Argoverse 2 split, remaining samples in

T

and

\mathrm{geomdiv}(T)

are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.

Sparsification Threshold	Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
–		22223	(100.00 %)		87302.62 m	(100.00 %)		49.53	( $\pm$ 0.00 %)
0.1		19777	(88.99 %)		87294.12 m	(99.99 %)		50.47	(+1.90 %)
0.2		19454	(87.54 %)		87289.24 m	(99.98 %)		50.13	(+1.21 %)
0.5		18445	(83.00 %)		87188.20 m	(99.87 %)		49.50	(-0.06 %)
1		16731	(75.29 %)		86747.95 m	(99.36 %)		50.18	(+1.31 %)
2		13827	(62.22 %)		84389.69 m	(96.66 %)		49.78	(+0.50 %)

Table 8: Geographically disjoint Argoverse 2 split from [ 24 ] (Near Extrapolation Split), remaining samples in

T

and

\mathrm{geomdiv}(T)

are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.

Sparsification Threshold	Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
–		23954	(100.00 %)		97153.16 m	(100.00 %)		57.61	( $\pm$ 0.00 %)
0.1		21391	(89.30 %)		97144.47 m	(99.99 %)		57.55	(-0.10 %)
0.2		21077	(87.99 %)		97138.62 m	(99.99 %)		57.74	(+0.23 %)
0.5		20106	(83.94 %)		97050.71 m	(99.89 %)		58.36	(+1.30 %)
1		18403	(76.83 %)		96625.41 m	(99.46 %)		57.88	(+0.47 %)
2		15348	(64.07 %)		93994.73 m	(96.75 %)		56.75	(-1.49 %)

Table 9: Geographically disjoint Argoverse 2 split from [ 42 ] , remaining samples in

T

and

\mathrm{geomdiv}(T)

are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.

Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
	28130	(100.00 %)		96828.00 m	(100.00 %)		60.95	( $\pm$ 0.00 %)
	23084	(82.06 %)		87906.66 m	(90.79 %)		59.79	(-1.90 %)
	22752	(80.88 %)		87715.78 m	(90.59 %)		59.50	(-2.38 %)
	21417	(76.14 %)		85104.96 m	(87.89 %)		59.36	(-2.61 %)
	18397	(65.40 %)		78757.05 m	(81.34 %)		58.48	(-4.05 %)
	13969	(49.66 %)		68450.78 m	(70.69 %)		56.79	(-6.83 %)

Table 10: Random sampling for remaining sample amounts from Tab. 4 from the original nuScenes split

Remaining Samples in $T$			$\mathrm{geomdiv}(T)$			Performance on $V$ (mAP)
	22279	(100.00 %)		91003.65 m	(100.00 %)		63.97	( $\pm$ 0.00 %)
	19892	(89.29 %)		86989.97 m	(95.59 %)		63.79	(-0.28 %)
	19580	(87.89 %)		86184.47 m	(94.70 %)		63.67	(-0.47 %)
	18633	(83.63 %)		84021.93 m	(92.33 %)		63.06	(-1.42 %)
	17159	(77.02 %)		80187.60 m	(88.11 %)		63.95	(-0.03 %)
	14440	(64.81 %)		73394.62 m	(80.65 %)		62.34	(-2.55 %)

Table 11: Random sampling for remaining sample amounts from Tab. 7 from the original Argoverse 2 split, remaining samples in

T

and

\mathrm{geomdiv}(T)

are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.

OOD Generalization

Out-of-distribution generalization studies how machine-learning models perform when test data, domains, tasks, or environments differ from training.

See all 395 papers

Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping
May 21, 2026Chouaib Bencheikh Lehocine, Adam Lilja, Junsheng Fu +1Autonomous Driving Perception
1Zenseact AB · 2Chalmers University of Technology Gothenburg, Sweden
Auditing Latent-Space Monitors for Autonomous Driving
Sep 24, 2026Nikhil Kamalkumar Advani, Vishwajeet Shivaji Hogale, Saurav KumarAutonomous Driving Safety Evaluation Autonomous Driving Planning
Independent Researcher · Noertheastern University · Bosch North America
Fail2Drive: Benchmarking Closed-Loop Driving Generalization
Apr 9, 2026Simon Gerstenecker, Andreas Geiger, Katrin RenzAutonomous Driving Benchmarks Autonomous Driving
University of Tübingen Tübingen AI Center

May 21, 2026J/K move · Enter open · S save

Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping

Chouaib Bencheikh Lehocine, Adam Lilja, Junsheng Fu, Lars Hammarstrand

1Zenseact AB · 2Chalmers University of Technology Gothenburg, Sweden

Autonomous Driving Perception

Open paper AskPDF arXiv

Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (SOTA) methods commonly predict map elements as ordered sequences of points that form polylines and polygons. The evaluation of these methods relies predominantly on mean average precision (mAP) based on thresholded Chamfer distance (CD). This framework lacks sensitivity to point ordering and provides limited granularity in assessing geometric quality, making it difficult to distinguish which methods truly excel over others. In this work, we address these limitations on two fronts. For the single-instance similarity measure, we introduce sequence optimal sub-pattern assignment (SOSPA), an order-aware metric that enables fine-grained evaluation of individual geometries while satisfying all metric axioms. For the multi-instance evaluation framework, we propose polyline localisation and detection (PLD), a soft metric that jointly captures detection quality and geometric accuracy, replacing the hard thresholding of mAP with a principled soft assignment. Through evaluations on nuScenes, we demonstrate that PLD effectively ranks SOTA online mapping methods (MapTRv2, StreamMapNet, MapTracker) while providing a decomposed error analysis. This analysis identifies detection capability as the dominant bottleneck in current methods, revealing a performance trend that mAP fails to capture. Code for evaluation using our metrics will be released.

Abstract

OOD Generalization

Explore similar work

Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping

Auditing Latent-Space Monitors for Autonomous Driving

Fail2Drive: Benchmarking Closed-Loop Driving Generalization