Map generation tasks feature extensive non-structural vectorized data (e.g., points, polylines, and polygons) and thus pose significant challenges to common pixel-wise generative models. Conventional approaches use multiple stages, first segmenting these features at the pixel level and then performing vectorized post-processing, with errors and complexity compounding at each stage. Motivated by the recent success of auto-regressive language modeling, we propose the first map foundation model, named Map Auto-Regressor (MARS), that is capable of generating both multi-polyline road networks and polygon buildings in a unified manner. For training MARS, we collected to our knowledge the largest multi-class map extraction dataset totaling 3.4M examples, which we call MAP-3M. Across four road and building datasets, MARS outperforms or matches the performance of multistage baselines. Additionally, we develop a "Chat with MARS" capability that enables interactive human-in-the-loop map generation and correction, supported by the auto-regressive nature of our end-to-end approach. We release our MAP-3M dataset and project demo page at (1) https://huggingface.co/datasets/bag-lab/MAP-3M and (2) https://huggingface.co/spaces/bag-lab/MARS, respectively.
Figures & tables
Figure 1: Examples of rasterized and vectorized map generation. Source: MARS.
Figure 2: Our stroke-based algorithm can decompose a non-sequential flat road networks to multiple single polylines, which can then be sequentialized for auto-regressive learning. Source: MARS.
Figure 3: MARS unifies vectorized map generation by end-to-end visual auto-regressive modeling. We adopt Swin Transformer as vision encoder to extract visual context features. The auto-regressive transformer cross-attends visual context features with map tokens, and then generates the sequences (points, polylines, polygons) in an auto-regressive manner. Benefited from teacher-forced training, we can also support human-in-the-loop map generation with prompts. Image source: MARS.
Figure 4: Chat with MARS . The user interactions are provided as single clicks, shown as yellow arrows. White dash-dotted line: Ground truth. Colored lines with asterisks: Predicted sequences. Black circles with white numbers: Object IDs. (a–b): Start-of-sequence interaction enables recovery of a missing building object (ID 77) in a NAIP image. (c–d): Similarly, a missing road object (ID 1) is recovered in a Cityscale image. (e–f): Mid-sequence interaction helps correct the prediction of a drifting road object (ID 10) in Cityscale. (g–h): Another mid-sequence example shows recovery of a drifting road segment (ID 3) in Cityscale. (i–j): End-of-sequence interaction allows the model to recover a missing road object (ID 6) in a NAIP image. (k–l): End-of-sequence interaction recovers a missing building object (ID 3) in the same NAIP image. Source: MARS.
Figure 5: MAP-3M features (a-b) wide map element density distribution per image, and (c) high-quality annotations with diverse vectorized map geometry coverage. Source: MARS.
Figure 6: MARS visualization. MARS is capable of handling diverse scenes and geometries without any post-processing, demonstrating the generalization and scalability of an end-to-end learning framework. White dash-dotted line: Ground truth. Colored lines with asterisks: Predicted sequences. Black circles with white numbers: Object IDs. Source: MARS.
Classes
MAP3M-VAL
CITYSCALE
SPACENET
AICROWD
P
R
F1
IOU
P
R
F1
P
R
F1
IOU
Road
89.8
85.6
87.7
-
85.1
78.2
81.5
79.3
83.3
81.2
-
Building
-
-
-
64.4
-
-
-
-
-
-
95.0
Both
90.1
77.1
83.1
61.0
84.3
81.5
82.9
79.7
84.6
82.1
97.3
Table 2: MARS architecture unifies single- or multi-classes learning by simply adding class tokens.
Model
CITYSCALE
SPACENET
P
R
F1
P
R
F1
Seg-UNet ( Ronneberger et al., 2015 )
75.34
65.99
70.36
68.96
66.32
67.61
Seg-DRM ( Máttyus et al., 2017 )
76.54
71.25
73.80
82.79
72.56
77.34
Seg-Improved ( Batra et al., 2019 )
75.83
68.90
72.20
81.56
71.38
76.13
Seg-DLA ( Yu et al., 2018 )
75.59
72.26
73.89
78.99
69.80
74.11
RoadTracer ( Bastani et al., 2018 )
78.00
57.44
66.16
78.61
62.45
69.90
Table 3: TOPO-based road performance comparison on Cityscale and SpaceNet.
Model
AICROWD-V1
AP
AP50
AP75
AR
AR50
AR75
bAP
IoU
C-IoU
PoLiS
N-ratio
PolyMapper
55.7
86.0
65.1
62.1
88.6
71.4
22.6
77.6
65.3
2.215
1.29
FFL*
67.0
92.1
75.6
73.2
93.5
81.1
34.4
84.3
73.8
1.945
1.13
PolyWorld
63.3
88.6
70.5
75.4
93.5
83.1
50.0
91.2
88.2
0.962
0.93
PolyBuilding
78.7
96.3
89.2
84.2
97.3
92.9
-
94.0
88.6
-
0.99
HiSup
79.4
92.7
85.3
81.5
93.1
86.7
66.5
94.3
89.6
0.726
-
Table 4: Building performance comparison on Aicrowd V1.
Model
SpaceNet
AICrowd
P
R
F1
IOU
W/O-Pretrain
77.62
64.48
70.45
95.09
W-Pretrain
79.68
84.56
82.05
95.24
Table 5: Effect of pretraining on downstream performance.
Model
CITYSCALE
SPACENET
AICROWD
P
R
F1
P
R
F1
IOU
MARS
84.28
81.53
82.88
79.68
84.56
82.31
97.32
Chat 1pt w MARS
84.95
82.66
83.79
81.40
84.65
82.95
97.35
Chat 2pts w MARS
85.15
83.14
84.13
81.74
85.12
83.17
97.40
Table 6: Performance of Chat with MARS across different datasets.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
AICROWD-V2
AP
AP50
AP75
AR
AR50
AR75
bAP
IoU
C-IoU
PoLiS
N-ratio
MARS
30.91
57.06
30.07
70.03
86.31
73.85
49.84
74.42
66.49
4.135
1.012
Appendix
Table 7: Building performance on Aicrowd V2
Figure 7: MARS visualization. Robust handling of complex road intersections by MARS. White dash-dotted line: Ground truth. Colored lines with asterisks: Predicted sequences. Black circles with white numbers: Object IDs. Source: MARS.
Figure 8: MARS visualization. Robust handling of occluded structures by MARS. White dash-dotted line: Ground truth. Colored lines with asterisks: Predicted sequences. Black circles with white numbers: Object IDs. Source: MARS.
Figure 9: MARS visualization. Difficult examples like (a–c) complex intersections and (d–f) dense canopies occluding roads and buildings where MARS finds it difficult to preserve high accuracy. White dash-dotted line: Ground truth. Colored lines with asterisks: Predicted sequences. Black circles with white numbers: Object IDs. Source: MARS.
Tsinghua University Baidu Beijing, China · University of Macau Macao, China · Institute of Information Engineering, Chinese Academy of Sciences Beijing, China +2