Polis: 3D Self-Supervision at City Scale

Alexander Rusnak; Sophia Kovalenko; Jingru Wang; Ismail Moudden; Xiru Wang; Frédéric Kaplan

cs.CVAug 29, 2026

Polis: 3D Self-Supervision at City Scale

Authors: Alexander Rusnak, Sophia Kovalenko, Jingru Wang, Ismail Moudden, Xiru Wang, Frédéric Kaplan

Organizations: École Polytechnique Fédérale de Lausanne

Abstract

Reliable semantic representations derived from city-scale 3D models are increasingly important for urban analysis, infrastructure monitoring, autonomous systems, and heritage conservation. However, urban scenes of large spatial extent captured through aerial surveying differ substantially from the indoor, object-level, and self-driving LiDAR data used to pretrain most 3D self-supervised models. We introduce Polis, to our knowledge the first application of Sketched Isotropic Gaussian Regularization (SIGReg) as an objective for a native point cloud encoder, and evaluate it through a frozen-feature benchmark spanning fourteen city- and building-scale corpora. Polis combines geometrically matched cosine invariance, SIGReg, and VICReg-style anti-collapse terms with a 12.8k-scene outdoor pretraining mixture and gravity-preserving spatial view sampling. Controlled ablations show that this objective outperforms student--teacher architecture alternatives, as well as Polis versions without anti-collapse terms, on the same representative outdoor corpus. On three pretraining-disjoint city datasets, Polis reaches $23.8\%$ mean mIoU versus $16.3\%$ for the next-best encoder under high-capacity frozen probing, and $17.3\%$ versus $16.1\%$ at a matched point and voxel budget. The same city-scale lead holds on datasets whose training sets were seen in pretraining. On localized terrestrial captures with fine-grained facade and streetscape labels, the ranking reverses. Our results show that distributionally-regularized joint embedding architectures can be successful on challenging city-scale 3D scenes, and that transfer improves when self-supervision is designed for the capture geometry and spatial context of this domain while also revealing the limits of this specialization.

Figures & tables

Figure 1: High-capacity frozen-probe mIoU on the three pretraining-disjoint city-scale datasets (UNS-Geo, HRHD-HK, and WHU-Urban3D) versus median whole-scene occupied-voxel capacity. Matched-budget scores and the remaining eleven corpora are in Sec. 4.3 . Polis combines the strongest held-out city transfer with the largest typical capacity.

Figure 2: Frozen-backbone feature PCA on a Zurich scene from Swiss3DCities, shown from a matched oblique view. Top: RGB and Polis (ours) . Bottom: Sonata, Concerto, and Utonia. All models use their high-budget point/voxel capacity. Polis produces more semantically and spatially coherent outdoor feature structure in this example, while the baselines exhibit weaker separation or visible spatial artifacts.

Figure 3: Polis training stack. An area crop from a sampled outdoor scene (or a persistent spatial chunk of a large scene) yields one global and two local spatial views (origin coordinates retained for matching), which receive independent gravity-preserving geometric and chromatic augmentations (differential yaw/tilt, elastic distortion, cuboid and color dropout, jitter/scale, and flips) before a shared LitePT-L encoder. Training combines matched cosine invariance $\mathcal{L}_{\mathrm{inv}}$ over origin-coordinate nearest-neighbor pairs with SIGReg and VICReg-style variance/covariance terms on the global and concatenated-local feature packs ( Eq. 1 ).

Model	Assets	Ep. / bs / GPUs	DINOv2	Domain
Sonata [ 46 ]	$\sim$ 140k scenes	200 / 96 / 32	no	indoor+sim
Concerto [ 54 ]	$\sim$ 40k clouds + 300k img	100 / – / 16 H20	yes	indoor
Utonia [ 55 ]	$\sim$ 250k + 1M CAD	100 † / 256 / 64 H20	yes	cross-dom.
Polis	$12.8$ k (22 sets)	50 ‡ / 512 / 128	no	outdoor/city

Table 1: Pretraining scale, schedule, and whether a pretrained DINOv2 image tower is used for bootstrapping representation quality. Baseline asset and schedule figures are from the cited papers; Polis figures are from the headline training run.

Figure 4: High-cap frozen-probe mIoU across 14 corpora (dark gray: cross-dataset mean; light gray: pretraining-disjoint held-out sets; top rule: building-scale corpora, the rest city-scale). Caps: Polis $1.5\mathrm{M}/1\mathrm{M}$ , Sonata $1.0\mathrm{M}/655\mathrm{k}$ , Concerto $600\mathrm{k}/393\mathrm{k}$ , Utonia $800\mathrm{k}/524\mathrm{k}$ . Polis leads on broad city-scale scenes; Utonia retains the building-scale edge.

City-scale
Dataset	Polis	Sonata	Concerto	Utonia
DALES [ 43 ]	23.5	9.0	8.2	15.3
Swiss3DCities [ 10 ]	20.7	13.4	17.4	14.7
Turin3D [ 4 ]	32.8	25.1	23.3	29.2
SUM [ 19 ]	28.2	20.8	11.3	16.3
YTU3D [ 6 ]	5.6	3.0	2.7	5.2

Table 3: Matched-budget, native-preprocessing mIoU (%) at 400k points / 262k voxels (10 probe epochs, best validation Acc. when available, otherwise best training Acc.). A superscript † marks a pretraining-disjoint held-out dataset; SensatUrban is validation-only because public test labels are unavailable.

Figure 5: Top-down frozen-backbone feature PCA on a Zug scene from Swiss3DCities under the same max-fill qualitative encode protocol as Fig. 2 using $k{=}32$ feature smoothing and 1-NN color fill for unsampled points. Labels use saturated categorical colors as a per-scene key, not a cross-model encoding. Sonata shows center color bias; Utonia shows axis-aligned patches. These visual artifacts, along with the clear height direction artifacts present in Fig. 2 for Utonia and Polis, motivate testing spatial layout and gravity-related cues as shortcuts in Sec. 4.5 .

Figure 6: Radial mIoU and geometric structure of frozen features. (a) Locked high-capacity one-shot test mIoU vs. XY-radius ring (macro-averaged over DALES, Swiss3DCities, SUM). (b) Feature variance explained by $z$ , $z_{\mathrm{rel}}$ , and $r_{xy}$ . (c) Ridge mIoU before/after residualizing height cues. (d) Shared building/vegetation apex-minus-body direction. Polis encodes stronger gravity-aligned structure, but its city advantage does not reduce to these measured directions.

Figure 7: Frozen-feature density sweep on four scenes from each of our corpora. Left: linear CKA vs. the $400\mathrm{k}$ -point encoding. Right: closed-form ridge probe mIoU. Polis moves more under decimation than Sonata / Concerto / Utonia while remaining competitive on ridge mIoU. Polis is more sensitive to decimation, yet demonstrates competitive transfer at practical point budgets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

City-scale
Dataset	Random-init	Polis
DALES [ 43 ]	5.7	23.5
Swiss3DCities [ 10 ]	6.8	20.7
Turin3D [ 4 ]	4.2	32.8
SUM [ 19 ]	7.0	28.2
YTU3D [ 6 ]	0.9	5.6

Appendix

Table 4: Matched-budget frozen-probe mIoU (%). Random-init is an untrained LitePT-L; Polis is the headline pretrained checkpoint. A superscript † marks a pretraining-disjoint held-out dataset; SensatUrban is validation mIoU because public test labels are unavailable.

City-scale
Dataset	Polis	Sonata	Concerto	Utonia
DALES [ 43 ]	28.0	11.8	10.3	15.4
Swiss3DCities [ 10 ]	23.7	10.2	11.6	15.9
Turin3D [ 4 ]	39.9	34.4	25.7	41.0
SUM [ 19 ]	37.7	24.6	15.9	18.0
YTU3D [ 6 ]	6.7	3.8	3.1	5.7

Appendix

Table 5: High-capacity, native-preprocessing mIoU (%) with model-specific one-shot caps (10 probe epochs, best validation Acc. when available, otherwise best training Acc.). A superscript † marks a pretraining-disjoint held-out dataset; SensatUrban is validation-only because public test labels are unavailable.

Parameter	Value
Grid size	$0.02\,\mathrm{m}$
Views	1 global + 2 local
Global / local scales	$(0.4,1.0)$ / $(0.02,0.25)$
Points per view	$100\mathrm{k}$ (headline run)
$\lambda$ (SIGReg)	$5{\times}10^{-4}$
$w_{\mathrm{f}}/w_{\mathrm{c}}$ (variance weights)	$0.3$ / $0.3$

Appendix

Table 6: Hyperparameters for Polis pretraining and frozen probing. The probe checkpoint, data sampling, objective weights, and resource settings are reported to make the training and evaluation protocol reproducible.

Point Cloud Learning

Point Cloud Learning develops methods for representing and interpreting unordered 3D point sets from LiDAR, depth sensors, radar, and 3D models.

See all 159 papers

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
Mar 24, 2026Yiping Chen, Jinpeng Li, Wenyu Ke +63D VLMs 3D Scene Understanding
School of Geospatial Engineering and Science, Sun Yat-sen University, Zhuhai, 519082, China. · College of Electronic Science, National University of Defense Technology, Changsha, 410000, China. · Department of Civil and Environmental Engineering, Norwegian University of Science and Technology, Trondheim, Norway. +1
Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds
Mar 26, 2026Bin Yang, Mohamed Abdelsamad, Miao Zhang +1Self-Supervised Point Cloud Learning Point Cloud Learning
Bosch Center for AI, Stuttgart, Germany · Institute for Neuro- and Bioinformatics, University of Lübeck, Lübeck, Germany · Robert Learning Lab, University of Freiburg, Freiburg, Germany
Utonia: Toward One Encoder for All Point Clouds
Mar 3, 2026Yujia Zhang, Xiaoyang Wu, Yunhan Yang +63D Foundation Models 3D ViTs
The University of Hong Kong · Xiaomi · The Chinese University of Hong Kong

Mar 24, 2026J/K move · Enter open · S save

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

Yiping Chen, Jinpeng Li, Wenyu Ke, Yang Luo+5

School of Geospatial Engineering and Science, Sun Yat-sen University, Zhuhai, 519082, China. · College of Electronic Science, National University of Defense Technology, Changsha, 410000, China. · Department of Civil and Environmental Engineering, Norwegian University of Science and Technology, Trondheim, Norway.+1

3D VLMs 3D Scene Understanding Multimodal Large Language Models 3D VQA

Open paper AskPDF arXiv

While multi-modality large language models excel in object-centric or indoor scenarios, scaling them to 3D city-scale environments remains a formidable challenge. To bridge this gap, we propose 3DCity-LLM, a unified framework designed for 3D city-scale vision-language perception and understanding. 3DCity-LLM employs a coarse-to-fine feature encoding strategy comprising three parallel branches for target object, inter-object relationship, and global scene. To facilitate large-scale training, we introduce 3DCity-LLM-1.2M dataset that comprises approximately 1.2 million high-quality samples across seven representative task categories, ranging from fine-grained object analysis to multi-faceted scene planning. This strictly quality-controlled dataset integrates explicit 3D numerical information and diverse user-oriented simulations, enriching the question-answering diversity and realism of urban scenarios. Furthermore, we apply a multi-dimensional protocol based on text-similarity metrics and LLM-based semantic assessment to ensure faithful and comprehensive evaluations for all methods. Extensive experiments on two benchmarks demonstrate that 3DCity-LLM significantly outperforms existing state-of-the-art methods, offering a promising and meaningful direction for advancing spatial reasoning and urban intelligence. The source code and dataset are available at https://github.com/SYSU-3DSTAILab/3D-City-LLM.

Abstract

Point Cloud Learning

Explore similar work

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds

Utonia: Toward One Encoder for All Point Clouds