Polis: 3D Self-Supervision at City Scale

Alexander Rusnak; Sophia Kovalenko; Jingru Wang; Ismail Moudden; Xiru Wang; Frédéric Kaplan

cs.CVAug 29, 2026

Polis: 3D Self-Supervision at City Scale

Authors: Alexander Rusnak, Sophia Kovalenko, Jingru Wang, Ismail Moudden, Xiru Wang, Frédéric Kaplan

Organizations: École Polytechnique Fédérale de Lausanne

Abstract

Reliable semantic representations derived from city-scale 3D models are increasingly important for urban analysis, infrastructure monitoring, autonomous systems, and heritage conservation. However, urban scenes of large spatial extent captured through aerial surveying differ substantially from the indoor, object-level, and self-driving LiDAR data used to pretrain most 3D self-supervised models. We introduce Polis, to our knowledge the first application of Sketched Isotropic Gaussian Regularization (SIGReg) as an objective for a native point cloud encoder, and evaluate it through a frozen-feature benchmark spanning fourteen city- and building-scale corpora. Polis combines geometrically matched cosine invariance, SIGReg, and VICReg-style anti-collapse terms with a 12.8k-scene outdoor pretraining mixture and gravity-preserving spatial view sampling. Controlled ablations show that this objective outperforms student--teacher architecture alternatives, as well as Polis versions without anti-collapse terms, on the same representative outdoor corpus. On three pretraining-disjoint city datasets, Polis reaches $23.8\%$ mean mIoU versus $16.3\%$ for the next-best encoder under high-capacity frozen probing, and $17.3\%$ versus $16.1\%$ at a matched point and voxel budget. The same city-scale lead holds on datasets whose training sets were seen in pretraining. On localized terrestrial captures with fine-grained facade and streetscape labels, the ranking reverses. Our results show that distributionally-regularized joint embedding architectures can be successful on challenging city-scale 3D scenes, and that transfer improves when self-supervision is designed for the capture geometry and spatial context of this domain while also revealing the limits of this specialization.

Figures & tables

Figure 1: High-capacity frozen-probe mIoU on the three pretraining-disjoint city-scale datasets (UNS-Geo, HRHD-HK, and WHU-Urban3D) versus median whole-scene occupied-voxel capacity. Matched-budget scores and the remaining eleven corpora are in Sec. 4.3 . Polis combines the strongest held-out city transfer with the largest typical capacity.

Figure 2: Frozen-backbone feature PCA on a Zurich scene from Swiss3DCities, shown from a matched oblique view. Top: RGB and Polis (ours) . Bottom: Sonata, Concerto, and Utonia. All models use their high-budget point/voxel capacity. Polis produces more semantically and spatially coherent outdoor feature structure in this example, while the baselines exhibit weaker separation or visible spatial artifacts.

Figure 3: Polis training stack. An area crop from a sampled outdoor scene (or a persistent spatial chunk of a large scene) yields one global and two local spatial views (origin coordinates retained for matching), which receive independent gravity-preserving geometric and chromatic augmentations (differential yaw/tilt, elastic distortion, cuboid and color dropout, jitter/scale, and flips) before a shared LitePT-L encoder. Training combines matched cosine invariance $\mathcal{L}_{\mathrm{inv}}$ over origin-coordinate nearest-neighbor pairs with SIGReg and VICReg-style variance/covariance terms on the global and concatenated-local feature packs ( Eq. 1 ).

Model	Assets	Ep. / bs / GPUs	DINOv2	Domain
Sonata [ 46 ]	$\sim$ 140k scenes	200 / 96 / 32	no	indoor+sim
Concerto [ 54 ]	$\sim$ 40k clouds + 300k img	100 / – / 16 H20	yes	indoor
Utonia [ 55 ]	$\sim$ 250k + 1M CAD	100 † / 256 / 64 H20	yes	cross-dom.
Polis	$12.8$ k (22 sets)	50 ‡ / 512 / 128	no	outdoor/city

Table 1: Pretraining scale, schedule, and whether a pretrained DINOv2 image tower is used for bootstrapping representation quality. Baseline asset and schedule figures are from the cited papers; Polis figures are from the headline training run.

Figure 4: High-cap frozen-probe mIoU across 14 corpora (dark gray: cross-dataset mean; light gray: pretraining-disjoint held-out sets; top rule: building-scale corpora, the rest city-scale). Caps: Polis $1.5\mathrm{M}/1\mathrm{M}$ , Sonata $1.0\mathrm{M}/655\mathrm{k}$ , Concerto $600\mathrm{k}/393\mathrm{k}$ , Utonia $800\mathrm{k}/524\mathrm{k}$ . Polis leads on broad city-scale scenes; Utonia retains the building-scale edge.

City-scale
Dataset	Polis	Sonata	Concerto	Utonia
DALES [ 43 ]	23.5	9.0	8.2	15.3
Swiss3DCities [ 10 ]	20.7	13.4	17.4	14.7
Turin3D [ 4 ]	32.8	25.1	23.3	29.2
SUM [ 19 ]	28.2	20.8	11.3	16.3
YTU3D [ 6 ]	5.6	3.0	2.7	5.2

Table 3: Matched-budget, native-preprocessing mIoU (%) at 400k points / 262k voxels (10 probe epochs, best validation Acc. when available, otherwise best training Acc.). A superscript † marks a pretraining-disjoint held-out dataset; SensatUrban is validation-only because public test labels are unavailable.

Figure 5: Top-down frozen-backbone feature PCA on a Zug scene from Swiss3DCities under the same max-fill qualitative encode protocol as Fig. 2 using $k{=}32$ feature smoothing and 1-NN color fill for unsampled points. Labels use saturated categorical colors as a per-scene key, not a cross-model encoding. Sonata shows center color bias; Utonia shows axis-aligned patches. These visual artifacts, along with the clear height direction artifacts present in Fig. 2 for Utonia and Polis, motivate testing spatial layout and gravity-related cues as shortcuts in Sec. 4.5 .

Figure 6: Radial mIoU and geometric structure of frozen features. (a) Locked high-capacity one-shot test mIoU vs. XY-radius ring (macro-averaged over DALES, Swiss3DCities, SUM). (b) Feature variance explained by $z$ , $z_{\mathrm{rel}}$ , and $r_{xy}$ . (c) Ridge mIoU before/after residualizing height cues. (d) Shared building/vegetation apex-minus-body direction. Polis encodes stronger gravity-aligned structure, but its city advantage does not reduce to these measured directions.

Figure 7: Frozen-feature density sweep on four scenes from each of our corpora. Left: linear CKA vs. the $400\mathrm{k}$ -point encoding. Right: closed-form ridge probe mIoU. Polis moves more under decimation than Sonata / Concerto / Utonia while remaining competitive on ridge mIoU. Polis is more sensitive to decimation, yet demonstrates competitive transfer at practical point budgets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

City-scale
Dataset	Random-init	Polis
DALES [ 43 ]	5.7	23.5
Swiss3DCities [ 10 ]	6.8	20.7
Turin3D [ 4 ]	4.2	32.8
SUM [ 19 ]	7.0	28.2
YTU3D [ 6 ]	0.9	5.6

Appendix

Table 4: Matched-budget frozen-probe mIoU (%). Random-init is an untrained LitePT-L; Polis is the headline pretrained checkpoint. A superscript † marks a pretraining-disjoint held-out dataset; SensatUrban is validation mIoU because public test labels are unavailable.

City-scale
Dataset	Polis	Sonata	Concerto	Utonia
DALES [ 43 ]	28.0	11.8	10.3	15.4
Swiss3DCities [ 10 ]	23.7	10.2	11.6	15.9
Turin3D [ 4 ]	39.9	34.4	25.7	41.0
SUM [ 19 ]	37.7	24.6	15.9	18.0
YTU3D [ 6 ]	6.7	3.8	3.1	5.7

Appendix

Table 5: High-capacity, native-preprocessing mIoU (%) with model-specific one-shot caps (10 probe epochs, best validation Acc. when available, otherwise best training Acc.). A superscript † marks a pretraining-disjoint held-out dataset; SensatUrban is validation-only because public test labels are unavailable.

Parameter	Value
Grid size	$0.02\,\mathrm{m}$
Views	1 global + 2 local
Global / local scales	$(0.4,1.0)$ / $(0.02,0.25)$
Points per view	$100\mathrm{k}$ (headline run)
$\lambda$ (SIGReg)	$5{\times}10^{-4}$
$w_{\mathrm{f}}/w_{\mathrm{c}}$ (variance weights)	$0.3$ / $0.3$

Appendix

Table 6: Hyperparameters for Polis pretraining and frozen probing. The probe checkpoint, data sampling, objective weights, and resource settings are reported to make the training and evaluation protocol reproducible.

Point Cloud Learning

Point Cloud Learning develops methods for representing and interpreting unordered 3D point sets from LiDAR, depth sensors, radar, and 3D models.

See all 159 papers

Mar 24, 2026

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

While multi-modality large language models excel in object-centric or indoor scenarios, scaling them to 3D city-scale environments remains a formidable challenge. To bridge this gap, we propose 3DCity-LLM, a unified framework designed for 3D city-scale vision-language perception and understanding. 3DCity-LLM employs a coarse-to-fine feature encoding strategy comprising three parallel branches for target object, inter-object relationship, and global scene. To facilitate large-scale training, we introduce 3DCity-LLM-1.2M dataset that comprises approximately 1.2 million high-quality samples across seven representative task categories, ranging from fine-grained object analysis to multi-faceted scene planning. This strictly quality-controlled dataset integrates explicit 3D numerical information and diverse user-oriented simulations, enriching the question-answering diversity and realism of urban scenarios. Furthermore, we apply a multi-dimensional protocol based on text-similarity metrics and LLM-based semantic assessment to ensure faithful and comprehensive evaluations for all methods. Extensive experiments on two benchmarks demonstrate that 3DCity-LLM significantly outperforms existing state-of-the-art methods, offering a promising and meaningful direction for advancing spatial reasoning and urban intelligence. The source code and dataset are available at https://github.com/SYSU-3DSTAILab/3D-City-LLM.

Yiping Chen, Jinpeng Li, Wenyu Ke +6

School of Geospatial Engineering and Science, Sun Yat-sen University, Zhuhai, 519082, China. · College of Electronic Science, National University of Defense Technology, Changsha, 410000, China. · Department of Civil and Environmental Engineering, Norwegian University of Science and Technology, Trondheim, Norway. +1

3D VLMs 3D Scene Understanding Multimodal Large Language Models 3D VQA

Ask PDF

Mar 26, 2026

Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds

Recent advances in self-supervised learning (SSL) for point clouds have substantially improved 3D scene understanding without human annotations. Existing approaches emphasize semantic awareness by enforcing feature consistency across augmented views or by masked scene modeling. However, the resulting representations transfer poorly to instance localization, and often require full finetuning for strong performance. Instance awareness is a fundamental component of 3D perception, thus bridging this gap is crucial for progressing toward true 3D foundation models that support all downstream tasks on 3D data. In this work, we introduce PointINS, an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning. PointINS employs an orthogonal offset branch to jointly learn high-level semantic understanding and geometric reasoning, yielding instance awareness. We identify two consistent properties essential for robust instance localization and formulate them as complementary regularization strategies, Offset Distribution Regularization (ODR), which aligns predicted offsets with empirically observed geometric priors, and Spatial Clustering Regularization (SCR), which enforces local coherence by regularizing offsets with pseudo-instance masks. Through extensive experiments across five datasets, PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation, paving the way for scalable 3D foundation models.

Bin Yang, Mohamed Abdelsamad, Miao Zhang +1

Bosch Center for AI, Stuttgart, Germany · Institute for Neuro- and Bioinformatics, University of Lübeck, Lübeck, Germany · Robert Learning Lab, University of Freiburg, Freiburg, Germany

Self-Supervised Point Cloud Learning Point Cloud Learning 3D Representation Learning 3D Foundation Models 3D Instance Segmentation

Ask PDF

Mar 3, 2026

Utonia: Toward One Encoder for All Point Clouds

We dream of a future where point clouds from all domains can come together to shape a single model that benefits them all. Toward this goal, we present Utonia, a first step toward training a single self-supervised point transformer encoder across diverse domains, spanning remote sensing, outdoor LiDAR, indoor RGB-D sequences, object-centric CAD models, and point clouds lifted from RGB-only videos. Despite their distinct sensing geometries, densities, and priors, Utonia learns a consistent representation space that transfers across domains. This unification improves perception capability while revealing intriguing emergent behaviors that arise only when domains are trained jointly. Beyond perception, we observe that Utonia representations can also benefit embodied and multimodal reasoning: conditioning vision-language-action policies on Utonia features improves robotic manipulation, and integrating them into vision-language models yields gains on spatial reasoning. We hope Utonia can serve as a step toward foundation models for sparse 3D data, and support downstream applications in AR/VR, robotics, and autonomous driving.

Yujia Zhang, Xiaoyang Wu, Yunhan Yang +6

The University of Hong Kong · Xiaomi · The Chinese University of Hong Kong

3D Foundation Models 3D ViTs Self-Supervised Pre-Training Point Cloud Learning Self-Supervised Point Cloud Learning

Ask PDF

Abstract

Point Cloud Learning

Explore similar work

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds

Utonia: Toward One Encoder for All Point Clouds