Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language
Authors: Yan Xia, Letian Shi, Yilin Di, Joao F. Henriques, Daniel Cremers
Organizations: the School of Artificial Intelligence and Data Science, University of Science and Technology of China, 230026 Hefei, China · Technical University of Munich, 80333 Munich, Germany · the Visual Geometry Group, University of Oxford, Oxford OX1 2JD, United Kingdom
We tackle the problem of localizing 3D point cloud submaps using complex and diverse natural language descriptions, and present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds in a coarse-to-fine localization pipeline. To support benchmarking, we introduce a new city-scale dataset covering both color and non-color point clouds from diverse urban scenes, and organize location descriptions into three levels of linguistic complexity. In the global place recognition stage, Text2Loc++ combines a pretrained language model with a Hierarchical Transformer with Max Pooling (HTM) for sentence-level semantics, and employs an attention-based point cloud encoder for spatial understanding. We further propose Masked Instance Training (MIT) to filter out non-aligned objects and improve multimodal robustness. To enhance the embedding space, we introduce Modality-aware Contrastive Learning (MCL), incorporating cross-modal, submap-, text-, and instance-level losses. In the fine localization stage, we completely remove explicit text-instance matching and design a lightweight yet powerful framework based on Prototype-based Map Cloning (PMC) and a Cascaded Cross-Attention Transformer (CCAT). Extensive experiments on the KITTI360Pose dataset show that Text2Loc++ outperforms existing methods by up to 15%. In addition, the proposed model exhibits robust generalization when evaluated on the new dataset, effectively handling complex linguistic expressions and a wide variety of urban environments.
Figures & tables
Fig. 1 : (Left) We present Text2Loc++, a framework developed for cross-city and cross-country position localization based on textual descriptions of varying complexity. Given a 3D point cloud representing the surrounding environment and a textual query describing a location from different urban or regional contexts, Text2Loc++ identifies the most probable position corresponding to the description within the map. The model demonstrates strong generalization across diverse language inputs and heterogeneous point cloud data. (Right) Localization results on the KITTI360Pose test set show that the proposed Text2Loc++ consistently outperforms existing baselines across all top-k retrieval thresholds, achieving up to 15% higher accuracy in localizing text queries within a 5m error range. The textual query here is expressed in the complex linguistic form.
Fig. 2 : The proposed Text2Loc++ architecture. It consists of two consecutive modules: Global place recognition and Fine localization. Global place recognition. Given a text-based position description, we first identify a set of coarse candidate locations, ”submaps,” potentially containing the target position. This is achieved by retrieving the top-k nearest submaps from a previously constructed database of submaps using our novel text-to-submap retrieval model. Fine localization. We then refine the center coordinates of the retrieved submaps via our designed matching-free position estimation module, which adjusts the target location to increase accuracy. The text description shown here is expressed in the complex linguistic form.
Fig. 3 : (left) The architecture of global place recognition and the training procedure in the global place recognition. (right) The architecture of instance encoder as well as intra/inter-text encoder architecture for point clouds. Note that the pre-trained/finetuned T5 model is frozen.
Fig. 4 : (top) Comparison for without and with masked instance modality-aware contrastive learning in the training process. T1 and T2 are different text embeddings. S1 and S2 are different submap embeddings from s1 and s2 . S1′ , S1′′ , and S2′ are submap embeddings with masked instance process. (bottom) Visualization of original submap and masked submap. The red circle points out the masked instance.
Fig. 5 : (top) The proposed matching-free fine localization architecture. It consists of two parallel branches: instance encoder as the top branch and text encoder as the bottom branch. Cascaded Cross-Attention Transformers (CCAT) fuse the multimodal information. (bottom) The procedure of the Prototype-based Map Cloning (PMC).
Baseline
Refinement
Dataset
Color
Training
Testing
Training
Testing
# submaps
# texts
# submaps
# texts
# submaps
# texts
# submaps
# texts
Kitti360
✓
11259
28807
4308
11505
11259
28807
4308
11505
Paris_CARLA
✓
-
-
1004
3857
2936
10855
1004
3857
Toronto
✓
-
-
685
2087
-
-
685
2087
Kitti360 (no color)
✗
11259
28807
4308
11505
11259
28807
4308
11505
TABLE I: Dataset information. Number of training and testing submaps and texts for our baseline and refinement networks. Color means whether the point cloud contains color information.
Fig. 6 : Samples of point cloud from different datasets. We present visualizations of the point clouds from the four constructed benchmark datasets. Subfigures CARLA - Paris_CARLA, Paris - Paris_CARLA, and Toronto depict datasets with color information, whereas other subfigures correspond to datasets without color attributes.
Simple
The pose is on-top of a green vegetation.
The pose is east of a dark-green fence.
The pose is north of a green sidewalk.
The pose is south of a green building.
The pose is north of a green terrain.
The pose is north of a green vegetation.
Moderate
Two vegetations, the sidewalk, the building and the terrain are green. One vegetation is below the pose. The sidewalk, the terrain and one vegetation are south of the pose. The building is north of the pose. They are east of the dark-green fence.
TABLE II : Textual descriptions of the same location in the submap at different levels of linguistic complexity.
Submap Retrieval Recall (%) ↑
Setting
Method
Simple
Moderate
Complex
k=1
k=3
k=5
k=1
k=3
k=5
k=1
k=3
k=5
KITTI360Pose [ 11 ]
Text2Pos [ 11 ]
11.8
24.6
32.5
8.6
18.1
24.0
3.3
7.8
11.0
Text2Loc [ 26 ]
29.3
47.7
56.9
21.6
39.3
48.2
17.0
32.0
40.1
Baseline
MambaPlace [ 17 ]
31.0
52.0
62.0
-
-
-
-
-
-
TABLE III : The evaluation results of the global place recognition. The ”setting” column (baseline or refinement) indicates which datasets are used to train the network, as detailed in Table. I . The terms simple, moderate, and complex refer to the linguistic complexity of the textual descriptions used during both training and testing.
Localization Recall (%) - ϵ<5/10/15m↑
Setting
Method
Simple
Moderate
Complex
k=1
k=5
k=10
k=1
k=5
k=10
k=1
k=5
k=10
KITTI360Pose [ 11 ]
Text2Pos [ 11 ]
12/21/24
32/47/51
42/60/64
7/16/20
20/38/44
28/50/57
3/6/8
9/19/24
14/29/36
Text2Loc [ 26 ]
34/47/50
61/74/76
71/83/85
28/39/43
54/67/70
65/77/80
14/29/34
33/56/61
42/67/72
Baseline
MambaPlace [ 17 ]
38/52/55
66/79/81
76/87/89
-
-
-
-
-
-
TABLE IV : The evaluation results of the fine localization. The ”setting” column (baseline or refinement) indicates which datasets are used to train the network, as detailed in Table. I . The terms simple, moderate, and complex refer to the linguistic complexity of the textual descriptions used during both training and testing.
Setting
Submap Retrieval Recall (%) ↑
k=1
k=3
k=5
w/o masked instance
32.1
53.5
62.6
matching
10.1
21.3
28.5
Text2Loc++
35.3 35.3 35.3 35.3
57.6 57.6 57.6 57.6
66.9 66.9 66.9 66.9
TABLE V : Performance analysis of the masked instance training. ”matching” refers to retaining only the instances in the submaps that are explicitly described in the text instead of masked instance training.
Setting
Submap Retrieval Recall (%) ↑
k=1
k=3
k=5
w/o instance loss
33.2
54.9
64.0
w/o text loss
33.4
56.2
65.3
w/o submap loss
32.5
54.1
63.6
Text2Loc++
35.3 35.3 35.3 35.3
57.6 57.6 57.6 57.6
66.9 66.9 66.9 66.9
TABLE VI : Performance analysis of the each component of the proposed modality-aware contrastive learning.
Level
TD
Extra
Submap Retrieval Recall (%) ↑
k=1
k=3
k=5
Moderate
✗
-
22.7
41.8
50.8
✗
Shuffle
23.1
41.7
50.5
✓
-
34.0 34.0 34.0 34.0
56.8 56.8 56.8 56.8
65.4 65.4 65.4 65.4
Complex
✗
-
19.0
35.5
44.1
✗
Translate
16.7
31.8
39.8
TABLE VII: Comparison of whether using text distillation for moderate and complex text on the KITTI360Pose benchmark. TD means text distillation. Several additional settings are applied during both training and testing phases. ”Shuffle” involves randomizing the format of moderate descriptions. For a specific position, we utilize various formats of descriptions during training. ”Translate” denotes the use of ChatGPT-4o [ 63 ] to simplify complex text descriptions before feeding them into the language encoder for both training and testing.
Parameter
Value
Submap Retrieval Recall (%) ↑
k=1
k=3
k=5
Loss
pairwise
31.5
53.3
62.5
Latent Space
dim = 64
28.3
48.5
57.4
dim = 128
32.9
54.8
64.1
dim = 512
35.6 35.6 35.6 35.6
58.3 58.3 58.3 58.3
67.9 67.9 67.9 67.9
HTM Layers
no HTM
31.0
51.0
59.8
TABLE VIII : Performance analysis of global place recognition in the different configurations in the simple text mode. The default config utilize InfoNCE loss [ 64 ] , 256 latent space, HTM (intra = 1 + inter = 1), max pooling in intra and inter encoder, T5_large [ 10 ] backbones in the text module, and number, color, position branches.
Localization Recall (%) - ϵ<5m↑
Method
k=1
k=5
k=10
Text2Pos [ 11 ]
12.2
319
43.0
Text2Pos*
36.6
647
72.8
Text2Loc++_PMC
36.8
646
72.9
Text2Loc++_CCAT
39.9
697
78.2
Text2Loc++_P+C
41.4
706
79.1
TABLE IX : Ablation study of the fine localization on the KITTI360Pose benchmark. * indicates the fine localization network from Text2Pose [ 11 ] , and the submaps retrieved through our global place recognition. Text2Loc++_CCAT indicates the removal of only the PMC while retaining the CCAT in our network. Conversely, Text2Loc++_PMC keeps the PMC but replaces the CCAT with the text-instance matcher in Text2Pos. Text2Loc++_P+C includes both PMC and CCAT, but excludes the MIT during training.
Method
Parameters (M)
Runtime (ms)
LocalizationRecall
Text2Loc++_Matcher
2.08
43.11
034
Text2Loc++ (Ours)
1.06
2.27
0.40
TABLE X : Computational cost requirement analysis of our fine localization network on the KITTI360Pose test dataset.
Fig. 7 : Qualitative localization results on the KITTI360Pose dataset: In global place recognition, the numbers in top3 retrieval submaps represent center distances between retrieved submaps and the ground truth. Green boxes indicate positive submaps containing the target location, while red boxes signify negative submaps. For fine localization, red and black dots represent the ground truth and predicted target locations, with the red number indicating the distance between them. (a), (b), and (c) uses simple, moderate, and complex text descriptions respectively.
Change Type
Hint
Original
The pose is east of a gray road.
Color
The pose is east of a green road.
Direction
The pose is west of a gray road.
Semantic class
The pose is east of a gray sidewalk .
Discard
(Sentence deleted)
TABLE XI : Change Instruction
Fig. 8 : Robustness analysis comparing different metrics.
Fig. 9 : T-SNE illustration comparison between with and without masked instance training (MIT), modality-aware contrastive learning (MCL), and text distillation (TD).
Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and dataset are available at repository.
Shuhao Kang, Youqi Liao, Peijie Wang +5
VCIP, CS, Nankai University · Wuhan University · CASIA +6
Text-to-point-cloud localization estimates a position in a city-scale 3D map from descriptions of surrounding objects. Existing coarse-to-fine methods retrieve submaps using aggregate learned compatibility and then localize within a selected submap. However, repetitive or similar urban objects can inflate the embedding similarity between the query and multiple submaps, even when the instance layout within a submap violates the query description. Meanwhile, query-relevant instances often span submap boundaries, leaving the retrieved submap with incomplete contextual evidence. We term these failure modes layout-inconsistent aliasing and boundary evidence incompleteness, respectively. To address them, we propose PARC-Loc, a coarse-to-fine localization framework built on Partial Assignment with Relational Consistency (PARC). PARC jointly models hint-object compatibility and pairwise spatial relations, allowing unmatched elements while favoring assignments consistent with the queried layout. At the coarse stage, its candidate-level assessment complements neural similarity for layout-consistent submap selection. At the fine stage, the context is expanded with query-relevant instances from adjacent submaps, while PARC yields object-level matching weights that guide cross-modal attention. Extensive experiments on KITTI360Pose and CityLoc show that PARC-Loc outperforms conventional coarse-to-fine baselines. On KITTI360Pose, our method improves Top-1 localization recall at 5 m from 0.50 to 0.67, achieving a 34% relative gain over the strongest baseline.
Language-based 3D localization retrieves the point-cloud submap containing a target position from descriptions of nearby objects and their spatial relations. Existing methods typically compress queries and submaps into global descriptors, potentially obscuring object-level semantics and cross-description spatial coherence. We propose Position-Conditioned Evidence Localization (PosEviLoc), a query-position-aware framework for coarse text-to-point-cloud localization. Instead of relying on global matching, PosEviLoc evaluates each candidate submap using explicit semantic and spatial evidence. It models direction as a relation jointly determined by an object position and a hypothetical query position. The resulting Query-Position Spatial Evidence Field (QSEF) measures the fraction of query descriptions supported at each hypothetical position, explicitly capturing their agreement without using the ground-truth query pose to construct the evidence field. A Multi-Level Evidence Readout (MER) summarizes this evidence in a compact representation, which a lightweight MLP converts into a retrieval score. Across five benchmarks, PosEviLoc outperforms MNCL by an average of 17 percentage points in Recall@1. When used as a plug-and-play reranker, it improves MNCL by an average of 16 percentage points. Moreover, PosEviLoc introduces substantially fewer parameters and achieves faster inference speed than existing methods.
Tianyi Shang, Yike Shi, Zhenyu Li
Purdue University · Xi’an Jiaotong-Liverpool University · Qilu University of Technology (Shandong Academy of Sciences)