Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language

Yan Xia; Letian Shi; Yilin Di; Joao F. Henriques; Daniel Cremers

cs.CVNov 19, 2025

Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language

Authors: Yan Xia, Letian Shi, Yilin Di, Joao F. Henriques, Daniel Cremers

Organizations: the School of Artificial Intelligence and Data Science, University of Science and Technology of China, 230026 Hefei, China · Technical University of Munich, 80333 Munich, Germany · the Visual Geometry Group, University of Oxford, Oxford OX1 2JD, United Kingdom

Abstract

We tackle the problem of localizing 3D point cloud submaps using complex and diverse natural language descriptions, and present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds in a coarse-to-fine localization pipeline. To support benchmarking, we introduce a new city-scale dataset covering both color and non-color point clouds from diverse urban scenes, and organize location descriptions into three levels of linguistic complexity. In the global place recognition stage, Text2Loc++ combines a pretrained language model with a Hierarchical Transformer with Max Pooling (HTM) for sentence-level semantics, and employs an attention-based point cloud encoder for spatial understanding. We further propose Masked Instance Training (MIT) to filter out non-aligned objects and improve multimodal robustness. To enhance the embedding space, we introduce Modality-aware Contrastive Learning (MCL), incorporating cross-modal, submap-, text-, and instance-level losses. In the fine localization stage, we completely remove explicit text-instance matching and design a lightweight yet powerful framework based on Prototype-based Map Cloning (PMC) and a Cascaded Cross-Attention Transformer (CCAT). Extensive experiments on the KITTI360Pose dataset show that Text2Loc++ outperforms existing methods by up to 15%. In addition, the proposed model exhibits robust generalization when evaluated on the new dataset, effectively handling complex linguistic expressions and a wide variety of urban environments.

Figures & tables

Fig. 1 : (Left) We present Text2Loc++, a framework developed for cross-city and cross-country position localization based on textual descriptions of varying complexity. Given a 3D point cloud representing the surrounding environment and a textual query describing a location from different urban or regional contexts, Text2Loc++ identifies the most probable position corresponding to the description within the map. The model demonstrates strong generalization across diverse language inputs and heterogeneous point cloud data. (Right) Localization results on the KITTI360Pose test set show that the proposed Text2Loc++ consistently outperforms existing baselines across all top-k retrieval thresholds, achieving up to 15% higher accuracy in localizing text queries within a $5\text{\,}\mathrm{m}$ error range. The textual query here is expressed in the complex linguistic form.

Fig. 2 : The proposed Text2Loc++ architecture. It consists of two consecutive modules: Global place recognition and Fine localization. Global place recognition. Given a text-based position description, we first identify a set of coarse candidate locations, ”submaps,” potentially containing the target position. This is achieved by retrieving the top-k nearest submaps from a previously constructed database of submaps using our novel text-to-submap retrieval model. Fine localization. We then refine the center coordinates of the retrieved submaps via our designed matching-free position estimation module, which adjusts the target location to increase accuracy. The text description shown here is expressed in the complex linguistic form.

Fig. 3 : (left) The architecture of global place recognition and the training procedure in the global place recognition. (right) The architecture of instance encoder as well as intra/inter-text encoder architecture for point clouds. Note that the pre-trained/finetuned T5 model is frozen.

Fig. 4 : (top) Comparison for without and with masked instance modality-aware contrastive learning in the training process. $T_{1}$ and $T_{2}$ are different text embeddings. $S_{1}$ and $S_{2}$ are different submap embeddings from $s_{1}$ and $s_{2}$ . $S_{1}^{\prime}$ , $S_{1}^{\prime\prime}$ , and $S_{2}^{\prime}$ are submap embeddings with masked instance process. (bottom) Visualization of original submap and masked submap. The red circle points out the masked instance.

Fig. 5 : (top) The proposed matching-free fine localization architecture. It consists of two parallel branches: instance encoder as the top branch and text encoder as the bottom branch. Cascaded Cross-Attention Transformers (CCAT) fuse the multimodal information. (bottom) The procedure of the Prototype-based Map Cloning (PMC).

Dataset	Color	Training		Testing		Training		Testing
		Baseline				Refinement
		# submaps	# texts	# submaps	# texts	# submaps	# texts	# submaps	# texts
Kitti360	✓	11259	28807	4308	11505	11259	28807	4308	11505
Paris_CARLA	✓	-	-	1004	3857	2936	10855	1004	3857
Toronto	✓	-	-	685	2087	-	-	685	2087
Kitti360 (no color)	✗	11259	28807	4308	11505	11259	28807	4308	11505

TABLE I: Dataset information. Number of training and testing submaps and texts for our baseline and refinement networks. Color means whether the point cloud contains color information.

Panel of Figure 7 — Fig. 6 : Samples of point cloud from different datasets. We present visualizations of the point clouds from the four constructed benchmark datasets. Subfigures CARLA - Paris_CARLA, Paris - Paris_CARLA, and Toronto depict datasets with color information, whereas other subfigures correspond to datasets without color attributes.

Simple	The pose is on-top of a green vegetation.
	The pose is east of a dark-green fence.
	The pose is north of a green sidewalk.
	The pose is south of a green building.
	The pose is north of a green terrain.
	The pose is north of a green vegetation.
Moderate	Two vegetations, the sidewalk, the building and the terrain are green. One vegetation is below the pose. The sidewalk, the terrain and one vegetation are south of the pose. The building is north of the pose. They are east of the dark-green fence.

TABLE II : Textual descriptions of the same location in the submap at different levels of linguistic complexity.

		Submap Retrieval Recall (%) $\uparrow$
Setting	Method	Simple			Moderate			Complex
		k=1	k=3	k=5	k=1	k=3	k=5	k=1	k=3	k=5
KITTI360Pose [ 11 ]
	Text2Pos [ 11 ]	11.8	24.6	32.5	8.6	18.1	24.0	3.3	7.8	11.0
	Text2Loc [ 26 ]	29.3	47.7	56.9	21.6	39.3	48.2	17.0	32.0	40.1
Baseline	MambaPlace [ 17 ]	31.0	52.0	62.0	-	-	-	-	-	-

TABLE III : The evaluation results of the global place recognition. The ”setting” column (baseline or refinement) indicates which datasets are used to train the network, as detailed in Table. I . The terms simple, moderate, and complex refer to the linguistic complexity of the textual descriptions used during both training and testing.

		Localization Recall (%) - $\epsilon<5/10/15m$ $\uparrow$
Setting	Method	Simple			Moderate			Complex
		k=1	k=5	k=10	k=1	k=5	k=10	k=1	k=5	k=10
KITTI360Pose [ 11 ]
	Text2Pos [ 11 ]	12/21/24	32/47/51	42/60/64	7/16/20	20/38/44	28/50/57	3/6/8	9/19/24	14/29/36
	Text2Loc [ 26 ]	34/47/50	61/74/76	71/83/85	28/39/43	54/67/70	65/77/80	14/29/34	33/56/61	42/67/72
Baseline	MambaPlace [ 17 ]	38/52/55	66/79/81	76/87/89	-	-	-	-	-	-

TABLE IV : The evaluation results of the fine localization. The ”setting” column (baseline or refinement) indicates which datasets are used to train the network, as detailed in Table. I . The terms simple, moderate, and complex refer to the linguistic complexity of the textual descriptions used during both training and testing.

Setting	Submap Retrieval Recall (%) $\uparrow$
Setting	$k=1$	$k=3$	$k=5$
w/o masked instance	32.1	53.5	62.6
matching	10.1	21.3	28.5
Text2Loc++	35.3 35.3 35.3 35.3	57.6 57.6 57.6 57.6	66.9 66.9 66.9 66.9

TABLE V : Performance analysis of the masked instance training. ”matching” refers to retaining only the instances in the submaps that are explicitly described in the text instead of masked instance training.

Setting	Submap Retrieval Recall (%) $\uparrow$
Setting	$k=1$	$k=3$	$k=5$
w/o instance loss	33.2	54.9	64.0
w/o text loss	33.4	56.2	65.3
w/o submap loss	32.5	54.1	63.6
Text2Loc++	35.3 35.3 35.3 35.3	57.6 57.6 57.6 57.6	66.9 66.9 66.9 66.9

TABLE VI : Performance analysis of the each component of the proposed modality-aware contrastive learning.

Level	TD	Extra	Submap Retrieval Recall (%) $\uparrow$
			$k=1$	$k=3$	$k=5$
Moderate	✗	-	22.7	41.8	50.8
	✗	Shuffle	23.1	41.7	50.5
	✓	-	34.0 34.0 34.0 34.0	56.8 56.8 56.8 56.8	65.4 65.4 65.4 65.4
Complex	✗	-	19.0	35.5	44.1
	✗	Translate	16.7	31.8	39.8

TABLE VII: Comparison of whether using text distillation for moderate and complex text on the KITTI360Pose benchmark. TD means text distillation. Several additional settings are applied during both training and testing phases. ”Shuffle” involves randomizing the format of moderate descriptions. For a specific position, we utilize various formats of descriptions during training. ”Translate” denotes the use of ChatGPT-4o [ 63 ] to simplify complex text descriptions before feeding them into the language encoder for both training and testing.

Parameter	Value	Submap Retrieval Recall (%) $\uparrow$
Parameter	Value	$k=1$	$k=3$	$k=5$
Loss	pairwise	31.5	53.3	62.5
Latent Space	dim = 64	28.3	48.5	57.4
	dim = 128	32.9	54.8	64.1
	dim = 512	35.6 35.6 35.6 35.6	58.3 58.3 58.3 58.3	67.9 67.9 67.9 67.9
HTM Layers	no HTM	31.0	51.0	59.8

TABLE VIII : Performance analysis of global place recognition in the different configurations in the simple text mode. The default config utilize InfoNCE loss [ 64 ] , 256 latent space, HTM (intra = 1

+

inter = 1), max pooling in intra and inter encoder, T5_large [ 10 ] backbones in the text module, and number, color, position branches.

	Localization Recall (%) - $\epsilon<5m$ $\uparrow$
Method	$k=1$	k=5	$k=10$
Text2Pos [ 11 ]	12.2	$\mathrm{3}\mathrm{1}\,\mathrm{9}$	43.0
Text2Pos*	36.6	$\mathrm{6}\mathrm{4}\,\mathrm{7}$	72.8
Text2Loc++_PMC	36.8	$\mathrm{6}\mathrm{4}\,\mathrm{6}$	72.9
Text2Loc++_CCAT	39.9	$\mathrm{6}\mathrm{9}\,\mathrm{7}$	78.2
Text2Loc++_P+C	41.4	$\mathrm{7}\mathrm{0}\,\mathrm{6}$	79.1

TABLE IX : Ablation study of the fine localization on the KITTI360Pose benchmark. * indicates the fine localization network from Text2Pose [ 11 ] , and the submaps retrieved through our global place recognition. Text2Loc++_CCAT indicates the removal of only the PMC while retaining the CCAT in our network. Conversely, Text2Loc++_PMC keeps the PMC but replaces the CCAT with the text-instance matcher in Text2Pos. Text2Loc++_P+C includes both PMC and CCAT, but excludes the MIT during training.

Method	Parameters (M)	Runtime (ms)	$\mathrm{L}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}$
Text2Loc++_Matcher	2.08	43.11	$\mathrm{0}\,\mathrm{3}\mathrm{4}$
Text2Loc++ (Ours)	1.06	2.27	0.40

TABLE X : Computational cost requirement analysis of our fine localization network on the KITTI360Pose test dataset.

Fig. 7 : Qualitative localization results on the KITTI360Pose dataset: In global place recognition, the numbers in top3 retrieval submaps represent center distances between retrieved submaps and the ground truth. Green boxes indicate positive submaps containing the target location, while red boxes signify negative submaps. For fine localization, red and black dots represent the ground truth and predicted target locations, with the red number indicating the distance between them. (a), (b), and (c) uses simple, moderate, and complex text descriptions respectively.

Change Type	Hint
Original	The pose is east of a gray road.
Color	The pose is east of a green road.
Direction	The pose is west of a gray road.
Semantic class	The pose is east of a gray sidewalk .
Discard	(Sentence deleted)

TABLE XI : Change Instruction

Fig. 8 : Robustness analysis comparing different metrics.

Fig. 9 : T-SNE illustration comparison between with and without masked instance training (MIT), modality-aware contrastive learning (MCL), and text distillation (TD).

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models
Mar 10, 2026Shuhao Kang, Youqi Liao, Peijie Wang +5Map-Based Localization 3D VLMs
VCIP, CS, Nankai University · Wuhan University · CASIA +6
PARC-Loc: Text-to-Point-Cloud Localization with Partial Assignment and Relational Consistency
Oct 7, 2026Shengkai Ma, Zhenyu Hou, Weihua CaoMap-Based Localization 3D Visual Grounding
PosEviLoc: Position-Conditioned Spatial Evidence for Language-Based 3D Localization
Sep 20, 2026Tianyi Shang, Yike Shi, Zhenyu LiMap-Based Localization 3D Visual Grounding
Purdue University · Xi’an Jiaotong-Liverpool University · Qilu University of Technology (Shandong Academy of Sciences)

Mar 10, 2026J/K move · Enter open · S save

VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

Shuhao Kang, Youqi Liao, Peijie Wang, Wenlong Liao+4

VCIP, CS, Nankai University · Wuhan University · CASIA+6

Map-Based Localization 3D VLMs

Open paper AskPDF arXiv

Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and dataset are available at repository.

Abstract

3D VLMs

Explore similar work

VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

PARC-Loc: Text-to-Point-Cloud Localization with Partial Assignment and Relational Consistency

PosEviLoc: Position-Conditioned Spatial Evidence for Language-Based 3D Localization