Vision-Wireless Fusion for Multi-User Localization: A Cross-Modal Transformer Approach

Can Zheng; Jiguang He; Guofa Cai; Henk Wymeersch; Merouane Debbah

cs.CVSep 20, 2026

Vision-Wireless Fusion for Multi-User Localization: A Cross-Modal Transformer Approach

Authors: Can Zheng, Jiguang He, Guofa Cai, Henk Wymeersch, Merouane Debbah

Organizations: School of Computing and Information Technology, Great Bay University, Dongguan Key Laboratory for Intelligence and Information Technology, and Great Bay Institute for Advanced Study (GBIAS), Dongguan 523000, China · School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China · Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden · Research Institute for Digital Future, Khalifa University, 127788 Abu Dhabi, UAE

Abstract

Accurate multi-user localization is challenging in complex urban environments, where wireless measurements can become ambiguous under noise, blockage, and multipath, while visual observations provide complementary spatial context. This paper presents a vision-wireless fusion framework for multi-user localization using pilot-indexed channel state information (CSI). Orthogonal pilot indices preserve the identities of the communicating UEs in the CSI token sequence and localization outputs. The model encodes each pilot-indexed CSI observation as a query token and uses cross-attention to retrieve user-specific information from spatial visual memory. Self-attention among CSI tokens further captures inter-user interactions, while the resulting multimodal representations are used for user-wise localization. Experiments on different datasets show consistent improvements over model-based, CSI-only, and multimodal-fusion baselines. Further experiments evaluate the model under different wireless and visual conditions.

Figures & tables

Fig. 1 : Multi-user V2I multimodal localization scenario. All localization targets are vehicular UEs. Vehicles without communication links may also appear in the FoV.

Fig. 2 : Architecture of the proposed multi-user vision–wireless localization model.

Panel of Figure 3 — Fig. 3 : Cross-modal alignment methods in the proposed model.

Group	Parameter	DV6G / RMT / MW
Data	Carrier frequency	60 / 60 / 28 GHz
	BS array	$4\times 4$ UPA
	Subcarriers	8 in 512
	# paths	10
	Image size	$3\times 270\times 480$
	$K_{\max}$	10

TABLE I : Experimental settings for all datasets.

Methods	Modalities	Layer sequence	Output size
3D-FFT	CSI	CSI $\rightarrow\mathrm{IFFT}_{2048}$ with zero padding $\rightarrow\mathrm{FFT2D}_{64\times 64}$ over unit-disk direction candidates $\rightarrow$ joint angle–delay peak $\rightarrow$ 3D mapping.	$3$
MUSIC	CSI	CSI $\rightarrow$ $16\times 16$ spatial covariance and $8\times 8$ frequency covariance $\rightarrow$ Eigenvalue decomposition (EVD) $\rightarrow$ direction spectrum and range spectrum $\rightarrow$ largest spatial and range peaks paired $\rightarrow$ 3D mapping.	$3$
MLP	CSI	Flatten $\rightarrow[\mathrm{FC}(256,512)+\mathrm{LN}+\mathrm{GELU}+\mathrm{Drop}(0.1)]\times 2\rightarrow\mathrm{FC}(512,256)\rightarrow$ CSI-Reg	$3$
CNN	CSI	$[\mathrm{C2D}(2,32,3)\rightarrow\mathrm{C2D}(32,64,3)\rightarrow\mathrm{C2D}(64,128,3)]+\mathrm{BN}+\mathrm{GELU}\rightarrow\mathrm{AAP}(2,2)\rightarrow\mathrm{FC}(512,256)+\mathrm{LN}+\mathrm{GELU}\rightarrow$ CSI-Reg	$3$
FNN	CSI	$\{\mathrm{C2D}(2,32,k)\rightarrow\mathrm{C2D}(32,64,k)+\mathrm{BN}+\mathrm{GELU}+\mathrm{GAP}\}_{k=(3,1),(1,3)}\rightarrow[\mathrm{Feat}_{3\times 1};\mathrm{Feat}_{1\times 3}]\rightarrow\mathrm{FC}(128,256)+\mathrm{LN}+\mathrm{GELU}\rightarrow$ CSI-Reg	$3$
L2L	CSI	angle-delay power $\rightarrow$ horizontal/vertical C3D refinement $\rightarrow\mathrm{C3D}(64,128,3)\rightarrow\mathrm{Inc3D}(64,128,256)\rightarrow\mathrm{GAP}\rightarrow\mathrm{FC}(256,256)+\mathrm{LN}+\mathrm{GELU}\rightarrow$ CSI-Reg	$3$

TABLE II : Layer configurations of all compared methods.

Panel of Figure 6 — Fig. 4 : Training, validation, and test splits and their number of users ( $K$ ) distributions. Numbers denote the number of samples.

Panel of Figure 7 — Fig. 5 : The CDFs compare 3D localization errors on the test sets of the three datasets. Shaded regions indicate one standard deviation above and below the mean.

Dataset	Method	Mean (m)	Median (m)	P95 (m)	Acc@1m (%)	Acc@2m (%)
DeepVerse 6G	3D-FFT	$3.93\pm 1.05$	$1.43\pm 0.45$	$13.84\pm 7.59$	$39.04\pm 10.48$	$61.58\pm 9.20$
	MUSIC	$4.15\pm 1.17$	$1.54\pm 0.64$	$14.26\pm 6.64$	$38.74\pm 12.88$	$58.29\pm 10.84$
	MLP	$2.20\pm 0.43$	$1.37\pm 0.21$	$6.15\pm 1.33$	$37.32\pm 7.20$	$64.59\pm 6.06$
	CNN	$1.77\pm 0.15$	$1.08\pm 0.02$	$5.03\pm 0.64$	$47.30\pm 1.22$	$73.63\pm 0.82$
	FNN	$2.19\pm 0.36$	$1.48\pm 0.15$	$5.89\pm 1.14$	$28.54\pm 1.88$	$68.38\pm 5.01$
	L2L	$1.73\pm 0.49$	$0.89\pm 0.23$	$5.48\pm 2.10$	$55.71\pm 11.97$	$79.78\pm 7.35$

TABLE III : Overall 3D localization performance on all datasets.

Dataset	CSI encoder	Mean (m)	Median (m)	P95 (m)	Acc@1m (%)	Acc@2m (%)
DeepVerse 6G	MLP	$1.29\pm 0.14$	$0.63\pm 0.07$	$4.65\pm 1.08$	$67.80\pm 3.21$	$84.41\pm 2.14$
	CNN	$\mathbf{1.27\pm 0.18}$	$0.65\pm 0.09$	$\mathbf{4.04\pm 0.93}$	$\mathbf{68.25\pm 4.16}$	$\mathbf{86.21\pm 1.80}$
	FNN	$1.36\pm 0.08$	$0.77\pm 0.18$	$4.35\pm 0.66$	$61.04\pm 9.70$	$82.87\pm 3.62$
	L2L	$1.44\pm 0.30$	$0.73\pm 0.10$	$4.34\pm 1.41$	$64.23\pm 5.14$	$85.10\pm 4.66$
Raymobtime	MLP	$1.88\pm 0.19$	$1.07\pm 0.15$	$6.39\pm 0.80$	$\mathbf{48.10\pm 5.46}$	$70.24\pm 6.16$
	CNN	$2.02\pm 0.16$	$1.35\pm 0.15$	$6.37\pm 0.10$	$38.33\pm 3.22$	$67.38\pm 5.36$

TABLE IV : Localization performance with different CSI encoders on all datasets.

Panel of Figure 10 — Fig. 6 : Localization performance under different SNRs and numbers of paths on all datasets.

Fig. 7 : Comparison of mean localization performance under wireless LoS and NLoS conditions on the Raymobtime dataset.

Condition	DeepVerse 6G	Raymobtime	MM-Wireless
Normal	$\mathbf{1.27\pm 0.18}$	$\mathbf{1.86\pm 0.03}$	$\mathbf{1.15\pm 0.57}$
Global shuffle	$2.37\pm 0.30$	$3.32\pm 0.31$	$1.32\pm 0.48$
RGB features zero	$2.87\pm 0.25$	$3.06\pm 0.30$	$1.34\pm 0.47$
Visual position zero	$2.34\pm 0.31$	$2.75\pm 0.09$	$1.50\pm 0.30$
Visual memory zero	$1.99\pm 0.15$	$3.08\pm 0.10$	$1.69\pm 0.13$

TABLE V : Visual-input ablation performance on all datasets.

Dataset	$K=1$	$K=2$	$K=3$	$K=4$
DeepVerse 6G	$0.92\pm 0.17$	$1.53\pm 0.43$	$1.77\pm 0.48$	$1.46\pm 0.31^{\dagger}$
Raymobtime	$1.99\pm 0.10$	$2.21\pm 0.32$	$1.54\pm 0.88^{\dagger}$	–
Multimodal-Wireless	$1.21\pm 0.30$	$1.06\pm 0.56$	$1.55\pm 0.90$	–

TABLE VI : Localization performance under different numbers of active users on all datasets.

Fig. 8 : Comparison of mean localization performance under different weather conditions.

Method	Params (M)	GFLOPs/ sample	Train/Inf. Time (s/ms)	Train/Inf. Mem. (MB)
3D-FFT	0	–	–/1.81	–/6.5
MUSIC	0	–	–/5.91	–/3.6
MLP	0.544	0.0011	0.04/0.68	75.3/68.2
CNN	0.242	0.024	0.05/1.06	183.8/67.9
FNN	0.063	0.0036	0.06/0.61	137.6/64.6
L2L	1.169	0.065	0.24/6.91	182.6/74.5

TABLE VII : Computational complexity on the DeepVerse 6G dataset.

Wireless Localization

Wireless localization uses radio signals and sensor fusion to estimate the positions and trajectories of people, devices, and transmitters.

See all 65 papers

RA-LWLM: Retrieval-Augmented In-Context Localization with Wireless Foundation Models
Jun 1, 2026Guangjin Pan, Hui Chen, Hei Victor Cheng +1Wireless Communications In-Context Learning
Department of Electrical Engineering, Chalmers University of Technology, 41296 Gothenburg, Sweden · Department of Electrical and Computer Engineering, Aarhus University, Denmark
PRI-Net: A Lightweight Multimodal Framework for 3D UAV Localization
Sep 13, 2026Zhixuan Chen, Jialiang Lu, Zhong Ye +2Robot Localization Camera-LiDAR Fusion
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China
Map as a Prompt: Learning Multi-Modal Spatial-Signal Foundation Models for Cross-scenario Wireless Localization
Jul 17, 2026Yong Chu, Xun Zhou, Zenglin Xu +2Wireless Foundation Models Prompt Tuning
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Shanghai Academy of AI for Science +1

Jun 1, 2026J/K move · Enter open · S save

RA-LWLM: Retrieval-Augmented In-Context Localization with Wireless Foundation Models

Guangjin Pan, Hui Chen, Hei Victor Cheng, Henk Wymeersch

Department of Electrical Engineering, Chalmers University of Technology, 41296 Gothenburg, Sweden · Department of Electrical and Computer Engineering, Aarhus University, Denmark

Wireless Communications In-Context Learning Wireless Localization Retrieval-Augmented ICL

Open paper AskPDF arXiv

Wireless localization is a fundamental capability of sixth-generation (6G) networks. Conventional model-based methods require accurate modeling of the propagation environment and degrade in complex multipath and non-line-of-sight scenarios, while learning-based methods couple model parameters tightly to the training scene, requiring costly retraining whenever the base station (BS) configuration or propagation environment changes. In this paper, we propose RA-LWLM, a retrieval-augmented in-context localization framework that achieves training-free cross-scene adaptation by externalizing scene-specific information into a per-scene fingerprint database rather than encoding it in model weights. The framework consists of three components: a frozen wireless foundation model (FM) encoder that maps raw channel state information into a scene-agnostic representation; a retrieval module that selects the most informative references from the per-scene database via similarity search in the representation space; and a transformer-based in-context learning (ICL) module that fuses the query with the retrieved references to predict the user equipment (UE) position. To accommodate varying retrieval quality and propagation complexity across queries, the ICL module adopts a mixture-of-experts design in which experts specialize in different context sizes and are softly combined by a learnable selector. Extensive ray-tracing-based experiments across heterogeneous scenes with diverse BS configurations show that RA-LWLM achieves nearly identical accuracy on seen and unseen scenes without any per-scene retraining, substantially outperforming end-to-end and FM-based baselines. These results validate the proposed retrieval-augmented in-context paradigm as a scalable solution for cross-scene localization in 6G networks.

Abstract

Wireless Localization

Explore similar work

RA-LWLM: Retrieval-Augmented In-Context Localization with Wireless Foundation Models

PRI-Net: A Lightweight Multimodal Framework for 3D UAV Localization

Map as a Prompt: Learning Multi-Modal Spatial-Signal Foundation Models for Cross-scenario Wireless Localization