LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning

Sadjad Alikhani; Akshay Malhotra; Shahab Hamidi-Rad; Ahmed Alkhateeb

cs.LGFeb 22, 2026

LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning

Authors: Sadjad Alikhani, Akshay Malhotra, Shahab Hamidi-Rad, Ahmed Alkhateeb

Abstract

LWM-Temporal is a new member of the Large Wireless Models (LWM) family that targets the spatiotemporal nature of wireless channels. Designed as a task-agnostic foundation model, LWM-Temporal learns universal channel embeddings that capture mobility-induced evolution and are reusable across various downstream tasks. To achieve this objective, LWM-Temporal operates in the angle-delay-time domain and introduces Sparse Spatio-Temporal Attention (SSTA), a propagation-aligned attention mechanism that restricts interactions to physically plausible neighborhoods, reducing attention complexity by an order of magnitude while preserving geometry-consistent dependencies. LWM-Temporal is pretrained in a self-supervised manner using a physics-informed masking curriculum that emulates realistic occlusions, pilot sparsity, and measurement impairments. Experimental results on channel prediction across multiple mobility regimes show consistent improvements over strong baselines, particularly under long horizons and limited fine-tuning data, highlighting the importance of geometry-aware architectures and geometry-consistent pretraining for learning transferable spatiotemporal wireless representations.

Figures & tables

Fig. 1 : This figure illustrates the geometry-driven evolution of wireless channels in dynamic environments and its role in efficient modeling. As users move, propagation paths evolve according to the underlying geometry, inducing structured changes in angle, delay, and Doppler. Transforming channels into the angle-delay-time domain exposes these physical and temporal structures in a more interpretable and sparser representation, enabling efficient tokenization and sparse attention over physically plausible interactions that are less explicit in the original space-frequency-time domain.

Fig. 2 : LWM-Temporal overview. Channels are transformed to angle-delay, tokenized, and pretrained with physics-informed masking and RoPE-based sparse spatiotemporal attention (SSTA) to produce embeddings. SSTA is bidirectional for reconstruction and causal for forecasting to prevent leakage.

Panel of Figure 3 — Fig. 3 : Attention patterns for angle-delay sequences. Colored strips show the flattened token order (time, angle, delay); each matrix shows allowable attention links. (a) Full attention: all-to-all, $\mathcal{O}(N^{2})$ . (b) SSTA: predefined local neighbors, $\mathcal{O}(KN)$ . (c) SSTA (routed): $K_{r}$ adaptive neighbors, $\mathcal{O}(K_{r}N)$ .

Hyperparameter	Value	Hyperparameter	Value
Depth $E$	12	Heads $H$	8
Embed dim $D$	32	MLP ratio $r$	4
Patch $(P_{h},P_{w})$	$(1,1)$	Activation	SwiGLU
Loss	NMSE	Mask ratio $\rho$	$0.60$ (curr.)
Same-frame window	$3\times 3$	Temporal offsets	$\pm\{1,2,3,4\}$
Top- $K$ (max)	64	Routing fraction $f$	0.2

TABLE I : Pretraining hyperparameters

Panel of Figure 5 — Fig. 4 : Visualization of four physics-informed masking strategies. The first three masks (rectangular, pilot-lattice, random) operate within individual frames, while the spatiotemporal tube mask creates a continuous occluded region that evolves across multiple time steps.

Velocity (m/s)	Model	Ray-Traced Fine-tuning Samples (%)						Stochastic Fine-tuning Samples (%)
		0	1	2	10	50	100	0	1	2	10	50	100
[0,10)	LWM	-15.218	-15.232	-15.297	-15.356	-15.618	-17.080	-6.777	-7.551	-8.506	-9.904	-15.512	-19.658
	WiFo-Tiny	11.242	3.591	3.586	0.621	0.070	0.070	3.693	2.777	2.467	1.398	0.193	-1.982
	WiFo-Base	1.105	0.039	0.037	-0.015	-6.891	-6.917	0.034	0.011	-0.484	-1.694	-7.557	-12.696
	WiFo-Tiny-RT	3.634	0.155	0.059	-0.014	-0.060	-3.918	3.154	2.225	1.646	0.862	0.073	-5.520
	WiFo-Base-RT	1.105	0.037	0.029	-0.119	-7.016	-7.365	0.034	0.010	-0.495	-3.109	-9.408	-13.101

TABLE II : Channel prediction NMSE (dB; lower is better) across training sample percentages for ray-traced and stochastic datasets

Spatio-Temporal Attention

See all 11 papers

Structured Latent Dynamics in Wireless CSI via Homomorphic World Models
Mar 20, 2026Salmane Naoumi, Mehdi Bennis, Marwa ChafiiLatent Dynamics Modeling Channel State Prediction
NYU WIRELESS, NYU Tandon School of Engineering, New York, USA. · Center for Wireless Communications, University of Oulu, Finland. · Engineering Division, New York University (NYU), Abu Dhabi, UAE.
LatentWave: JEPA Pretraining for Wireless Foundation Models
Jun 4, 2026Ahmed Mohamed, Ahmed Aboulfotouh, Hatem Abou-ZeidWireless Foundation Models Self-Supervised Pre-Training
Department of Electrical and Software Engineering, University of Calgary, Canada
PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels
May 19, 2026Berkay Guler, Giovanni Geraci, Hamid JafarkhaniRepresentation Learning Masked Autoencoders
Center for Pervasive Communications and Computing, University of California, Irvine CA, USA · Nokia and Universitat Pompeu Fabra, Spain

Mar 20, 2026J/K move · Enter open · S save

Structured Latent Dynamics in Wireless CSI via Homomorphic World Models

Salmane Naoumi, Mehdi Bennis, Marwa Chafii

NYU WIRELESS, NYU Tandon School of Engineering, New York, USA. · Center for Wireless Communications, University of Oulu, Finland. · Engineering Division, New York University (NYU), Abu Dhabi, UAE.

Latent Dynamics Modeling Channel State Prediction World Model Learning Topological Representation Learning

Open paper AskPDF arXiv

We introduce a self-supervised framework for learning predictive and structured representations of wireless channels by modeling the temporal evolution of channel state information (CSI) in a compact latent space. Our method casts the problem as a world modeling task and leverages the Joint Embedding Predictive Architecture (JEPA) to learn action-conditioned latent dynamics from CSI trajectories. To promote geometric consistency and compositionality, we parameterize transitions using homomorphic updates derived from Lie algebra, yielding a structured latent space that reflects spatial layout and user motion. Evaluations on the DICHASUS dataset show that our approach outperforms strong baselines in preserving topology and forecasting future embeddings across unseen environments. The resulting latent space enables metrically faithful channel charts, offering a scalable foundation for downstream applications such as mobility-aware scheduling, localization, and wireless scene understanding.

Abstract

Spatio-Temporal Attention

Explore similar work

Structured Latent Dynamics in Wireless CSI via Homomorphic World Models

LatentWave: JEPA Pretraining for Wireless Foundation Models

PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels