cs.CVNov 15, 2026

Latent World Modeling for Point-Cloud Video Pretraining

Authors: Zhuoyue Zhang, Jian Liu, Jihua Zhu, Ajmal Saeed Mian

Organizations: Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence · School of Artificial Intelligence and Robotics, Hunan University · University of Western Australia

Abstract

Masked modeling is the dominant self-supervised paradigm for point-cloud videos. However, it leaks the information it is meant to predict, whether it reconstructs masked points as in MAE-style methods or predicts their latent features. Because point-cloud tokens are anchored at points sampled from each frame, querying a target at its own position reveals where its geometry has moved, in reconstruction and latent prediction alike. For latent predictors, random spatio-temporal masks introduce a further shortcut, as visible frames on both sides of each target allow interpolation rather than forecasting. The pretext task thus reduces to completion. Once the target positions are withheld, a masked latent predictor performs only slightly better than copying the last frame. To remove this leakage, we propose PointWM, a latent world model that turns completion into forecasting by predicting the features and geometry of future frames from strictly earlier context. Each target is queried only with its last-seen position and timestamp, while a latent action learned without annotations is meant to capture the part of the future that the past leaves undetermined. On HOI4D action segmentation with a P4Transformer backbone, PointWM improves F1@50 over training from scratch by 24.9 offline and 29.1 under strictly causal inference. These results show the promise of latent world models as a leakage-free pretraining paradigm for point-cloud videos.Our code is publicly available at https://github.com/InitalZ/PointWM.

Figures & tables

Explore similar work

CardsList
  1. Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos

    Aug 25, 2026Jheng-Ling Lee, Shang-Tse ChenSelf-Supervised Point Cloud LearningSelf-Supervised Pre-Training

  2. Diffusion Masked Pretraining for Dynamic Point Cloud

    May 5, 2026Zhuoyue Zhang, Yiding Sun, Chaowei Fang +4Self-Supervised Pre-TrainingPoint Cloud Learning

  3. Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding

    Aug 31, 2026Wei Wang, Yiding Sun, Yuyan Wang +4Masked ModelingPoint Cloud Learning