Latent World Modeling for Point-Cloud Video Pretraining
Organizations: Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence · School of Artificial Intelligence and Robotics, Hunan University · University of Western Australia
Abstract
Masked modeling is the dominant self-supervised paradigm for point-cloud videos. However, it leaks the information it is meant to predict, whether it reconstructs masked points as in MAE-style methods or predicts their latent features. Because point-cloud tokens are anchored at points sampled from each frame, querying a target at its own position reveals where its geometry has moved, in reconstruction and latent prediction alike. For latent predictors, random spatio-temporal masks introduce a further shortcut, as visible frames on both sides of each target allow interpolation rather than forecasting. The pretext task thus reduces to completion. Once the target positions are withheld, a masked latent predictor performs only slightly better than copying the last frame. To remove this leakage, we propose PointWM, a latent world model that turns completion into forecasting by predicting the features and geometry of future frames from strictly earlier context. Each target is queried only with its last-seen position and timestamp, while a latent action learned without annotations is meant to capture the part of the future that the past leaves undetermined. On HOI4D action segmentation with a P4Transformer backbone, PointWM improves F1@50 over training from scratch by 24.9 offline and 29.1 under strictly causal inference. These results show the promise of latent world models as a leakage-free pretraining paradigm for point-cloud videos.Our code is publicly available at https://github.com/InitalZ/PointWM.
Figures & tables
| Skill vs. baseline | Feature change | ||||||
|---|---|---|---|---|---|---|---|
| Pretraining | CL ∗ | CL | CM | LE | mov. | cos | |
| Block | 0.81 | 0.03 | 0.35 | 0.12 | 0.05 | 0.04 | 0.26 |
| Leak | 0.85 | 0.07 | 0.18 | 0.12 | 0.08 | 0.12 | 0.09 |
| Teacher act. | – | 0.27 | 0.15 | 0.12 | 0.30 | 0.26 | 0.52 |
| Last | – | 0.42 | 0.29 | 0.25 | 0.47 | 0.39 | 0.75 |
| PointWM | 0.54 | 0.52 | 0.40 | 0.37 | 0.58 | 0.49 | 0.81 |
| Method | Acc | Edit | F1@10 | F1@25 | F1@50 |
|---|---|---|---|---|---|
| PPTr [ 37 ] | 77.4 | 80.1 | 81.7 | 78.5 | 69.5 |
| + STRL [ 13 ] | 78.4 | 79.1 | 81.8 | 78.6 | 69.7 |
| + VideoMAE [ 34 ] | 78.6 | 80.2 | 81.9 | 78.7 | 69.9 |
| + C2P [ 39 ] | 81.1 | 84.0 | 85.4 | 82.5 | 74.1 |
| + DiMP [ 40 ] | 84.0 | 86.2 | 89.1 | 86.6 | 81.4 |
| + PointWM | 84.2 0.3 | 88.0 0.8 | 92.3 0.5 | 89.4 0.4 | 84.0 0.5 |
| Method | Acc | Method | Acc |
|---|---|---|---|
| PSTNet [ 8 ] | 91.20 | P4Transformer [ 7 ] | 90.94 |
| + PointCMP [ 26 ] | 93.27 | + MaST-Pre [ 25 ] | 91.29 |
| PST-Transformer [ 9 ] | 93.73 | + M2PSC [ 12 ] | 93.03 |
| + MaST-Pre [ 25 ] | 94.08 | + Disco4D [ 41 ] | 93.38 |
| + M2PSC [ 12 ] | 94.84 | + DiMP [ 40 ] | 93.97 |
| + DiMP [ 40 ] | 95.51 | + PointTube-JEPA [ 14 ] | 94.08 |
| Method | Acc | Edit | F1@10 | F1@25 | F1@50 |
|---|---|---|---|---|---|
| PPTr [ 37 ] | 69.7 | 64.5 | 69.4 | 64.8 | 52.9 |
| P4Mamba [ 20 ] | 71.0 | 78.5 | 77.8 | 73.4 | 61.5 |
| PointNet4D [ 20 ] | 70.3 | 72.3 | 73.7 | 69.1 | 57.2 |
| 4DMAP [ 20 ] | 72.4 | 78.7 | 78.8 | 74.8 | 63.6 |
| P4Transformer [ 7 ] | 66.7 | 62.0 | 65.3 | 59.8 | 46.3 |
| + DiMP [ 40 ] | 80.4 | 82.2 | 84.8 | 83.2 | 73.5 |
| Pretext | Downstream | |||||
| Pretraining | Skill | Rank | Seg 10% | Online | MSR | |
| (a) | Leak | 0.07 | 94.3 | 54.2 | 56.1 | 92.37 |
| (b) | Last | 0.42 | 138.7 | 63.4 | 68.6 | 93.94 |
| (c) | (b) block masks | 0.04 | 12.6 | 48.6 | 52.8 | 87.09 |
| (d) | (c) frame VICReg | 0.43 | 178.2 | 65.1 | 69.4 | 93.60 |
| (e) | (d) position head | 0.48 | 182.6 | 67.0 | 72.5 | 94.39 |
| Skill at | |||||
|---|---|---|---|---|---|
| Latent action | Prior | Post. | Clip var. | Seg 10% | |
| None, Table 5 (e) | 0.40 | – | – | – | 67.0 |
| Fixed prior | 0.40 | 0.57 | 0.06 | 34.5 | 67.8 |
| IDM on | 0.42 | 0.69 | 0.12 | 71.4 | 66.9 |
| PointWM | 0.45 | 0.61 | 0.10 | 23.8 | 69.2 |