MS-Point: Multi-Modal Hierarchical Spectral Learning for Point Cloud Understanding
Organizations: Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Abstract
Point-image alignment learns useful object representations, but different local feature patterns can produce the same pooled descriptor and receive the same alignment signal. We introduce MS-Point, a cross-modal pretraining framework that directly supervises intermediate point features and their neighborhood responses. At selected encoder stages, we compute neighborhood averages and residuals before pooling and align their branch descriptors with image features. Branch-level objectives align each representation with its visual target, while cross-level consistency and adaptive fusion coordinate information across branches. A common interface of intermediate features and spatial neighborhoods supports both graph encoders and point Transformers. Evaluations across object classification, few-shot adaptation, dense prediction, and low-label scene learning demonstrate the transfer value of this design. Same-backbone comparisons and capacity-matched and computation-matched controls connect these improvements to the structure of supervision. Together, our findings support directly supervising intermediate features and neighborhood variation to learn transferable, geometry-aware representations across modalities.
Figures & tables
Appendix figures & tables53 assets
Supplementary material from the paper’s appendix.
Appendix
| Cumulative change | Acc. |
| Global only | 81.7 |
| + Intermediate targets | 84.5 |
| + Cross-level coordination | 85.8 |
| + Adaptive fusion | 86.5 |
| Response | Sources | Branches | Acc. |
| Hierarchy only | None | 0 | 86.5 |
| Both | 2 | 87.4 | |
| Both | 2 | 87.2 | |
| Both | 2 | 87.0 | |
| Both | 4 | 88.3 | |
| Smoothing | 3 | 87.9 |
| Objective | MN40 | Scan | Part | H100 h |
| Global only | 93.2 | 88.3 | 85.7 | 27.4 |
| + Hierarchy | 93.6 | 89.0 | 86.1 | 28.5 |
| + Responses | 94.0 | 89.8 | 86.5 | 29.6 |
| + Fusion (MS-Point-T) | 94.3 | 90.4 | 86.9 | 30.4 |
| Construction | Same graph cost | Same active branches | MN40 | Scan | Part | min/ epoch |
| Duplicate | 92.9 | 88.2 | 85.5 | 50.1 | ||
| Random graph | 93.0 | 88.3 | 85.6 | 50.3 | ||
| Shuffled features | 93.1 | 88.5 | 85.7 | 50.2 | ||
| Smooth only | One active | 93.2 | 88.8 | 85.7 | – | |
| Deviation only | One active | 93.3 | 89.0 | 85.8 | – | |
| Pair | 93.6 | 89.6 | 86.1 | 50.0 |
| Method | 3D/2D passes | Params. M | GFLOPs | GiB | min/epoch | GPU h |
| Global | 2/1 | 71.84 | 13.1 | 9.72 | 40.3 | 24.6 |
| MM-Point | 2/4 | 73.10 | 21.8 | 12.80 | 59.8 | 36.9 |
| MS-Point | 2/1 | 73.37 | 15.2 | 10.03 | 50.0 | 30.5 |
| Point-MAE | 1/0 | 22.1 | 17.9 | 11.40 | 105.0 | 72.0 |
| Inter-MAE | 1/0 | 29.6 | 19.6 | 12.10 | 111.0 | 76.5 |
| Metric | Global | MM-Point | MS-Point |
| 5 | 5 | 5 | |
| MN40 | |||
| Scan | |||
| Part | |||
| S3DIS |
| Pretraining / initialization | 1% | 5% | 10% | 100% |
| Random initialization | 34.2 | 48.8 | 53.7 | 63.4 |
| Scene-only SSL | 40.8 | – | – | – |
| Global MS-Point | 44.6 | 53.9 | 58.1 | 64.7 |
| MS-Point + local | 47.3 | 55.9 | 59.8 | 65.2 |
| Dataset | Use / evaluation |
| ShapeNet ( Chang et al., 2015 ) | SSL 2D–3D pretraining; two point views + rendered image; pretraining loss |
| ModelNet40 / ModelNet10 ( Wu et al., 2015 ) | Downstream object classification; frozen backbone + linear SVM; accuracy |
| ScanObjectNN ( Uy et al., 2019 ) | Real-world object classification / ablation; frozen backbone + linear SVM; accuracy |
| ShapeNetPart ( Yi et al., 2016 ) | Downstream part segmentation; fine-tuning; OA / mIoU |
| S3DIS ( Armeni et al., 2016 ) | Downstream indoor semantic segmentation; fine-tuning; OA / mIoU |
| Pairing | Acc. | |
| None | 0.0 | 84.5 |
| Adjacent | 0.4 | 85.8 |
| All pairs | 0.4 | 85.2 |
| All pairs | 0.8 | 84.6 |
| 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | |
| Acc. | 88.2 | 88.9 | 89.6 | 89.1 | 88.4 |
| Group | Weight |
| Shallow | 0.18 |
| Middle | 0.21 |
| Deep/global | 0.24 |
| Smoothing | 0.20 |
| Deviation | 0.17 |
| Variant | Accuracy (%) | Runtime (s) | Peak memory (GiB) |
| Duplicated-feature heads | 91.4222 | 442.78 | 10.026 |
| Smoothing/deviation | 91.5559 | 551.73 | 10.028 |
| Response | Val. (%) | Parameters | Min. |
| Fixed pair | 90.930 | 73,367,890 | 7.36 |
| Learnable | 91.082 | 73,367,896 | 7.63 |
| Polynomial | 91.184 | 73,367,908 | 8.58 |
| Three-way | 90.955 | 87,574,488 | 8.72 |
| Field | |||
| Point channels | 64 | 256 | 1024 |
| Point pooled dim. | 128 | 512 | 2048 |
| Image stage | C3 | C5 | C5 |
| Image channels | 512 | 2048 | 2048 |
| Image pooled dim. | 1024 | 4096 | 4096 |
| Config. | Hier. | Resp. | Params. (M) | Hours | MN40 | Scan | Part |
| G | – | – | 71.84 | 24.6 | 92.8 | 87.9 | 85.4 |
| H | – | 72.45 | 26.5 | 93.1 | 88.7 | 85.7 | |
| R | – | 72.76 | 27.8 | 93.2 | 88.9 | 85.8 | |
| H+R | 73.12 | 29.3 | 93.4 | 89.2 | 85.9 | ||
| Full | 73.37 | 30.5 | 93.6 | 89.6 | 86.1 |
| Configuration | Smooth | Deviation | MN40 | Scan | Part |
| Global | – | – | 92.8 | 87.9 | 85.4 |
| Smooth only | – | 93.1 | 88.6 | 85.7 | |
| Dev only | – | 93.2 | 88.9 | 85.8 | |
| Pair | 93.6 | 89.6 | 86.1 |
| Images/update | MN40 | Scan | Relative time |
| 1 | 93.6 | 89.6 | 1.00 |
| 2 | 93.7 | 89.8 | 1.11 |
| 3 | 93.7 | 89.8 | 1.24 |
| Metric | Object | Pooled | Geometry |
| MN40 | 93.6 | 93.7 | 93.9 |
| Scan | 89.6 | 89.8 | 90.2 |
| Part class | 84.7 | 84.9 | 85.4 |
| Part instance | 86.1 | 86.3 | 86.8 |
| S3DIS | 67.5 | 67.9 | 69.0 |
| Relative time | 1.00 | 1.01 | 1.18 |
| Token relation | Uses attention | MN40 | Scan | Part | Hours |
| Attention-affinity | Yes | 93.8 | 89.7 | 86.4 | 31.6 |
| Feature kNN | No | 94.0 | 90.0 | 86.6 | 31.3 |
| Radius | No | 94.1 | 90.1 | 86.7 | 30.8 |
| Center kNN | No | 94.3 | 90.4 | 86.9 | 30.4 |
| Map for l2/l4/g | Mapping | MN40 | Scan | Part |
| C5/C5/C5 | High-level | 93.1 | 88.6 | 85.7 |
| C3/C4/C5 | Strict stage | 93.4 | 89.1 | 85.9 |
| C3/C5/C5 | Default | 93.6 | 89.6 | 86.1 |
| C5/C3/C3 | Reversed | 92.7 | 88.0 | 85.3 |
| Width | Params. (M) | MN40 | Scan | Part |
| 128 | 72.62 | 93.3 | 89.1 | 85.8 |
| 256 | 73.37 | 93.6 | 89.6 | 86.1 |
| 512 | 75.10 | 93.6 | 89.5 | 86.1 |
| Replicate | Duplicated features | Responses | Difference |
| A | 91.0454 | 91.3290 | +0.2836 |
| B | 91.2075 | 91.6937 | +0.4862 |
| C | 91.2480 | 91.7342 | +0.4862 |
| D | 91.4100 | 91.6937 | +0.2836 |
| E | 91.5721 | 91.4100 | -0.1621 |
| F | 91.1264 | 90.9643 | -0.1621 |
| Response | Replicate A | Replicate B | Mean |
| Fixed pair | 90.7012 | 91.1585 | 90.9299 |
| Learnable first order | 91.0061 | 91.1585 | 91.0823 |
| Polynomial | 91.4634 | 90.9045 | 91.1839 |
| Three responses | 90.8028 | 91.1077 | 90.9553 |
| Variant | Replicate A | Replicate B | Parameters | Minutes |
| Inactive local-head control | 90.8537 | 90.8028 | 73,450,066 | 7.23 |
| Pooled set matching | 91.0061 | 91.1077 | 73,450,066 | 7.26 |
| Three responses + set matching | 91.2602 | 91.2602 | 87,656,664 | 8.76 |
| Variant | Best validation | Terminal test | Hours |
| Fixed responses | 91.8191 | 90.9238 | 9.63 |
| Polynomial responses | 91.3618 | 90.4781 | 12.55 |
| Pooled set matching | 91.5142 | 91.8152 | 9.67 |
| Three responses + set matching | 91.4126 | 91.2480 | 12.91 |
| Tokenization | 64 patch centers per view by farthest point sampling, with -NN point grouping |
| Supervised blocks | Blocks 4, 8, and 12 of the 12-block point Transformer |
| Response graph | nearest patch centers among all 64 unmasked tokens |
| Method | Pretraining | Backbone | MN40 | Scan | Protocol |
| Point-BERT | Masked token | Transf. | 87.4 | – | P0 |
| Point-MAE | Reconstruction | Transf. | 91.0 | 77.7 | P1 |
| Point-M2AE | Multiscale MAE | Transf. | 92.9 | 84.1 | P0 |
| CrossPoint | Cross-modal | DGCNN | 91.2 | 81.7 | P1 |
| I2P-MAE | Image-guided MAE | Transf. | 93.4 | 87.1 | P0 |
| MM-Point | Multi-view contrast | DGCNN | 92.4 | 87.8 | P2 |
| 3D encoder | 2D encoder | MN40 | Scan | Part | H100 h |
| DGCNN | ResNet-50 | 93.6 | 89.6 | 86.1 | 30.5 |
| Point Transformer | ResNet-50 | 94.3 | 90.4 | 86.9 | 30.4 |
| DGCNN | ConvNeXt-B | 93.9 | 90.0 | 86.5 | 32.1 |
| Point Transformer | ConvNeXt-B | 94.6 | 90.8 | 87.2 | 32.3 |
| Initialization | Freq. (8) | Mid. (6) | Rare (5) | Overall |
| Random initialization | 45.8 | 31.1 | 19.4 | 34.2 |
| Scene-only SSL | 52.4 | 38.8 | 24.7 | 40.8 |
| Global MS-Point | 56.0 | 42.7 | 28.6 | 44.6 |
| MS-Point + local | 58.2 | 45.1 | 32.3 | 47.3 |
Point Cloud Learning
Point Cloud Learning develops methods for representing and interpreting unordered 3D point sets from LiDAR, depth sensors, radar, and 3D models.
See all 170 papers