cs.AINov 18, 2026

MS-Point: Multi-Modal Hierarchical Spectral Learning for Point Cloud Understanding

Authors: Hai-tao Yu, Jun Xia

Organizations: Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China

Abstract

Point-image alignment learns useful object representations, but different local feature patterns can produce the same pooled descriptor and receive the same alignment signal. We introduce MS-Point, a cross-modal pretraining framework that directly supervises intermediate point features and their neighborhood responses. At selected encoder stages, we compute neighborhood averages and residuals before pooling and align their branch descriptors with image features. Branch-level objectives align each representation with its visual target, while cross-level consistency and adaptive fusion coordinate information across branches. A common interface of intermediate features and spatial neighborhoods supports both graph encoders and point Transformers. Evaluations across object classification, few-shot adaptation, dense prediction, and low-label scene learning demonstrate the transfer value of this design. Same-backbone comparisons and capacity-matched and computation-matched controls connect these improvements to the structure of supervision. Together, our findings support directly supervising intermediate features and neighborhood variation to learn transferable, geometry-aware representations across modalities.

Figures & tables

Appendix figures & tables53 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

    Jul 1, 2026Trung Thanh Nguyen, Hai Nguyen-Truong, Tu Vo +2Cross-Modal Knowledge DistillationPoint Cloud Learning

  2. Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understanding

    Jul 21, 2026Junlin Chang, Longhao Zou, Rui Li3D Point Cloud SegmentationPoint Cloud Classification

  3. Utonia: Toward One Encoder for All Point Clouds

    Mar 3, 2026Yujia Zhang, Xiaoyang Wu, Yunhan Yang +63D Foundation Models3D ViTs