PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models

Yuanhao Su; Shaofeng Zhang; Xiaosong Jia; Qi Fan

cs.CVFeb 28, 2026

PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models

Authors: Yuanhao Su, Shaofeng Zhang, Xiaosong Jia, Qi Fan

Organizations: University of Science and Technology of China · Fuzhou University · Fudan University · Nanjing University

Abstract

The development of 3D Vision-Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-token prediction loss, using only language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable geometric information in intermediate representations. To address these limitations, we propose {\mname}, a novel feature-level alignment regularization method. {\mname} explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information throughout the language modeling process. Specifically, we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss. By training only a lightweight alignment projector and LoRA adapters, {\mname} achieves explicit feature-level supervision with minimal computational overhead, effectively preventing geometric degradation. Extensive experiments on ModelNet40 and Objaverse datasets demonstrate that our method achieves 2.08 pp improvement on average for classification tasks, with a substantial 7.50 pp gain on the challenging open-vocabulary Objaverse classification task and 4.88 pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of {\mname}. Code is publicly available at https://github.com/yharoldsu0627/PointAlign.

Figures & tables

Figure 1: Overview of our method. Stage 1 adopts the three training recipes of MiniGPT-3D for pre-training. Stage 2 freezes the point cloud encoder, MLP, Q-Former, and modality projector, and only trains the LoRA layers of the LLM and the alignment projector. The alignment projector aligns the latent representations of point cloud tokens in the LLM with the Q-Former output through cosine similarity loss. Flame icons indicate trainable modules, and snowflakes indicate frozen modules.

Model	Reference	LLM Size	3D Data Size	Input	ModelNet40			Objaverse			Average
Model	Reference	LLM Size	3D Data Size	Input	(I)	(C)	Average	(I)	(C)	Average	Average
InstructBLIP-7B [ 3 ]	NeurIPS,23	7B	0K	Single-V. Img.	17.67	22.81	20.24	21.50	26.00	23.75	22.00
InstructBLIP-13B [ 3 ]	NeurIPS,23	13B	0K	Single-V. Img.	21.56	21.92	21.74	21.50	21.50	21.50	21.62
LLaVA-7B [ 24 ]	CVPR,24	7B	0K	Single-V. Img.	27.11	21.68	24.40	37.50	30.00	33.75	29.07
LLaVA-13B [ 24 ]	CVPR,24	13B	0K	Single-V. Img.	27.71	27.76	27.74	39.50	35.50	37.50	32.62
GPT-4o mini [ 16 ]	OpenAI	-	0K	Single-V. Img.	22.00	23.10	22.55	39.00	35.00	37.00	29.78

Table 1: Performance evaluation of generative 3D object classification on ModelNet40 test set and Objaverse dataset. Classification accuracy (%) is evaluated using two prompt formats: Instruction-based (I) “What is this?” and Completion-based (C) “This is an object of”, with average scores provided. Best and second-best performances are highlighted in bold and underlined, respectively.

Model	Reference	LLM Size	3D Data Size	Input	Qwen2-72B-Instruct	Sentence-BERT	SimCSE	Average
InstructBLIP-7B [ 3 ]	NIPS,23	7B	0K	Single-V. Img.	16.10	35.79	36.67	29.52
InstructBLIP-13B [ 3 ]	NIPS,23	13B	0K	Single-V. Img.	13.79	33.52	35.60	27.64
LLaVA-1.5-7B [ 24 ]	CVPR,24	7B	0K	Single-V. Img.	17.80	39.32	41.08	32.73
LLaVA-1.5-13B [ 24 ]	CVPR,24	13B	0K	Single-V. Img.	16.00	39.64	40.90	32.18
GPT-4o mini [ 16 ]	OpenAI	-	0K	Single-V. Img.	26.00	38.70	39.13	34.61
Point-Bind LLM [ 10 ]	arXiv,23	7B	-	Point Cloud	1.93	27.29	25.35	18.19

Table 2: Performance evaluation of 3D object captioning on the Objaverse dataset. Evaluation metrics include Qwen2 evaluation and traditional metrics. Bold and underlined values represent the best and second-best performances, respectively.

Figure 2: Examples of 3D Object Understanding with Our Model. The figure demonstrates our model’s capabilities in the context of 3D object understanding, showcasing its performance on tasks such as 3D object recognition, description generation, and 3D VQA.

Table 3: Qualitative comparison of model outputs for classification and captioning on the Objaverse datasets.

Loss	ModelNet40		Objaverse		Average
Loss	(I)	(C)	(I)	(C)	Average
l1	61.30	60.58	71.50	69.50	65.72
l2	61.51	61.75	69.50	69.50	65.57
cosine	61.55	60.78	72.50	69.50	66.08

Table 4: Ablation study on loss functions. We evaluate the impact of different loss functions at 16-th layer on classification accuracy.

Layer Index	ModelNet40		Objaverse		Average
Layer Index	(I)	(C)	(I)	(C)	Average
8	61.47	60.86	69.50	69.50	65.33
12	61.71	60.33	70.50	68.00	65.14
16	61.55	60.78	72.50	69.50	66.08
20	61.30	60.09	70.50	69.00	65.22
24	61.71	60.86	71.00	66.00	64.89

Table 5: Ablation study on target layers. We evaluate the impact of different layer indices on classification accuracy (%).

$\lambda$	ModelNet40		Objaverse		Average
$\lambda$	(I)	(C)	(I)	(C)	Average
0	60.58	60.29	69.50	68.50	64.72
0.1	61.55	60.78	72.50	69.50	66.08
0.3	61.75	60.21	69.50	67.50	64.74
0.5	61.39	60.94	69.50	68.50	65.08
0.7	61.10	60.94	67.00	68.50	64.39

Table 6: Ablation study on the weight of the alignment loss

\lambda

. We evaluate the impact of different

\lambda

at 16-th layer.

Figure 3: KNN classification accuracy of point cloud tokens extracted from different LLM layers on ModelNet40. We compare the baseline model and our aligned model using K=1 and K=10.

Figure 4: Impact of training data fraction on 3D object captioning performance. We evaluate baseline and aligned models using 10%, 30%, 50%, 70%, and 100% of training data on Objaverse dataset.

Num. Layers	ModelNet40		Objaverse		Average
Num. Layers	(I)	(C)	(I)	(C)	Average
1	61.47	60.82	70.50	65.00	64.45
2	61.26	61.67	71.00	68.00	65.48
3	61.55	60.78	72.50	69.50	66.08
4	61.43	61.02	70.00	68.50	65.24

Table 7: Ablation study on the number of linear layers in the alignment projector. We evaluate the impact at layer 16 on classification accuracy using cosine similarity loss with

\lambda=0.1

Loss	ModelNet40		Objaverse		Average
Loss	(I)	(C)	(I)	(C)	Average
q-former	61.55	60.78	72.50	69.50	66.08
Projector-mid	62.16	60.70	69.50	68.00	65.09
Projector	61.79	60.33	70.00	67.50	64.91

Table 8: Ablation study on the alignment target. We evaluate the impact of different alignment targets at layer 16 on classification accuracy (%) using cosine similarity loss with

\lambda=0.1

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	[email protected]	[email protected]	[email protected]	[email protected]
3D-LLaVA ∗	76.1	36.3	27.0	57.3
+ Ours	78.8	37.1	27.2	57.6

Appendix

Table 9: Scene-level 3D dense captioning on Scan2Cap [ 2 ] . Our alignment regularization is applied to 3D-LLaVA [ 5 ] , which does not use Q-Former. (* denotes reproduced results.)

Layer	ModelNet40		Objaverse		Average
Layer	(I)	(C)	(I)	(C)	Average
MiniGPT-3D	62.50	56.50	57.13	48.03	56.04
8	65.00	56.50	57.18	49.59	57.07
12	66.00	56.00	58.74	49.35	57.52
16	65.50	56.00	58.69	48.66	57.21
20	66.00	56.00	58.51	49.27	57.45

Appendix

Table 10: Layer selection ablation on Phi-3 backbone. Middle layers generally achieve better performance, aligning with the Phi-2 results in Table 5 .

Explore similar work

From Alignment to Fusion in 3D Vision-Language
Sep 23, 2026Xueqi Qiu, Xingyu Miao, Jingjing Deng +33D Representation Learning Vision-Language Alignment
School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK · Department of Automation, Tsinghua University, Beijing, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing 100049, China
P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Models
Apr 17, 2026Geunyoung Jung, Soohong Kim, Kyungwoo Song +1Point Cloud Classification VLM Adaptation
Department of Artificial Intelligence, University of Seoul, South Korea · Department of Applied Statistics, Yonsei University, South Korea
Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment
Apr 23, 2026Jingkun Chen, Ruoshi Xu, Mingqi Gao +23D Spatial Reasoning VLM Hallucination
Northwestern Polytechnical University China · Southern University of Science and Technology China · The University of Sheffield United Kingdom +2

Sep 23, 2026J/K move · Enter open · S save

From Alignment to Fusion in 3D Vision-Language

Xueqi Qiu, Xingyu Miao, Jingjing Deng, Haoran Duan+2

School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK · Department of Automation, Tsinghua University, Beijing, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing 100049, China

3D Representation Learning Vision-Language Alignment Multimodal Fusion 3D Instance Segmentation 3D VLMs 3D Visual Grounding

Open paper AskPDF arXiv

Unified 3D vision-language systems must combine complementary geometry, scale, and appearance cues while supporting tasks from instance segmentation to language-guided reasoning. Existing methods often process point clouds, voxel grids, and multi-view images independently; directly combining these heterogeneous representations may leave substantial feature discrepancy unresolved, while subsequent unconstrained adaptation may distort their internal geometry. We propose an align-then-fuse framework that first applies triple pairwise cosine alignment to establish segment-level correspondence across the three representations and then retrieves task-conditioned features with a prompt-guided query decoder. Before fusion, representation-specific query features are transformed by learnable mappings constrained to the special orthogonal group. These mappings preserve inner products and Euclidean distances within each representation, permitting controlled representation-specific re-parameterisation without arbitrarily distorting its internal geometry. The transformed features are subsequently combined through Adaptive Fusion under downstream task supervision. Experiments cover eight datasets for instance segmentation, visual grounding, question answering, and dense captioning. Compared with PQ3D, the model improves average precision by 3.2 points on ScanNet200 and grounding accuracy by 2.9, 10.6, 4.6, and 4.1 points on ScanRefer, Nr3D, Sr3D, and Multi3DRefer, respectively, while also improving performance on ScanQA, SQA3D, and Scan2Cap. Ablations further support the complementary roles of alignment and orthogonal re-parameterisation and the effectiveness of Adaptive Fusion.

Abstract

Explore similar work

From Alignment to Fusion in 3D Vision-Language

P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Models

Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment