Cross-Attentive Multiview Fusion of Vision-Language Embeddings

Tomas Berriel Martins; Martin R. Oswald; Javier Civera

cs.CVApr 14, 2026

Cross-Attentive Multiview Fusion of Vision-Language Embeddings

Authors: Tomas Berriel Martins, Martin R. Oswald, Javier Civera

Organizations: University of Zaragoza, Spain · University of Amsterdam, Netherlands

Abstract

Vision-language models have been key to the development of open-vocabulary 2D semantic segmentation. Lifting these models from 2D images to 3D scenes, however, remains a challenging problem. Existing approaches typically back-project and average 2D descriptors across views, or heuristically select a single representative one, often resulting in suboptimal 3D representations. In this work, we introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints and fuses them into a unified per-3D-instance embedding. As a second contribution, we leverage multiview consistency as a self-supervision signal for this fusion, which significantly improves performance when added to a standard supervised target-class loss. Our Cross-Attentive Multiview Fusion, which we denote with its acronym CAMFusion, not only consistently outperforms naive averaging or single-view descriptor selection, but also achieves state-of-the-art results on 3D semantic and instance classification benchmarks, including zero-shot evaluations on out-of-domain datasets.

Figures & tables

Figure 1: CAMFusion overview. We propose a method to fuse vision-language descriptors from multiple views. Given masks of an object in $n$ images (red cameras), we extract per-view vision-language features ( $F_{1}\dots F_{n}$ ) and aggregate them using our CAMFusion to produce a unified descriptor $F_{mv}$ . We also introduce a multiview contrastive loss that enforces consistency between the fused descriptor $F_{mv}$ and those from unseen viewpoints (blue cameras, $F_{n+1}\dots F_{n+m}$ ), complementing the supervised class contrastive loss typically used for this task.

Figure 2: CAMFusion architecture. Given a set of vision-language descriptors $F_{1},...,F_{n}$ of a 3D instance at $n$ views, these are processed by a multi-view transformer. At each block $d$ , each embedding $E_{i}^{d}$ alternates between attending to it self and attending to its memory $M_{i}^{d}$ made of embeddings from other views. Finally, a learned latent pooling computes the final multi-view vision-language feature $F_{mv}$ .

Panel of Figure 3 — Figure 3: Multiview contrastive loss w/o (a) and w/ (b) the class mask. The mask sets as positive target instances of the same class, preventing the contrastive loss from pushing their descriptors apart.

Figure 4: Qualitative results for open-vocabulary semantic instance classification on Replica (top) and ScanNet200 (bottom), using GT instance masks for all methods. We visualize results of our CAMFusion against the ground-truth and the baselines OV-3DIS and Open YOLO 3D. Observe how CAMFusion produces sharper and more coherent object boundaries filtering out the segmentation noise observed in the baselines, resulting in 3D masks that closely match the ground truth.

Figure 5: 3D Instance segmentation vs number of views . CAMFusion trained with 5 views and evaluated in Replica and ScanNet200 using Mask3D’s masks.

MV method	IoU	Acc	mAP	mAP h	mAP c	mAP t
Naive fusion baselines
AvgPool	55.4	69.5	64.2	67.6	61.8	63.3
L1 Medoid	52.2	68.6	64.1	66.9	62.6	62.7
Our fusion method
class loss	31.4	43.7	37.3	55.3	34.8	21.8
+ weigh. resampling	51.9	63.7	58.7	65.6	57.8	52.6

Table 1: Ablation of loss components on Replica. We compare naïve fusion strategies (Average Pooling, L1 Medoid) against our CAMFusion, incrementally validating the impact of our weighted resampling and multiview constrastive loss.

	All		Head		Common		Tail
Method	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc
OpenScene(OSeg) [ 25 ]	15.9	31.7	35.4	45.6	20.1	31.3	5.8	17.6
HOV-SG [ 41 ]	22.5	34.2	35.9	44.2	23.6	42.3	8.0	16.1
Open3DIS [ 22 ] (SigLip)	25.6	38.7	49.7	64.4	22.1	42.4	4.9	9.4
OV3R [ 10 ]	30.7	41.3	43.1	55.3	27.1	38.0	21.8	30.6
OVO [ 20 ]	27.0	39.1	45.0	59.9	25.1	38.5	11.0	18.8

Table 2: Open-Vocabulary 3D Semantic Segmentation on Replica. CAMFusion integrated with OVO outperforms all previous approaches by a significant margin, even against the same single-view descriptors (“OVO w/ PE+TR”).

	ScanNet20				ScanNet200
Method	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc
OpenScene(LSeg) [ 25 ]	52.7	65.5	67.7	79.6	8.1	12.0	52.0	66.9
OpenScene(OSeg) [ 25 ]	47.0	70.3	57.7	69.8	11.6	22.8	24.5	29.2
Open3DIS [ 22 ]	24.7	40.9	32.5	45.3	9.4	17.0	22.9	32.2
OVO w/ PE+TR [ 20 ]	31.8	61.2	49.1	63.3	16.0	42.8	38.5	49.9
OVO w/ ours	35.7	62.5	54.5	68.3	17.7	45.3	45.2	56.8

Table 3: Open-Vocabulary 3D Semantic Segmentation on ScanNet 20 and 200 . OVO with CAMFusion significantly improves on the challenging ScanNet200 vocabulary.

		Replica				ScanNet200
	Method	mAP	mAP h	mAP c	mAP t	mAP	mAP h	mAP c	mAP t
Top-k	OpenYolo3D [ 2 ]	44.0	47.5	49.0	35.5	39.6	43.3	36.8	38.5
	OV-3DIS [ 13 ]	44.3	46.9	47.2	38.7	30.9	33.9	26.5	32.5
	OV-3DIS w/ PE+TR	52.3	67.3	47.8	45.4	46.6	51.1	38.9	50.5
	AvgPool w/ PE+TR	63.1	74.2	62.4	52.8	52.3	57.7	42.3	58.0
	Ours	73.7	84.2	81.0	55.9	62.1	70.3	51.2	65.6

Table 4: 3D Instance Segmentation on Replica and ScanNet200 using ground-truth 3D masks .

	Method	mAP	mAP h	mAP c	mAP t	mAP new
Top-k	OpenYolo3D [ 2 ]	26.9	18.9	25.7	28.9	17.1
	OV-3DIS [ 13 ]	16.7	33.4	12.1	17.1	8.1
	OV-3DIS w/ PE+TR	30.1	32.2	25.7	32.7	19.7
	AvgPool w/ PE+TR	37.1	34.6	36.7	37.8	30.3
	Ours	42.9	35.0	39.3	46.5	31.8
Top-1	OpenYolo3D [ 2 ]	17.6	13.3	17.7	18.2	12.3

Table 5: 3D Instance Segmentation on 3RScan using ground-truth 3D masks . CAMFusion achieves the overall best performance despite the blurry input images.

	Method	3D mask	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
	Method	proposals	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
Top-k	Open3DIS [ 22 ]	2D+3D	18.5	24.5	28.2	-	-	-
	OV-3DIS [ 13 ]	2D+3D	25.7	34.9	42.3	-	-	-
	OV-3DIS [ 13 ]	3D	18.9	24.4	32.1	-	-	-
	Open-Yolo3D [ 2 ]	3D	17.7	22.6	29.3	18.5	23.1	11.6
	PE+TR+AvgPool	3D	20.8	26.7	39.0	23.6	22.2	16.5

Table 6: 3D Instance Segmentation on Replica. We integrate CAMFusion with Mask3D.

	Method	3D mask	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
	Method	proposal	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
	Mask3D [ 11 ] (c.v.)	3D	26.9	39.8	21.7	17.9	-	-
Top-k	Open3DIS [ 22 ]	2D+3D	23.7	29.4	32.8	27.8	21.2	21.8
	Any3DIS [ 21 ]	2D+3D	25.8	-	-	27.4	23.8	26.4
	OV-3DIS [ 13 ]	2D+3D	32.7	41.4	45.3	34.5	30.7	33.1
	OV-3DIS [ 13 ]	3D	29.0	37.6	42.8	33.0	28.1	25.3

Table 7: 3D Instance Segmentation on ScanNet200. Thanks to its novel instance segmentation, OV-3DIS is the only approach that competes with CAMFusion.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Replica				ScanNet200				ScanNet++
	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc
SigLip SO400M/16
MaskCrop	25.8	46.3	31.1	50.8	14.4	26.8	19.4	36.1	19.7	39.6	48.4	57.0
BBoxCrop	22.1	33.7	36.9	55.1	14.6	22.9	23.6	35.9	18.5	49.0	29.7	43.2
HOV-SG	30.1	53.1	46.1	63.8	18.5	32.4	27.1	42.5	26.1	53.3	41.0	53.9
TextRegion	33.6	45.7	46.0	63.7	22.2	34.9	25.1	37.6	29.1	58.4	42.7	54.4

Appendix

Table A-1: Single-view 2D semantic segmentation metrics on Replica, ScanNet200, ScanNet++ using ground-truth segmentation masks.

	Replica				ScanNet200				ScanNet++
Method	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc
SigLIP-SO400M
MV Fusion: AvgPooling
BboxCrop	30.9	46.8	60.9	71.0	25.1	38.6	32.3	43.2	24.9	41.9	36.9	51.2
MaskCrop	30.3	43.2	62.8	66.2	12.7	18.4	17.8	22.6	24.0	32.7	52.3	59.6
HOVSG	32.9	43.9	65.7	73.8	24.3	33.9	32.5	41.9	32.6	48.4	48.6	62.2

Appendix

Table A-2 : Cross multiview/single-view methods per class instance classification on Replica using GT 3D instances and PE-Core-L14-336 and SigLIP-So400M descriptors. Highlighted in bold the best from each multi view subgroup, and in first , second , and third best overall.

Figure A-1 : Qualitative comparison of 3D instance semantic classification with ground-truth 3D masks on Replica.

Figure A-2 : Qualitative comparison of 3D instance semantic classification with ground-truth 3D masks on ScanNet200.

Multi-View Feature Fusion

See all 25 papers

From Alignment to Fusion in 3D Vision-Language
Sep 23, 2026Xueqi Qiu, Xingyu Miao, Jingjing Deng +33D Representation Learning Vision-Language Alignment
School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK · Department of Automation, Tsinghua University, Beijing, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing 100049, China
Multi-View Foundation Models
Dec 17, 2025Leo Segre, Or Hirschorn, Shai AvidanMulti-View Consistency Vision Foundation Model Adaptation
Tel Aviv University
FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder
Apr 2, 2026Wei Li, Yufan Ren, Hanqing Jiang +6Multi-View Learning Cross-Modal Retrieval
IROOTECH TECHNOLOGY, Hangzhou, Zhejiang, China · Wolf 1069 b Lab, Sany Group, Hangzhou, Zhejiang, China · Zhejiang University, Hangzhou, Zhejiang, China +4

Sep 23, 2026J/K move · Enter open · S save

From Alignment to Fusion in 3D Vision-Language

Xueqi Qiu, Xingyu Miao, Jingjing Deng, Haoran Duan+2

School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK · Department of Automation, Tsinghua University, Beijing, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing 100049, China

3D Representation Learning Vision-Language Alignment Multimodal Fusion 3D Instance Segmentation 3D VLMs 3D Visual Grounding

Open paper AskPDF arXiv

Unified 3D vision-language systems must combine complementary geometry, scale, and appearance cues while supporting tasks from instance segmentation to language-guided reasoning. Existing methods often process point clouds, voxel grids, and multi-view images independently; directly combining these heterogeneous representations may leave substantial feature discrepancy unresolved, while subsequent unconstrained adaptation may distort their internal geometry. We propose an align-then-fuse framework that first applies triple pairwise cosine alignment to establish segment-level correspondence across the three representations and then retrieves task-conditioned features with a prompt-guided query decoder. Before fusion, representation-specific query features are transformed by learnable mappings constrained to the special orthogonal group. These mappings preserve inner products and Euclidean distances within each representation, permitting controlled representation-specific re-parameterisation without arbitrarily distorting its internal geometry. The transformed features are subsequently combined through Adaptive Fusion under downstream task supervision. Experiments cover eight datasets for instance segmentation, visual grounding, question answering, and dense captioning. Compared with PQ3D, the model improves average precision by 3.2 points on ScanNet200 and grounding accuracy by 2.9, 10.6, 4.6, and 4.1 points on ScanRefer, Nr3D, Sr3D, and Multi3DRefer, respectively, while also improving performance on ScanQA, SQA3D, and Scan2Cap. Ablations further support the complementary roles of alignment and orthogonal re-parameterisation and the effectiveness of Adaptive Fusion.

Abstract

Multi-View Feature Fusion

Explore similar work

From Alignment to Fusion in 3D Vision-Language

Multi-View Foundation Models

FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder