Cross-Attentive Multiview Fusion of Vision-Language Embeddings

Tomas Berriel Martins; Martin R. Oswald; Javier Civera

cs.CVApr 14, 2026

Cross-Attentive Multiview Fusion of Vision-Language Embeddings

Authors: Tomas Berriel Martins, Martin R. Oswald, Javier Civera

Organizations: University of Zaragoza, Spain · University of Amsterdam, Netherlands

Abstract

Vision-language models have been key to the development of open-vocabulary 2D semantic segmentation. Lifting these models from 2D images to 3D scenes, however, remains a challenging problem. Existing approaches typically back-project and average 2D descriptors across views, or heuristically select a single representative one, often resulting in suboptimal 3D representations. In this work, we introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints and fuses them into a unified per-3D-instance embedding. As a second contribution, we leverage multiview consistency as a self-supervision signal for this fusion, which significantly improves performance when added to a standard supervised target-class loss. Our Cross-Attentive Multiview Fusion, which we denote with its acronym CAMFusion, not only consistently outperforms naive averaging or single-view descriptor selection, but also achieves state-of-the-art results on 3D semantic and instance classification benchmarks, including zero-shot evaluations on out-of-domain datasets.

Figures & tables

Figure 1: CAMFusion overview. We propose a method to fuse vision-language descriptors from multiple views. Given masks of an object in $n$ images (red cameras), we extract per-view vision-language features ( $F_{1}\dots F_{n}$ ) and aggregate them using our CAMFusion to produce a unified descriptor $F_{mv}$ . We also introduce a multiview contrastive loss that enforces consistency between the fused descriptor $F_{mv}$ and those from unseen viewpoints (blue cameras, $F_{n+1}\dots F_{n+m}$ ), complementing the supervised class contrastive loss typically used for this task.

Figure 2: CAMFusion architecture. Given a set of vision-language descriptors $F_{1},...,F_{n}$ of a 3D instance at $n$ views, these are processed by a multi-view transformer. At each block $d$ , each embedding $E_{i}^{d}$ alternates between attending to it self and attending to its memory $M_{i}^{d}$ made of embeddings from other views. Finally, a learned latent pooling computes the final multi-view vision-language feature $F_{mv}$ .

Panel of Figure 3 — Figure 3: Multiview contrastive loss w/o (a) and w/ (b) the class mask. The mask sets as positive target instances of the same class, preventing the contrastive loss from pushing their descriptors apart.

Figure 4: Qualitative results for open-vocabulary semantic instance classification on Replica (top) and ScanNet200 (bottom), using GT instance masks for all methods. We visualize results of our CAMFusion against the ground-truth and the baselines OV-3DIS and Open YOLO 3D. Observe how CAMFusion produces sharper and more coherent object boundaries filtering out the segmentation noise observed in the baselines, resulting in 3D masks that closely match the ground truth.

Figure 5: 3D Instance segmentation vs number of views . CAMFusion trained with 5 views and evaluated in Replica and ScanNet200 using Mask3D’s masks.

MV method	IoU	Acc	mAP	mAP h	mAP c	mAP t
Naive fusion baselines
AvgPool	55.4	69.5	64.2	67.6	61.8	63.3
L1 Medoid	52.2	68.6	64.1	66.9	62.6	62.7
Our fusion method
class loss	31.4	43.7	37.3	55.3	34.8	21.8
+ weigh. resampling	51.9	63.7	58.7	65.6	57.8	52.6

Table 1: Ablation of loss components on Replica. We compare naïve fusion strategies (Average Pooling, L1 Medoid) against our CAMFusion, incrementally validating the impact of our weighted resampling and multiview constrastive loss.

	All		Head		Common		Tail
Method	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc
OpenScene(OSeg) [ 25 ]	15.9	31.7	35.4	45.6	20.1	31.3	5.8	17.6
HOV-SG [ 41 ]	22.5	34.2	35.9	44.2	23.6	42.3	8.0	16.1
Open3DIS [ 22 ] (SigLip)	25.6	38.7	49.7	64.4	22.1	42.4	4.9	9.4
OV3R [ 10 ]	30.7	41.3	43.1	55.3	27.1	38.0	21.8	30.6
OVO [ 20 ]	27.0	39.1	45.0	59.9	25.1	38.5	11.0	18.8

Table 2: Open-Vocabulary 3D Semantic Segmentation on Replica. CAMFusion integrated with OVO outperforms all previous approaches by a significant margin, even against the same single-view descriptors (“OVO w/ PE+TR”).

	ScanNet20				ScanNet200
Method	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc
OpenScene(LSeg) [ 25 ]	52.7	65.5	67.7	79.6	8.1	12.0	52.0	66.9
OpenScene(OSeg) [ 25 ]	47.0	70.3	57.7	69.8	11.6	22.8	24.5	29.2
Open3DIS [ 22 ]	24.7	40.9	32.5	45.3	9.4	17.0	22.9	32.2
OVO w/ PE+TR [ 20 ]	31.8	61.2	49.1	63.3	16.0	42.8	38.5	49.9
OVO w/ ours	35.7	62.5	54.5	68.3	17.7	45.3	45.2	56.8

Table 3: Open-Vocabulary 3D Semantic Segmentation on ScanNet 20 and 200 . OVO with CAMFusion significantly improves on the challenging ScanNet200 vocabulary.

		Replica				ScanNet200
	Method	mAP	mAP h	mAP c	mAP t	mAP	mAP h	mAP c	mAP t
Top-k	OpenYolo3D [ 2 ]	44.0	47.5	49.0	35.5	39.6	43.3	36.8	38.5
	OV-3DIS [ 13 ]	44.3	46.9	47.2	38.7	30.9	33.9	26.5	32.5
	OV-3DIS w/ PE+TR	52.3	67.3	47.8	45.4	46.6	51.1	38.9	50.5
	AvgPool w/ PE+TR	63.1	74.2	62.4	52.8	52.3	57.7	42.3	58.0
	Ours	73.7	84.2	81.0	55.9	62.1	70.3	51.2	65.6

Table 4: 3D Instance Segmentation on Replica and ScanNet200 using ground-truth 3D masks .

	Method	mAP	mAP h	mAP c	mAP t	mAP new
Top-k	OpenYolo3D [ 2 ]	26.9	18.9	25.7	28.9	17.1
	OV-3DIS [ 13 ]	16.7	33.4	12.1	17.1	8.1
	OV-3DIS w/ PE+TR	30.1	32.2	25.7	32.7	19.7
	AvgPool w/ PE+TR	37.1	34.6	36.7	37.8	30.3
	Ours	42.9	35.0	39.3	46.5	31.8
Top-1	OpenYolo3D [ 2 ]	17.6	13.3	17.7	18.2	12.3

Table 5: 3D Instance Segmentation on 3RScan using ground-truth 3D masks . CAMFusion achieves the overall best performance despite the blurry input images.

	Method	3D mask	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
	Method	proposals	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
Top-k	Open3DIS [ 22 ]	2D+3D	18.5	24.5	28.2	-	-	-
	OV-3DIS [ 13 ]	2D+3D	25.7	34.9	42.3	-	-	-
	OV-3DIS [ 13 ]	3D	18.9	24.4	32.1	-	-	-
	Open-Yolo3D [ 2 ]	3D	17.7	22.6	29.3	18.5	23.1	11.6
	PE+TR+AvgPool	3D	20.8	26.7	39.0	23.6	22.2	16.5

Table 6: 3D Instance Segmentation on Replica. We integrate CAMFusion with Mask3D.

	Method	3D mask	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
	Method	proposal	mAP	mAP 50	mAP 25	mAP h	mAP c	mAP t
	Mask3D [ 11 ] (c.v.)	3D	26.9	39.8	21.7	17.9	-	-
Top-k	Open3DIS [ 22 ]	2D+3D	23.7	29.4	32.8	27.8	21.2	21.8
	Any3DIS [ 21 ]	2D+3D	25.8	-	-	27.4	23.8	26.4
	OV-3DIS [ 13 ]	2D+3D	32.7	41.4	45.3	34.5	30.7	33.1
	OV-3DIS [ 13 ]	3D	29.0	37.6	42.8	33.0	28.1	25.3

Table 7: 3D Instance Segmentation on ScanNet200. Thanks to its novel instance segmentation, OV-3DIS is the only approach that competes with CAMFusion.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Replica				ScanNet200				ScanNet++
	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc
SigLip SO400M/16
MaskCrop	25.8	46.3	31.1	50.8	14.4	26.8	19.4	36.1	19.7	39.6	48.4	57.0
BBoxCrop	22.1	33.7	36.9	55.1	14.6	22.9	23.6	35.9	18.5	49.0	29.7	43.2
HOV-SG	30.1	53.1	46.1	63.8	18.5	32.4	27.1	42.5	26.1	53.3	41.0	53.9
TextRegion	33.6	45.7	46.0	63.7	22.2	34.9	25.1	37.6	29.1	58.4	42.7	54.4

Appendix

Table A-1: Single-view 2D semantic segmentation metrics on Replica, ScanNet200, ScanNet++ using ground-truth segmentation masks.

	Replica				ScanNet200				ScanNet++
Method	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc	IoU	Acc	f-IoU	f-Acc
SigLIP-SO400M
MV Fusion: AvgPooling
BboxCrop	30.9	46.8	60.9	71.0	25.1	38.6	32.3	43.2	24.9	41.9	36.9	51.2
MaskCrop	30.3	43.2	62.8	66.2	12.7	18.4	17.8	22.6	24.0	32.7	52.3	59.6
HOVSG	32.9	43.9	65.7	73.8	24.3	33.9	32.5	41.9	32.6	48.4	48.6	62.2

Appendix

Table A-2 : Cross multiview/single-view methods per class instance classification on Replica using GT 3D instances and PE-Core-L14-336 and SigLIP-So400M descriptors. Highlighted in bold the best from each multi view subgroup, and in first , second , and third best overall.

Figure A-1 : Qualitative comparison of 3D instance semantic classification with ground-truth 3D masks on Replica.

Figure A-2 : Qualitative comparison of 3D instance semantic classification with ground-truth 3D masks on ScanNet200.

Multi-View Feature Fusion

See all 23 papers

Sep 23, 2026

From Alignment to Fusion in 3D Vision-Language

Unified 3D vision-language systems must combine complementary geometry, scale, and appearance cues while supporting tasks from instance segmentation to language-guided reasoning. Existing methods often process point clouds, voxel grids, and multi-view images independently; directly combining these heterogeneous representations may leave substantial feature discrepancy unresolved, while subsequent unconstrained adaptation may distort their internal geometry. We propose an align-then-fuse framework that first applies triple pairwise cosine alignment to establish segment-level correspondence across the three representations and then retrieves task-conditioned features with a prompt-guided query decoder. Before fusion, representation-specific query features are transformed by learnable mappings constrained to the special orthogonal group. These mappings preserve inner products and Euclidean distances within each representation, permitting controlled representation-specific re-parameterisation without arbitrarily distorting its internal geometry. The transformed features are subsequently combined through Adaptive Fusion under downstream task supervision. Experiments cover eight datasets for instance segmentation, visual grounding, question answering, and dense captioning. Compared with PQ3D, the model improves average precision by 3.2 points on ScanNet200 and grounding accuracy by 2.9, 10.6, 4.6, and 4.1 points on ScanRefer, Nr3D, Sr3D, and Multi3DRefer, respectively, while also improving performance on ScanQA, SQA3D, and Scan2Cap. Ablations further support the complementary roles of alignment and orthogonal re-parameterisation and the effectiveness of Adaptive Fusion.

Xueqi Qiu, Xingyu Miao, Jingjing Deng +3

School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK · Department of Automation, Tsinghua University, Beijing, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing 100049, China

3D Representation Learning Vision-Language Alignment Multimodal Fusion 3D Instance Segmentation 3D VLMs 3D Visual Grounding

Ask PDF

Dec 17, 2025

Multi-View Foundation Models

Foundation models are vital tools in various Computer Vision applications. They take as input a single RGB image and output a deep feature representation that is useful for various applications. However, in case we have multiple views of the same 3D scene, they operate on each image independently and do not always produce consistent features for the same 3D point. We propose a way to convert a Foundation Model into a Multi-View Foundation Model. Such a model takes as input a set of images and outputs a feature map for each image such that the features of corresponding points are as consistent as possible. This approach bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space. Specifically, we show how to augment Transformers-based foundation models (i.e., DINO, SAM, CLIP) with intermediate 3D-aware attention layers that help match features across different views. As leading examples, we show surface normal estimation and multi-view segmentation tasks. Quantitative experiments show that our method improves feature matching considerably compared to current foundation models.

Leo Segre, Or Hirschorn, Shai Avidan

Tel Aviv University

Multi-View Consistency Vision Foundation Model Adaptation

Ask PDF

Apr 2, 2026

FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder

We propose FusionBERT, a novel multi-view visual fusion framework for image--3D multimodal retrieval. Existing image--3D representation learning methods predominantly focus on feature alignment of a single object image and its 3D model, limiting their applicability in realistic scenarios where an object is typically observed and captured from multiple viewpoints. Although multi-view observations naturally provide complementary geometric and appearance cues, existing multimodal large models rarely explore how to effectively fuse such multi-view visual information for better cross-modal retrieval. To address this limitation, we introduce a multi-view image--3D retrieval framework named FusionBERT, which innovatively utilizes a cross-attention-based multi-view visual aggregator to adaptively integrate features from multi-view images of an object. The proposed multi-view visual encoder fuses inter-view complementary relationships and selectively emphasizes informative visual cues across multiple views to get a more robustly fused visual feature for better 3D model matching. Furthermore, FusionBERT proposes a normal-aware 3D model encoder that can further enhance the 3D geometric feature of an object model by jointly encoding point normals and 3D positions, enabling a more robust representation learning for textureless or color-degraded 3D models. Extensive image--3D retrieval experiments on both synthetic 3D models and real-world industrial mechanical objects demonstrate that FusionBERT achieves significantly higher retrieval accuracy than SOTA multimodal large models under both single-view and multi-view settings, establishing a strong baseline for multi-view multimodal retrieval.

Wei Li, Yufan Ren, Hanqing Jiang +6

IROOTECH TECHNOLOGY, Hangzhou, Zhejiang, China · Wolf 1069 b Lab, Sany Group, Hangzhou, Zhejiang, China · Zhejiang University, Hangzhou, Zhejiang, China +4

Multi-View Learning Cross-Modal Retrieval 3D Representation Learning Multimodal Fusion Multi-View Feature Fusion

Ask PDF

Abstract

Multi-View Feature Fusion

Explore similar work

From Alignment to Fusion in 3D Vision-Language

Multi-View Foundation Models

FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder