VeriFuse: Bounded Vision-Language Arbitration and Reason-Guided Refinement for Cooperative 3D Perception

Hongyi Lin; Yiyao Liu; Qi Kang; Heye Huang; Yang Liu; Haris Koutsopoulos; Jinhua Zhao

cs.CVSep 18, 2026

VeriFuse: Bounded Vision-Language Arbitration and Reason-Guided Refinement for Cooperative 3D Perception

Authors: Hongyi Lin, Yiyao Liu, Qi Kang, Heye Huang, Yang Liu, Haris Koutsopoulos, Jinhua Zhao

Organizations: Tsinghua University · MIT · UESTC · Northeastern University · KAIST

Abstract

Vision-language models (VLMs) have demonstrated strong scene understanding and semantic judgment across diverse tasks, but their appropriate role in cooperative perception remains unclear. Directly asking a VLM to regress 3D detections is unreliable and computationally expensive, whereas using it to select the output of a single source discards useful information from other agents. We introduce VeriFuse, a bounded arbitration framework for vehicle-infrastructure cooperative 3D detection. Each agent first produces detections independently. Around each vehicle and roadside proposal, VeriFuse generates source-conditioned geometric candidates and combines the original detections, their perturbations, and cross-source hypotheses into a unified candidate pool. A frozen VLM then chooses among three admissible actions: SELECT an adequate candidate; REFINE an existing anchor when an object is supported but all candidates are geometrically inadequate; or REJECT an unsupported infrastructure-only proposal. Experiments on the DAIR-V2X dataset show that VeriFuse achieves 0.494/0.357 cooperative 3D AP50/AP70 and limits the relative vehicle-side BEV AP50 drop under a 300 ms delay to 1.7%. Overall, VeriFuse assigns the VLM a clear and constrained role in cooperative perception: semantic reasoning resolves ambiguity among cross-agent hypotheses, while deterministic constraints determine the final 3D geometry.

Figures & tables

Fig. 1: Role of the VLM in cooperative 3D perception. (a) Direct VLM regression predicts box coordinates directly. (b) VeriFuse uses specialized detectors to generate metric hypotheses, a frozen VLM to select an action, and a deterministic executor to constrain the final box.

Fig. 2: Overview of VeriFuse. Source-conditioned proposals form one mixed pool; the VLM selects, rejects, or returns one anchor–reason–token tuple, while deterministic compilation and re-verification bound geometric execution.

Setting	Fixed value
Data split	1,200 / 200 / 600 paired frames; 544 test frames after fixed ego-only filtering
Detectors	Separate PointPillars [ 20 ] ; best development 3D AP 50 checkpoint
Range / class	$[0,100]\!\times\![-40,40]\!\times\![-3,1]$ m; Car
Association	$(\lambda_{\rm I},\lambda_{\rm p},\lambda_{\rm m}){=}(0.45,0.35,0.20)$ ; $R_{\rm p}{=}3$ m, $R_{\rm m}{=}4$ m; IoU gate 0.10; 3 frames
Perturbation / bounds	Vehicle: 0.25 m / $3^{\circ}$ / 3%; RSU: 0.75 m / $6^{\circ}$ / 6%; signed half-step tokens; root-center bounds 0.75/1.50 m
Candidate cap	$K_{\max}{=}16$ ; originals-first, source-balanced truncation

TABLE I: Fixed implementation settings.

		Vehicle-side		Cooperative
Level	Method	3D AP ${}_{50}\uparrow$	3D AP ${}_{70}\uparrow$	3D AP ${}_{50}\uparrow$	3D AP ${}_{70}\uparrow$	[email protected] $\uparrow$	FP/fr. $\downarrow$
None	Vehicle detector	0.636	0.528	0.456	0.312	0.477	0.37
None	RSU detector	0.421	0.283	0.401	0.242	0.292	2.18
Early	Point-cloud concatenation	0.455	0.301	0.335	0.205	0.286	2.94
Intermediate	Where2comm [ 3 ]	0.617	0.406	0.517	0.315	0.438	4.83
Intermediate	CoAlign [ 11 ]	0.628	0.432	0.529	0.342	0.462	3.76

TABLE II: Detection performance on the fixed 544-frame test set after ego-only scene filtering.

Fig. 3: Candidate coverage, refinement, and delay robustness of VeriFuse. (a) Cooperative 3D AP 70 versus initial-pool oracle coverage

\mathrm{OC}_{70}^{0}

with the same frozen VLM arbiter and refinement disabled;

n

is the mean candidate count per cluster. (b) Refinement on positive clusters with

\mathrm{OC}_{50}^{0}=0

: Succ. 50 measures recovery over the entire subset. (c) Vehicle-side BEV AP 50 on the same 1,592 common valid frames at RSU delays of 0, 100, 300, and 500 ms. The gray dashed line marks the vehicle-only baseline (0.612). Percentage labels show relative AP decreases from 0 to 300 ms.

Arbitration rule	Pos. succ. $\uparrow$	IoU regret $\downarrow$	3D AP ${}_{50}\uparrow$	3D AP ${}_{70}\uparrow$
Random selection	0.312	0.214	0.401	0.236
Maximum confidence	0.598	0.126	0.452	0.291
Minimum geom. cost	0.641	0.108	0.459	0.304
Learned geom. selector	0.735	0.071	0.476	0.323
VLM arbiter (ours)	0.836	0.041	0.486	0.342

TABLE III: Arbitration ablation on the fixed full candidate pool.

Method	OC ${}_{50}^{\Gamma}\uparrow$	Succ. ${}_{50}\uparrow$	$\Delta$ IoU $\uparrow$	FB(%) $\downarrow$
Select / Reject	–	0.000	–	–
VLM 7-DoF regression	–	0.091	$-0.018$	23.4
Fixed expansion	0.691	0.421	0.081	0.0
Bounded token w/o verify	0.824	0.564	0.126	2.2
Bounded token + verify	0.824	0.548	0.141	7.4

TABLE IV: Refinement ablation on the initially uncovered hard subset.

Fig. 4: Illustration of VeriFuse on DAIR-V2X-C camera images. Box overlays illustrate the decision paths: (a) vehicle-only and (b) infrastructure-only detections; (c) the mixed candidate pool; (d) Select of an adequate candidate; (e,f) an inadmissible Reject response for a vehicle-supported cluster, followed by fallback to its original vehicle proposal; (g) a Refine diagnosis of a center offset; and (h) the refined box after deterministic compilation.

Explore similar work

ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models
Jul 14, 2026Haojie Ren, Songrui Luo, Lingfeng Wang +6Lidar-Based 3D Object Detection Vision Foundation Models
University of Science and Technology of China
CoGoal3D: Collaborative 3D Object Detection with 3D-Aware Fusion and Refinement
Jul 21, 2026Zhihao Yang, Zhiyu Xiang, Peng Xu +5Point Cloud Surface Reconstruction 3D Object Detection
Zhejiang University, Hangzhou, China · Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing, Hangzhou, China · China Jiliang University, Hangzhou, China
Robust Fusion of Object-Level V2X for Learned 3D Object Detection
May 1, 2026Lukas Ostendorf, Lennart Reiher, Onn Haran +13D Object Detection Cooperative Perception
Institute for Automotive Engineering, RWTH Aachen University, 52074 Aachen, Germany · Qualcomm Technologies, Inc., Kfar-Netter, Israel

Jul 14, 2026, cs.CVJ/K move · Enter open · S save

ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models

Haojie Ren, Songrui Luo, Lingfeng Wang, Yan Xia+5

University of Science and Technology of China

Lidar-Based 3D Object Detection Vision Foundation Models Bird's-Eye-View Representation 3D Object Detection Cooperative Perception

Open paper AskPDF arXiv

LiDAR-based collaborative 3D perception in Vehicle-to-Everything (V2X) systems typically relies on fusing bird's-eye-view (BEV) features across agents. However, current BEV representations, typically extracted by LiDAR backbones trained from scratch, are geometry-dominated and lack general semantic priors, inherently limiting the efficacy of feature-level collaboration. Meanwhile, vision foundation models (VFMs) pretrained on large-scale image data have demonstrated strong capability in learning general-purpose and informative visual representations for 2D tasks, and have the potential to enhance agent-wise LiDAR BEV representations for collaboration. Despite this potential, adapting VFMs to LiDAR-based 3D detection remains challenging due to the substantial image-point cloud modality gap. To bridge this gap, we propose ViCo3D, a collaborative 3D object detection framework powered by VFMs. Specifically, ViCo3D adapts VFMs to LiDAR-based collaborative perception from three aspects: First, ViCo3D projects point clouds onto the BEV plane as three-channel images, enabling DINOv2 to extract BEV-space visual features from LiDAR inputs. Besides, to effectively integrate these DINOv2-derived features with LiDAR geometric features, ViCo3D introduces a multi-scale BEV fusion module within the single-agent encoder. In addition, ViCo3D adopts an ego-centric cross-agent fusion strategy to aggregate complementary information from multiple agents. Experiments on DAIR-V2X and V2XSet demonstrate that ViCo3D achieves state-of-the-art 3D detection performance. Remarkably, it delivers up to 1.8x greater collaborative gains than prior methods on DAIR-V2X. The code will be made public available for future investigation.

Abstract

Explore similar work

ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models

CoGoal3D: Collaborative 3D Object Detection with 3D-Aware Fusion and Refinement

Robust Fusion of Object-Level V2X for Learned 3D Object Detection