calibfusion: Transformer-Based Differentiable Calibration for Radar-Camera Fusion Detection in Water-Surface Environments

Yuting Wan; Liguo Sun; Jiuwu Hao; Pin LV

cs.CVMar 3, 2026

calibfusion: Transformer-Based Differentiable Calibration for Radar-Camera Fusion Detection in Water-Surface Environments

Authors: Yuting Wan, Liguo Sun, Jiuwu Hao, Pin LV

Abstract

Millimeter-wave (mmWave) Radar--Camera fusion improves perception under adverse illumination and weather, but its performance is sensitive to Radar--Camera extrinsic calibration: residual misalignment biases Radar-to-image projection and degrades cross-modal aggregation for downstream 2D detection. Existing calibration and auto-calibration methods are mainly developed for road and urban scenes with abundant structures and object constraints, whereas water-surface environments feature large textureless regions, sparse and intermittent targets, and wave-/specular-induced Radar clutter, which weakens explicit object-centric matching. We propose CalibFusion, a calibration-conditioned Radar--Camera fusion detector that learns implicit extrinsic refinement end-to-end with the detection objective. CalibFusion builds a multi-frame persistence-aware Radar density representation with intensity weighting and Doppler-guided suppression of fast-varying clutter. A cross-modal transformer interaction module predicts a confidence-gated refinement of the initial extrinsics, which is integrated through a differentiable projection-and-splatting operator to generate calibration-conditioned image-plane Radar features. Experiments on WaterScenes and FLOW show improved fusion-based 2D detection and robustness under synthetic miscalibration, supported by sensitivity analyses and qualitative Radar-to-image overlays. Results on nuScenes indicate that the refinement mechanism transfers beyond water-surface scenarios.

Figures & tables

Fig. 1: Overall architecture of CalibFusion. CalibFusion includes: (1) Doppler-Guided Persistence Density for multi-frame Radar density construction; (2) Cross-Modal Token Interaction between image and Radar tokens; (3) Confidence-Gated Extrinsic Refinement that updates the initial extrinsic $T_{0}$ ; and (4) Calibration-Conditioned Projection that generates image-plane Radar features for fusion-based 2D detection. The model is trained end-to-end with detection supervision backpropagated through the differentiable projection.

Fig. 2: Cross-Modal Token Interaction via bi-directional cross-attention. (a) Radar-to-image attention updates Radar tokens using Radar queries and image keys/values. (b) Image-to-Radar attention updates image tokens using image queries and Radar keys/values. Layer normalization and residual updates are applied around each attention block.

Camera-only
Performance	Sensor	mAP 50	mAP 50:95
YOLOv8-N [ 20 ]	Camera	75.2	29.9
YOLOv8-S [ 20 ]	Camera	79.1	32.9
YOLOv8-M [ 20 ]	Camera	79.5	33.1
YOLOv8-L [ 20 ]	Camera	75.6	34.0
YOLOv8-X [ 20 ]	Camera	79.7	34.3

TABLE I: Performance comparison on the FLOW dataset under different sensor settings.

Range	Methods	Rotation ( ∘ )				Translation (cm)
Range	Methods	Mean	Roll	Pitch	Yaw	Mean	X	Y	Z
R1 ( $\pm 10^{\circ}$ , $\pm 0.25$ m)	LCCNet-1	1.603	0.123	3.130	1.556	16.531	22.992	17.648	8.954
	NetCalib2	1.205	0.387	2.289	0.941	12.297	12.532	12.076	12.284
	CalibDepth	0.807	0.390	0.345	1.686	12.608	12.860	12.250	12.715
	Coarse	2.035	0.581	1.519	4.004	–	–	–	–
	Fine	1.692	0.442	0.939	3.695	–	–	–	–

TABLE II: Comparison with LiDAR–Camera-based and Radar–Camera-based auto-calibration methods on the nuScenes dataset.

Fig. 3: Calibration-conditioned projection results. (a,d) Miscalibrated Radar-to-image projections under $T_{0}^{\prime}$ . (b,e) Refined projections after CalibFusion predicts $T_{t}$ , improving cross-modal alignment. (c,f) Corresponding multi-frame Radar density maps used for refinement.

Fig. 4: Calibration error statistics on the WaterScenes test set under a broad range of initial perturbations: (a) distribution of translation errors; (b) distribution of rotation errors.

Fig. 5: Refinement error under axis-wise injected extrinsic offsets. Top row: translation error (m); bottom row: rotation error (deg). In each subplot, the solid curve corresponds to the perturbed axis (X or Y), while the dashed curve (“Compared”) reports the error on the other axis to indicate cross-axis coupling. (a,c) Y-axis offsets; (b,d) X-axis offsets.

Fig. 6: Representative detection results of CalibFusion on WaterScenes, illustrating Radar–Camera fusion detection across typical targets and water-surface environments.

Image Backbone.	Radar Backbone	mAP 50	mAP 50:95
ResNet101	PointPillar	94.2	46.8
ResNet101	PointNet++	94.5	46.7
SwinT.	PointPillar	94.7	46.6
SwinT.	PointNet++	95.3	47.1

TABLE III: Ablation study of image backbone and Radar backbone (Image Bb. )

Camera-Radar Fusion

Camera-radar fusion combines visual semantics with radar’s range, velocity, and all-weather sensing for robust perception.

See all 49 papers

physfusion: A Transformer-based Dual-Stream Radar and Vision Fusion Framework for Open Water Surface Object Detection
Mar 2, 2026Yuting Wan, Liguo Sun, Jiuwu Hao +2Camera-Radar Fusion Object Detection
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, China.
Adversarial Robustness of Camera-Radar Sensor Fusion for Autonomous Driving
Oct 12, 2026Ce Zhou, Yubo Wu, Khang Nguyen +3Camera-Radar Fusion Adversarial Robustness
Missouri University of Science and Technology, USA · Virginia Tech, USA · Michigan State University, USA
Online Target-less Radar-LiDAR-Camera Extrinsic Calibration via Joint Optimization
Oct 3, 2026Gunhee Shin, Yunsoo Kim, Chanhyuk Lee +7Lidar-Camera Extrinsic Calibration Radar-LiDAR Fusion
School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon, 34141, Korea · Kookmin University, Seoul, 02707, Korea · LG Innotek, Seoul, 07796, Korea

Mar 2, 2026J/K move · Enter open · S save

physfusion: A Transformer-based Dual-Stream Radar and Vision Fusion Framework for Open Water Surface Object Detection

Yuting Wan, Liguo Sun, Jiuwu Hao, Zao Zhang+1

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, China.

Camera-Radar Fusion Object Detection Millimeter-Wave Radar

Open paper AskPDF arXiv

Detecting water-surface targets for Unmanned Surface Vehicles (USVs) is challenging due to wave clutter, specular reflections, and weak appearance cues in long-range observations. Although 4D millimeter-wave radar complements cameras under degraded illumination, maritime radar point clouds are sparse and intermittent, with reflectivity attributes exhibiting heavy-tailed variations under scattering and multipath, making conventional fusion designs struggle to exploit radar cues effectively. We propose PhysFusion, a physics-informed radar-image detection framework for water-surface perception. The framework integrates: (1) a Physics-Informed Radar Encoder (PIR Encoder) with an RCS Mapper and Quality Gate, transforming per-point radar attributes into compact scattering priors and predicting point-wise reliability for robust feature learning under clutter; (2) a Radar-guided Interactive Fusion Module (RIFM) performing query-level radar-image fusion between semantically enriched radar features and multi-scale visual features, with the radar branch modeled by a dual-stream backbone including a point-based local stream and a transformer-based global stream using Scattering-Aware Self-Attention (SASA); and (3) a Temporal Query Aggregation module (TQA) aggregating frame-wise fused queries over a short temporal window for temporally consistent representations. Experiments on WaterScenes and FLOW demonstrate that PhysFusion achieves 59.7% mAP50:95 and 90.3% mAP50 on WaterScenes (T=5 radar history) using 5.6M parameters and 12.5G FLOPs, and reaches 94.8% mAP50 and 46.2% mAP50:95 on FLOW under radar+camera setting. Ablation studies quantify the contributions of PIR Encoder, SASA-based global reasoning, and RIFM.

Abstract

Camera-Radar Fusion

Explore similar work

physfusion: A Transformer-based Dual-Stream Radar and Vision Fusion Framework for Open Water Surface Object Detection

Adversarial Robustness of Camera-Radar Sensor Fusion for Autonomous Driving

Online Target-less Radar-LiDAR-Camera Extrinsic Calibration via Joint Optimization