CalibSwitch: Observation-Driven Camera-LiDAR Extrinsic Recovery and Selective Maintenance for Multimodal Perception

Haifa Zhang; Yijing Wang; Zhiqiang Zuo

cs.ROOct 30, 2026

CalibSwitch: Observation-Driven Camera-LiDAR Extrinsic Recovery and Selective Maintenance for Multimodal Perception

Authors: Haifa Zhang, Yijing Wang, Zhiqiang Zuo

Organizations: School of Electrical and Information Engineering, Tianjin University, Tianjin, China

Abstract

Camera--LiDAR perception on mobile robots and intelligent vehicles depends on reliable sensor geometry throughout deployment. Installation variations create uncertainty in the initial extrinsics, while changing observation quality makes frame-wise estimates fluctuate during operation. This paper presents CalibSwitch, an observation-driven framework that separates absolute extrinsic recovery from selective state maintenance. A cross-modal absolute extrinsic generator (CAEG) combines projection-free bidirectional feature interaction with geometry-regularized learning to recover complete SE(3) candidates from image--point observations and coarse view context, without a supplied numerical initial transform. A selective recurrent calibration policy (SRCP) combines observation quality and temporal history to determine when a candidate should replace the working geometry consumed by a frozen multimodal detector. On nuScenes, CAEG achieves mean translation and rotation errors of 0.248 m and 5.884 $^\circ$ . Under sustained geometric transitions, SRCP reaches a mean composite error of 1.46 with 9.70% candidate commitments, outperforming quality-based and temporal-consistency gating at comparable commitment rates. Relative to unconditional per-frame updating, it improves frozen-detector mAP and NDS by 1.43 and 0.92 points. Paired geometry--observation uncertainty tests, two real-vehicle platforms, and 13.5-FPS single-camera Jetson execution demonstrate the system value of selective geometric-state maintenance for continuous perception.

Figures & tables

Fig. 1: Overview of CalibSwitch . CAEG forms an absolute camera-to-LiDAR $\mathrm{SE}(3)$ candidate $\widehat{T}_{t}$ from image–point evidence and coarse view context, without a supplied numerical initial transform. SRCP selects keep or update between the previous working state $T_{t-1}$ and current candidate; the committed state $T_{t}$ supplies the frozen 3D detector. Camera superscripts are omitted.

Fig. 2: Detailed SRCP architecture. The CAEG descriptor and quality/history statistics form $x_{t}$ . The GRU memory update gate $z_{t}$ and reset gate $r_{t}$ update $h_{t}$ ; the actor’s extrinsic commitment action $a_{t}$ selects $\widehat{T}_{t}$ or $T_{t-1}$ to establish $T_{t}$ . The critic and candidate-error head provide training auxiliaries. Camera superscripts are omitted.

Fig. 3: Recovery and maintenance. (a) Frozen detection with four extrinsic sources. (b) Gate threshold sweeps and reference policies; the commitment axis uses symmetric-log scaling with a linear 0–1% region to display zero-commitment Hold. Lines connect tested thresholds. (c) Full-sequence corruption. (d) Paired joint geometry–observation uncertainty.

With numerical initial transform
Method	Numerical initial $T$	$t_{\rm mean}$ (m) $\downarrow$	$t_{\rm P95}$ (m) $\downarrow$	$R_{\rm mean}$ (deg) $\downarrow$	$R_{\rm P95}$ (deg) $\downarrow$
Training-rig mean	Yes	1.725	2.566	23.997	34.493
CalibNet [ 12 ]	Yes	0.423	1.110	9.180	24.40
LCCNet [ 25 ]	Yes	0.351	0.940	7.760	21.10
CalibFormer [ 36 ]	Yes	0.284	0.780	6.440	17.80
Without a numerical transformation matrix

TABLE I: Absolute Extrinsic Accuracy Under Virtual Rig Reconfiguration. Methods are grouped by transformation-matrix input; lower is better.

Method	Init. $T$	$\alpha=1.00$	$\alpha=1.25$	$\alpha=1.50$
CalibNet [ 12 ]	Yes	0.283/6.12	0.423/9.18	0.782/17.43
LCCNet [ 25 ]	Yes	0.231/5.11	0.351/7.76	0.638/14.26
CalibFormer [ 36 ]	Yes	0.190/4.30	0.284/6.44	0.517/11.91
Direct pooled regressor	No	0.251/5.57	0.338/7.47	0.471/10.63
CAEG (ours)	No	0.183/4.16	0.248/5.88	0.341/7.96

TABLE II: Mean Pose Error Beyond the Training Perturbation Range. Each entry is

t

(m)/

R

(deg);

\alpha>1

is OOD.

Policy	Commits	Commit (%) $\downarrow$	Harmful (%) $\downarrow$	Mean $E$ $\downarrow$	Final $E$ $\downarrow$	Mean Peak $E$ $\downarrow$	mAP $\uparrow$	NDS $\uparrow$
First-frame hold	0	0.00	–	2.61	3.98	4.05	62.42	67.87
Periodic-5	6522	18.52	48.10	1.76	2.49	4.17	64.15	68.94
Per-frame update	35214	100.00	49.20	1.61	2.34	3.76	64.61	69.19
Quality Gate	3518	9.99	3.24	1.72	2.13	2.84	65.04	69.41
Temporal Gate	3429	9.74	2.22	1.63	2.07	2.78	65.12	69.56
SRCP (ours)	3415	9.70 ( $\downarrow\,90.3\%$ )	0.54 ( $\downarrow\,98.9\%$ )	1.46 ( $\downarrow\,9.3\%$ )	1.86	2.41	66.04 ( $\uparrow\,1.43$ pts)	70.11 ( $\uparrow\,0.92$ pts)

TABLE III: Working-state maintenance in Protocol I with a shared candidate stream and frozen detector. Harmful rate is the fraction of worsening commitments; “–” denotes no commitments. Green annotations indicate gains over per-frame updating; mAP/NDS gains are in points.

Condition	Policy	Mean $E$ $\downarrow$	Mean Peak $E$ $\downarrow$	Commit (%) $\downarrow$	Harmful (%) $\downarrow$	$D_{\mathrm{rec}}$ $\downarrow$	NDS $\uparrow$
Stable / Clean	Hold	0.79	0.79	0.00	–	–	69.72
	Per-frame	0.84	2.16	100.00	47.83	–	70.06
	Quality Gate	0.77	1.02	7.36	3.12	–	69.98
	SRCP (ours)	0.76	0.91	1.83	0.63	–	70.04
Stable / Degraded	Hold	0.79	0.79	0.00	–	–	68.11
	Per-frame	1.43	4.86	100.00	53.77	–	66.92

TABLE IV: Paired joint uncertainty in Protocol II. Clean/degraded pairs share geometry and initialization.

D_{\mathrm{rec}}

: recovery delay (frames); harmful rate is conditional on commitment; “–”: undefined or inapplicable. Green annotations are relative to per-frame updating within each condition.

Fig. 4: Candidate estimates and working geometry in paired chronological examples. Clean/degraded panels share geometry and initialization within each row. Curves show candidate, Hold-reference, and maintained-extrinsic error. Shading denotes observation degradation, dotted lines mark persistent geometric changes, and purple ticks in the lower action strip mark commitments. A reference change alters held-state error. Panels use different error ranges to resolve each condition.

Fig. 5: Geometric reliability and frozen perception on five common Protocol-I candidate-error strata. (a) mAP and (b) NDS for per-frame CAEG, Quality Gate, and CalibSwitch. Each stratum uses the same frames across policies;

n

denotes its frame count. Lines connect discrete strata. Candidate error defines grouping, while the detector consumes each policy’s working geometry.

Source	Init. $T$	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mAOE $\downarrow$
GT extrinsics	GT	68.35	71.18	0.2792	0.2990
Rig nominal	Yes	68.14	71.01	0.2798	0.3003
CalibFormer	Yes	65.29	69.62	0.2855	0.3081
Pooled regressor	No	64.26	68.87	0.2889	0.3146
LiDAR-only	–	62.20	67.62	0.2924	0.3248
Identity $\mathrm{SE}(3)$	No	61.84	67.31	0.2941	0.3304

TABLE V: Frozen-detector results with alternative extrinsic sources. Bold marks the best method without a numerical initial transform.

Table 11

Panel of Figure 12 — Fig. 6: Real-vehicle platforms. Both use Jetson Orin; this work evaluates their camera–LiDAR subset.

Fig. 7: Natural two-platform deployment. Panels (a)–(b) show UGV garage frames; (c)–(d) show cross-date road scenes. Fixed Extrinsics uses the original calibration-file transform unchanged throughout the sequence; Ours uses CalibSwitch. Input denotes the raw view. All branches share the field of view and privacy masks.

Platform	Policy	Commit (%)	$\overline{\delta R}$ (deg)	$\overline{\delta t}$ (m)	TCS	$\bar{t}$ (m)	$\bar{R}$ (deg)
I: UGV	Hold	0.00	0.000	0.000	0.932	–	–
I: UGV	Per-frame	100.00	2.810	0.184	0.918	–	–
I: UGV	CalibSwitch	1.47	0.043	0.006	0.949	–	–
II: Road	Hold	0.00	0.000	0.000	0.903	0.312	4.906
II: Road	Per-frame	100.00	1.740	0.076	0.917	0.266	4.349
II: Road	CalibSwitch	2.18	0.061	0.004	0.938	0.263	3.813

TABLE VIII: Working-state continuity and perception in natural deployment. Pose errors use the road-platform reference.

Scene	FPS $\uparrow$	E2E (ms) $\downarrow$	CAEG (ms) $\downarrow$	Commit (%)	TCS $\uparrow$
Hallway	13.8	72.5/80.4	7.8/8.6	0.67	0.944
Garage	13.3	75.2/84.7	8.2/9.1	1.86	0.958
Stair	13.7	73.1/81.6	7.9/8.8	1.34	0.931
Overall	13.5	74.1/83.9	8.0/9.0	1.47	0.949

TABLE IX: Platform-I Jetson AGX Orin execution (latency: mean/P95; TCS: temporal consistency).

Lidar-Camera Extrinsic Calibration

See all 19 papers

VeriCalib: Verifiable Target-Based LiDAR-Camera Calibration
Oct 20, 2026Xiangcheng Hu, Xieyuanli Chen, Feiyi Chen +8Lidar-Camera Extrinsic Calibration Camera Calibration
Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China · X Square Robot Technology (Shenzhen) Co., Ltd., Shenzhen, China · College of Intelligence Science and Technology at National University of Defense Technology, Changsha, Hunan, China +4
Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement
May 29, 2026Aziz Al-Najjar, Marzieh Amini, James R. Green +1Multi-View Geometry Bundle Adjustment
Department of System and Computer Engineering, Carleton University, Ottawa, Canada
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
Mar 31, 2026Ni Ou, Zhuo Chen, Xinru Zhang +1Lidar-Camera Extrinsic Calibration Camera Calibration
Ni Ou, Junzheng Wang are with the School of Automation, Beijing Institute of Technology, Beijing, 100081, China. · Zhuo Chen is with the Robot Perception Lab, Centre for Robotics Research, Department of Engineering, King’s College London, London WC2R 2LS, United Kingdom. · Xinru Zhang is with the School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, 100081, China.

Oct 20, 2026J/K move · Enter open · S save

VeriCalib: Verifiable Target-Based LiDAR-Camera Calibration

Xiangcheng Hu, Xieyuanli Chen, Feiyi Chen, Shengli Ren+7

Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China · X Square Robot Technology (Shenzhen) Co., Ltd., Shenzhen, China · College of Intelligence Science and Technology at National University of Defense Technology, Changsha, Hunan, China+4

Lidar-Camera Extrinsic Calibration Camera Calibration Uncertainty Quantification

Open paper AskPDF arXiv

Target-based extrinsic calibration estimates the rigid transform between sensors from a few observations of a known target, and its result is commonly assessed by the root-mean-square error of the fitted residuals. However, this scalar error is computed on the same observations and therefore cannot indicate whether to accept the result, re-extract, recollect or add observations. This paper presents VeriCalib, which formulates extrinsic calibration as a verifiable estimation problem and reports a calibration state. The key insight is a two-level extrinsic compensation mechanism, in which the extrinsic compensates the rigid part of each observation error at the residual level, and rotation and translation compensate each other at the parameter level. Consequently, the fitted residual can reveal neither an inconsistency between the observations and the model nor whether the estimated extrinsic is reliable in every direction. Accordingly, we assess the calibration through a structured covariance that jointly propagates the uncertainty of the camera and LiDAR board features with a known-geometry prior, and estimate its scale as a variance component from the residuals. On this covariance, we decouple rotation and translation to resolve the precision along each direction and guide data collection toward the weakest one. Since this precision presumes consistent observations, we further develop within-view and cross-view checks that examine the internal residual and the rigid discrepancy between observations, respectively, with thresholds that hold their significance level under an unknown noise level. We also extend the minimal detectable bias to the extrinsic to bound the smallest detectable fault along each direction. These criteria form a calibration state that indicates whether to accept the result or how to collect the data...

Abstract

Lidar-Camera Extrinsic Calibration

Explore similar work

VeriCalib: Verifiable Target-Based LiDAR-Camera Calibration

Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement

Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations