MAC-I$^2$: Learned Metrics-Aware Covariance for Robust Visual-Inertial Fusion in Initialization and Calibration

Xiang Fei; Yuheng Qiu; Can Xu; Yutian Chen; Ruogu Li; Xingxing Zuo; Wenshan Wang; Sebastian Scherer

cs.ROSep 7, 2026

MAC-I $^2$ : Learned Metrics-Aware Covariance for Robust Visual-Inertial Fusion in Initialization and Calibration

Authors: Xiang Fei, Yuheng Qiu, Can Xu, Yutian Chen, Ruogu Li, Xingxing Zuo, Wenshan Wang, Sebastian Scherer

Organizations: The Robotics Institute, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 15213 USA. · University of Toronto Institute for Aerospace Studies (UTIAS), University of Toronto, 4925 Dufferin St, Toronto, ON M3H 5T6, Canada. · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Masdar City, Abu Dhabi, UAE.

Abstract

Visual-Inertial (VI) fusion is fundamental to accurate and robust state estimation, where camera and IMU measurements are combined according to their respective uncertainties. Existing methods, however, fuse the two modalities with predefined uncertainties, regardless of how reliable each is in the local context, and thus often struggle under challenging environments involving illumination changes, dynamic objects, and textureless regions. In this paper, we present MAC-I $^2$ , which achieves robust VI fusion through learned metric-aware covariance for both modalities, so that vision and IMU compete on their own merits rather than relying on predefined uncertainties. Here, metrics-aware means that each predicted covariance faithfully reflects the actual magnitude of the corresponding measurement noise. On the visual side, we propagate learned feature-matching uncertainties into pose covariances for the fusion. On the inertial side, motivated by the observation that integration error accumulates sharply at the early stage and grows slowly afterward, we design a learned IMU model with a learnable initial covariance, and propose a dedicated fine-tuning strategy on a held-out training subset to enable the metrics-aware covariance on unseen sequences. As a showcase, we build a VI initialization and calibration system, since accurate and robust initialization and calibration are the prerequisite for any reliable VI system. Experiments on EuRoC, and VBR show that MAC-I $^2$ substantially outperforms existing methods: it achieves a 99.9% initialization success rate on EuRoC, reducing gravity and velocity errors by about 60% and 42% over the strongest baseline, and maintains 80% success rate on challenging VBR sequences where baseline methods such as VINS-Mono drop below 10%.

Figures & tables

Fig. 2: Translation and rotation pose errors versus the predicted visual pose covariance on EuRoC. Blue bars denote the mean error within each bin, and grey bars denote the sample counts. The estimated uncertainty exhibits metrics-aware behavior, closely following the optimal $y=x$ .

Fig. 3: Average IMU integration error with ground-truth bias correction on the EuRoC MH_04 sequence.

Fig. 4: Comparison of IMU covariance estimation on unseen EuRoC sequences MH_02 , MH_04 , V1_01 , V1_03 , and V2_02 . Our model produces metrics-aware uncertainties that follow the optimal $y=x$ line, while AirIMU is overconfident. Blue bars denote the mean error in each bin, and grey bars denote the sample counts.

Fig. 5: Overview of the VI initialization and calibration system built on MAC-I 2 . (a) With the learned IMU model: corrected IMU measurements and metrics-aware uncertainties. (b) Without the learned IMU model: raw IMU measurements with explicit gyroscope bias estimation.

Trajectory	MH02		MH04		V101		V103		V202		Avg.
	Vel(m/s)	G.Dir( $\degree$ )	Vel(m/s)	G.Dir( $\degree$ )	Vel(m/s)	G.Dir( $\degree$ )	Vel(m/s)	G.Dir( $\degree$ )	Vel(m/s)	G.Dir( $\degree$ )	Vel(m/s)	G.Dir( $\degree$ )	SR $\uparrow$
VINS-Mono	0.085	1.093	0.367	1.447	0.107	1.582	0.187	2.748	0.100	1.182	0.169	1.611	0.318
DRT-t	0.104	0.962	0.261	1.159	0.080	0.966	0.145	1.679	0.081	0.918	0.134	1.137	0.633
DRT-l	0.065	0.967	0.189	1.091	0.057	1.018	0.269	2.235	0.109	1.206	0.138	1.303	0.640
sqrt-VINS mono ⋆	0.059	1.792	0.121	0.857	0.048	0.841	0.075	1.119	0.062	0.753	0.073	1.072	0.704
FF-VIO-Init ⋆	0.054	0.934	0.139	0.966	0.025	0.884	0.032	1.045	0.028	0.740	0.055	0.914	0.866

TABLE I: Initialization errors and success rate (SR) for the 10KFs setting in EuRoC dataset.

Fig. 6: Comparison of gyroscope bias magnitude estimation on all the sequences of EuRoC dataset.

Panel of Table 7 — TABLE II: Detailed initialization results for the 10KFs setting in VBR dataset. Vel is in m/s, G.Dir in degrees.

Fig. 7: Comparison of camera-IMU extrinsic calibration on TUM-VI calib-imu sequences.

Metrics	Methods			MH02	MH04	V101	V103	V202	Avg
Velocity (m/s)	Id Vis	+	Id IMU	0.013	0.031	0.010	0.026	0.022	0.021
	Id Vis	+	Learned IMU	0.040	0.073	0.037	0.061	0.056	0.053
	Learned Vis	+	Id IMU	0.011	0.021	0.011	0.027	0.029	0.020
	Learned Vis	+	Learned IMU	0.010	0.023	0.010	0.025	0.024	0.018
Gravity Direction (deg)	Id Vis	+	Id IMU	0.239	0.284	0.353	0.671	0.766	0.463
	Id Vis	+	Learned IMU	0.284	0.459	0.385	0.655	0.865	0.530

TABLE III: Initialization errors on the EuRoC dataset under different covariance settings.

Metrics	Methods	MH02	MH04	V101	V103	V202	Avg
Velocity (m/s)	w/o Learned IMU	0.011	0.022	0.010	0.025	0.020	0.017
	AirIMU	0.013	0.031	0.008	0.028	0.020	0.020
	Our Learned IMU	0.010	0.023	0.010	0.025	0.024	0.018
Gravity Direction (deg)	w/o Learned IMU	0.820	0.800	0.872	1.149	0.804	0.889
	AirIMU	0.442	0.438	0.589	0.794	0.791	0.611
	Our Learned IMU	0.166	0.208	0.332	0.581	0.804	0.418

TABLE IV: Initialization errors on the EuRoC dataset with and without the learned IMU model, and against AirIMU.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 8: Network architecture of the learned imu model developed in this paper.

Metrics	Methods	MH02	MH04	V101	V103	V202	Avg
Rotation Error ( $\degree$ )	MAC-VO	0.211	0.189	0.195	0.541	0.459	0.319
	MAC-I 2 w/o Learned IMU	0.132	0.145	0.185	0.410	0.383	0.251
	MAC-I 2 w/ Learned IMU	0.082	0.095	0.159	0.317	0.313	0.193
Translation Error (m)	MAC-VO	0.010	0.033	0.008	0.043	0.024	0.024
	MAC-I 2 w/o Learned IMU	0.009	0.031	0.008	0.038	0.021	0.021
	MAC-I 2 w/ Learned IMU	0.009	0.031	0.008	0.035	0.019	0.020

Appendix

TABLE V: Pose estimation comparison on EuRoC between MAC-VO (vision only) and MAC-I 2 (visual-inertial fusion).

Fig. 9: Per-segment pose error comparison between MAC-VO and our method on EuRoC MH02 (top) and V103 (bottom). In segments where MAC-VO exhibits large errors, our method produces more accurate estimates by shifting trust toward IMU measurements through covariance weighting.

Panel of Figure 14 — Fig. 10: Visual pose covariance binning analysis on (a) TartanAirV2 and (b) TUM-VI. Despite being trained only on synthetic TartanAir with pinhole cameras, the covariance model remains metrics-aware across diverse scenes, sensor types, and motion regimes.

Fig. 11: IMU covariance binning analysis on held-out TUM-VI room sequences. The predicted covariance remains metrics-aware on a different sensor and motion regime than EuRoC.

KFs	Metrics	Methods	MH02	MH04	V101	V103	V202	Avg
5	Vel (m/s)	w/ Learned IMU	0.012	0.029	0.011	0.031	0.025	0.022
	Vel (m/s)	w/o Learned IMU	0.012	0.025	0.010	0.029	0.022	0.020
	G.Dir ( $\degree$ )	w/ Learned IMU	0.267	0.432	0.395	0.785	0.999	0.576
	G.Dir ( $\degree$ )	w/o Learned IMU	0.842	0.854	0.914	1.197	0.871	0.936
10	Vel (m/s)	w/ Learned IMU	0.010	0.023	0.010	0.025	0.024	0.018
	Vel (m/s)	w/o Learned IMU	0.011	0.022	0.010	0.025	0.020	0.018

Appendix

TABLE VI: Sensitivity to keyframe count on EuRoC.

Multimodal Sensor Fusion

Multimodal sensor fusion combines data from different sensors to improve perception, tracking, localization, and prediction.

See all 409 papers

Sep 12, 2026

When Do Learned Priors Help Visual Inertial Estimation? A Controlled Study of Prior Integration, Calibration, Initialization, and Backend Consistency

Learned components are increasingly integrated into geometric visual--inertial estimators to provide motion, depth, bias, uncertainty, or confidence cues. Yet it remains unclear whether gains arise from useful learned priors or from changes in the backend, calibration, initialization, temporal association, or evaluation gauge. We present a controlled framework for learning-augmented visual--inertial estimation that separates fusion gain from the incremental value of a learned prior and evaluates four evidence layers: local motion consistency, global trajectory accuracy, physical-state correctness, and numerical consistency. We instantiate the framework with a MonoViT-based monocular motion prior added as a local relative-motion factor to an unchanged VINS backend. We compare Original VINS and learned-prior VINS under matched sensor streams, timestamps, initialization, frontend/backend settings, and camera--IMU extrinsics, while probing calibration, initialization, state coupling, scale, bundle adjustment, and loop closure. On KITTI, with fixed reference extrinsics, translation APE RMSE is 31.4 m for Original VINS and 31.8 m with the learned prior. Across four recordings, the prior changes mean APE by only -0.2%, while a five-times-higher weight worsens it by 8.2%. Online extrinsic updates increase mean APE by 45.7% and 52.1%, respectively, while mean RPE changes by less than 2%. These results show that fusion performance alone cannot establish the value of learned priors. Reliable evaluation requires same-backend controls and joint analysis of prior compatibility, calibration, initialization, global drift, physical-state error, and backend consistency.

Jinchang Zhang, Guoyu Lu

Department of Computer Science, Indiana University, Bloomington, IN 47405, USA.

Visual-Inertial Odometry Multimodal Sensor Fusion

Ask PDF

Sep 30, 2026

LBDU-VIO: Learned Bias Dynamics and Uncertainty for Visual-Inertial Odometry with Unreliable Vision

Visual-inertial odometry (VIO) for aerial robots relies on high rate inertial measurement unit (IMU) propagation between visual updates. However, conventional multi state constraint Kalman filters (MSCKFs) use random walk bias assumptions and fixed noise parameters, which can limit robustness when visual information is unreliable. To address this problem, we propose LBDU-VIO, a learning-augmented MSCKF with learned continuous time bias dynamics and an IMU uncertainty model. A neural ordinary differential equation (ODE) models continuous time bias dynamics to propagate the filter's bias states, replacing their random walk model. The IMU uncertainty model predicts motion adaptive measurement noise covariances for covariance propagation. Both models are trained with pose supervision without direct labels. Experiments on real world EuRoC and TUM-VI benchmarks show lower errors than representative visual-inertial baselines, including a 25.1% reduction in mean relative position error compared with S-MSCKF on EuRoC sequences with 10s visual outage.

Qizhi Guo, Junning Lyu, Defu Lin +1

School of Aerospace Engineering and the Beijing Key Laboratory of UAV Autonomous Control, Beijing Institute of Technology, Beijing 100081, China

Kalman Filtering Visual-Inertial Odometry Robot State Estimation

Ask PDF

Jun 3, 2026

Uncertainty-Aware Adaptive Sensor Fusion for Autonomous Navigation

This work introduces a hybrid deep learning approach integrated with an Unscented Kalman Filter (UKF) to enhance pose estimation accuracy in Visual-Inertial Odometry (VIO) for autonomous navigation. The proposed model employs a Vision Transformer (ViT) network to effectively capture temporal dependencies from inertial measurement unit (IMU) data and utilizes a Multiscale Convolutional Neural Network (MCNN) to learn optical flow-based motion cues from visual data. An adaptive sensor fusion module dynamically weights IMU and visual features by leveraging estimated uncertainty, thus improving robustness in diverse and challenging environmental conditions. Additionally, a novel uncertainty-aware loss function is proposed to explicitly incorporate prediction uncertainty into the learning process, enabling robust and accurate navigation under noisy, incomplete, or unreliable sensor inputs. Comprehensive evaluations of the KITTI dataset demonstrate that the proposed method significantly outperforms baseline approaches, achieving superior performance in terms of Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). The lightweight and computationally efficient model processes data at 155 FPS on an NVIDIA A100 GPU, making it highly suitable for deployment in resource-constrained autonomous systems.

Simegnew Yihunie Alaba, Yuichi Motai

Department of Electrical and Computer Engineering, College of Engineering, Virginia Commonwealth University, Richmond, VA 23220, USA

Unscented Kalman Filtering Visual-Inertial Odometry Multimodal Sensor Fusion

Ask PDF

Abstract

Multimodal Sensor Fusion

Explore similar work

When Do Learned Priors Help Visual Inertial Estimation? A Controlled Study of Prior Integration, Calibration, Initialization, and Backend Consistency

LBDU-VIO: Learned Bias Dynamics and Uncertainty for Visual-Inertial Odometry with Unreliable Vision

Uncertainty-Aware Adaptive Sensor Fusion for Autonomous Navigation