MomentBA: Second-order Spatial Moments for Anisotropic Correspondence Uncertainty in Differentiable Bundle Adjustment

Yuqing Wang; Xiaoji Niu; Yan Wang; Hailiang Tang; Jian Kuang; Tisheng Zhang

cs.CVSep 12, 2026

MomentBA: Second-order Spatial Moments for Anisotropic Correspondence Uncertainty in Differentiable Bundle Adjustment

Authors: Yuqing Wang, Xiaoji Niu, Yan Wang, Hailiang Tang, Jian Kuang, Tisheng Zhang

Organizations: GNSS Research Center, Wuhan University, Wuhan 430079, China · Electronic Information School, Wuhan University, Wuhan 430079, China · Hubei Technology Innovation Center for Spatiotemporal Information and Positioning Navigation, Wuhan 430079, China · Hubei Luojia Laboratory, Wuhan 430079, China

Abstract

Most existing visual odometry (VO) systems treat feature correspondences as deterministic measurements or assign uniform uncertainty, ignoring the inherent localization ambiguity of different observations. However, correspondence uncertainty is often anisotropic due to image structures such as edges, repetitive patterns, and motion blur, which can significantly affect geometric optimization. In this work, we propose MomentBA, a geometry-aware bundle adjustment framework that derives anisotropic correspondence uncertainty from second-order spatial moments of local similarity responses. Instead of introducing additional covariance prediction networks, the proposed method directly converts matching response distributions into interpretable covariance estimates and incorporates them into bundle adjustment as correspondence-specific information matrices for uncertainty-aware residual weighting. Furthermore, the proposed formulation is integrated into a differentiable optimization framework, establishing a direct connection between correspondence uncertainty and geometric estimation. Experiments on the EuRoC MAV and TartanAir v1 Hard datasets demonstrate that MomentBA improves monocular visual odometry accuracy compared with existing feature-based and learning-based approaches. The proposed anisotropic covariance model achieves lower rotational errors and more robust trajectory estimation than fixed and isotropic uncertainty models, validating the effectiveness of geometry-induced uncertainty modeling for challenging visual environments.

Figures & tables

Fig. 1: We present MomentBA , a differentiable bundle adjustment framework that models anisotropic correspondence uncertainty from similarity responses via second-order spatial moments. The left shows correspondence images and uncertainty visualization, the middle illustrates geometry-induced uncertainty estimation and its integration into bundle adjustment, and the right demonstrates improved trajectory estimation under challenging scenes.

Fig. 2: Overview of MomentBA. Similarity responses from the tracking network are used to derive anisotropic covariances via spatial moments, which are incorporated into bundle adjustment for uncertainty-aware optimization and learning.

Fig. 3: Statistics of MomentBA covariance across feature types. Distributions of anisotropy ratio $\lambda_{1}/\lambda_{2}$ , covariance trace $\mathrm{tr}(\mathbf{\Sigma})$ , and similarity entropy $H(P)$ show increasing geometric ambiguity from corners to edges and repetitive regions.

Method	MH03		MH05		V102		V103		V202		V203		Avg.
Method	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$
VO
TartanVO	0.0302	0.2791	0.0193	0.0604	0.0251	0.1244	0.0263	0.1552	0.0171	0.1251	0.0303	0.2986	0.0247	0.1738
MAC-VO	0.0023	0.0238	0.0025	0.0216	0.0029	0.0434	0.0032	0.0580	0.0018	0.0406	0.0049	0.1284	0.0029	0.0526
Feature
ORB	0.0064	0.0200	0.0085	0.0147	0.0067	0.0406	0.0186	0.0481	0.0061	0.0411	0.0123	0.0628	0.0098	0.0379

TABLE I: Relative pose error comparison on EuRoC dataset.

t_{rel}

denotes translational error and

r_{rel}

denotes rotational error.

Method	H01		H03		H05		H07		H09		H011		Avg.
Method	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$	$t_{rel}$	$r_{rel}$
VO
TartanVO	0.2918	0.0877	0.3275	0.1391	0.3644	0.1899	0.2605	0.0338	0.6182	0.0731	0.2204	0.0287	0.3471	0.0921
MAC-VO	0.3440	0.1052	0.2507	0.0778	0.1485	0.0352	0.0985	0.0418	0.3848	0.0628	0.1292	0.1414	0.2260	0.0774
Feature
ORB	0.1163	0.0667	0.2212	0.0241	0.2464	0.0298	0.2613	0.0325	0.3454	0.0416	0.1826	0.0438	0.2289	0.0398

TABLE II: Relative pose error comparison on TartanAir v1 Hard dataset.

t_{rel}

denotes translational error and

r_{rel}

denotes rotational error.

Fig. 4: Visualization of the estimated anisotropic correspondence covariance on the EuRoC MAV dataset.

Fig. 5: Trajectory comparison of different methods on the TartanAir v1 Hard dataset.

Relative Translation Error ( $t_{\mathrm{rel}}$ , m/frame)
Trajectory	K01	K03	K05	K07	K09
constant	0.1089	0.0321	0.1820	0.1252	0.1224
isotropic	0.2499	0.0344	0.1916	0.0779	0.1424
anisotropic	0.0802	0.0290	0.1394	0.0650	0.0996

TABLE III: Performance comparison of different ablation setups on the KITTI Dataset.

Visual Odometry

Visual odometry estimates camera or robot motion from successive images, enabling navigation and mapping when external positioning is unavailable.

See all 55 papers

ViBA: Implicit Bundle Adjustment with Geometric and Temporal Consistency for Robust Visual Matching
Apr 3, 2026Xiaoji Niu, Yuqing Wang, Yan Wang +2Bundle Adjustment Visual Odometry
GNSS Research Center, Wuhan University, Wuhan 430079, China · Electronic Information School, Wuhan University, Wuhan 430079, China · Hubei Technology Innovation Center for Spatiotemporal Information and Positioning Navigation +1
DAPEVO: Deep Adaptive Patch Frame-Event Visual Odometry
Sep 25, 2026Luca Gandolfi, Simone Nascivera, Roberto Pellerito +3Visual Odometry RGB-Event Sensor Fusion
Robotics and Perception Group, University of Zurich, Switzerland · Bocconi University, Italy
EpiDiffVO: Geometry-Aware Epipolar Diffusion for Robust Visual Odometry
May 19, 2026Prateeth RaoMulti-View Geometry Relative Pose Estimation
International Institute of Information Technology Bangalore, Bengaluru, India

Apr 3, 2026J/K move · Enter open · S save

ViBA: Implicit Bundle Adjustment with Geometric and Temporal Consistency for Robust Visual Matching

Xiaoji Niu, Yuqing Wang, Yan Wang, Hailiang Tang+1

GNSS Research Center, Wuhan University, Wuhan 430079, China · Electronic Information School, Wuhan University, Wuhan 430079, China · Hubei Technology Innovation Center for Spatiotemporal Information and Positioning Navigation+1

Bundle Adjustment Visual Odometry Online Learning

Open paper AskPDF arXiv

Most existing image keypoint detection and description methods rely on datasets with accurate pose and depth annotations, limiting scalability and generalization, and often degrading navigation and localization performance. We propose ViBA, a sustainable learning framework that integrates geometric optimization with feature learning for continuous online training on unconstrained video streams. Embedded in a standard visual odometry pipeline, it consists of an implicitly differentiable geometric residual framework: (i) an initial tracking network for inter-frame correspondences, (ii) depth-based outlier filtering, and (iii) differentiable global bundle adjustment that jointly refines camera poses and feature positions by minimizing reprojection errors. By combining geometric consistency from BA with long-term temporal consistency across frames, ViBA enforces stable and accurate feature representations. We evaluate ViBA on EuRoC and UMA datasets. Compared with state-of-the-art methods such as SuperPoint+SuperGlue, ALIKED, and LightGlue, ViBA reduces mean absolute translation error (ATE) by 12-18% and absolute rotation error (ARE) by 5-10% across sequences, while maintaining real-time inference speeds (FPS 36-91). When evaluated on unseen sequences, it retains over 90% localization accuracy, demonstrating robust generalization. These results show that ViBA supports continuous online learning with geometric and temporal consistency, consistently improving navigation and localization in real-world scenarios.

Abstract

Visual Odometry

Explore similar work

ViBA: Implicit Bundle Adjustment with Geometric and Temporal Consistency for Robust Visual Matching

DAPEVO: Deep Adaptive Patch Frame-Event Visual Odometry

EpiDiffVO: Geometry-Aware Epipolar Diffusion for Robust Visual Odometry