HyVGGT-VO: Tightly Coupled Hybrid Dense Visual Odometry with Feed-Forward Models

Junxiang Pan; Lipu Zhou; Baojie Chen

cs.ROApr 2, 2026

HyVGGT-VO: Tightly Coupled Hybrid Dense Visual Odometry with Feed-Forward Models

Authors: Junxiang Pan, Lipu Zhou, Baojie Chen

Organizations: The authors are with the School of Instrument Science and Opto-electronics Engineering, Beihang University, Beijing 100191, China

Abstract

Dense visual odometry (VO), which provides pose estimation and dense 3D reconstruction, serves as the cornerstone for applications ranging from robotics to augmented reality. Recently, feed-forward models have demonstrated remarkable capabilities in dense mapping. However, when these models are used in dense visual SLAM systems, their heavy computational burden restricts them to yielding sparse pose outputs at keyframes while still failing to achieve real-time pose estimation. In contrast, traditional sparse methods provide high computational efficiency and high-frequency pose outputs, but lack the capability for dense reconstruction. To address these limitations, we propose HyVGGT-VO, a novel framework that combines the computational efficiency of sparse VO with the dense reconstruction capabilities of feed-forward models. To the best of our knowledge, this is the first work to tightly couple a traditional VO framework with VGGT, a state-of-the-art feed-forward model. Specifically, we design an adaptive hybrid tracking frontend that dynamically switches between traditional optical flow and the VGGT tracking head to ensure robustness. Furthermore, we introduce a hierarchical optimization framework that jointly refines VO poses and the scale of VGGT predictions to ensure global scale consistency. Our approach achieves an approximately 5x processing speedup compared to existing VGGT-based methods, while reducing the average trajectory error by 85% on the indoor EuRoC dataset and 12% on the outdoor KITTI benchmark. Our code will be publicly available upon acceptance. Project page: https://geneta2580.github.io/HyVGGT-VO.io.

Figures & tables

Fig. 1: Motivation and qualitative comparison of dense 3D reconstructions on the EuRoC dataset. Due to its reliance on heavy sub-graph inference, VGGT-SLAM 2.0 [ 7 ] yields sparse, delayed poses at 3.3 FPS and exhibits severe global scale drift, as seen in the MH01 (b) and V101 (d) sequences. In contrast, our HyVGGT-VO achieves real-time execution at 16.1 FPS, producing globally consistent dense maps for both MH01 (a) and V101 (c) while delivering continuous, high-frequency poses and smoother trajectories.

Fig. 2: Overall architecture of the proposed HyVGGT-VO framework. Taking a monocular image stream as input, the system features a hybrid tracking frontend and an asynchronous hierarchical optimization backend. The frontend adaptively couples efficient KLT optical flow with a robust VGGT tracking head to handle visual degradation. In the backend, the first stage performs covisibility-based local BA for metric precision, while the second stage executes an asynchronous local PGO incorporating VGGT-predicted relative poses and an explicitly optimized scale factor to ensure consistency.

Fig. 3: Structure of our local PGO. The first frame of the first sub-graph is fixed as the world reference frame.

Method	Sequence											Avg
Method	MH01	MH02	MH03	MH04	MH05	V101	V102	V103	V201	V202	V203	Avg
DeepFactors	1.587	1.479	3.139	5.331	4.002	1.520	0.679	0.900	0.876	1.905	1.021	2.040
DeepV2D	0.739	1.144	0.752	1.492	1.567	0.981	0.801	1.570	0.290	2.202	2.743	1.298
TartanVO	0.639	0.325	0.550	1.153	1.021	0.447	0.389	0.622	0.433	0.749	1.152	0.680
DROID-SLAM (odometry only)	0.163	0.121	0.242	0.399	0.270	0.103	0.165	0.158	0.102	0.115	0.204	0.186
MASt3R-SLAM (odometry only)	0.237	0.558	0.341	2.043	0.776	0.126	0.147	0.136	0.117	0.353	0.644	0.498

TABLE I: Euroc Dataset Evaluation (ATE[M]

\downarrow

)

Fig. 4: Qualitative comparison of dense 3D indoor reconstructions on the EuRoC V101 sequence. Prominently, (b) VGGT-SLAM exhibits severe scale drift and structural distortion due to the propagation of $Sim(3)$ alignment errors, while (d) DROID-SLAM yields a highly sparse map caused by its aggressive confidence filtering and down-sample strategy. In contrast, both (a) MASt3R-SLAM and our proposed method (c) maintain global structural integrity. However, as highlighted in the enlarged detailed views, our method preserves significantly richer structural details and superior geometric consistency compared to (a).

Method	Sequence											Avg
Method	00	01	02	03	04	05	06	07	08	09	10	Avg
DROID-SLAM (odometry only)	98.430	84.200	108.800	2.580	0.930	59.270	64.400	24.200	64.550	71.800	16.910	54.188
MASt3R-SLAM (odometry only)	$\times$ *	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
VGGT-SLAM (Sim3)	120.844	144.136	132.728	8.431	3.977	38.376	73.745	29.691	203.832	94.950	22.280	79.363
VGGT-SLAM 2.0 (odometry only)	122.820	155.546	127.870	12.020	4.136	57.155	77.368	26.302	69.657	68.727	23.321	67.720
Ours (high-freq)	114.905	49.097	140.522	2.247	0.691	69.012	62.387	14.799	112.541	80.393	20.558	60.650

TABLE II: KITTI Dataset Evaluation (ATE[M]

\downarrow

)

Fig. 5: The reconstructed global map encompasses approximately 20.1 million 3D points over a 920 meter trajectory (visualized after voxel downsampling). As shown in the zoomed-in views, fine details of colorful architectural structures and vegetation are preserved well, demonstrating the capability of our approach to maintain high-fidelity mapping performance in large-scale outdoor environments.

Method	KITTI		EuRoC
Method	00	01	V103	V203
KLT Only	$\times$	$\times$	$\times$	$\times$
Adaptive Hybrid Frontend	114.29	47.22	0.222	1.514

TABLE III: Ablation Study of Adaptive Hybrid Frontend (ATE[M]

\downarrow

)

Fig. 6: Qualitative trajectory comparison with ATE heatmaps on the KITTI 01 sequence. Left: Result using the proposed adaptive hybrid frontend. Right: Result relying solely on the KLT baseline.

Method	MH04	MH05	V103	V203	Average
w/o Local PGO	0.247	0.170	0.289	1.539	0.561
w Local PGO	0.237	0.163	0.222	1.514	0.534

TABLE IV: Ablation Study of Local PGO (ATE[M]

\downarrow

)

Fig. 7: Qualitative trajectory ATE comparison on the EuRoC V103 sequence. Left: w/ local PGO. Right: w/o local PGO.

Method	Average Processing FPS	GPU Memory Consumption (GB)
DROID-SLAM	OOM	OOM
MASt3R-SLAM	OOM	OOM
VGGT-SLAM	OOM	OOM
VGGT-SLAM2	3.301	6.59
Ours (HyVGGT-VO)	16.07	6.51

TABLE V: System Speed and GPU Memory Consumption Comparison (RTX 4060 Laptop, 8GB VRAM)

Thread	Module	Time (ms)
Frontend	Tracking	$54.11$
	Keyframe Creation	$2.33$
	Total Latency	$56.44$
Backend	First Stage Opt.	$32.70$
Backend	Second Stage Opt.	$9.98$
Async. VGGT	VGGT Inference *	$2807.56$

TABLE VI: Average Execution Time of Individual Modules

Explore similar work

Sep 18, 2026

SFVO: Decoupled Confidence-Guided Stereo-Flow Visual Odometry with Bidirectional PnP

Deep learning-based visual odometry (VO) has achieved significant progress, yet most existing methods focus on a monocular approach, which suffers from scale ambiguity. Stereo VO provides real metric by its nature, but remains less studied in deep learning VO due to its high computational cost and modeling complexity. Recent advances in stereo matching and optical flow estimation have made dense visual correspondence increasingly accurate and reliable, but their complementary geometric information has not been fully exploited for VO. In this paper, we present SFVO, a correspondence-driven stereo VO framework that directly builds upon pretrained stereo matching and optical flow models. SFVO exploits pretrained stereo matching and optical flow models to estimate stereo and temporal correspondences. Instead of learning pose directly from images, SFVO maps learned correspondences into geometric constraints and predicts which points are trustworthy. To improve the reliability of visual correspondence-based geometric constraints, we introduce decoupled confidence maps for rotation and translation. This design better aligns the characteristics of visual correspondence and 6-DoF transformations. Extensive experiments on outdoor and indoor datasets demonstrate that SFVO achieves robust and accurate pose estimation with strong generalization capability. The code will be released.

Kai Zhang, Guoyang Zhao, Jun Ma

Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology, Hong Kong SAR, China

Visual Odometry Camera Pose Estimation

Ask PDF

Sep 5, 2026

NSFlow: End-to-End Differentiable Neuro-Symbolic Optical Flow for Visual Odometry

Sparse optical flow provides stable inter-frame correspondence, playing a key role in Visual Odometry (VO) and Visual-Inertial Odometry (VIO). Classical optimization-based methods, such as Lucas-Kanade (LK), perform well under small displacements but are sensitive to large motions and illumination changes. Modern regression-based learning methods, while more robust in complex scenes, are often computationally heavy and lack explicit geometric consistency, making them less suitable for efficient VO/VIO front-ends. To bridge this gap, we propose a hybrid neuro-symbolic framework that combines the strengths of both paradigms. Our method uses a Convolutional Neural Network (CNN) to extract robust feature representations, which is fed into a differentiable LK optimizer to estimate optical flow in an end-to-end trainable manner. Through implicit differentiation, gradients are propagated across the iterative solver, enabling joint optimization of feature extraction and flow estimation. The resulting system integrates seamlessly into existing VO/VIO pipelines and runs in real-time on embedded platforms. Experiments show that our method outperforms conventional optimization-based flow in challenging conditions such as dynamic lighting and low texture, while also achieving higher accuracy and lower latency than purely regression-based alternatives. When deployed in a VIO system, our method demonstrates significant performance improvement, achieving an average error reduction of 42% on challenging datasets while enhancing tracking stability. The code is publicly available.

Yicheng Lin, Zhipeng Fei, Yuxiu Xu +3

Guangzhou Purpleriver Electronic Technology Co., Ltd, Guangzhou, China · State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China

Optical Flow Visual Odometry Neurosymbolic Learning Visual-Inertial Odometry

Ask PDF

Sep 12, 2026

FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry

Stable and reliable 4D spatial understanding is fundamental for autonomous driving systems. While feedforward reconstruction networks can estimate camera motion and 3D structure in one pass, pose estimation over long videos remains challenged by computational cost, long-context ambiguity, and temporal instability. To address these challenges, we propose Feedforward Visual Odometry (FFVO), a pose-specialized adaptation of joint reconstruction architectures for efficient and temporally stable camera-pose estimation. FFVO uses (i) a compact camera-token representation for computationally efficient temporal aggregation, (ii) a hierarchical local-to-global temporal decoder that mitigates geometric ambiguity by separating short-range motion aggregation from sequence-level integration, and (iii) intermediate trajectory supervision that promotes temporal stability. Extensive evaluation on the Waymo Open Dataset (WOD), KITTI, and a large-scale proprietary benchmark demonstrates that our method performs favorably against existing feedforward approaches, and greatly reduces jitter and drift. These results support FFVO as an effective feedforward camera-pose decoder in long-horizon visual odometry settings.

Meng-Li Shih, Shih-Yang Su, Yuliang Zou +7

Waymo LLC · University of Washington · Google DeepMind

Camera Pose Estimation Visual Odometry Feed-Forward 3D Reconstruction

Ask PDF