KLTNet: Learning Sparse Feature Tracking for Robust and Accurate Monocular Visual-Inertial Odometry

Renbiao Jin; Danping Zou; Wenxian Yu

cs.CVAug 25, 2026

KLTNet: Learning Sparse Feature Tracking for Robust and Accurate Monocular Visual-Inertial Odometry

Authors: Renbiao Jin, Danping Zou, Wenxian Yu

Organizations: Shanghai Jiao Tong University, Shanghai 200240, China

Abstract

Many feature-based visual-inertial odometry (VIO) systems rely on sparse feature tracking, whose accuracy and robustness directly affect state estimation. Classical KLT trackers rely primarily on local image patches and can become unreliable under rapid motion or in low-texture environments. We propose KLTNet, a lightweight learning-based, plug-and-play sparse feature tracker designed to replace classical KLT trackers in KLT-based VIO front ends. KLTNet follows a coarse-to-fine, dense-to-sparse architecture that combines low-resolution dense optical flow for robust global motion initialization with triplet-patch refinement for accurate and temporally consistent tracking. A fixed reference patch provides a stable anchor throughout each feature track and helps reduce accumulated tracking drift. In addition, KLTNet predicts anisotropic confidence weights supervised through differentiable multi-view triangulation, which can be used as observation weights in compatible VIO estimators. Experiments with VINS-Mono and OpenVINS on public benchmarks and a self-collected low-texture dataset demonstrate improved tracking and odometry accuracy over classical KLT, while maintaining real-time performance on an embedded platform.

Figures & tables

Fig. 1: Trajectory comparison in a low-texture environment using VINS-Mono with the classical KLT and the proposed KLTNet tracker. (a) KLTNet closely follows ground truth (GT), while classical KLT drifts under challenging scenarios. (b) During rapid rotation (T1-T4) in low-texture regions, KLT exhibits large drift (red dashed circle), but KLTNet remains stable. (c) In steady translation (green dashed box), both perform similarly, demonstrating that KLTNet performs comparably under moderate motion and remains robust when classical KLT degrades.

Fig. 2: (a) Overview of the KLTNet tracker for sparse feature tracking, illustrated with a single point for clarity. Given a feature $p_{0}$ with its reference patch $X_{0}$ , and its tracked position $p_{j-1}$ with corresponding patch $X_{j-1}$ , CoarseFlowNet first estimates a coarse flow $f_{\text{coarse}}$ between previous frame $I_{j-1}$ and the current frame $I_{j}$ to predict an initial position $p_{j}^{\text{coarse}}$ . The TriPatchRefiner then processes the triplet image patch $(X_{0},X_{j-1},X_{j})$ , where $X_{j}$ is extracted around $p_{j}^{\text{coarse}}$ , along with flow priors to regress a fine offset $\Delta f_{j}$ and a confidence weight matrix $W_{j}$ . The final tracked feature position is obtained by adding the refined offset to the initial prediction. (b) CoarseFlowNet architecture. It downscales input images, extracts features using FEN and CEN, constructs a cost volume, and iteratively refines flow using an RNN, followed by upsampling and bilinear sampling to get the coarse flow at the tracked point. (c) TriPatchRefiner architecture. It utilizes FeatEnc to encode triplet image patches. CorrEnc processes cost volumes constructed between patch features. CxtEnc encodes context from patch $X_{j}$ , and PriorEnc encodes flow priors. The concatenated features are fed to FlowDec and ConfDec to predict offset and confidence weights, respectively.

Fig. 3: Feature tracking evaluation on the Replica Dataset. (a) Tracking Precision: the percentage of all evaluated feature observations with tracking error below a specified pixel threshold. KLTNet consistently achieves the highest precision, especially at low error thresholds (1–3 pixels). (b) Temporal Tracking Error: distributions of tracking errors across five equal frame intervals of each 200-frame sub-sequence. Box plots show the median and quartiles of the tracking errors. KLTNet exhibits the slowest error growth over time. (c) Qualitative comparison of tracked feature positions across methods.

Method	Tracker	MH01	MH02	MH03	MH04	MH05	V101	V102	V103	V201	V202	V203	Average
VINS-Mono	KLT	0.157	0.179	0.195	0.316	0.302	0.089	0.113	0.188	0.086	0.154	0.277	0.187
	RAFT-Sparse	0.218	0.248	0.224	0.369	0.305	0.076	0.089	0.129	0.110	0.089	0.236	0.190
	SEA-RAFT-Sparse	0.221	0.158	0.223	0.278	0.324	0.098	0.070	0.100	0.084	0.100	0.166	0.166
	KLTNet (Ours)	0.092	0.088	0.162	0.233	0.207	0.075	0.070	0.074	0.089	0.110	0.149	0.123
OpenVINS	KLT	0.099	0.206	0.150	0.188	0.508	0.069	0.058	0.074	0.105	0.060	0.122	0.149
	RAFT-Sparse	0.239	0.310	0.235	0.404	0.636	0.073	0.077	0.093	0.112	0.084	0.153	0.219

TABLE I: Absolute Trajectory Error (ATE) Comparison on the EuRoC Dataset. Results are in meters. Bold indicates best per block.

Method	Tracker	Room1	Room2	Room3	Room4	Room5	Room6	Average
VINS-Mono	KLT	0.067	0.069	0.121	0.047	0.203	0.075	0.097
	RAFT-Sparse	0.120	0.088	0.095	0.074	0.069	0.130	0.096
	SEA-RAFT-Sparse	0.067	0.058	0.120	0.047	0.032	0.109	0.072
	KLTNet (Ours)	0.052	0.047	0.061	0.029	0.081	0.026	0.049
OpenVINS	KLT	0.055	0.103	0.081	0.041	0.067	0.086	0.072
	RAFT-Sparse	0.116	0.179	0.102	0.098	0.123	0.096	0.119

TABLE II: Absolute Trajectory Error (ATE) Comparison on the TUM-VI Dataset. Results are in meters. Bold indicates best per block.

No.	CoarseFlowNet	TriPatchRefiner	Observation Weight	Weight Supervision	Max Points	Avg. ATE
R1	$\checkmark$	$\times$	Fixed Isotropic	-	150	0.317
R2	$\checkmark$	$\checkmark$	Fixed Isotropic	-	150	0.153
R3	$\checkmark$	$\checkmark$	Isotropic	Reprojection	150	0.139
R4	$\checkmark$	$\checkmark$	Anisotropic	Reprojection	150	0.140
R5	$\checkmark$	$\checkmark$	Isotropic	Triangulation	150	0.131
R6	$\checkmark$	$\checkmark$	Anisotropic	Triangulation	150	0.123

TABLE III: Ablation study of KLTNet components integrated with VINS-Mono on EuRoC. Results are the average ATE in meters over the 11 sequences.

Input Patches	Median Tracking Error (px)		VIO ATE (m)
Input Patches	Short interval	Long interval	VINS-Mono	OpenVINS
two-patch ( $X_{j-1},X_{j}$ )	0.83	1.22	0.135	0.134
triplet-patch ( $X_{0},X_{j-1},X_{j}$ )	0.25	0.59	0.123	0.099

TABLE IV: Impact of the reference patch

X_{0}

on tracking accuracy (Replica) and VIO accuracy (EuRoC).

KITTI-360

UMA-VI

Tracker

09 ∗

KLT

1.17/0.15

1.10/0.22

1.33/ 0.10

1.02/0.21

1.02/0.26

0.99/0.17

1.06/0.14

2.00/0.24

4.50

1.41

0.81

0.24

0.22

SEA-RAFT-Sparse

1.99/0.21

1.30/0.27

1.08/0.15

1.86/0.24

1.09/0.23

1.34/0.21

1.60/0.20

1.88/0.27

7.89

1.07

0.18

0.27

0.41

KLTNet (Ours)

0.72 / 0.12

0.65 / 0.19

0.78 /0.11

0.82 / 0.18

0.88 / 0.22

0.81 / 0.15

0.69 / 0.13

1.65 / 0.21

1.93

0.49

0.16

0.18

0.16

TABLE V: Tracker comparison within VINS-Mono on KITTI-360 and the illumination-change subset of UMA-VI. KITTI-360:

t_{rel}(\%)

r_{rel}(^{\circ}/100m)

; UMA-VI: start-to-end errors in meters. Best results are in bold.

Fig. 4: VIO performance on a self-collected low-texture dataset. (a) A view of the challenging low-texture environment. (b) The handheld data acquisition platform. (c)-(e) Estimated trajectories for KLTNet-VINS-Mono compared against KLT-VINS-Mono, DM-VIO, and DPVO. (f) Absolute Trajectory Error (ATE) in meters. Note that DM-VIO failed to track in seq2.

Fig. 5: Throughput and GPU Memory Comparison. KLTNet, RAFT-Sparse, and SEA-RAFT-Sparse are evaluated within the VINS-Mono visual front end, with throughput measured for the complete front-end pipeline. KLTNet is evaluated on a PC (NVIDIA A6000; LibTorch without TensorRT) and a Jetson AGX Orin (with TensorRT) with maximum track counts of 25, 50, 100, and 150. End-to-end learned odometry systems are shown as system-level reference points.

Point Tracking

Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.

See all 33 papers

ROFT-VINS: Robust Feature Tracking-based Visual-Inertial State Estimation for Harsh Environment
Mar 19, 2026Sanghyun Park, Soohee HanVisual-Inertial Odometry Visual Object Tracking
Department of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea · Department of Electrical Engineering and of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea
MDE-VIO: Enhancing Visual-Inertial Odometry Using Learned Depth Priors
Feb 11, 2026Arda Alniak, Sinan Kalkan, Mustafa Mert Ankarali +2Inertial Odometry Visual-Inertial Odometry
Dept. of Electrical & Electronics Engineering · Center for Image Analysis (OGAM), METU, Ankara, Türkiye · Dept. of Computer Engineering
Robust Structureless Monocular Visual Inertial Initialization Exploiting Line Features and Vanishing Points
Sep 18, 2026Junwan Choi, Woongrae Jo, Dong-Uk Seo +2Visual-Inertial Odometry
School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon, 34141, Republic of Korea

Mar 19, 2026J/K move · Enter open · S save

ROFT-VINS: Robust Feature Tracking-based Visual-Inertial State Estimation for Harsh Environment

Sanghyun Park, Soohee Han

Department of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea · Department of Electrical Engineering and of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea

Visual-Inertial Odometry Visual Object Tracking

Open paper AskPDF arXiv

SLAM (Simultaneous Localization and Mapping) and Odometry are important systems for estimating the position of mobile devices, such as robots and cars, utilizing one or more sensors. Particularly in camera-based SLAM or Odometry, effectively tracking visual features is important as it significantly impacts system performance. In this paper, we propose a method that leverages deep learning to robustly track visual features in monocular camera images. This method operates reliably even in textureless environments and situations with rapid lighting changes. Additionally, we evaluate the performance of our proposed method by integrating it into VINS-Fusion (Monocular-Inertial), a commonly used Visual-Inertial Odometry (VIO) system.

Abstract

Point Tracking

Explore similar work

ROFT-VINS: Robust Feature Tracking-based Visual-Inertial State Estimation for Harsh Environment

MDE-VIO: Enhancing Visual-Inertial Odometry Using Learned Depth Priors

Robust Structureless Monocular Visual Inertial Initialization Exploiting Line Features and Vanishing Points