Many feature-based visual-inertial odometry (VIO) systems rely on sparse feature tracking, whose accuracy and robustness directly affect state estimation. Classical KLT trackers rely primarily on local image patches and can become unreliable under rapid motion or in low-texture environments. We propose KLTNet, a lightweight learning-based, plug-and-play sparse feature tracker designed to replace classical KLT trackers in KLT-based VIO front ends. KLTNet follows a coarse-to-fine, dense-to-sparse architecture that combines low-resolution dense optical flow for robust global motion initialization with triplet-patch refinement for accurate and temporally consistent tracking. A fixed reference patch provides a stable anchor throughout each feature track and helps reduce accumulated tracking drift. In addition, KLTNet predicts anisotropic confidence weights supervised through differentiable multi-view triangulation, which can be used as observation weights in compatible VIO estimators. Experiments with VINS-Mono and OpenVINS on public benchmarks and a self-collected low-texture dataset demonstrate improved tracking and odometry accuracy over classical KLT, while maintaining real-time performance on an embedded platform.
Figures & tables
Fig. 1: Trajectory comparison in a low-texture environment using VINS-Mono with the classical KLT and the proposed KLTNet tracker. (a) KLTNet closely follows ground truth (GT), while classical KLT drifts under challenging scenarios. (b) During rapid rotation (T1-T4) in low-texture regions, KLT exhibits large drift (red dashed circle), but KLTNet remains stable. (c) In steady translation (green dashed box), both perform similarly, demonstrating that KLTNet performs comparably under moderate motion and remains robust when classical KLT degrades.
Fig. 2: (a) Overview of the KLTNet tracker for sparse feature tracking, illustrated with a single point for clarity. Given a feature p0 with its reference patch X0 , and its tracked position pj−1 with corresponding patch Xj−1 , CoarseFlowNet first estimates a coarse flow fcoarse between previous frame Ij−1 and the current frame Ij to predict an initial position pjcoarse . The TriPatchRefiner then processes the triplet image patch (X0,Xj−1,Xj) , where Xj is extracted around pjcoarse , along with flow priors to regress a fine offset Δfj and a confidence weight matrix Wj . The final tracked feature position is obtained by adding the refined offset to the initial prediction. (b) CoarseFlowNet architecture. It downscales input images, extracts features using FEN and CEN, constructs a cost volume, and iteratively refines flow using an RNN, followed by upsampling and bilinear sampling to get the coarse flow at the tracked point. (c) TriPatchRefiner architecture. It utilizes FeatEnc to encode triplet image patches. CorrEnc processes cost volumes constructed between patch features. CxtEnc encodes context from patch Xj , and PriorEnc encodes flow priors. The concatenated features are fed to FlowDec and ConfDec to predict offset and confidence weights, respectively.
Fig. 3: Feature tracking evaluation on the Replica Dataset. (a) Tracking Precision: the percentage of all evaluated feature observations with tracking error below a specified pixel threshold. KLTNet consistently achieves the highest precision, especially at low error thresholds (1–3 pixels). (b) Temporal Tracking Error: distributions of tracking errors across five equal frame intervals of each 200-frame sub-sequence. Box plots show the median and quartiles of the tracking errors. KLTNet exhibits the slowest error growth over time. (c) Qualitative comparison of tracked feature positions across methods.
Method
Tracker
MH01
MH02
MH03
MH04
MH05
V101
V102
V103
V201
V202
V203
Average
VINS-Mono
KLT
0.157
0.179
0.195
0.316
0.302
0.089
0.113
0.188
0.086
0.154
0.277
0.187
RAFT-Sparse
0.218
0.248
0.224
0.369
0.305
0.076
0.089
0.129
0.110
0.089
0.236
0.190
SEA-RAFT-Sparse
0.221
0.158
0.223
0.278
0.324
0.098
0.070
0.100
0.084
0.100
0.166
0.166
KLTNet (Ours)
0.092
0.088
0.162
0.233
0.207
0.075
0.070
0.074
0.089
0.110
0.149
0.123
OpenVINS
KLT
0.099
0.206
0.150
0.188
0.508
0.069
0.058
0.074
0.105
0.060
0.122
0.149
RAFT-Sparse
0.239
0.310
0.235
0.404
0.636
0.073
0.077
0.093
0.112
0.084
0.153
0.219
TABLE I: Absolute Trajectory Error (ATE) Comparison on the EuRoC Dataset. Results are in meters. Bold indicates best per block.
Method
Tracker
Room1
Room2
Room3
Room4
Room5
Room6
Average
VINS-Mono
KLT
0.067
0.069
0.121
0.047
0.203
0.075
0.097
RAFT-Sparse
0.120
0.088
0.095
0.074
0.069
0.130
0.096
SEA-RAFT-Sparse
0.067
0.058
0.120
0.047
0.032
0.109
0.072
KLTNet (Ours)
0.052
0.047
0.061
0.029
0.081
0.026
0.049
OpenVINS
KLT
0.055
0.103
0.081
0.041
0.067
0.086
0.072
RAFT-Sparse
0.116
0.179
0.102
0.098
0.123
0.096
0.119
TABLE II: Absolute Trajectory Error (ATE) Comparison on the TUM-VI Dataset. Results are in meters. Bold indicates best per block.
No.
CoarseFlowNet
TriPatchRefiner
Observation Weight
Weight Supervision
Max Points
Avg. ATE
R1
✓
×
Fixed Isotropic
-
150
0.317
R2
✓
✓
Fixed Isotropic
-
150
0.153
R3
✓
✓
Isotropic
Reprojection
150
0.139
R4
✓
✓
Anisotropic
Reprojection
150
0.140
R5
✓
✓
Isotropic
Triangulation
150
0.131
R6
✓
✓
Anisotropic
Triangulation
150
0.123
TABLE III: Ablation study of KLTNet components integrated with VINS-Mono on EuRoC. Results are the average ATE in meters over the 11 sequences.
Input Patches
Median Tracking Error (px)
VIO ATE (m)
Short interval
Long interval
VINS-Mono
OpenVINS
two-patch ( Xj−1,Xj )
0.83
1.22
0.135
0.134
triplet-patch ( X0,Xj−1,Xj )
0.25
0.59
0.123
0.099
TABLE IV: Impact of the reference patch X0 on tracking accuracy (Replica) and VIO accuracy (EuRoC).
KITTI-360
UMA-VI
Tracker
00
02
03
04
05
06
09 ∗
10
C1
C2
C3
T1
T2
KLT
1.17/0.15
1.10/0.22
1.33/ 0.10
1.02/0.21
1.02/0.26
0.99/0.17
1.06/0.14
2.00/0.24
4.50
1.41
0.81
0.24
0.22
SEA-RAFT-Sparse
1.99/0.21
1.30/0.27
1.08/0.15
1.86/0.24
1.09/0.23
1.34/0.21
1.60/0.20
1.88/0.27
7.89
1.07
0.18
0.27
0.41
KLTNet (Ours)
0.72 / 0.12
0.65 / 0.19
0.78 /0.11
0.82 / 0.18
0.88 / 0.22
0.81 / 0.15
0.69 / 0.13
1.65 / 0.21
1.93
0.49
0.16
0.18
0.16
TABLE V: Tracker comparison within VINS-Mono on KITTI-360 and the illumination-change subset of UMA-VI. KITTI-360: trel(%) / rrel(∘/100m) ; UMA-VI: start-to-end errors in meters. Best results are in bold.
Fig. 4: VIO performance on a self-collected low-texture dataset. (a) A view of the challenging low-texture environment. (b) The handheld data acquisition platform. (c)-(e) Estimated trajectories for KLTNet-VINS-Mono compared against KLT-VINS-Mono, DM-VIO, and DPVO. (f) Absolute Trajectory Error (ATE) in meters. Note that DM-VIO failed to track in seq2.
Fig. 5: Throughput and GPU Memory Comparison. KLTNet, RAFT-Sparse, and SEA-RAFT-Sparse are evaluated within the VINS-Mono visual front end, with throughput measured for the complete front-end pipeline. KLTNet is evaluated on a PC (NVIDIA A6000; LibTorch without TensorRT) and a Jetson AGX Orin (with TensorRT) with maximum track counts of 25, 50, 100, and 150. End-to-end learned odometry systems are shown as system-level reference points.
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
SLAM (Simultaneous Localization and Mapping) and Odometry are important systems for estimating the position of mobile devices, such as robots and cars, utilizing one or more sensors. Particularly in camera-based SLAM or Odometry, effectively tracking visual features is important as it significantly impacts system performance. In this paper, we propose a method that leverages deep learning to robustly track visual features in monocular camera images. This method operates reliably even in textureless environments and situations with rapid lighting changes. Additionally, we evaluate the performance of our proposed method by integrating it into VINS-Fusion (Monocular-Inertial), a commonly used Visual-Inertial Odometry (VIO) system.
Sanghyun Park, Soohee Han
Department of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea · Department of Electrical Engineering and of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea
Traditional monocular Visual-Inertial Odometry (VIO) systems struggle in low-texture environments where sparse visual features are insufficient for accurate pose estimation. To address this, dense Monocular Depth Estimation (MDE) has been widely explored as a complementary information source. While recent Vision Transformer (ViT) based complex foundational models offer dense, geometrically consistent depth, their computational demands typically preclude them from real-time edge deployment. Our work bridges this gap by integrating learned depth priors directly into the VINS-Mono optimization backend. We propose a novel framework that enforces affine-invariant depth consistency and pairwise ordinal constraints, explicitly filtering unstable artifacts via variance-based gating. This approach strictly adheres to the computational limits of edge devices while robustly recovering metric scale. Extensive experiments on the TartanGround and M3ED datasets demonstrate that our method prevents divergence in challenging scenarios and delivers significant accuracy gains, reducing Absolute Trajectory Error (ATE) by up to 28.3%. Code will be made available.
Arda Alniak, Sinan Kalkan, Mustafa Mert Ankarali +2
Dept. of Electrical & Electronics Engineering · Center for Image Analysis (OGAM), METU, Ankara, Türkiye · Dept. of Computer Engineering
Accurate initialization is essential for reliable visual-inertial odometry (VIO), but it is often ill-conditioned under degenerate motions. Existing methods typically require restrictive excitation motions to ensure sufficient observability or rely on computationally expensive 3D structure reconstruction, limiting efficient and practical deployment. To address these limitations, we propose SLIM-init, a structureless monocular VIO initializer that directly exploits geometric constraints from tracked 2D line features without explicit 3D landmark reconstruction. Specifically, SLIM-init leverages line-derived vanishing points (VPs) as translation-invariant orientation cues to provide robust rotation-only constraints under degenerate scenarios such as low-parallax or translation-dominant motions. It further incorporates a line epipolar residual to constrain translation and a line-normal projection residual to improve the conditioning of linear alignment, enhancing the accuracy and robustness of initial state estimation. Extensive experiments on a public benchmark and challenging custom degenerate-motion sequences demonstrate improved accuracy and robustness over state-of-the-art initialization methods. The source code is available at: https://github.com/cjunwan/SLIM-init.
Junwan Choi, Woongrae Jo, Dong-Uk Seo +2
School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon, 34141, Republic of Korea