Benchmarking Visual Feature Representations for LiDAR-Inertial-Visual Odometry Under Challenging Conditions
Authors: Eunseon Choi, Junwoo Hong, Daehan Lee, Sanghyun Park, Hyunyoung Jo, Sunyoung Kim, Changho Kang, Seongsam Kim, Yonghan Jung, Jungwook Park, Seul Koo, Soohee Han
Organizations: Department of Covergence IT Engineering, Pohang University of Science and Technology(POSTECH), Pohang 37673, Republic of Korea · School of Mechanical Engineering, Kunsan National University, Jeonbuk 54150, Republic of Korea · Department of Artificial Intelligence and Robotics, Sejong University, Seoul 05006, Republic of Korea · Disaster Scientific Investigation Division Disaster Investigation Technology Team, National Disaster Management Research Institute, Ministry of the Interior and Safety, Ulsan 44429, Republic of KOREA · Department of Electrical Engineering, POSTECH, Pohang 37673, Republic of Korea
Accurate localization in autonomous driving is critical for successful missions including environmental mapping and survivor searches. In visually challenging environments, including low-light conditions, overexposure, illumination changes, and high parallax, the performance of conventional visual odometry methods significantly degrade undermining robust robotic navigation. Researchers have recently proposed LiDAR-inertial-visual odometry (LIVO) frameworks, that integrate LiDAR, IMU, and camera sensors, to address these challenges. This paper extends the FAST-LIVO2-based framework by introducing a hybrid approach that integrates direct photometric methods with descriptor-based feature matching. For the descriptor-based feature matching, this work proposes pairs of ORB with the Hamming distance, SuperPoint with SuperGlue, SuperPoint with LightGlue, and XFeat with the mutual nearest neighbor. The proposed configurations are benchmarked by accuracy, computational cost, and feature tracking stability, enabling a quantitative comparison of the adaptability and applicability of visual descriptors. The experimental results reveal that the proposed hybrid approach outperforms the conventional sparse-direct method. Although the sparse-direct method often fails to converge in regions where photometric inconsistency arises due to illumination changes, the proposed approach still maintains robust performance under the same conditions. Furthermore, the hybrid approach with learning-based descriptors enables robust and reliable visual state estimation across challenging environments.
Figures & tables
Fig. 1: Framework for the LiDAR-inertial-visual odometry (LIVO) system built on FAST-LIVO2 to integrate four pairs of visual feature extractors and matchers
Fig. 2: Step-by-step hybrid visual measurement update process
Dataset
Sequence
Visual Degeneration
LiDAR Degeneration
NC
Quad-Hard
Planar surfaces
Well-constrained
Maths-Hard
Planar surfaces
Well-constrained
STM
Cave1
Illumination change
Partial-constrained
Cave2
Illumination change
Partial-constrained
ML
AMvalley02
Low parallax
Vertical structure loss
AMvalley03
Low parallax
Vertical structure loss
TABLE I: Factors in degeneracy of Newer College (NC) [ 35 ] , SubT-MRS (STM) [ 36 ] , and MARS-LVIG (ML) datasets
Method
Feature Type
Feature + Matcher
SD-only (baseline)
None
None (photometric only)
hybrid SD+ORB+HD
Handcrafted
ORB [ 27 ] +HD
hybrid SD+SP+SG
Learning-based
SP [ 14 ] +SG [ 15 ]
hybrid SD+SP+LG
Learning-based
SP [ 14 ] +LG [ 16 ]
hybrid SD+XFeat+MNN
Learning-based
XFeat [ 28 ] +MNN
TABLE II: Visual feature extractor–matcher pairs integrated into the LiDAR–inertial–visual odometry framework.
Algorithm
Parameter
Value
Description
Sparse-Direct
Pyramid levels
4
Number of image pyramid levels
Patch size
8 px
Patch size for photometric error calculation
Photometric outlier threshold
1000
Threshold for outlier rejection based on the squared photometric error
ORB
Max features
1000
Maximum number of features per frame
Pyramid levels
8
Number of pyramid levels
Scale factor
1.2
Scaling ratio between pyramid levels
TABLE III: Main parameters used in the experiments. For sparse-direct [ 29 ] , ORB [ 27 ] , SuperPoint [ 14 ] , SuperGlue [ 15 ] and LightGlue [ 16 ] , the parameter settings primarily follow the recommended presets provided by the original authors, with minimal adjustments to fit the LIVO experimental environment. All parameters were applied consistently to ensure a fair comparison.
Newer College
Super-Loc
MARS-LVIG (w sliding window)
Visual Module Configuration
Quad-Hard
Maths-Hard
Cave1
Cave2
AMvalley02
AMvalley03
RMSE
Max
RMSE
Max
RMSE
Max
RMSE
Max
RMSE
Max
RMSE
Max
SD-only (Baseline)
0.587
1.072
0.555
1.049
0.255
1.321
12.128
21.403
1.834
12.721
8.283
13.351
hybrid SD+ORB+HD
0.594
0.986
0.556
1.063
0.221
0.398
11.641
20.764
1.831
9.193
5.214
15.620
hybrid SD+SP+SG
0.590
0.974
0.557
1.078
0.216
0.398
6.206
11.154
1.854
11.952
7.327
11.824
hybrid SD+SP+LG
0.591
0.975
0.557
1.037
0.221
0.418
4.292
8.194
1.779
6.913
5.892
11.093
TABLE IV: Comparison of absolute pose error in translation [m] across datasets for different visual feature extractor–matcher pairs, evaluated after full SE(3) alignment.
Fig. 3: Mapping results on the AMValley03 sequence using sliding-window local mapping: (a) sparse-direct only, (b) ORB [ 27 ] + Hamming distance, (c) SuperPoint [ 14 ] + SuperGlue [ 15 ] , (d) SuperPoint [ 14 ] + LightGlue [ 16 ] , and (e) XFeat [ 28 ] + mutual nearest neighbor search. Square red insets reveal zoomed-in point-cloud details, and the blue line traces the UAV’s flight path.
Fig. 4: Trajectories on the Cave2 sequence. The sparse-direct (SD)-only method exhibits pronounced deviations at specific segments, which are caused by high parallax in (A) and illumination changes in (B)–(D). The color bar indicates the absolute trajectory error (m), and trajectories are aligned by translating the initial poses (origin alignment) to emphasize where and how drift accumulates without global alignment.
Fig. 5: Resulting trajectories on the Cave1 sequence. The estimated trajectories were aligned with the ground truth using full SE(3) alignment (rotation and translation). Gray segments indicate regions where the position error falls within the range of −0.1mto0.1m , and in the yaw plot, gray bands correspond to yaw errors from −1∘to1∘ . Here, SD, HD, SP, SG, LG, and MNN refer to Sparse-Direct [ 29 ] , hamming distance, SuperPoint [ 14 ] , SuperGlue [ 15 ] , LightGlue [ 16 ] , and mutual nearest neighbor search, respectively.
Fig. 6: Comparison of (a) CPU usage, (b) memory usage, (c) GPU usage, and (d) GPU memory usage on the Cave1 sequence. The SD-only and hybrid SD+ORB+HD are purely CPU-based methods and do not use a GPU (no GPU using or GPU memory consumption). The CPU usage is reported as a percentage of total CPU utilization, where 100% corresponds to the full usage of a single core. Memory usage indicates the resident set size (RSS) divided by the total physical memory (64 GB RAM in this experiment). Here, SD, HD, SP, SG, LG, and MNN refer to Sparse-Direct [ 29 ] , hamming distance, SuperPoint [ 14 ] , SuperGlue [ 15 ] , LightGlue [ 16 ] , and mutual nearest neighbor search, respectively.
Fig. 7: Frame-wise processing time for each visual feature extractor-matcher configuration on the Cave1 sequence, decomposed into five key stages (summarized in Table. VI ). The y-axis represents the average processing time per frame for each stage on a logarithmic scale. The total time shown corresponds to the overall computation time for all VIO processing stages, excluding visualization overhead. Here, SD, HD, SP, SG, LG, and MNN refer to Sparse-Direct [ 29 ] , hamming distance, SuperPoint [ 14 ] , SuperGlue [ 15 ] , LightGlue [ 16 ] , and mutual nearest neighbor search, respectively.
Method
Peak CPU Memory (%)
Peak GPU Memory (MiB)
SD only
3.3
–
hybrid SD+ORB+HD
3.0
–
hybrid SD+SP+LG
4.7
3058
hybrid SD+SP+SG
6.0
15742
hybrid SD+XFeat+MNN
6.1
–
TABLE V: Peak memory consumption of different methods on the Cave1 sequence. Peak values correspond to the maximum observed memory usage during runtime. GPU memory is reported only for GPU-enabled methods, including the hybrid SD+SP+SG and SD+SP+LG configurations. Here, SD, HD, SP, SG, LG, and MNN refer to Sparse-Direct [ 29 ] , hamming distance, SuperPoint [ 14 ] , SuperGlue [ 15 ] , LightGlue [ 16 ] , and mutual nearest neighbor search, respectively.
Functions
Description
retrieveFromVisualSparseMap()
Extract candidate visual map points from the current frame
computeJacobianAndUpdateEKF()
Compute Jacobian for photometric error and update EKF state
generateVisualMapPoints()
Generate new visual map points via voxel raycasting and outlier rejection
updateVisualMapPoints()
Add visual map points to the voxel map
updateReferencePatch()
Update reference patch for selected map points based on patch score
TABLE VI: Stage-wise functions of the visual module execution process. Each function corresponds to a major processing stage, with its role summarized in the right column. The same modular structure is applied in Fig. 7 to analyze the frame processing time of different feature extractor–matcher pairs.
Fig. 8: Comparison of mapping efficiency and matching stability for each feature extractor–matcher combination. (a) Number of accumulated 3D visual map points (SparseMapSize), (b) number of features detected per frame, and (c) temporal changes in the number of initial and inlier matches. The SD-only configuration does not extract features; hence, plots for (b) and (c) are unavailable. Dashed lines indicate the number of initial matches; solid lines with markers represent inlier matches. Inliers are obtained via RANSAC-based geometric verification using a 2D–3D reprojection error under the predicted camera pose. Here, SD, HD, SP, SG, LG, and MNN refer to Sparse-Direct [ 29 ] , hamming distance, SuperPoint [ 14 ] , SuperGlue [ 15 ] , LightGlue [ 16 ] , and mutual nearest neighbor search, respectively.
Key Laboratory of Space Active Opto-Electronics Technology, Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai 200083, China. · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China.
Department of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea · Department of Electrical Engineering and of Convergence IT Engineering, POSTECH, Pohang, 37673, Korea