Organizations: Active Intelligent Systems Lab., Toyohashi University of Technology, Japan · Dept. of Information Technology and Human Factors, AIST, Japan
LiDAR-Inertial-Visual Odometry (LIVO) integrates complementary LiDAR, camera, and IMU measurements to achieve robust state estimation. However, most existing methods assume a fixed relative pose and accurate time synchronization between sensors. Toward modular and heterogeneous robotic systems, this paper presents FREE-LIVO, a tightly coupled odometry method that requires neither a rigid LiDAR-camera rig nor LiDAR-camera time synchronization. The proposed method uses a pretrained image feature matcher to establish correspondences between a camera image and a dense LiDAR intensity image generated from a single LiDAR scan using a generative model. From the resulting 3D point-to-pixel correspondences, we construct reprojection factors that directly constrain LiDAR and camera poses at different acquisition times. These factors are jointly optimized in a single factor graph together with constraints from point cloud registration, visual feature tracking, and the IMU measurements associated with each sensor. On six sequences from the public M3DGR dataset, FREE-LIVO achieves an average absolute trajectory error (ATE) RMSE of 0.45 m without provided LiDAR-camera extrinsic parameters and exhibits minimal accuracy degradation under time synchronization errors of up to 0.5 s. Furthermore, experiments with the LiDAR and camera mounted on separate robots demonstrate that the method can jointly estimate the trajectories of independently moving sensors.
Figures & tables
Fig. 1: FREE-LIVO estimation results with a physically decoupled LiDAR and camera. (a) The LiDAR and camera are held in separate hands. (b) A colored point cloud generated using the estimated sensor poses. (c) Jointly estimated trajectories of the independently moving LiDAR and camera.
Fig. 2: Overview of FREE-LIVO. Upsampled LiDAR intensity images are matched with camera images to obtain 2D–3D correspondences. These are integrated with LiDAR, visual, and IMU constraints in a single factor graph to jointly estimate sensor trajectories without a fixed LiDAR–camera rig or time synchronization.
Fig. 3: LiDAR–camera matching. (a) Sparse intensity image from a single LiDAR scan. (b) Dense intensity image generated by Conditional Rectified Flow. (c) Feature correspondences between the dense intensity image and the camera image.
Fig. 4: Sliding-window factor graph of FREE-LIVO. LiDAR–camera reprojection factors connect the LiDAR and camera state sequences. States outside the optimization window are marginalized. For clarity, only factors involving the latest LiDAR or camera state are shown.
Fig. 5: Factor graph during operation. LiDAR–camera reprojection factors connect the LiDAR and camera trajectories. The colored point clouds show the reprojection points used by these factors.
Fig. 6: LiDAR–camera reprojection factor. The reprojection error between a projected 3D LiDAR point and its corresponding image feature jointly constrains the LiDAR and camera poses.
Sequence
Metric
VINS-Mono
GLIM
LVI-SAM
R3LIVE
FAST-LIVO2
FREE-LIVO
FREE-LIVO
FREE-LIVO
(Loose)
(NN)
Outdoor01
ATE [m]
3.23
0.21
4.94
0.85
0.15
0.18
1.28
0.20
Outdoor04
ATE [m]
5.04
0.39
15.74
5.10
0.58
0.40
5.13
0.54
Dynamic01
ATE [m]
0.33
1.72
0.50
4.43
7018.72 †
0.98
1.21
1.19
Dynamic02
ATE [m]
0.34
1.31
0.27
1.21
1746.95 †
0.56
0.41
0.56
Dynamic03
ATE [m]
1.34
0.26
4.94
0.39
0.14
0.25
2.22
0.33
TABLE I: Evaluation results on M3DGR.
Sequence
LVI-SAM
R3LIVE
FAST-LIVO2
FREE-LIVO
0.0 s
0.1 s
0.3 s
0.5 s
0.0 s
0.1 s
0.3 s
0.5 s
0.0 s
0.1 s
0.3 s
0.5 s
0.0 s
0.1 s
0.3 s
0.5 s
Outdoor01+04
10.34
10.80
11.72
28.18
2.98
1.73
8.72
2.28
0.36
0.39
0.36
0.40
0.29
0.24
0.29
0.29
Dynamic01+02
0.38
1.06
2.04
2.53
2.82
9.80
4.92
1.37
–
–
–
–
0.77
1.23
1.39
1.83
Dynamic03+04
7.69
8.32
15.47
25.80
1.20
1.34
1.83
0.89
0.18
0.17
0.20
0.28
0.29
0.29
0.29
0.31
TABLE II: ATE [m] on M3DGR for selected delay offsets (pair-averaged by sequence groups).
Fig. 7: Colored point clouds generated by R3LIVE, FAST-LIVO2, and FREE-LIVO on the FAST-LIVO2 dataset, with and without an added 0.3 s camera delay. FREE-LIVO maintains alignment between image colors and LiDAR geometry under the delay.
Fig. 8: Colored point clouds generated by FREE-LIVO with the LiDAR and camera held in separate hands, as in Fig. 1 (a).
Fig. 9: FREE-LIVO estimation with two independently moving robots. (a) LiDAR and camera units mounted on separate robots. (b) Online estimation during operation. (c) Final estimated trajectories.
Light detection and ranging (LiDAR)-inertial-visual odometry fuses geometric depth with visual measurements, but its sensors fail independently: LiDAR in under-constrained geometry, vision under poor illumination or texture. Weak information is harmless, since the filter then relies on the other sensor or the inertial prediction; over-confident information is not. Existing countermeasures act on weak information or switch vision by hand-set thresholds. This article presents SA-LIVO, which fuses vision through explicit, online-estimated error models. A frame-level noise scale, estimated by covariance matching, absorbs whole-image errors, and Schmidt filtering carries the correlation of reused observations. Subspace-aware information fusion models the error shared by a frame's observations as a pose offset capping the visual information per direction; the exact fusion attenuates vision only above that cap and reduces to the information sum when the visual model is consistent. LiDAR and visual residuals share one invariant extended Kalman filter loop. On 24 handheld and aerial sequences, SA-LIVO is the most accurate or tied on 22 and stays bounded where others diverge; on the HILTI 2022 benchmark it needs 13.4 ms per frame on a laptop processor and 2.0-6.2x less memory than FAST-LIVO2, R3LIVE, and SR-LIVO.
Yinong Cao, Xin He, Yijie Wang +11
Key Laboratory of Space Active Opto-Electronics Technology, Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai 200083, China. · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China.
Measurement-level LiDAR-Inertial-Visual Odometry (LIVO) can achieve high accuracy, and its practical resource requirements depend on the adopted association and map-maintenance strategy. Persistently attaching dense photometric patches or appearance attributes to mapped LiDAR points can introduce additional storage, association, and update costs. This paper presents F-LIVO, a filter-based LIVO framework that couples a visual-inertial subsystem and a LiDAR-inertial subsystem through feature-level point-to-plane constraints between visual points and LiDAR planes. To preserve cross-modal coupling when visual triangulation is weak, we further recover visual feature depth from the LiDAR map through local plane fitting and ray-plane intersection. In the LiDAR update, the covariance of each point-to-plane residual is adaptively scaled according to local planar reliability, thereby reducing overconfident updates on poorly conditioned surfaces. Because the proposed method relies on sparse geometric features rather than persistent dense photometric map attributes, it retains a compact map representation. Experiments on multiple public datasets show competitive, sequence-dependent accuracy. Across the evaluated degradation cases, F-LIVO maintains continuous estimation with comparatively low accumulated end-to-end drift. Under the evaluated resource configuration, it achieves the lowest mean and peak central processing unit (CPU) usage, the lowest peak resident set size (RSS), and the lowest RSS growth rate among the evaluated implementations, while R2LIVE retains a lower mean RSS. The source code will be made publicly available upon acceptance to facilitate reproducible research.
Xinye Dai, Yapeng Wang, Shujie Zhou +6
The authors are with the GNSS Center, Wuhan University, Wuhan 430079, China.
Robust state estimation and mapping in long-term, large-scale, and highly dynamic environments remains a key challenge in robotics. Existing LiDAR-Inertial-Visual Odometry (LIVO) systems achieve strong local accuracy but suffer from accumulated drift over long distances and may fail in geometrically degraded or textureless scenes. Meanwhile, GNSS-aided fusion frameworks often rely on LiDAR or visual odometry for state prediction and outlier rejection, making them vulnerable when odometry degenerates. To address these limitations, we propose a tightly coupled LiDAR-Inertial-Visual-GNSS fusion framework based on an Error-State Iterated Kalman Filter. An online spatiotemporal alignment module using Dynamic Time Warping is introduced for highly dynamic conditions. To better exploit GNSS precision, we develop observation models based on Doppler shifts and fixed-anchor Time-Differenced Carrier Phase, providing millimeter-level relative constraints without augmenting historical anchor states. We further design a degeneracy-aware dual-mode outlier rejection strategy that switches between LIVO-prior-guided rejection and GNSS-aided recovery according to the LIVO degeneracy level. Experiments on the public M3DGR dataset and a custom 20~m/s fixed-wing UAV dataset demonstrate that our system reduces accumulated drift and map ghosting, outperforming state-of-the-art methods in accuracy and robustness.
Zhiyu Chen, Chunran Zheng, Jiayu Wen +4
College of Mechatronics and Control Engineering, Shenzhen University, Shenzhen 518060, China · Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · College of Automation, Harbin Engineering University, Harbin 150001, China