NSFlow: End-to-End Differentiable Neuro-Symbolic Optical Flow for Visual Odometry
Authors: Yicheng Lin, Zhipeng Fei, Yuxiu Xu, WenDong Chen, Cong Li, Bin Han
Organizations: Guangzhou Purpleriver Electronic Technology Co., Ltd, Guangzhou, China · State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China
Sparse optical flow provides stable inter-frame correspondence, playing a key role in Visual Odometry (VO) and Visual-Inertial Odometry (VIO). Classical optimization-based methods, such as Lucas-Kanade (LK), perform well under small displacements but are sensitive to large motions and illumination changes. Modern regression-based learning methods, while more robust in complex scenes, are often computationally heavy and lack explicit geometric consistency, making them less suitable for efficient VO/VIO front-ends. To bridge this gap, we propose a hybrid neuro-symbolic framework that combines the strengths of both paradigms. Our method uses a Convolutional Neural Network (CNN) to extract robust feature representations, which is fed into a differentiable LK optimizer to estimate optical flow in an end-to-end trainable manner. Through implicit differentiation, gradients are propagated across the iterative solver, enabling joint optimization of feature extraction and flow estimation. The resulting system integrates seamlessly into existing VO/VIO pipelines and runs in real-time on embedded platforms. Experiments show that our method outperforms conventional optimization-based flow in challenging conditions such as dynamic lighting and low texture, while also achieving higher accuracy and lower latency than purely regression-based alternatives. When deployed in a VIO system, our method demonstrates significant performance improvement, achieving an average error reduction of 42% on challenging datasets while enhancing tracking stability. The code is publicly available.
Figures & tables
Fig. 1: A Framework for End-to-End Trained Neuro-Symbolic Optical Flow. In the forward pass, a convolutional network extracts image features for an iteratively optimized LK algorithm to solve optical flow; during backpropagation, gradients through this optimization are computed via the implicit function theorem to enable error propagation and joint training.
Fig. 2: The pipeline of the proposed method. Images IA and IB are two consecutive frames in a sequence. Sparse keypoints are first detected in IA using the GFTT [ 30 ] detector. A shared-weight convolutional network is then used to extract descriptor maps. In the forward flow estimation, a standard LM optimization routine computes the flow from the descriptor maps, where the two orange-colored steps are non-differentiable. For backward propagation, derivatives are calculated by leveraging the optimality condition (where the gradient of the objective w.r.t. flow equals zero). The dashed orange arrows illustrate the corresponding gradient flow. The loss is computed from the error between the estimated optical flow and the ground truth.
Fig. 3: Comparison between the original and the proposed learnable front-end in VINS-Fusion. The original system (top) employs a hand-crafted LK optical flow for feature tracking, while the proposed version (bottom) replaces it with a learnable LM-optimized optical flow module guided by a descriptor network, enabling end-to-end trainability and improved robustness.
Fig. 4: Comparison of trackable regions among different methods. Red areas indicate regions that are easy to track, while blue areas denote low-texture or ambiguous regions that are difficult to track. Compared with the Original LK Method (a) and LET-Net (b), our approach (c) produces a broader and more consistent distribution of trackable regions, demonstrating stronger robustness under varying illumination and surface texture conditions.
Seq.
Trackable Regions @1/@2/@3 ↑
LK Method
LET-Net [ 34 ]
Ours
MH000
0.02 / 0.05 / 0.08
6.78 / 8.31 / 8.69
19.90 / 24.48 / 27.12
MH001
0.35 / 0.79 / 1.03
13.70 / 17.43 / 18.48
13.29 / 17.76 / 20.41
MH002
0.01 / 0.02 / 0.02
10.79 / 12.38 / 12.64
17.08 / 20.03 / 22.04
MH003
0.12 / 0.21 / 0.25
10.93 / 13.10 / 13.58
26.11 / 31.04 / 33.90
MH004
0.02 / 0.09 / 0.13
2.60 / 3.65 / 3.95
3.55 / 4.75 / 5.51
TABLE I: Median proportion (%) of trackable regions under different pixel error thresholds. Each cell shows the results for < 1 px / < 2 px / < 3 px.
Fig. 5: Descriptor map visualization . The top two images are the original images, and the bottom two are the descriptor maps, corresponding to an indoor scene and an underwater weak-texture environment. From the magnified regions, it can be observed that the proposed method enhances the nearly textureless areas of the original images, strengthens the image gradient information, and propagates these gradients to the surrounding regions.
Fig. 6: Optical flow error curves over iterations. The plot compares the convergence behavior of different methods (LK Method [ 12 ] , LET-NET [ 34 ] , and Ours). The proposed method achieves faster error reduction and lower final error, demonstrating better stability and accuracy during iterative refinement.
Method
Params/M ↓
GFLOPs ↓
Interface time (ms) ↓
GPU
CPU
SuperPoint [ 24 ]
22.23
1.30
8.22
114.72
ALIKE-T [ 43 ]
1.82
0.83
49.95
94.81
DISK [ 44 ]
98.97
1.09
116.36
589.78
XFeat [ 45 ]
1.10
0.64
4.26
11.17
Ours
1.01e-3
0.27
0.84
5.21
TABLE II: INFERENCE EFFICIENCY COMPARISON
%Inlier ↑
Error ↓
H00
H01
H02
H03
H04
H05
H06
H07
H00
H01
H02
H03
H04
H05
H06
H07
RAFT [ 15 ]
23.3
17.9
17.1
12.1
11.9
20.2
15.7
22.3
12.02
15.48
16.29
25.81
31.46
15.33
28.17
15.82
SEA-RAFT(M) [ 46 ]
20.6
16.0
14.8
10.7
12.4
17.3
17.4
22.1
15.30
16.28
19.55
19.89
19.99
21.16
16.46
14.22
SEA-RAFT(S) [ 46 ]
21.1
16.0
16.9
11.9
11.4
20.5
17.8
22.9
14.45
16.19
16.06
17.81
22.13
16.63
16.21
12.71
Key.Net(MS) [ 47 ]
25.5
20.0
23.4
16.8
14.6
26.8
20.0
25.3
10.14
11.74
9.47
13.71
23.19
7.89
12.62
10.52
ALIKE-T [ 43 ]
25.3
22.6
24.0
16.5
14.9
26.3
19.6
25.6
9.58
10.94
9.33
13.95
21.23
7.90
12.90
10.47
TABLE III: evaluation results of feature matching
Fig. 7: Boxplot of optical flow tracking matching results. The validation was carried out in illumination, viewpoint, and all images, respectively, based on the HPatches [ 38 ] dataset. The smaller tracking error means that the proposed features are easier to track. The proposed method achieves lower error results in both illumination and viewpoint varying datasets.
Fig. 8: Schematic diagram of the position change of sparse keypoints in continuous tracking. The middle image shows the initial keypoint positions. The left and right images respectively display the keypoint positions after 60 consecutive tracking iterations using our method and the latest optical flow regression estimation algorithm, SEA_RAFT [ 46 ] . The sparse keypoints estimated by SEA_RAFT [ 46 ] exhibit noticeable drift (as shown in the zoomed-in images). In contrast, the proposed method maintains keypoint accuracy even after 60 tracking iterations.
Fig. 9: Illumination augmentation on the EuRoC dataset. The last row shows the sequences with large illumination disturbances, the middle row shows those with small disturbances, and the bottom row shows the original sequences. The illumination perturbations are generated by applying absolute and relative pixel transformations to the images using two random variables.
Seq.
None (m)
Small (m)
Large (m)
VINS [ 4 ]
LET-VINS [ 34 ]
Ours
VINS [ 4 ]
LET-VINS [ 34 ]
Ours
VINS [ 4 ]
LET-VINS [ 34 ]
Ours
MH_01
0.24/0.26
0.26/0.27
0.22/0.24
0.27/0.26
0.25/0.24
0.25/0.25
0.51/0.49
0.48/0.45
0.39/0.37
MH_02
0.17/0.17
0.18/0.18
0.17/0.17
0.18/0.16
0.21/0.23
0.17/0.16
0.48/0.47
0.41/0.39
0.32/0.30
MH_03
0.24/0.22
0.22/0.22
0.23/0.23
0.25/0.23
0.27/0.25
0.23/0.25
1.27/1.13
0.38/0.38
0.26/0.26
MH_04
0.42/0.43
0.35/0.29
0.42/0.44
0.57/0.53
0.50/0.48
0.41/0.41
F
0.81/0.80
0.45/0.46
MH_05
0.28/0.25
0.29/0.26
0.28/0.24
0.30/0.26
0.35/0.30
0.30/0.28
F
0.67/0.63
0.35/0.28
TABLE IV: Comparison of mean/median ATE (m) on EuRoC sequences under different illumination disturbances. All values are in meters (m). Each cell shows mean/median ATE, and the best result is in bold .
Sequence
PL-SLAM [ 51 ]
ORB-SLAM3 [ 52 ]
Basalt [ 53 ]
OKVIS [ 54 ]
DROID [ 9 ]
AirSLAM [ 20 ]
VINS [ 4 ]
Ours
two-floors-csc1
F
F
0.760
0.154
0.341
0.066
0.335
0.150
two-floors-csc2
F
F
1.211
0.679
0.299
0.190
1.108
0.700
third-floor-csc1
4.478
0.863
0.420
0.287
0.048
0.070
0.047
0.047
third-floor-csc2
6.068
0.149
0.590
0.271
0.890
0.127
0.106
0.068
long-walk-eng
F
F
5.046
3.005
F
1.801
2.382
0.353
lab-module-csc
F
F
0.403
0.579
0.319
0.979
0.570
0.173
TABLE V: Comparison of mean/median ATE (m) on UMA-VI sequences under different illumination and environmental conditions. All values are in meters (m). Each cell shows mean/median ATE, the best result is marked in red , and the second-best result is marked in green .
Method
01
02
03
04
05
06
ORB-SLAM3 [ 52 ]
F
F
F
F
0.063
0.910
VINS [ 4 ]
0.633
0.437
F
F
0.849
0.963
LET-VINS [ 34 ]
0.672
0.684
1.774
0.911
0.527
F
Ours
0.564
0.350
0.667
0.641
0.255
0.784
TABLE VI: Comparison of mean ATE across different methods on the AQUALOC dataset. All values are in meters (m). DROID and DPVO are evaluated without metric scale. The best results are in bold .
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology, Hong Kong SAR, China