SF3D-RGB: Scene Flow Estimation from Monocular Camera and Sparse LiDAR

Rajai Alhimdiat; Ramy Battrawy; René Schuster; Didier Stricker; Wesam Ashour

cs.CVFeb 25, 2026

SF3D-RGB: Scene Flow Estimation from Monocular Camera and Sparse LiDAR

Authors: Rajai Alhimdiat, Ramy Battrawy, René Schuster, Didier Stricker, Wesam Ashour

Organizations: Islamic University of Gaza, Gaza, Palestine · German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Abstract

Scene flow estimation is an extremely important task in computer vision to support the perception of dynamic changes in the scene. For robust scene flow, learning-based approaches have recently achieved impressive results using either image-based or LiDAR-based modalities. However, these methods have tended to focus on the use of a single modality. To tackle these problems, we present a deep learning architecture, SF3D-RGB, that enables sparse scene flow estimation using 2D monocular images and 3D point clouds (e.g., acquired by LiDAR) as inputs. Our architecture is an end-to-end model that first encodes information from each modality into features and fuses them together. Then, the fused features enhance a graph matching module for better and more robust mapping matrix computation to generate an initial scene flow. Finally, a residual scene flow module further refines the initial scene flow. Our model is designed to strike a balance between accuracy and efficiency. Furthermore, experiments show that our proposed method outperforms single-modality methods and achieves better scene flow accuracy on real-world datasets while using fewer parameters compared to other state-of-the-art methods with fusion.

Figures & tables

Figure 1: SF3D-RGB consists of a Pointwise Feature Extraction (FE) module, a Feature Pyramid Network (FPN), a Fusion Module (FM), a Graph Matching Module (GM), and a Refinement Module (RF). We denote the feature extraction convolutions by $g(\cdot)$ .

Figure 2: Pointwise Feature Extraction Module based on graph convolution.

Method	Modality	Evaluated	Domain	FT3D		Model size and runtime
		Evaluated		EPE3D	EPE2D	P	R	TFLOPS	Device
		Points		[m] $\downarrow$	[px] $\downarrow$	[M] $\downarrow$	[ms] $\downarrow$
PWOC-3D Saxena et al. (2019)	Stereo only	all	2D	–	6.97	8.05	130	11.34	GTX1080Ti
LiDAR-Flow Battrawy et al. (2019)	Stereo+LiDAR	5000	2D	–	29.97	–	65900	-	CPU
DeepLiDARFlow Rishav et al. (2020)	RGB+LiDAR	2048	2D	–	6.41	8.29	82	13.45	RTX2080Ti
RAFT-3D Teed and Deng (2021)	RGB-D	all	2D	0.094	2.37	45.00	386	11.34	GTX1080Ti

Table 1: Quantitative comparison on the FlyingThings3D (FT3D) test set.

\downarrow

indicates lower is better. We categorize the table into three groups based on the scene flow estimation domain ( i.e. , 2D, 2D/3D, or 3D). Model size is evaluated using the number of parameters (P). Runtime (R) is analyzed across different hardware devices, and the (TFLOPS) column gives the tera-floating-point-operations per second; higher values indicate more powerful GPUs. Best values in each category, with and without fine-tuning, are highlighted in bold.

Method	Modality	FT	KITTId			lidarKITTI
			EPE3D	Acc3DR	Acc3DS	EPE2D	EPE3D	Acc3DR
			[m] $\downarrow$	[%] $\uparrow$	[%] $\uparrow$	[px] $\downarrow$	[m] $\downarrow$	[%] $\uparrow$
FLOT Puy et al. (2020)	LiDAR only	$\times$	0.311	48.5	25.4	18.19	0.501	33.7
FlowStep3D Kittenplon et al. (2021)	LiDAR only	$\times$	0.303	57.8	38.2	–	0.479	20.4
SF3D-RGB (ours)	RGB+LiDAR	$\times$	0.227	63.1	40.1	14.46	0.395	49.8
FLOT Puy et al. (2020)	LiDAR only	$\checkmark$	0.101	80.2	57.8	5.48	0.289	66.2

Table 2: Metrics and comparison on KITTId and lidarKITTI. FT denotes fine-tuning. Best values in each category, with and without fine-tuning, are highlighted in bold.

Figure 3: Qualitative scene flow results of SF3D-RGB on the KITTId dataset, compared with LiDAR-only and early fusion baselines. LiDAR points are overlaid on the images to improve visualization clarity of the input scenes. Note that FLOT Puy et al. (2020) uses only LiDAR as input, whereas both the early fusion and our SF3D-RGB use LiDAR combined with a monocular camera as inputs. In the error maps, dark blue indicates lower error, dark red indicates higher error, and black regions indicate unavailable LiDAR points due to removal of ground points.

Figure 4: Qualitative comparison of SF3D-RGB and DeepLiDARFlow Rishav et al. (2020) on the KITTId and lidarKITTI test datasets. Our SF3D-RGB results show strong scene flow accuracy across both datasets compared to DeepLiDARFlow. LiDAR points are overlaid on the images to improve visualization clarity of the input scenes. Note that DeepLiDARFlow and our SF3D-RGB use LiDAR combined with a monocular camera as inputs. In the error maps, dark blue indicates lower error, dark red indicates higher error, and black regions indicate unavailable LiDAR points due to removal of ground points. Note that DeepLiDARFlow does not exclude ground points in its scene flow estimation.

Method	$\mathbf{\dfrac{\lambda}{\lambda+\varepsilon}}$	EPE3D	OUT3D	Acc3DR	Acc3DS
Method	$\mathbf{\dfrac{\lambda}{\lambda+\varepsilon}}$	[m] $\downarrow$	[%] $\downarrow$	[%] $\uparrow$	[%] $\uparrow$
FLOT	0.608	0.138	68.1	66.2	33.2
Early Fusion	0.549	0.129	64.2	70.3	36.1
SF3D-RGB	0.587	0.102	57.6	77.5	45.8

Table 3: Ablation study on FT3D (late fusion, number of Sinkhorn iterations

k

fixed to 1). Best values are in bold.

LiDAR	RGB	Early	Late	1 $\times$ MLP	2 $\times$ MLP	$\mathbf{\dfrac{\lambda}{\lambda+\varepsilon}}$	EPE3D	OUT3D
Modality	Modality	Fusion	Fusion	256	256	$\mathbf{\dfrac{\lambda}{\lambda+\varepsilon}}$	[m] $\downarrow$	[%] $\downarrow$
✓						0.524	0.101	31.8
✓	✓	✓				0.372	0.092	29.4
✓	✓		✓		✓	0.307	0.098	27.6
✓	✓		✓	✓		0.443	0.078	25.5

Table 4: Verification of SF3D-RGB fusion on KITTId. Best values are in bold.

Scene Flow Estimation

See all 21 papers

UniFlow: Zero-Shot LiDAR Scene Flow for Autonomous Vehicles
Nov 23, 2025Siyi Li, Qingwen Zhang, Ishan Khatri +4Point Cloud Learning Autonomous Driving Perception
University of Pennsylvania · KTH Royal Institute of Technology · StackAV +1
CorrelationFlow: A Training-Free Geometric Approach for LiDAR Scene Flow Estimation
Jul 31, 2026Minh-Quan Dao, Yancong Lin, Shiming Wang +2Scene Flow Estimation
Nantes Université, LS2N · University of Nottingham · TU Delft +1
$x^2$ -Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space
Mar 17, 2026Ruishan Guo, Ciyu Ruan, Haoyang Wang +3Scene Flow Estimation Optical Flow
Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology · The University of Hong Kong

Nov 23, 2025J/K move · Enter open · S save

UniFlow: Zero-Shot LiDAR Scene Flow for Autonomous Vehicles

Siyi Li, Qingwen Zhang, Ishan Khatri, Kyle Vedder+3

University of Pennsylvania · KTH Royal Institute of Technology · StackAV+1

Point Cloud Learning Autonomous Driving Perception Cross-Domain Generalization 3D Scene Understanding

Open paper AskPDF arXiv

LiDAR scene flow is the task of estimating per-point 3D motion between consecutive point clouds. Recent methods achieve centimeter-level accuracy on popular autonomous vehicle (AV) datasets, but are typically only trained and evaluated on a single sensor. In this paper, we aim to learn general motion priors that transfer to diverse and unseen LiDAR sensors. However, prior work in LiDAR semantic segmentation and 3D object detection demonstrate that naively training on multiple datasets yields worse performance than single dataset models. Interestingly, we find that this conventional wisdom does not hold for motion estimation, and that state-of-the-art scene flow methods greatly benefit from cross-dataset training without architectural modification. We posit that low-level tasks such as motion estimation may be less sensitive to sensor configuration; indeed, our analysis shows that models trained on fast-moving objects (e.g., from highway datasets) perform well on fast-moving objects, even across different datasets. Informed by our analysis, we propose UniFlow, a feedforward model that unifies and trains on multiple large-scale LiDAR scene flow datasets with diverse sensor placements and point cloud densities. Our frustratingly simple solution establishes a new state-of-the-art on Waymo and nuScenes, improving over prior work by 5.1% and 35.2% respectively. Moreover, UniFlow achieves state-of-the-art accuracy on unseen datasets like TruckScenes and AEVAScenes, outperforming prior dataset-specific models by 30.1% and 22.5% respectively.

Abstract

Scene Flow Estimation

Explore similar work

UniFlow: Zero-Shot LiDAR Scene Flow for Autonomous Vehicles

CorrelationFlow: A Training-Free Geometric Approach for LiDAR Scene Flow Estimation

x2x^2x2-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space

$x^2$ -Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space