FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow

Sadra Safadoust; Fabio Tosi; Matteo Poggi; Fatma Güney

cs.CVMar 30, 2026

FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow

Authors: Sadra Safadoust, Fabio Tosi, Matteo Poggi, Fatma Güney

Organizations: Department of Computer Engineering and KUIS AI Center, Koç University, Istanbul, Turkey · Department of Computer Science and Engineering (DISI), University of Bologna, Italy

Abstract

We present FlowIt, a novel architecture for optical flow estimation that combines global matching with confidence and occlusion-guided refinement. At its core, FlowIt leverages a hierarchical transformer architecture that captures extensive global context, enabling the model to effectively model long-range correspondences. To overcome the limitations of localized matching, we formulate the flow initialization as an optimal transport problem. This formulation yields a highly robust initial flow field, alongside explicitly derived occlusion and confidence maps. These cues are then seamlessly integrated into a guided refinement stage, where the network actively propagates reliable motion estimates from high-confidence regions into ambiguous, low-confidence areas. Extensive experiments across the Sintel, KITTI, Spring, and LayeredFlow datasets validate the effectiveness of our approach. FlowIt achieves state-of-the-art results on the competitive Sintel benchmark and establishes new state-of-the-art cross-dataset zero-shot generalization performance on Sintel, Spring, and LayeredFlow, while also delivering competitive performance on both the KITTI benchmark and KITTI zero-shot generalization settings.

Figures & tables

Figure 2: Architecture Overview. FlowIt extracts multi-scale features from images using a CNN encoder followed by a Feature Pyramid Network (FPN). These features are processed with one or more Multi-Resolution Transformer (MRT) blocks. A 4D correlation volume is constructed using the $\frac{1}{4}$ resolution features, and optimal transport is applied to produce a 4D probability map. Initial flow, occlusion, and confidence maps are derived using the probability map. These predictions are refined through three refinement iterations to obtain the final outputs.

Panel of Figure 3 — Figure 3: Correlation and probability visualizations for a single pixel. (a) and (b) show the correlation values in the pixel’s neighborhood for SEA-RAFT and FlowIt, respectively. (c) shows the corresponding probability map obtained by applying optimal transport to FlowIt’s correlation volume.

	Method	Sintel (test)		KITTI (test)
	Method	Clean $\downarrow$	Final $\downarrow$	Fl-all $\downarrow$	Non-Occ $\downarrow$	Fl-bg $\downarrow$	Fl-fg $\downarrow$
(A)	MemFlow [ 11 ]	1.05	1.91	4.10	2.56	3.67	6.27
	StreamFlow [ 67 ]	1.04	1.87	4.24	-	-	-
	ARFlow [ 37 ]	0.96	1.79	2.85	1.91	2.48	4.69
	VideoFlow [ 58 ]	0.99	1.65	3.65	-	-	-
	MEM-FOF [ 3 ]	0.93	1.89	2.94	1.97	2.60	4.66

Table 1: Results on Sintel [ 7 ] and KITTI [ 14 ] benchmarks. FlowIt (XL) ranks first on both Clean and Final splits of Sintel. On KITTI, it establishes a new state-of-the-art on non-occluded pixels, while remaining competitive with top-performing methods on other metrics.

Panel of Figure 4 — Figure 4: Qualitative Results on KITTI [ 14 ] Test Set. Visualizations and error metrics are obtained directly from the official evaluation server. For WAFT, we report results using the DAv2-a2 variant, which achieves the best performance among WAFT models on the KITTI benchmark.

Method	Sintel (train)		KITTI (train)
Method	Clean $\downarrow$	Final $\downarrow$	Fl-epe $\downarrow$	Fl-all $\downarrow$
PWC-Net [ 64 ]	2.55	3.93	10.4	33.7
FlowNet2 [ 22 ]	2.02	3.14	10.1	30.4
RAFT [ 69 ]	1.43	2.71	5.04	17.4
GMA [ 27 ]	1.30	2.74	4.69	17.1
SKFlow [ 66 ]	1.22	2.46	4.27	15.5

Table 2: Zero-Shot Generalization on Sintel (train) and KITTI (train).

Panel of Figure 5 — Figure 5: Qualitative Results on Sintel [ 7 ] Training Set. From left to right: first frame, flow by WAFT-DAv2-a1 and FlowIt (XL), ground-truth flow. The top four rows are from the clean split, and the bottom four rows are from the final split.

Method Spring (train) 1px

\downarrow

EPE

\downarrow

MatchFlow(G) [ 10 ] 4.504 0.407 Flowformer

++

[ 59 ] 4.482 0.447 MS-RAFT+ [ 24 ] 3.577 0.397 RAFT [ 69 ] 4.788 0.448 RPKNet [ 50 ] 4.472 0.416 DIP [ 85 ] 4.273 0.463 SKFlow [ 66 ] 4.521 0.408 GMFlow [ 76 ] 29.49 0.930 GMFlow

+

[ 77 ] 4.292 0.433 Flowformer [ 17 ] 4.508 0.470 SEA-RAFT (S) [ 73 ] 4.161 0.410 SEA-RAFT (M) [ 73 ] 3.888 0.406 SEA-RAFT (L) [ 73 ] 3.842 0.426 FlowSeek (T) [ 53 ] 4.111 0.410 FlowSeek (S) [ 53 ] 4.058 0.406 FlowSeek (M) [ 53 ] 3.941 0.419 FlowSeek (L) [ 53 ] 3.838 0.402 WAFT-DAv2-a1 [ 72 ] 3.811 0.433 WAFT-Twins-a2 [ 72 ] 5.611 0.557 WAFT-DAv2-a2 [ 72 ] 5.340 0.487 WAFT-DINOv3-a2 [ 72 ] 4.006 0.458 FlowIt (S) 3.937 0.391 FlowIt (M) 4.017 0.398 FlowIt (L) 3.450 0.377 FlowIt (XL) 3.474 0.371

Method LayeredFlow (All pixels) EPE

\downarrow

1px

\downarrow

3px

\downarrow

5px

\downarrow

FlowNet-C [ 12 ] 9.71 89.07 61.51 43.93 FlowNet2 [ 22 ] 10.07 77.56 54.22 42.13 PWC-Net [ 64 ] 9.49 74.93 50.47 39.05 GMA [ 27 ] 9.77 72.46 46.93 36.97 SKFlow [ 66 ] 9.86 72.02 47.44 36.88 CRAFT [ 61 ] 10.36 72.34 47.54 37.00 GMFlow [ 76 ] 9.09 81.99 51.79 37.75 GMFlow+ [ 77 ] 9.46 82.71 53.14 39.70 FlowFormer [ 17 ] 10.20 73.59 48.97 38.56 RAFT [ 69 ] 9.38 71.98 46.46 36.15 SEA-RAFT (S) [ 73 ] 10.05 71.48 46.90 36.32 SEA-RAFT (M) [ 73 ] 10.17 69.73 45.94 34.78 SEA-RAFT (L) [ 73 ] 10.99 69.46 45.59 34.78 FlowSeek (T) [ 53 ] 9.09 70.82 43.74 32.36 FlowSeek (S) [ 53 ] 9.16 69.99 43.67 31.90 FlowSeek (M) [ 53 ] 8.30 68.85 41.81 32.09 FlowSeek (L) [ 53 ] 8.30 68.98 41.49 31.64 WAFT-DAv2-a1 [ 72 ] 10.37 68.23 43.73 33.38 WAFT-Twins-a2 [ 72 ] 12.35 70.93 47.11 36.51 WAFT-DAv2-a2 [ 72 ] 10.75 73.06 51.40 40.85 WAFT-DINOv3-a2 [ 72 ] 9.30 73.05 48.25 38.22 FlowIt (S) 9.56 70.41 45.45 35.62 FlowIt (M) 10.07 69.83 46.47 36.08 FlowIt (L) 7.93 66.55 43.20 33.09 FlowIt (XL) 7.26 65.62 38.78 28.23

Table 3: Generalization on Spring and LayeredFlow.

Experiment	Configuration	EPE $\downarrow$	3px $\downarrow$	#GMACS	latency (ms)
Model Size	(S)	0.617	2.295	344	99
	(M)	0.595	2.209	764	120
	(L)	0.499	1.635	1412	146
	(XL)	0.447	1.328	3092	216
Transformer Architecture	GMFlow	0.652	2.457	397	155
Transformer Architecture	MRT	0.617	2.295	344	99

Table 4: Ablation study on FlyingChairs validation set. We ablate model size , tranformer architecture , flow initialization using optimal transport , conditioning refinement on confidence and occlusion maps , number of refinement steps , axis-wise refinement , and loss function . Configurations used in our final model are underlined. Latencies are measured on an NVIDIA L40S GPU.

Panel of Figure 6 — Figure 6: Qualitative Results of Zero-Shot Generalization on LayeredFlow [ 75 ] . From top to bottom: first frame, flow by SEA-RAFT (L) [ 73 ] , WAFT-DAv2-a1 [ 72 ] , FlowSeek (L) [ 53 ] , and FlowIt (XL).

Panel of Figure 10 — Figure S1: Qualitative Results of Zero-Shot Generalization on KITTI [ 14 ] Training Set. From top to bottom: first frame, flow by SEA-RAFT (L) [ 73 ] , FlowSeek (L) [ 53 ] , WAFT-Twins-a2 [ 72 ] , and FlowIt (XL).

Panel of Figure 11 — Figure S2: Qualitative Results on Sintel (clean) [ 7 ] Test Set. From top to bottom: first frame, flow predicted by SEA-RAFT (L) [ 73 ] , WAFT-DINOv3-a2 [ 72 ] , and FlowIt (XL), followed by the corresponding error maps. Results are obtained directly from the official Sintel evaluation server.

Panel of Figure 12 — Figure S3: Qualitative Results on Sintel (final) [ 7 ] Test Set. From top to bottom: first frame, flow predicted by SEA-RAFT (L) [ 73 ] , WAFT-DINOv3-a2 [ 72 ] , and FlowIt (XL), followed by the corresponding error maps. Results are obtained directly from the official Sintel evaluation server.

Panel of Figure 13 — Figure S4: Qualitative Results of Zero-Shot Generalization on Spring [ 45 ] Training Set. From top to bottom: first frame, flow by SEA-RAFT (L) [ 73 ] , WAFT-DAv2-a1 [ 72 ] , FlowSeek (L) [ 53 ] , FlowIt (XL), and ground-truth.

Extra data	Method	Sintel (test)		KITTI (test)
Extra data	Method	Clean $\downarrow$	Final $\downarrow$	Fl-all $\downarrow$	Non-Occ $\downarrow$	Fl-bg $\downarrow$	Fl-fg $\downarrow$
Multi-frame	SplatFlow [ 70 ]	1.12	2.07	4.61	2.96	4.26	6.34
	MemFlow [ 11 ]	1.05	1.91	4.10	2.56	3.67	6.27
	StreamFlow [ 67 ]	1.04	1.87	4.24	-	-	-
	ARFlow [ 37 ]	0.96	1.79	2.85	1.91	2.48	4.69
	VideoFlow [ 58 ]	0.99	1.65	3.65	-	-	-

Table S1: Results on Sintel [ 7 ] and KITTI [ 14 ] benchmarks – impact of extra data. On the left-most column, we report extra data used at inference in general (i.e., multi-frame) or for training 2-frame methods (e.g., TartanAir). The top-three scores achieved by 2-frame methods trained with and without external data are highlighted separately.

Explore similar work

MegaFlow: Zero-Shot Large Displacement Optical Flow
Mar 26, 2026Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys +1Optical Flow
ETH Zurich, Switzerland · Microsoft, Switzerland · University of Tübingen, Tübingen AI Center, Germany
FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation
Sep 10, 2026Vladislav Bargatin, Alexander Yakovenko, Khaled Abud +1Optical Flow Vision Transformer
AI Center, Lomonosov MSU, Moscow, Russia · Lomonosov Moscow State University, Moscow, Russia · MSU Institute for Artificial Intelligence, Moscow, Russia
FlowPainter: Inpainting Optical Flow via Confidence-Guided Completion
Jul 11, 2026Yuang Meng, Chenyang Wu, Xianshun Liu +7Optical Flow Diffusion Model Guidance
VCIP, CS, Nankai University · OPPO Research Institute · The Hong Kong Polytechnic University

Mar 26, 2026J/K move · Enter open · S save

MegaFlow: Zero-Shot Large Displacement Optical Flow

Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys, Haofei Xu

ETH Zurich, Switzerland · Microsoft, Switzerland · University of Tübingen, Tübingen AI Center, Germany

Optical Flow

Open paper AskPDF arXiv

Accurate estimation of large displacement optical flow remains a critical challenge. Existing methods typically rely on iterative local search or/and domain-specific fine-tuning, which severely limits their performance in large displacement and zero-shot generalization scenarios. To overcome this, we introduce MegaFlow, a simple yet powerful model for zero-shot large displacement optical flow. Rather than relying on highly complex, task-specific architectural designs, MegaFlow adapts powerful pre-trained vision priors to produce temporally consistent motion fields. In particular, we formulate flow estimation as a global matching problem by leveraging pre-trained global Vision Transformer features, which naturally capture large displacements. This is followed by a few lightweight iterative refinements to further improve the sub-pixel accuracy. Extensive experiments demonstrate that MegaFlow achieves state-of-the-art zero-shot performance across multiple optical flow benchmarks. Moreover, our model also delivers highly competitive zero-shot performance on long-range point tracking benchmarks, demonstrating its robust transferability and suggesting a unified paradigm for generalizable motion estimation. Our project page is at: https://kristen-z.github.io/projects/megaflow.

Abstract

Explore similar work

MegaFlow: Zero-Shot Large Displacement Optical Flow

FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation

FlowPainter: Inpainting Optical Flow via Confidence-Guided Completion