GenOpticalFlow: A Generative Approach to Unsupervised Optical Flow Learning

Yixuan Luo; Feng Qiao; Zhexiao Xiong; Yanjing Li; Nathan Jacobs

cs.CVMar 23, 2026

GenOpticalFlow: A Generative Approach to Unsupervised Optical Flow Learning

Authors: Yixuan Luo, Feng Qiao, Zhexiao Xiong, Yanjing Li, Nathan Jacobs

Organizations: University of Chicago · Washington University in St. Louis

Abstract

Optical flow estimation is a fundamental problem in computer vision, yet the reliance on expensive ground-truth annotations limits the scalability of supervised approaches. Although unsupervised and semi-supervised methods alleviate this issue, they often suffer from unreliable supervision signals based on brightness constancy and smoothness assumptions, leading to inaccurate motion estimation in complex real-world scenarios. To overcome these limitations, we introduce \modelname, a novel framework that synthesizes large-scale, perfectly aligned frame--flow data pairs for supervised optical flow training without human annotations. Specifically, our method leverages a pre-trained depth estimation network to generate pseudo optical flows, which serve as conditioning inputs for a next-frame generation model trained to produce high-fidelity, pixel-aligned subsequent frames. This process enables the creation of abundant, high-quality synthetic data with precise motion correspondence. Furthermore, we propose an inconsistent pixel filtering strategy that identifies and removes unreliable pixels in generated frames, effectively enhancing fine-tuning performance on real-world datasets. Extensive experiments on KITTI2012, KITTI2015, and Sintel demonstrate that \modelname achieves competitive or superior results compared to existing unsupervised and semi-supervised approaches, highlighting its potential as a scalable and annotation-free solution for optical flow learning. We will release our code upon acceptance.

Figures & tables

Figure 1: Visualization of the synthesized data triplet on KITTI2012, including the reference frame $I_{t}$ , the artificially generated optical flow $\tilde{F}_{t\rightarrow t+1}$ , and the conditioned next-frame generation prediction result $\tilde{I}_{t+1}$ .

Figure 2: Overview of the conditioned next-frame framework and artificial optical flow generation. Given an input view and its corresponding optical flow, our framework constructs two types of embeddings: a 2D coordinate embedding of the input view and a warped coordinate embedding of the target view derived from the optical flow. A semantic preserver network extracts high-level semantic features from the input view, while a diffusion model conditioned on these embeddings learns the geometric warping necessary to generate the novel view and accurately align pixel-level motion. To further enhance spatial correspondence, we augment the self-attention mechanism with cross-view attention and jointly aggregate features across views. Notably, ground-truth optical flow is used only during pre-training, while synthetic datasets are constructed using artificial optical flow produced via a geometry-based camera-motion model.

Methods	Pretrained	Middlebury 2014			KITTI 2015
Methods	Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Leave blank	-	11.328	0.315	0.450	12.980	0.374	0.313
Stretch	-	14.842	0.432	0.285	14.757	0.429	0.212
3D Photography [ 44 ]	-	14.190	0.427	0.275	14.540	0.398	0.210
RePaint [ 30 ]	-	15.102	0.462	0.311	15.056	0.462	0.251
SD-Inpainting [ 47 ]	-	15.740	0.412	0.311	9.792	0.230	0.652

Table 1: Quantitative comparison of motion-conditioned image generation methods on the Middlebury 2014 and KITTI 2015 datasets. Evaluation metrics include PSNR (

\uparrow

), SSIM (

\uparrow

), and LPIPS (

\downarrow

). For the Middlebury 2014 dataset, we report results using optical flow estimated by RAFT and WAFT, while the KITTI 2015 dataset provides ground-truth optical flow for evaluation. Qualitative results can be found in Supp A

Method	Without GenOpticalFlow		With GenOpticalFlow
Method	EPE $\downarrow$	Fl-all $\downarrow$	EPE $\downarrow$	Fl-all $\downarrow$
RAFT-chairs [ 52 ]	9.82	37.55	8.18	20.59
GMFlowNet-things [ 71 ]	4.43	16.70	4.24	16.68
FlowFormer-chairs [ 15 ]	10.14	34.46	7.45	21.19
Flowformer++-chairs [ 43 ]	10.45	36.97	7.98	22.01
FlowDiffuser-things [ 31 ]	3.61	11.8	3.52	10.69

Table 2: Quantitative zero-shot evaluation on the KITTI 2015 benchmark. We report the EPE and Fl-all metrics for six representative optical flow models, both with and without integrating our proposed GenOpticalFlow. Across all methods, incorporating GenOpticalFlowconsistently reduces error, demonstrating its strong generalization ability and compatibility with diverse architectures. Lower values indicate better performance.

	Middlebury			KITTI 2015
	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
w/o CVA	17.93	.417	.314	15.36	.291	.290
w/o Coord	17.15	.508	.291	14.63	.384	.314
Ours	19.24	.653	.217	17.85	.552	.268

Table 3: Ablation Study.

Table 6

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Qualitative results of optical flow generation on the KITTI (top) and Sintel (bottom) datasets. In each panel, the first row displays the conditioning input frame, while the second row visualizes a corresponding optical flow map randomly sampled from our model. The results highlight the model’s ability to generate dense, structurally aligned flow predictions across varying scene complexities.

Figure 4: Visualization of next-frame generation results. The figure demonstrates the temporal consistency of our method on the KITTI and Sintel benchmarks. The top rows show the reference input frames, while the bottom rows display the synthesized next frames. Our framework effectively preserves texture details and object geometry during the generation process.

Explore similar work

U $^{2}$ Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation
Apr 11, 2026Xunpei Sun, Wenwei Lin, Yi Chang +1Optical Flow
Sun Yat-sen University, Guangzhou, China · Huazhong University of Science and Technology, Wuhan, China
MegaFlow: Zero-Shot Large Displacement Optical Flow
Mar 26, 2026Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys +1Optical Flow
ETH Zurich, Switzerland · Microsoft, Switzerland · University of Tübingen, Tübingen AI Center, Germany
FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow
Mar 30, 2026Sadra Safadoust, Fabio Tosi, Matteo Poggi +1Optical Flow
Department of Computer Engineering and KUIS AI Center, Koç University, Istanbul, Turkey · Department of Computer Science and Engineering (DISI), University of Bologna, Italy

Apr 11, 2026J/K move · Enter open · S save

U $^{2}$ Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation

Xunpei Sun, Wenwei Lin, Yi Chang, Gang Chen

Sun Yat-sen University, Guangzhou, China · Huazhong University of Science and Technology, Wuhan, China

Optical Flow

Open paper AskPDF arXiv

Unsupervised optical flow methods typically lack reliable uncertainty estimation, limiting their robustness and interpretability. We propose U

^{2}

Flow, the first recurrent unsupervised framework that jointly estimates optical flow and per-pixel uncertainty. The core innovation is a decoupled learning strategy that derives uncertainty supervision from augmentation consistency via a Laplace-based maximum likelihood objective, enabling stable training without ground truth. The predicted uncertainty is further integrated into the network to guide adaptive flow refinement and dynamically modulate the regional smoothness loss. Furthermore, we introduce an uncertainty-guided bidirectional flow fusion mechanism that enhances robustness in challenging regions. Extensive experiments on KITTI and Sintel demonstrate that U

^{2}

Flow achieves state-of-the-art performance among unsupervised methods while producing highly reliable uncertainty maps, validating the effectiveness of our joint estimation paradigm. The code is available at https://github.com/sunzunyi/U2FLOW.

Abstract

Explore similar work

U2^{2}2Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation

MegaFlow: Zero-Shot Large Displacement Optical Flow

FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow

U $^{2}$ Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation