Foundation Model Driven Optical Flow

Praroop Chanda; Suryansh Kumar

cs.CVOct 19, 2026

Foundation Model Driven Optical Flow

Authors: Praroop Chanda, Suryansh Kumar

Organizations: Visual and Spatial AI Lab, VCCM Section, Texas A&M University, College Station, Texas, USA · Department of ECEN, Texas A&M University, College Station, Texas, USA · College of PVFA, Texas A&M University, College Station, Texas, USA · Department of CSCE, Texas A&M University, College Station, Texas, USA

Abstract

We investigate how frozen pretrained visual and depth representations support dense optical flow when coupled with a supervised matching network. Building on TTSFlow, this paper studies Dense Prediction Transformer (DPT) feature decoding and one coarse-to-fine refinement stage. Our method fuses features from frozen DINOv2 and Depth Anything V2 models and estimates optical flow through a trainable transformer-based matching pipeline. After optical-flow training on FlyingChairs and FlyingThings3D, it achieves 2.37 endpoint error (EPE) on the Sintel training split, Final pass, compared with 2.52 for the FlowSeek (T) configuration reported here. Our RoCo-Spring 2026 entry, RoCo-29.VSAI, obtains a robustness-balanced score of 1.1038, Spring EPE of 1.411, and RobustSpring proxy EPE of 4.739, with a lower balanced score than seven of the twelve listed reference baselines. These results characterize the complete pipeline, which reuses frozen foundation backbones and performs one refinement stage. Code is available at: https://github.com/Visual-and-Spatial-AI-Lab/FFlow

Figures & tables

Panel of Figure 1 — Figure 1: Overall pipeline, including refinement. Each feature branch is applied to both RGB frames. The TTSFlow feature-fusion design [ 3 ] is extended with visual DPT decoding and two feature scales. Matching, propagation, refinement, and convex upsampling follow GMFlow [ 55 ] . The transformer is shared across scales. Fusion and projection weights are shared. Frozen labels apply to the pretrained foundation models. The visual DPT gradient status in the released code is documented in Appendix G.2 .

Method	#refine	Things (val)	Sintel (train, clean)				Sintel (train, final)
Method	#refine	EPE	EPE	s 0-10	s 10-40	s 40+	EPE	s 0-10	s 10-40	s 40+
RAFT [ 47 ]	32	4.25	1.43	0.33	1.54	9.04	2.71	0.51	2.98	17.62
GMFlow [ 55 ]	0	3.48	1.50	0.46	1.77	8.26	2.96	0.72	3.45	17.70
GMFlow [ 55 ]	1	2.80	1.08	0.30	1.25	6.26	2.48	0.51	2.81	15.76
SEA-RAFT (S) [ 51 ]	4	-	1.27	-	-	-	4.32	-	-	-
FlowSeek (T) [ 34 ]	4	3.94	1.12	0.23	1.27	7.12	2.52	0.45	2.78	16.60

Table 1: Generalization performance after training on Chairs and Things train sets. The 2.37 EPE is on the Sintel training split, Final pass, and is the lowest among these listed configurations. The TTSFlow row is reproduced from [ 3 ] ; refinement counts are not runtime measurements.

Method	Extra flow data	#refine	Sintel (train)		KITTI (train)
			Clean EPE	Final EPE	EPE	F1-all (%)
RAFT [ 47 ]	–	32	0.768	1.217	0.63	1.47
GMFlow [ 55 ]	–	0	0.947	1.276	2.06	7.57
GMFlow [ 55 ]	–	1	0.762	1.110	1.36	5.17
SEA-RAFT (S) [ 51 ]	TartanAir	4	0.546	0.782	0.93	2.65
FlowSeek (T) [ 34 ]	TartanAir	4	0.710	1.280	1.26	3.90

Table 2: Performance on the Sintel and KITTI training sets. Sintel results are obtained after training on Chairs, Things, and the mixed TSKH datasets. Training configurations differ; the comparison does not isolate architecture or training-scale effects. Extra flow data denotes additional optical-flow pretraining, not foundation-model pretraining. The TTSFlow row is reproduced from [ 3 ] . #refine denotes the number of refinement steps. Lower values indicate better performance for all metrics.

Method	RbS-Score	Spring EPE	Proxy EPE	$\Delta$
SEA-RAFT [ 51 ]	0.4584	0.363	3.323	2.960
MS-RAFT+ [ 14 ]	0.6774	0.643	4.263	3.620
FlowFormer [ 12 ]	0.7368	0.723	4.493	3.770
GMFlow [ 55 ]	0.8015	0.945	3.925	2.980
GMA [ 15 ]	0.8704	0.914	4.944	4.030
VSAI (RoCo-29, Method 1)	1.1038	1.411	4.739	3.328

Table 3: RoCo-Spring optical flow result. The reported columns compare the submitted entry with challenge reference methods.

\Delta=E_{R,\mathrm{proxy}}-E_{S}

is shown for interpretation. All metrics are lower-is-better; EPE and disagreement are in pixels. Reference baselines are unranked challenge references, not competing team ranks.

Figure 2: Qualitative results on Spring Benchmark Dataset [ 41 ] .

Method	Things (val, clean)	Sintel (train, clean)				Sintel (train, final)
Method	EPE	EPE	s 0-10	s 10-40	s 40+	EPE	s 0-10	s 10-40	s 40+
(w/o) Fusion	3.52	1.575	0.45	2.01	8.53	3.12	0.685	3.57	19.37
(w/) Fusion	3.02	1.46	0.39	2.05	7.74	2.81	0.64	3.36	16.99

Table 4: Fusion ablation reproduced from TTSFlow [ 3 ] , Table 4, with 800k Things-stage updates following 200k Chairs updates, and no DPT or refinement. This is prior evidence for the base fusion design, not a new ablation of the submitted configuration.

Method	Sintel (train, clean)				Sintel (train, final)
Method	EPE	$s_{0-10}$	$s_{10-40}$	$s_{40+}$	EPE	$s_{0-10}$	$s_{10-40}$	$s_{40+}$
without DPT	1.33	0.40	1.61	7.26	2.73	0.62	3.16	16.69
with DPT	1.26	0.34	1.52	7.25	2.37	0.55	2.92	14.05

Table 5: DPT comparison on the Sintel training split, Clean and Final passes, after 360k Things-stage updates following Chairs training. Both configurations use one refinement stage.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	EPE	Subset	EPE
Overall	1.411	Non-rigid	3.804
Low-detail	1.374	Not sky	1.406
High-detail	7.193	Sky	1.489
Matched	1.288	$s_{0-10}$	0.898
Unmatched	6.500	$s_{10-40}$	1.340
Rigid	1.094	$s_{40+}$	5.887

Appendix

Table 6: Clean Spring EPE by evaluation subset. Values are in pixels.

Corruption	$\Delta_{c}$	Corruption	$\Delta_{c}$
Brightness	0.308	JPEG compression	1.671
Contrast	3.213	Motion blur	1.538
Defocus blur	0.592	Pixelate	0.756
Elastic transform	1.189	Rain	13.034
Fog	1.382	Saturate	0.415
Frost	8.579	Shot noise	1.980

Appendix

Table 7: RobustSpring disagreement for all twenty corruptions.

\Delta_{c}

is in pixels. Lower values indicate less change in predicted flow. These are not corrupted-input ground-truth EPE values.

Figure 3: Sintel qualitative comparison. Two examples from the Sintel benchmark [ 1 ] . The circles are annotations retained from the reported experiment. The visual comparisons illustrate local differences in predicted motion, without establishing a quantitative occlusion or boundary advantage.

Figure 4: KITTI qualitative comparison. Two examples from the KITTI training split [ 10 ] . These examples belong to the target-adapted evaluation and should not be interpreted as held-out test results.

Panel of Figure 5 — Figure 5: Predicted flow for the component-removal configurations in Table 8 , without DPT or refinement. Colors encode flow, not ground-truth error.

Method	Sintel (train, clean)				Sintel (train, final)
Method	EPE	s 0-10	s 10-40	s 40+	EPE	s 0-10	s 10-40	s 40+
Ours (Depth)	122.87	103.32	152.01	197.83	120.12	99.42	152.38	196.38
Ours (DINOv2)	36.15	26.49	47.55	79.76	36.42	25.33	49.20	87.06
Ours (Depth + DINOv2)	1.46	0.39	2.05	7.74	2.81	0.64	3.36	16.99

Appendix

Table 8: Ablation of DINOv2 and depth features. Results on the Sintel training set for the reported component-removal configurations of a Chairs+Things model, without DPT or refinement. These are not a capacity-matched comparison of independently trained single-branch models.

	(w/o) Depth	(w/o) DINO	Depth+DINO
EPE	1.77	1.07	0.87

Appendix

Table 9: Three feature configurations after 100k FlyingChairs updates, without DPT or refinement. The 1.77 and 0.87 entries are retained from TTSFlow [ 3 ] . The historical evaluation split is unspecified.

	DepthAnythingV2 (Small)	DepthAnythingV2 (Base)
EPE (FlyingChairs)	0.995	0.87

Appendix

Table 10: DepthAnythingV2 backbone comparison. Results on the FlyingChairs dataset after training for 100k iterations.

	DINOv2 (Small)	DINOv2 (Base)
EPE (FlyingChairs)	1.091	1.056

Appendix

Table 11: DINOv2 backbone comparison. Results on the FlyingChairs dataset after training for 50k iterations.

	RAFT	GMFlow	GMFlow	FlowSeek[T]	SEA-RAFT[S]	Ours
#Refine	32	0	1	4	4	1
Time (ms)	229	194	295	144	63	362

Appendix

Table 12: Inference time comparison of different optical flow methods.

Method	#refine	KITTI EPE	KITTI F1-all
Ours (without DPT)	1	1.337	4.74
Ours (with DPT)	1	1.29	4.29

Appendix

Table 13: Performance on the KITTI training set after 140k KITTI fine-tuning updates, following Chairs, Things, and mixed TSKH training. Both configurations use one refinement stage; this is separate from the 200k setting in Table 2 .

Method	#refine	Things (val)	Sintel (train, clean)				Sintel (train, final)
Method	#refine	EPE	EPE	s 0-10	s 10-40	s 40+	EPE	s 0-10	s 10-40	s 40+
RAFT [ 47 ]	32	4.25	1.43	0.33	1.54	9.04	2.71	0.51	2.98	17.62
GMFlow [ 55 ]	0	3.48	1.50	0.46	1.77	8.26	2.96	0.72	3.45	17.70
GMFlow [ 55 ]	1	2.80	1.08	0.30	1.25	6.26	2.48	0.51	2.81	15.76
SEA-RAFT (S) [ 51 ]	4	-	1.27	-	-	-	4.32	-	-	-
FlowSeek (T) [ 34 ]	4	3.94	1.12	0.23	1.27	7.12	2.52	0.45	2.78	16.60

Appendix

Table 14: Generalization after Chairs training and the indicated Things-stage training duration. The 560k checkpoint improves Things and Sintel Clean EPE relative to the 360k checkpoint, while Sintel Final EPE increases from 2.37 to 2.42.

Module	Parameters	Status in released flow path
DINOv2-S encoder	22,056,576	Frozen
Added visual DPT head	4,124,576	No flow gradient; flag true
Depth Anything V2-B encoder	86,580,480	Frozen
Depth Anything V2 DPT decoder	10,890,305	Frozen
Depth downsampling	959,040	Gradient-connected
Quarter-scale depth projection	32,896	Gradient-connected

Appendix

Table 15: Source-derived parameter inventory of the released two-scale configuration. Counts include biases and embedding/normalization parameters; shared modules are counted once. “No flow gradient” describes the released computation, not a reconstruction of historical training. No experimental checkpoint was loaded to obtain these counts.

Stage	Updates	Global batch	Crop $(H\times W)$	Peak learning rate
FlyingChairs	200k	8	$384\times 512$	$4\times 10^{-4}$
FlyingThings3D	360k	8	$384\times 768$	$2\times 10^{-4}$
TSKH mixture	320k	4	$320\times 896$	$2\times 10^{-4}$
KITTI	200k	4	$320\times 1152$ (code)	$2\times 10^{-4}$
Spring	400k	4	$544\times 960$	$1\times 10^{-4}$

Appendix

Table 16: Reported stage durations and released launcher settings. Durations denote updates within each stage, rather than cumulative updates over all stages. The launcher uses step limits one larger than the displayed round values and names checkpoints at the round values. The KITTI crop discrepancy is recorded explicitly below.

Optical Flow

Optical flow estimates the apparent motion of pixels between images, supporting video understanding, tracking, robotics, and image synthesis.

See all 58 papers

Rethinking Dense Optical Flow without Test-Time Scaling
May 8, 2026Praroop Chanda, Suryansh KumarOptical Flow
Visual and Spatial AI Lab1, VCCM Section College of PVFA2, Department of ECEN3, Department of CSCE4, Texas A&M University, College Station, Texas, USA
FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation
Sep 10, 2026Vladislav Bargatin, Alexander Yakovenko, Khaled Abud +1Optical Flow Vision Transformer
AI Center, Lomonosov MSU, Moscow, Russia · Lomonosov Moscow State University, Moscow, Russia · MSU Institute for Artificial Intelligence, Moscow, Russia
FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow
Mar 30, 2026Sadra Safadoust, Fabio Tosi, Matteo Poggi +1Optical Flow
Department of Computer Engineering and KUIS AI Center, Koç University, Istanbul, Turkey · Department of Computer Science and Engineering (DISI), University of Bologna, Italy

May 8, 2026J/K move · Enter open · S save

Rethinking Dense Optical Flow without Test-Time Scaling

Praroop Chanda, Suryansh Kumar

Visual and Spatial AI Lab1, VCCM Section College of PVFA2, Department of ECEN3, Department of CSCE4, Texas A&M University, College Station, Texas, USA

Optical Flow

Open paper AskPDF arXiv

Recent progress in dense optical flow has been driven by increasingly complex architectures and multi-step refinement for test-time scaling. While these approaches achieve strong benchmark performance, they also require substantial computation during inference. This raises a fundamental question: Is scaling test-time computation the only way to improve dense optical flow accuracy? We argue that it is not. Instead, powerful visual semantic and geometric priors encoded in modern foundation models can reduce, if not overcome, the need for computationally expensive iterative refinement at test-time. In this paper, we present a framework that estimates dense optical flow in a single forward pass, leveraging pretrained foundation representations, while avoiding iterative refinement and additional inference-time computation, thus offering an alternative to test-time scaling. Our method extracts visual semantic features from a frozen DINO-v2 backbone and combines them with geometric cues from a monocular depth foundation model. We fuse these complementary priors into a unified representation and apply a global matching formulation to estimate dense correspondences without recurrent updates or test-time optimization. Despite avoiding iterative refinement, our approach achieves strong cross-dataset generalization across challenging benchmarks. On Sintel Final, we obtain 2.81 EPE without refinement, significantly improving over state-of-the-art (SOTA) SEA-RAFT under comparable training conditions and outperforming RAFT, GMFlow (without refinement), and recent FlowSeek in the same setting. These results suggest that strong foundation priors can substitute for test-time scaling, offering a computationally efficient alternative to refinement-heavy pipelines.

Abstract

Optical Flow

Explore similar work

Rethinking Dense Optical Flow without Test-Time Scaling

FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation

FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow