State Space Models (SSMs) have surfaced as a promising architecture in Video Frame Interpolation (VFI), as they can capture long-range dependencies with linear computational complexity. However, their predefined scanning order limits their effectiveness in modeling the dynamic motion trajectories inherent in VFI problems. To tackle this challenge, we propose Motion-Guided Mamba for Video Frame Interpolation (MGMVFI), an adaptation of the selective state space model tailored explicitly for VFI. MGMVFI introduces Motion-Guided Serialization (MGS), which leverages optical flow to define a motion-adaptive 1D input order for the SSM. This aligns the causal state updates with semantically related tokens, enabling motion-consistent feature propagation, particularly for large and dynamic motions. Additionally, to mitigate the unreliable feature representations caused by inaccurate optical flow estimates, we introduce contextual synthesis that utilizes the surrounding spatial context for robust inter-frame feature synthesis. These components are seamlessly integrated within our tailored Mamba architecture, which also employs a lightweight refinement block to enhance local detail reconstruction at a reduced computational cost. Extensive experiments on standard VFI benchmarks demonstrate that MGMVFI achievesstate-of-the-artperformance,particularly on complex and dynamic motions, thereby establishing a new direction for sequence modeling in video interpolation.
Figures & tables
Figure 1: We present Motion-Guided Serialization (MGS), which uses optical flow to sample from motion-aligned feature maps F0′ & F1′ . MGS results in consecutive semantically consistent tokens (Patch 7 & 9 ) for a. Horizontal and b. Vertical scans.
Figure 2: Overall pipeline of our model. Our MGMVFI block first rearranges input features based on their motion trajectory. It then augments features in regions with unreliable optical flow using spatial context from the surrounding (highlighted) patches. The Contextual Synthesis is visualized for a 3 × 3 hollow kernel case.
Train
Vimeo-90K [ 46 ]
UCF101 [ 43 ]
SNU-FILM [ 4 ]
Dataset
Easy
Medium
Hard
Extreme
DAIN [ 1 ]
V
34.71/0.9756
34.99/0.9683
39.73/0.9902
35.46/0.9780
30.17/0.9335
25.09/0.8584
AdaCof [ 23 ]
V
34.47/0.9730
34.90/0.9680
39.80/0.9900
35.05/0.9754
29.46/0.9244
24.31/0.8439
CAIN [ 4 ]
V
34.65/0.9730
34.91/0.9690
39.89/0.9900
35.61/0.9776
29.90/0.9292
24.78/0.8507
Softsplat [ 37 ]
V
36.13/0.9805
35.39/0.9697
40.26/0.9911
36.07/0.9798
30.53/0.9365
25.16/0.8604
XVFI [ 42 ]
V
35.09/0.9759
35.17/0.9685
39.93/0.9907
35.37/0.9782
29.58/0.9276
24.17/0.8450
Table 1: Quantitative Evaluation across three benchmark datasets. The best and second best results are indicated by bold and underline .
Xiph 2K
Xiph 4K
Params(M)
RT(ms)
SGM-VFI
36.57
0.9424
34.23
0.9021
20.8
1168.9
EMA-VFI
36.74
0.9445
34.54
0.9054
65.6
332.3
EDEN
33.36
0.9043
26.84
0.7303
157.8
1304.4
VFIMamba
37.13
0.9451
34.62
0.9059
66.1
414.7
LCMamba(B)
36.90
0.9456
34.26
0.9040
16.2
785.6
Ours
37.15
0.9454
34.65
0.9059
78.4
427.5
Table 2: Quantitative evaluation on Xiph [ 35 ] dataset and efficiency comparison. Runtimes (RT) are measured for a 2048 × 1024 resolution input on an RTX A6000.
Figure 3: Visual comparison of the interpolated results from the SNU-FILM [ 4 ] Hard and Vimeo90K [ 46 ] dataset. Best viewed zoomed in.
Figure 4: Visual comparison of the progressive module analysis from Vimeo-90K [ 46 ] .
Video frame interpolation (VFI) remains a challenging task, particularly when dealing with large, non-linear motions and complex occlusions. While flow-based methods are prevalent, they often struggle with ambiguous correspondences. Recent VFI methods based on selective State Space Models (SSMs) are still limited by static grid-based scanning that misaligns with physical motion. In this paper, we propose Motion-Aligned Selective Scan (MASS), a novel framework that reformulates feature scanning from static spatial grids to dynamic motion trajectories. MASS builds a feature sequence along each pixel's flow-guided trajectory and aggregates it with an SSM. Specifically, we introduce a learnable non-linear path integration to approximate complex curved trajectories via residual velocity updates, and a velocity-aware SSM that dynamically adjusts the sampling budget and step size based on motion magnitude. This adaptive strategy allocates denser sampling to fast-motion regions while keeping static regions efficient. Furthermore, the aggregated states guide a refinement module to rectify intermediate flows and masks in an end-to-end manner. Extensive experiments indicate that MASS achieves highly competitive overall performance on standard benchmarks, establishing state-of-the-art results particularly in challenging scenarios with large displacements and complex dynamics.
Jun-Sang Yoo, Seung-Won Jung
Department of Electrical Engineering, Korea University, Seoul, Korea
Large pre-trained video diffusion models excel in video frame interpolation but struggle to generate high fidelity frames due to reliance on intrinsic generative priors, limiting detail preservation from start and end frames. Existing methods often depend on motion control for temporal consistency, yet dense optical flow is error-prone, and sparse points lack structural context. In this paper, we propose FC-VFI for faithful and consistent video frame interpolation, supporting 4×x and 8× interpolation, boosting frame rates from 30 FPS to 120 and 240 FPS at 2560×1440resolution while preserving visual fidelity and motion consistency. We introduce a temporal modeling strategy on the latent sequences to inherit fidelity cues from start and end frames and leverage semantic matching lines for structure-aware motion guidance, improving motion consistency. Furthermore, we propose a temporal difference loss to mitigate temporal inconsistencies. Extensive experiments show FC-VFI achieves high performance and structural integrity across diverse scenarios.
Ganggui Ding, Hao Chen, Xiaogang Xu
⋆Zhejiang University · † Chinese University of Hong Kong
We propose Symmetric Nonlinear Motion-guided Generative Video Frame Interpolation (SNM-VFI), a training-free framework for motion-controllable generative video frame interpolation with pre-trained optical flow and video diffusion models. Unlike conventional diffusion-based VFI methods that synthesize intermediate frames from random noise, SNM-VFI guides the generative process with correspondence-aware frames produced by a symmetric nonlinear motion model. Specifically, we first utilize a pre-trained optical flow model to construct multi-frame nonlinear flow-based intermediate frames and confidence maps. These flow-guided frames are then encoded as latent priors to initialize and iteratively guide a pre-trained Video Diffusion model, enabling the diffusion model to preserve dense motion correspondence while improving perceptual realism. To further enhance output quality, we employ confidence maps to fuse structurally reliable flow-based predictions with diffusion-generated details in uncertain regions such as occlusions and object boundaries. Extensive evaluations on challenging benchmarks, including DAVIS, Sintel, and KITTI, demonstrate that SNM-VFI achieves strong perceptual quality, competitive reconstruction accuracy, and robust temporal coherence across diverse motion scenarios.
Jisoo Jeong, Hong Cai, Jamie Menjay Lin +5
Qualcomm AI Research · Qualcomm Technologies, Inc.