Organizations: Nanjing University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · The Chinese University of Hong Kong · The University of Hong Kong · Shenzhen University of Advanced Technology · INSAIT, Sofia University “St. Kliment Ohridski”
Video out-painting is an important problem, yet existing research focuses primarily on short clips, leaving long video out-painting largely underexplored. Extending video out-painting to long videos poses substantial challenges. A natural solution is iterative out-painting, i.e., generating videos chunk by chunk, but this introduces two major issues: generation instability or even failure caused by inter-chunk inconsistencies and error accumulation, and semantic drift over long temporal horizons. In this paper, we propose LongSCOP, short for Semantically Consistent long Video Out-Painting. First, we introduce a training recipe with three components for smooth and stable long-horizon generation: self-forcing training, latent calibration, and future references. Beyond generation stability, we recognize that the rich temporal context of long videos provides valuable references for maintaining semantic consistency. We therefore develop a VLM-based agent that automatically selects the most informative reference frames from the entire video to maintain consistent subjects, scenes, and backgrounds throughout long-horizon generation. We also construct a high-quality training dataset of 62K video clips. To facilitate research on this underexplored problem, we introduce LVO-Bench, a benchmark curated by human experts to assess long-horizon stability and semantic consistency. Extensive experiments demonstrate that our method outperforms existing approaches in both generation stability and semantic coherence.
Figures & tables
Figure 1: Illustration of the two main challenges in long video out-painting: (a) unstable generation including abrupt transitions, brightness flickering, and even complete generation collapse caused by accumulated errors, and (b) semantic drift like changed scene layouts, unstable object identities, or inconsistent subject appearances over long time. Both cases are generated by VACE model [ 22 ] .
Figure 2: Illustration of self-forcing training. During training, we use the model-predicted frames as reference frames, which could reduce the gap between training and inference.
Figure 3: Illustration of the importance of suitable reference frame. Adding only one reference frame could significantly improve the out-painting performance of the whole video by nearly 70 FVD points and more than 1.1 dB PSNR on average.
Figure 4: Illustration of key reference frame retrieval framework. Given query frames from current query chunk to be out-painted and candidate frames from both history chunk and future chunk, we aim to find the most suitable reference frames to out-paint current chunk. Specifically, we calculate the registration maps and scores for all query-candidate pairs, and then input them into a VLM, Qwen3.5-VL-27B [ 32 ] . The VLM is requested to understand the whole video, speculate the missing contents ( i.e. , contents to out-paint like the two women and the indoor background), analyze which candidate frames may contain such missing contents, finally select the most suitable reference frames.
DAVIS [ 30 ]
YouTube-VOS [ 43 ]
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
Dehan [ 8 ]
0.627
17.96
0.233
363.1
0.720
18.25
0.228
149.7
SDM [ 33 ]
0.708
20.02
0.217
334.6
0.728
19.91
0.200
94.8
M3DDM [ 12 ]
0.708
20.26
0.203
300.0
0.731
20.20
0.185
66.6
Infinite-Canvas [ 7 ]
0.726
20.80
0.160
242.8
-
-
-
-
OutDreamer [ 50 ]
0.757
20.30
0.174
268.9
0.764
20.21
0.183
56.0
Table 1: Results of video out-painting performance on the DAVIS [ 30 ] and YouTube-VOS [ 43 ] benchmarks across multiple metrics. The best and second best results are bold and underlined . Note that the two benchmarks mainly contain short clips where most samples only have one chunk ( i.e. , ≤ 3 seconds for DAVIS and 3 to 5 seconds for YouTube-VOS). Though specifically designed for long videos with multiple chunks, we still achieve comparable results on such short clips.
Figure 5: Qualitative comparison. Our LongSCOP could achieve stable video out-painting results and maintain the semantic consistency over a long time horizon. The region circled by the yellow box indicates the original input region. Zoom in for better views.
Table 7
Stable Gen.
Refer. Retrieval
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
✗
✗
0.725
18.59
0.238
225.5
✓
✗
0.751
19.47
0.208
208.6
✓
✓
0.749
19.87
0.207
197.3
Table 4: Ablation of stable generation and reference retrieval on LVO-Bench.
Method
Hit@1 ↑
Hit@3 ↑
Recall@3 ↑
Recall@5 ↑
GT rank@1 ↓
GT rank@3 ↓
nDCG@3 ↑
Variant (1)
0.1394
0.6653
0.3320
0.5252
6.7849
7.2363
0.6066
Variant (2)
0.1163
0.5814
0.3036
0.4774
7.5349
7.4496
0.5612
Variant (3)
0.1395
0.6162
0.3023
0.4942
7.0076
7.4199
0.5815
Variant (4)
0.1240
0.5890
0.3056
0.4941
6.8992
7.4670
0.5590
Variant (5)
0.1434
0.6550
0.3198
0.5110
6.9729
7.4477
0.5931
Variant (6)
0.1589
0.6279
0.3114
0.4994
6.9884
7.4625
0.5768
Table 5: Quantitative results of various implementation variants of reference frame retrieval methods.
Table 10
Method
Time (min) ↓
Overhead
VACE
13.5
–
LongSCOP (9B)
16.5
+22%
LongSCOP (27B)
20.1
+49%
Table 8: Average inference time per 15-second video (Mast3R retrieval).