Organizations: Nanjing University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · The Chinese University of Hong Kong · The University of Hong Kong · Shenzhen University of Advanced Technology · INSAIT, Sofia University “St. Kliment Ohridski”
Video out-painting is an important problem, yet existing research focuses primarily on short clips, leaving long video out-painting largely underexplored. Extending video out-painting to long videos poses substantial challenges. A natural solution is iterative out-painting, i.e., generating videos chunk by chunk, but this introduces two major issues: generation instability or even failure caused by inter-chunk inconsistencies and error accumulation, and semantic drift over long temporal horizons. In this paper, we propose LongSCOP, short for Semantically Consistent long Video Out-Painting. First, we introduce a training recipe with three components for smooth and stable long-horizon generation: self-forcing training, latent calibration, and future references. Beyond generation stability, we recognize that the rich temporal context of long videos provides valuable references for maintaining semantic consistency. We therefore develop a VLM-based agent that automatically selects the most informative reference frames from the entire video to maintain consistent subjects, scenes, and backgrounds throughout long-horizon generation. We also construct a high-quality training dataset of 62K video clips. To facilitate research on this underexplored problem, we introduce LVO-Bench, a benchmark curated by human experts to assess long-horizon stability and semantic consistency. Extensive experiments demonstrate that our method outperforms existing approaches in both generation stability and semantic coherence.
Figures & tables
Figure 1: Illustration of the two main challenges in long video out-painting: (a) unstable generation including abrupt transitions, brightness flickering, and even complete generation collapse caused by accumulated errors, and (b) semantic drift like changed scene layouts, unstable object identities, or inconsistent subject appearances over long time. Both cases are generated by VACE model [ 22 ] .
Figure 2: Illustration of self-forcing training. During training, we use the model-predicted frames as reference frames, which could reduce the gap between training and inference.
Figure 3: Illustration of the importance of suitable reference frame. Adding only one reference frame could significantly improve the out-painting performance of the whole video by nearly 70 FVD points and more than 1.1 dB PSNR on average.
Figure 4: Illustration of key reference frame retrieval framework. Given query frames from current query chunk to be out-painted and candidate frames from both history chunk and future chunk, we aim to find the most suitable reference frames to out-paint current chunk. Specifically, we calculate the registration maps and scores for all query-candidate pairs, and then input them into a VLM, Qwen3.5-VL-27B [ 32 ] . The VLM is requested to understand the whole video, speculate the missing contents ( i.e. , contents to out-paint like the two women and the indoor background), analyze which candidate frames may contain such missing contents, finally select the most suitable reference frames.
DAVIS [ 30 ]
YouTube-VOS [ 43 ]
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
Dehan [ 8 ]
0.627
17.96
0.233
363.1
0.720
18.25
0.228
149.7
SDM [ 33 ]
0.708
20.02
0.217
334.6
0.728
19.91
0.200
94.8
M3DDM [ 12 ]
0.708
20.26
0.203
300.0
0.731
20.20
0.185
66.6
Infinite-Canvas [ 7 ]
0.726
20.80
0.160
242.8
-
-
-
-
OutDreamer [ 50 ]
0.757
20.30
0.174
268.9
0.764
20.21
0.183
56.0
Table 1: Results of video out-painting performance on the DAVIS [ 30 ] and YouTube-VOS [ 43 ] benchmarks across multiple metrics. The best and second best results are bold and underlined . Note that the two benchmarks mainly contain short clips where most samples only have one chunk ( i.e. , ≤ 3 seconds for DAVIS and 3 to 5 seconds for YouTube-VOS). Though specifically designed for long videos with multiple chunks, we still achieve comparable results on such short clips.
Figure 5: Qualitative comparison. Our LongSCOP could achieve stable video out-painting results and maintain the semantic consistency over a long time horizon. The region circled by the yellow box indicates the original input region. Zoom in for better views.
Table 7
Stable Gen.
Refer. Retrieval
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
✗
✗
0.725
18.59
0.238
225.5
✓
✗
0.751
19.47
0.208
208.6
✓
✓
0.749
19.87
0.207
197.3
Table 4: Ablation of stable generation and reference retrieval on LVO-Bench.
Method
Hit@1 ↑
Hit@3 ↑
Recall@3 ↑
Recall@5 ↑
GT rank@1 ↓
GT rank@3 ↓
nDCG@3 ↑
Variant (1)
0.1394
0.6653
0.3320
0.5252
6.7849
7.2363
0.6066
Variant (2)
0.1163
0.5814
0.3036
0.4774
7.5349
7.4496
0.5612
Variant (3)
0.1395
0.6162
0.3023
0.4942
7.0076
7.4199
0.5815
Variant (4)
0.1240
0.5890
0.3056
0.4941
6.8992
7.4670
0.5590
Variant (5)
0.1434
0.6550
0.3198
0.5110
6.9729
7.4477
0.5931
Variant (6)
0.1589
0.6279
0.3114
0.4994
6.9884
7.4625
0.5768
Table 5: Quantitative results of various implementation variants of reference frame retrieval methods.
Table 10
Method
Time (min) ↓
Overhead
VACE
13.5
–
LongSCOP (9B)
16.5
+22%
LongSCOP (27B)
20.1
+49%
Table 8: Average inference time per 15-second video (Mast3R retrieval).
Video outpainting generates plausible visual content beyond the original spatial extent of a video, playing a key role in adapting videos to diverse display formats. To support such use cases, it must enable large spatial extrapolation over long sequences. However, most existing methods address only one of these challenges or lack explicit mechanisms for ensuring global spatio-temporal consistency, leading to notable limitations. In this paper, we propose HL-OutPaint, a high-resolution video outpainting framework for long sequences. Our approach follows a coarse-to-fine strategy with a two-stage pipeline. We first construct Global Coarse Guidance (GCG), a low-resolution representation that captures global structure and dominant motion across the video. Unlike naive downsampling, GCG is built via a novel global-local frame swapping mechanism that couples sparse global keyframes with local temporal windows and exchanges information during sampling. This enables GCG to encode both long-term structural consistency and short-term temporal dynamics in a unified representation. Guided by this representation, HL-OutPaint then performs high-resolution outpainting to generate spatially detailed and temporally consistent content. By separating global structure modeling from fine-grained synthesis, our framework achieves stable, coherent generation for large spatial expansion and long video sequences. Extensive experiments show that HL-OutPaint outperforms existing methods in challenging scenarios involving wide spatial extrapolation and long video sequences.
Jeongeun Park, Janghyeok Han, Geonung Kim +4
POSTECH, Republic of Korea · Visual Display Business, Samsung Electronics, Republic of Korea
Synthesizing consistent and coherent long video remains a fundamental challenge. Existing methods suffer from semantic drift and narrative collapse over long horizons. We present A2RD, an Agentic Auto-Regressive Diffusion architecture that decouples creative synthesis from consistency enforcement. A2RD formulates long video synthesis as a closed-loop process that synthesizes and self-improves video segment-by-segment through a Retrieve--Synthesize--Refine--Update cycle. It comprises three core components: (i) Multimodal Video Memory that tracks video progression across modalities; (ii) Adaptive Segment Generation that switches among generation modes for natural progression and visual consistency; and (iii) Hierarchical Test-Time Self-Improvement that self-improves each segment at frame and video levels to prevent error propagation. We further introduce LVBench-C, a challenging benchmark with non-linear entity and environment transitions to stress-test long-horizon consistency. Across public and LVBench-C benchmarks spanning one- to ten-minute videos, A2RD outperforms state-of-the-art baselines by up to 30% in consistency and 20% in narrative coherence. Human evaluations corroborate these gains while also highlighting notable improvements in motion and transition smoothness.
Do Xuan Long, Yale Song, Min-Yen Kan +2
Google Cloud AI Research · National University of Singapore
Long-horizon video generation requires retaining an ever-expanding history, but conditioning on its growing token sequence is costly. Existing methods restrict or compress historical context, potentially losing information needed for future generation. This work proposes ReCam, which groups local spatiotemporal blocks of historical latents into vectors and projects them directly into the high-dimensional hidden space of a Diffusion Transformer. The model learns to use the resulting memory tokens through native self-attention, reducing token count while retaining block-level access to the full history. To mitigate temporal drift during autoregressive rollout, we further introduce dual-pointer denoising with separate noise schedules for recent context and target frames. Among the evaluated methods, ReCam achieves the highest VBench-Long overall score of 80.1 and competitive static-scene revisitation performance on R2M-Bench. It also leads on all four consistency metrics for off-screen subject reappearance, including clothing consistency of 4.09 versus 3.03 for the best baseline.
Yukuo Ma, Cong Liu, Junke Wang +4
Fudan University · Institute of Artificial Intelligence (TeleAI), China Telecom