LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generation

Byoungwoo Park; Jaemoo Choi; Juho Lee; Yongxin Chen

cs.CVSep 29, 2026

LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generation

Authors: Byoungwoo Park, Jaemoo Choi, Juho Lee, Yongxin Chen

Organizations: KAIST · Georgia Tech

Abstract

World models, game simulators, and long-take video creation require coherent scene evolution and sustained dynamics over extended durations. Autoregressive (AR) video diffusion provides a natural framework for long-horizon generation, yet extended rollouts often become near-static or lose visual quality. We hypothesize that these failures reflect the limited guidance provided by short-video supervision on how ongoing scene dynamics develops over longer durations. This motivates us to introduce LongTake, a two-stage training pipeline built around Long-Horizon Teacher Forcing (TF) on curated real long videos. Long-Horizon TF trains the AR model to predict later frames conditioned on long ground-truth video prefixes, extending direct supervision beyond the short training horizon. This supervision is designed to help the model sustain dynamics and preserve visual quality during long-horizon generation. Our central finding is that this training stage strengthens direct initialization for distribution matching distillation (DMD) under student self-rollout, without the intermediate few-step distillation stage used in standard pipelines. Under the same five-second DMD training setup, our initialization yields substantially higher dynamic degree than short horizon TF initialization on 30-second rollouts at comparable aesthetic quality, and surpasses the evaluated baselines in both measures. Hybrid DMD further reuses this teacher to extend supervision to later frames of the self-rollout while retaining bidirectional joint supervision over the initial window. On long-horizon self-rollouts, LongTake lies on the Pareto front of dynamic degree and aesthetic quality, and Hybrid DMD attains the highest dynamic degree among evaluated methods at both 30s and 60s.

Figures & tables

Figure 1: LongTake learns long-horizon generation through a two-stage training pipeline. (Left) Stage 1: Long-Horizon TF learns from real long videos and provides a strong direct initialization for self-rollout DMD. (Right) Stage 2: Self-rollout DMD uses this initialization. Hybrid DMD further adds conditional supervision of later frames from the AR teacher, while retaining joint supervision from the frozen bidirectional teacher over the initial window of the student rollout.

Figure 2: Three-stage pipeline for few-step AR video diffusion ( Zheng et al., 2026 ) .

Figure 3: Teacher and student gaps in KV-cache states.

Figure 4: Teacher gap with a bounded KV-cache. With sink retention, the temporal separation between sink frames and recent frames increases as the video prefix grows. Short-horizon TF directly supervises prediction only within ${\color[rgb]{0.1875,0.6289,0.4336}{N}}$ , while generating later frames toward ${\color[rgb]{0.4688,0.4336,0.6953}{M}}$ relies on generalization.

Figure 5: Long-Horizon TF enables effective two-stage training. All models use ${\color[rgb]{0.1875,0.6289,0.4336}{5}}$ s self-rollout DMD and are evaluated on ${\color[rgb]{0.4688,0.4336,0.6953}{30}}$ s rollouts. Long TF initialization improves long-horizon dynamics over Short TF while maintaining comparable visual quality, showing that stronger TF initialization supports effective long-horizon generation without a separate CD stage.

Figure 6: Two-stage training

Figure 7: Long-horizon generation with sustained dynamics. Qualitative comparisons over 30s (top) and 60s (bottom). Baseline rollouts exhibit appearance degradation or limited scene progression. LongTake with Hybrid DMD sustains scene evolution with coherent appearance in later frames.

	30s						60s
Method	Subj. $\uparrow$	Dyn. $\uparrow$	Aesth. $\uparrow$	Quality $\uparrow$	$\Delta$ IQ $\downarrow$	VLM $\uparrow$	Subj. $\uparrow$	Dyn. $\uparrow$	Aesth. $\uparrow$	Quality $\uparrow$	$\Delta$ IQ $\downarrow$	VLM $\uparrow$
Short-video supervision
Self Forcing ( Huang et al., 2025a )	$97.43$	$40.42$	$60.66$	$82.34$	$5.82$	$3.13$	$97.27$	$44.21$	$56.58$	$81.89$	$6.65$	2.69
Causal Forcing ( Zhu et al., 2026 )	$96.10$	$69.38$	$57.41$	$82.19$	$7.34$	$2.45$	$95.96$	$59.86$	$51.83$	$80.93$	$12.97$	2.07
LongLive ( Yang et al., 2026 )	$98.03$	$41.25$	$63.21$	$83.25$	$2.96$	$3.90$	$98.40$	$37.04$	$62.23$	$82.98$	$2.69$	3.83
Reward Forcing ( Lu et al., 2026 )	$97.26$	$66.98$	$62.42$	$84.36$	$2.58$	$3.79$	$97.29$	$80.23$	$62.08$	$85.54$	$2.45$	3.75

Table 1: Evaluation on 30s and 60s AR rollouts. Quality aggregates seven video-quality dimensions with standard VBench normalization and weights.

\Delta

IQ and VLM assess visual stability. Best and second-best results are highlighted for each duration. LongTake uses Long-Horizon TF followed by 5s joint DMD. Full results appear in Table 5 .

\dagger

checkpoint from ( Zhu et al., 2026 ) .

Figure 8: Motion-quality trade-off at 30s. Gray line: Pareto front.

$\lambda$	Subj. $\uparrow$	Dyn. $\uparrow$	Aesth. $\uparrow$	Quality $\uparrow$
$0.0$	$96.83$	$90.73$	$62.51$	$85.79$
$0.1$	$96.78$	$90.00$	$62.64$	$85.83$
$0.2$	$96.32$	$96.30$	$62.17$	$85.97$
$0.3$	$96.38$	$95.57$	$61.49$	$85.85$
$0.4$	$96.39$	$95.42$	$61.91$	$85.61$

Table 2: Hybrid DMD with shared initialization.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Parallel cache construction for Long-Horizon TF. (Left) At each transformer layer, all known prefix chunks are projected together to

Q_{i}

and

C_{i}=(K_{i},V_{i})

. For each query, the cache operator constructs a conditioning KV-cache

\mathcal{C}_{i}

, which enters causal attention together with the current chunk’s Q/K/V. The resulting hidden states feed the next layer. (Right) Two alternative operators construct the KV-cache states of all queries in parallel. Operator (a) selects the initial sink and the most recent preceding chunks, and operator (b) additionally merges evicted entries into the sink through an associative weighted prefix scan. The example uses seven chunks with one sink and two recent slots. After prefill,

\mathcal{C}_{8}

at each layer is saved for the subsequent target-window TF pass. Parallelism is across chunks within each layer, while transformer layers remain sequential.

Figure 10: Hybrid DMD. A bidirectional teacher jointly supervises the initial window. The Long-Horizon AR teacher supervises later frames using the conditioning KV-cache updated during student self-rollout for each target chunk.

Setting	Long-Horizon TF	Joint DMD	Hybrid DMD
Iterations	3,000	1,200 / 800	400
Generator learning rate	$5\times 10^{-7}$	$2\times 10^{-6}$	$2\times 10^{-6}$
Fake-score learning rate	—	$4\times 10^{-7}$	$4\times 10^{-7}$
Effective batch size	64	8	8
Gradient accumulation	8	1	1
Optimizer	AdamW	AdamW	AdamW

Appendix

Table 3: Training configuration. LongTake uses 1,200 joint DMD iterations. The Hybrid DMD variant uses 800 joint DMD iterations followed by 400 Hybrid DMD iterations.

Source	Prefix length $s$	Probability
Causal Forcing clean GT	0	0.10
OpenVidHD	0–15	0.20
OpenVidHD	18–39	0.45
OpenVidHD	42–60	0.20
OpenVidHD	63–99	0.05

Appendix

Table 4: Prefix sampling for Long-Horizon TF. Prefix lengths are measured in latent frames and sampled in multiples of three. Probabilities refer to the complete training mixture.

Figure 11: Dataset duration.

Short-video supervision
	30s
Method	Subj. $\uparrow$	Backg. $\uparrow$	Motion $\uparrow$	Dyn. $\uparrow$	Aesth. $\uparrow$	Imaging $\uparrow$	Flicker $\uparrow$	Quality $\uparrow$
Self Forcing ( Huang et al., 2025a )	$97.43$	$96.50$	$98.55$	$40.42$	$60.66$	$69.27$	$98.86$	$82.34$
Causal Forcing ( Zhu et al., 2026 )	$96.10$	$95.76$	$96.99$	$69.38$	$57.41$	$66.66$	$98.24$	$82.19$
LongLive ( Yang et al., 2026 )	$98.03$	$97.01$	$98.78$	$41.25$	$63.21$	$69.50$	$99.06$	$83.25$
Reward Forcing ( Lu et al., 2026 )	$97.26$	$96.49$	$98.15$	$66.98$	$62.42$	$69.64$	$98.61$	$84.36$

Appendix

Table 5: Full evaluation on 30s and 60s autoregressive rollouts. Quality aggregates seven video-quality dimensions using the standard VBench normalization and weights. LongTake uses Long-Horizon TF and joint DMD on 5s rollouts. The best and second-best results are highlighted.

Dimension	30s	60s
Subject consistency	128	72
Background consistency	128	86
Motion smoothness	128	72
Dynamic degree	128	72
Aesthetic quality	128	93
Imaging quality	128	93

Appendix

Table 6: Evaluation prompt counts.

Score	Exposure criterion
0	Near-total whiteout or blackout makes the scene unreadable.
1	Widespread exposure failure severely reduces visibility.
2	Persistent highlight or shadow clipping causes substantial detail loss.
3	Exposure problems have limited spatial extent or duration.
4	Occasional local exposure flaws cause little visibility loss.
5	Balanced exposure preserves visibility without disruptive clipping or darkening.

Appendix

Table 7: VLM exposure-stability levels, summarized from the original MemRoPE rubric.

Figure 12: Effect of the conditional DMD weight $\lambda$ . Frames sampled from 30s rollouts show how $\lambda$ affects scene progression. With $\lambda=0.2$ , both the viewpoint and surrounding scene evolve while preserving a coherent appearance of the central subject throughout the rollout.

Prefix (video frames)	Teacher	VR $\uparrow$	IF $\uparrow$	DINO $\uparrow$	RGB L1 $\downarrow$
81 $(\approx 5s)$	Short TF	17.26	96	0.9904	0.0202
	Long TF	18.28	100	0.9919	0.0176
321 $(\approx 20s)$	Short TF	16.60	96	0.9930	0.0183
	Long TF	17.83	100	0.9941	0.0166

Appendix

Table 8: Teacher generation before DMD on held-out real long videos. Each teacher constructs its conditioning KV-cache from the same real video prefix and generates 81 additional video frames. Quality and prompt alignment are evaluated on generated frames, while boundary metrics assess appearance preservation at the immediate transition from observed to generated video frames.

Figure 13: Motion-quality trade-off at (Left) 30s and (Right) 60s. The gray line joins the Pareto-optimal methods. No other method has both higher dynamic degree and higher aesthetic quality. Both LongTake variants lie on front at both durations.

Figure 14: Motion vs. non-motion quality Non-motion quality is the VBench Quality score recomputed without dynamic degree with equal weights. Both LongTake variants lie on the front at both durations and are the only methods on it with dynamic degree above

90

Figure 15: 30s generation with the same prompt across evaluated methods.

Figure 16: 30s generation with the same prompt across evaluated methods.

Figure 17: 60s generation with the same prompt across evaluated methods.

Figure 18: Long-Horizon TF initialization under the same 5s joint DMD.

Figure 19: Long-Horizon TF initialization under the same 5s joint DMD.

Figure 20: Long-Horizon TF initialization under the same 5s joint DMD.

Figure 21: Hybrid DMD with shared initialization and equal distillation iterations.

Figure 22: Hybrid DMD with shared initialization and equal distillation iterations.

Explore similar work

Aug 6, 2026cs.CV

Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model

Recently, diffusion models have made great progress in video generation. However, most existing video diffusion models are trained with short videos, and degrade when extrapolated to long videos, struggling to maintain long-range temporal coherence while retaining diverse motions. To generate consistent, high-quality and dynamic long videos, we propose Diff-VF, a training-free, plug-and-play and model-agnostic framework that converts existing short-video diffusion backbones into long-video generators without modifying or fine-tuning the base model. Diff-VF couples three complementary strategies: Hybrid Noise Initialization (HNI) to constrain global semantics, Weighted Window Sampling (WWS) to remove inter-window discontinuities, and Temporal Extended Sampling (TES) to establish long-range dependencies with a timestep-varying fusion. We further extend Diff-VF to long-video enhancement via Skip Residual Guidance that balances fidelity and realism through timestep-dependent guidance. VBench-Long evaluation results show that Diff-VF achieves a more favorable balance between temporal coherence and motion diversity than base models and recent training-free long video generation baselines, including FreeNoise, FreeLong, and RIFLEx, while maintaining competitive frame-wise quality. Experiments on two base models demonstrate the applicability to video diffusion models with different spatial-temporal modeling strategies. Extensive ablations validate the contribution of each component and hyperparameters.

Haoning Yang, Xinyuan Chen, Yaohui Wang +1

Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Video Diffusion Models Interactive Video Generation Video Generation Diffusion Models Long-Range Coherence

Ask PDF

Jun 1, 2026cs.CV

LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation

Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory. Code is available at https://github.com/qixinhu11/LongLive-RAG.

Qixin Hu, Shuai Yang, Wei Huang +2

NVIDIA · USC · MIT

Long Video Generation Hievi-Rag Latent Variable

Ask PDF

Sep 29, 2026cs.CV

Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation

Autoregressive (AR) video diffusion enables low-latency, streamable video generation, but prediction errors often accumulate over long rollouts. Training the generator on its own rollouts exposes it to these imperfect histories. However, existing video-level distribution matching distillation (DMD) scores the whole rollout jointly. Because a chunk is evaluated together with its past and future, its correction can favor matching artifacts in the surrounding context merely to preserve temporal consistency. To provide a clearer visual-quality signal, we introduce Rollout-Marginal Distillation (RMD). RMD retains the generated history for AR prediction but scores each chunk independently against a chunk teacher, ensuring its quality correction is not compromised by an imperfect temporal context. To compensate for the lack of temporal context in independent chunk scoring, RMD subsequently applies video-level DMD to restore temporal coherence. Extensive experiments demonstrate that RMD maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines. Code and video results are available at https://cjeen.github.io/RMD

Chenjian Gao, Zhihao Hu, Jianqi Ma +3

MMLab, The Chinese University of Hong Kong · Tencent AIPD

Autoregressive Video Diffusion Models Autoregressive Video Generation Few-Step Distillation Autoregressive Model

Ask PDF