Attn-Forcing: Learning Full-Context Attention Proxies for Streaming Video Generation

Junyao Gao; Sibo Liu; Jiaxing Li; Yanan Sun; Hanxin Zhu; Weidong Zhang; Jun Zhang; Cairong Zhao

cs.CVOct 21, 2026

Attn-Forcing: Learning Full-Context Attention Proxies for Streaming Video Generation

Authors: Junyao Gao, Sibo Liu, Jiaxing Li, Yanan Sun, Hanxin Zhu, Weidong Zhang, Jun Zhang, Cairong Zhao

Organizations: Tongji University · Tencent · Nanyang Technological University · Hong Kong University of Science and Technology

Abstract

Increasing the attention context length significantly improves streaming video generation quality, while incurring prohibitive memory and computational costs as the context grows. Existing methods attempt to capture this contextual information through attention selection or compression, but remain limited by bounded representations and lack explicit attention-level supervision to retain historical contributions to current queries. To address this, we introduce Attn-Forcing, a novel framework that preserves full-context attention contributions for streaming video generation under bounded memory and computation. We achieve this by introducing attention proxies that continuously approximate the attention responses induced by the historical context. Specifically, we implement the proxies as learnable visual latents and employ a Proxy Update Module to incorporate visual context from historical latents. Moreover, we propose an Attention Matching Loss to explicitly constrain the attention behavior of the learned proxies using full-context attention supervision. Extensive qualitative and quantitative experiments demonstrate that Attn-Forcing consistently improves streaming video generation quality and achieves state-of-the-art performance over existing baseline methods.

Figures & tables

Figure 1: VBench-Long score and peak GPU memory of Self-Forcing ( Huang et al., 2026 ) during 30s video inference with different local attention sizes. A larger attention window generally improves generation quality, but at the cost of increased peak GPU memory.

Figure 2: The framework of Attn-Forcing. During training, learnable proxy latents are continuously updated using visual latents evicted from the local attention window through a Proxy Update Module. Attention Matching Loss further aligns the proxy-induced attention responses with historical attention responses approximated using online accumulated statistics.

Diffusion models
Model	#Params	Resolution	Evaluation Scores on 5s $\uparrow$			Evaluation Scores on 30s $\uparrow$
Model	#Params	Resolution	Total	Quality	Semantic	Total	Quality	Semantic
LTX-Video	1.9B	$768{\times}512$	79.31	81.51	70.53	–	–	–
Wan-2.1	1.3B	$832{\times}480$	82.84	83.64	79.67	–	–	–
Autoregressive models
Self-Forcing	1.3B	$832{\times}480$	83.76	84.43	81.06	81.92	82.98	77.68

Table 1: Quantitative comparison on 5s short videos and 30s long videos generated from the official VBench prompts. Best results are shown in bold , and second-best results are underlined .

Figure 3: Qualitative comparison on 60s videos generated using MovieGen prompts. Attn-Forcing achieves stronger generation quality and motion dynamics.

Model	Evaluation Scores on 60s $\uparrow$
Model	Total	Subject	Background	Smoothness	Dynamic	Aesthetic	Imaging Quality
Self-Forcing	78.29	97.07	96.17	98.51	34.53	54.28	67.32
Causal-Forcing	79.05	95.78	95.80	97.36	61.20	50.46	64.78
LongLive	79.26	97.93	96.60	98.85	35.55	55.99	68.81
Attn-Forcing	82.37	97.11	95.84	98.56	72.14	56.03	69.42

Table 2: Quantitative comparison on 60s long videos generated from the official MovieGen prompts. Best results are shown in bold , and second-best results are underlined .

Figure 4: ( Left ) Qualitative comparison of ablations on our proposed modules. ( Right ) The orange and blue curves show the cosine similarity between proxy and exact historical attention with and without $\mathcal{L}_{\mathrm{AM}}$ , respectively. The green dashed curve shows the similarity between exact historical attention and its approximation $\widehat{\mathbf{A}}_{\mathrm{hist}}$ .

Improvement
Model	Evaluation Scores on 60s $\uparrow$
Model	Total	Background	Smooth	Dynamic	Quality
Baseline	80.03	96.25	98.57	46.30	68.70
+ proxies	80.16	96.13	98.35	46.82	69.41
+ $\mathcal{L}_{\mathrm{AM}}$	82.37	95.84	98.56	72.14	69.42
Impact of $M$

Table 3: Ablation studies on key components using 60s videos generated from the 128 MovieGen prompts. The best results for the “Improvement” module are indicated in bold .

Model	Evaluation Scores on 60s $\uparrow$
Model	Total	Background	Smooth	Dynamic	Quality
LongLive	79.26	96.60	98.85	35.55	68.81
w/ Attn. Size = 120	75.01	97.08	99.37	11.69	57.27
Attn-Forcing	82.37	95.84	98.56	72.14	69.42

Table 4: Quantitative comparison of LongLive, LongLive with a local attention size of 120, and Attn-Forcing. The best results are indicated in bold .

Figure 5: Qualitative results of directly using $\widehat{\mathbf{A}}_{\mathrm{hist}}$ for generation. Direct use of the approximation results in increasing blur over time.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Additional qualitative comparison on 60s videos generated using MovieGen prompts. Attn-Forcing achieves stronger generation quality and motion dynamics.

Streaming Video Generation

Streaming video generation creates video incrementally as frames or chunks arrive, enabling low-latency interactive synthesis and editing.

See all 99 papers

Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation
Apr 11, 2026Ruibin Li, Tao Yang, Fangzhou Ai +4Long-Horizon Video Generation Streaming Video Generation
The Hong Kong Polytechnic University · ByteDance · City University of Hong Kong
Recency Forcing: Bridging the Long-Horizon Gap in Autoregressive Video Generation
Sep 17, 2026Tri Cao, Hung Nguyen, Phong Nguyen +1Long-Horizon Video Generation KV-Cache Management
Qualcomm AI Research
Stream Forcing: Constructing Unified Training Trajectory for Robust Streaming Video Generation
Aug 11, 2026Yueting Zhu, Yuehao Song, Kaicheng Zhang +5Video Diffusion Models Long-Horizon Video Generation
Huazhong University of Science & Technology · Anyverse Dynamics · Horizon Robotics

Apr 11, 2026J/K move · Enter open · S save

Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation

Ruibin Li, Tao Yang, Fangzhou Ai, Tianhe Wu+3

The Hong Kong Polytechnic University · ByteDance · City University of Hong Kong

Long-Horizon Video Generation Streaming Video Generation Autoregressive Video Generation Hybrid Attention Diffusion Model Distillation

Open paper AskPDF arXiv

Streaming video generation (SVG) distills a pretrained bidirectional video diffusion model into an autoregressive model equipped with sliding window attention (SWA). However, SWA inevitably loses distant history during long video generation, and its computational overhead remains a critical challenge to real-time deployment. In this work, we propose Hybrid Forcing, which jointly optimizes temporal information retention and computational efficiency through a hybrid attention design. First, we introduce lightweight linear temporal attention to preserve long-range dependencies beyond the sliding window. In particular, we maintain a compact key-value state to incrementally absorb evicted tokens, retaining temporal context with negligible memory and computational overhead. Second, we incorporate block-sparse attention into the local sliding window to reduce redundant computation within short-range modeling, reallocating computational capacity toward more critical dependencies. Finally, we introduce a decoupled distillation strategy tailored to the hybrid attention design. A few-step initial distillation is performed under dense attention, then the distillation of our proposed linear temporal and block-sparse attention is activated for streaming modeling, ensuring stable optimization. Extensive experiments on both short- and long-form video generation benchmarks demonstrate that Hybrid Forcing consistently achieves state-of-the-art performance. Notably, our model achieves real-time, unbounded 832x480 video generation at 29.5 FPS on a single NVIDIA H100 GPU without quantization or model compression. The source code and trained models are available at https://github.com/leeruibin/hybrid-forcing.

Abstract

Streaming Video Generation

Explore similar work

Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation

Recency Forcing: Bridging the Long-Horizon Gap in Autoregressive Video Generation

Stream Forcing: Constructing Unified Training Trajectory for Robust Streaming Video Generation