MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens

Youngrae Kim; Qixin Hu; C. -C. Jay Kuo; Peter A. Beerel

cs.CVMar 12, 2026

MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens

Authors: Youngrae Kim, Qixin Hu, C. -C. Jay Kuo, Peter A. Beerel

Organizations: University of Southern California

Abstract

Autoregressive diffusion enables real-time frame streaming, yet existing sliding-window caches discard past context, causing fidelity degradation, identity drift, and motion stagnation over long horizons. Current approaches preserve a fixed set of early tokens as attention sinks, but this static anchor cannot reflect the evolving content of a growing video. We introduce MemRoPE, a training-free framework with two co-designed components. Memory Tokens continuously compress all past keys into dual long-term and short-term streams via exponential moving averages, maintaining both global identity and recent dynamics within a fixed-size cache. Online RoPE Indexing caches unrotated keys and applies positional embeddings dynamically at attention time, ensuring the aggregation is free of conflicting positional phases. These two mechanisms are mutually enabling: positional decoupling makes temporal aggregation well-defined, while aggregation makes fixed-size caching viable for unbounded generation. Extensive experiments validate that MemRoPE outperforms existing methods in temporal coherence, visual fidelity, and subject consistency across minute- to hour-scale generation.

Figures & tables

Figure 2: KV cache structures for long video generation. (a) FIFO eviction [ 19 , 50 , 44 , 47 ] maintains an initial sink frame while discarding the oldest remaining frames when the cache is full, losing distant context. (b) Deep Forcing [ 48 ] dedicates over half the cache to static sink tokens and selects a small number of compressed tokens via attention-based importance scoring, which often causes temporal instability in the generated sequence (see Fig. 3 ). (c) MemRoPE preserves a sink frame and manages distinct long- and short-term memories ( Sec. 4.1 ). By storing all keys without RoPE , it prevents positional interference from corrupting the stored features, thereby enabling stable memory aggregation ( Sec. 4.2 ).

Panel of Figure 3 — Figure 3: Failure mode of Participative Compression. (a) Participative Compression (PC), proposed in Deep Forcing [ 48 ] , rapidly converges to retaining the same long-persisted tokens in the compressed frames, discarding most newly arriving tokens. (b) The few newly admitted tokens carry high attention scores, so each rare cache update exerts a disproportionately strong influence on generation. (c, d) This causes frame-to-frame instability: consecutive SSIM drops and LPIPS spikes indicate abrupt visual shifts whenever the cache content changes. Our EMA memory evolves continuously, maintaining smooth transitions.

Figure 4: Overview of MemRoPE. (a) At each autoregressive step, the three-tier KV cache (sink, memory, and recent tokens) is concatenated with the current noisy chunk and fed into the DiT. When the local window is full, the oldest frames are absorbed into the long- and short-term memory tokens via dual EMA updates. (b) All cached keys are stored without RoPE, enabling temporal aggregation for memory tokens. At attention time, contiguous block-relative indices are assigned to the full sequence, and RoPE is applied on the fly, ensuring that indices never exceed the training range.

Method	Aesthetic Quality $\uparrow$	Background Consistency $\uparrow$	Imaging Quality $\uparrow$	Motion Smoothness $\uparrow$	Subject Consistency $\uparrow$	Temporal Flickering $\uparrow$	Average $\uparrow$	$\Delta$
120 seconds
Results on Self-Forcing [ 19 ]
Self-Forcing [ 19 ]	55.11	95.41	65.95	97.17	95.35	96.53	84.25	–
Deep Forcing [ 48 ]	56.86	94.58	65.02	97.07	94.17	95.33	83.84	$-$ 0.41
$\infty$ -RoPE [ 47 ]	52.82	95.66	61.07	98.48	96.30	97.50	83.64	$-$ 0.61
MemRoPE (Ours)	56.77	95.54	68.44	97.90	96.37	96.33	85.23	$+$ 0.98

Table 1: Quantitative comparison on long video generation. We report VBench-Long [ 20 ] metrics. Within each base model group, best results are bold with darker background and second best are with lighter background.

\Delta

denotes the average improvement over the respective base model.

Figure 5: Qualitative comparison on 2-minute video generation. MemRoPE maintains subject identity and background consistency throughout, whereas baselines exhibit progressive degradation, including structural collapse and color corruption.

Method	Aesthetic Quality $\uparrow$	Background Consistency $\uparrow$	Imaging Quality $\uparrow$	Motion Smoothness $\uparrow$	Subject Consistency $\uparrow$	Temporal Flickering $\uparrow$	Average $\uparrow$
480 seconds
Results on Self-Forcing [ 19 ]
$\infty$ -RoPE [ 47 ]	48.35	95.45	56.00	98.59	96.46	97.71	82.09
MemRoPE (Ours)	56.12	95.65	68.23	97.96	96.87	96.42	85.21
Results on LongLive [ 44 ]
$\infty$ -RoPE [ 47 ]	56.77	96.25	61.32	99.05	97.16	97.95	84.75

Table 2: Ultra-long video generation. We report VBench-Long [ 20 ] metrics. Within each base model group, best results are bold with darker background.

Baseline	Color Cons.	Subject Cons.	Bg. Cons.	Text Align.	Motion Smo.	Overall Pref.
Self-Forcing [ 19 ]	93.0	93.0	97.4	91.6	95.7	98.3
Rolling Forcing [ 27 ]	90.4	77.4	79.1	80.7	88.7	82.6
Results on LongLive [ 44 ]
LongLive [ 44 ]	81.6	70.2	83.3	66.1	71.9	71.1
Deep Forcing [ 48 ]	79.0	75.2	81.9	70.5	72.4	72.6
$\infty$ -RoPE [ 47 ]	81.3	76.4	77.6	74.5	73.8	70.1

Table 3: User study (% favoring Ours).

Figure 6: Spatial attention distribution between the dual memory streams. Yellow indicates higher attention to long-term memory. Red indicates higher attention to short-term memory. Temporally consistent regions attend more to long-term memory, while rapidly changing regions attend more to short-term memory.

Method	Stability
Self-Forcing [ 19 ]	1.55
Rolling Forcing [ 27 ]	3.40
Results on LongLive [ 44 ]
LongLive [ 44 ]	4.10
Deep Forcing [ 48 ]	3.90
$\infty$ -RoPE [ 47 ]	4.05

Table 4: Visual stability (VLM).

Long-term Memory ( $\mu_{L}$ )	Short-term Memory ( $\mu_{S}$ )	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	Subject Consistency $\uparrow$	Avg $\uparrow$
Results on LongLive [ 44 ]
		58.09	65.29	97.40	85.48
✓		58.69	66.31	97.62	85.84
	✓	58.73	66.63	97.61	85.95
✓	✓	58.76	67.09	97.61	85.97

Table 5: Ablation on memory components. While using only long-term memory (

\mu_{L}

) achieves the highest Subject Consistency, combining both memories yields the highest scores in overall average.

Figure 7: Sensitivity to EMA decay rates. All metrics remain stable across 12 combinations of

\alpha_{L}

and

\alpha_{S}

, with the average score varying by less than 0.7.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Aesthetic Quality $\uparrow$	Background Consistency $\uparrow$	Imaging Quality $\uparrow$	Motion Smoothness $\uparrow$	Subject Consistency $\uparrow$	Temporal Flickering $\uparrow$	Average $\uparrow$	$\Delta$
30 seconds
CausVid [ 50 ]	57.25	96.77	65.16	98.08	97.87	96.86	85.33	–
Rolling Forcing [ 27 ]	58.45	95.77	69.35	98.23	96.78	97.13	85.95	–
Results on Self-Forcing [ 19 ]
Self-Forcing [ 19 ]	58.28	95.14	66.92	98.02	95.50	96.87	85.12	–
Deep Forcing [ 48 ]	57.83	94.86	66.29	97.53	94.98	95.71	84.53	$-$ 0.59

Appendix

Table 6: Quantitative comparison on short video generation (30 and 60 seconds). We report VBench-Long [ 20 ] metrics. Within each base model group, best results are bold with darker background and second best are with lighter background.

\Delta

denotes the average improvement over the respective base model. ∗ Results from [ 47 ] .

Method	Aesthetic Quality $\uparrow$	Subject Consistency $\uparrow$	Imaging Quality $\uparrow$	Motion Smoothness $\uparrow$	Background Consistency $\uparrow$	Temporal Flickering $\uparrow$	Average $\uparrow$	$\Delta$
Results on Self-Forcing [ 19 ]
Self-Forcing [ 19 ]	56.96	95.08	66.36	97.35	95.15	96.66	84.57	–
Aggregation w/ RoPE	56.55	96.44	68.25	98.19	95.60	96.66	85.28	$+$ 0.71
MemRoPE (Ours)	57.77	95.58	68.54	97.93	96.29	96.35	85.41	$+$ 0.84
Results on LongLive [ 44 ]
LongLive [ 44 ]	57.93	96.10	67.41	98.57	97.10	97.15	85.71	–

Appendix

Table 7: Ablation on position-free caching. Comparison between MemRoPE (position-free EMA) and EMA with RoPE-rotated keys at 60 seconds.

\Delta

denotes the average improvement over the respective base model.

Figure 8: VBench-Long average score vs. video duration on Self-Forcing [ 19 ] . MemRoPE maintains higher scores across all durations, with the gap widening as videos grow longer.

Figure 9: Qualitative comparison at 2-minute generation on Self-Forcing [ 19 ] and LongLive [ 44 ] . MemRoPE preserves subject identity and visual fidelity more consistently than existing methods across both prompts.

Figure 10: 1-hour generation comparison on LongLive [ 44 ] . Top: $\infty$ -RoPE [ 47 ] . Bottom: MemRoPE (Ours). Frames sampled at 12-minute intervals. MemRoPE maintains more consistent subject appearance and background color tone over the full hour.

Figure 11: Qualitative memory component ablation on LongLive [ 44 ] . From top to bottom: no memory, long-term memory only, short-term memory only, and both (ours). Only the full dual-stream configuration preserves background structures and subject appearance throughout.

Method	Video FPS
Self-Forcing [ 19 ] ( $C=21$ )	3.631
+ MemRoPE ( $C=12$ )	5.115
LongLive [ 44 ] ( $C=12$ )	4.409
+ MemRoPE ( $C=12$ )	4.376

Appendix

Table 8: Inference speed. Video FPS measured on a single A6000 GPU.

Figure 12: Example VLM evaluation. Gemini 3.1 Pro [ 14 ] rates each method on a 5-point scale with detailed reasoning.

Figure 13: User study interface. (a) Welcome page with instructions. (b) Evaluation page showing side-by-side video comparison with per-dimension preference questions.

Long-Horizon Video Generation

Long-horizon video generation creates coherent, controllable videos over extended durations despite limited context and accumulating errors.

See all 164 papers

Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation
Jun 3, 2026Yuxuan Bian, Zeyue Xue, Songchun Zhang +9Long Video Generation Memory-Augmented Video Generation
1The Chinese University of Hong Kong · 2Joy Future Academy, JD · 3The Hong Kong University of Science and Technology +4
Memory Forcing: Attendable Mid-Horizon History for Streaming Video Generation
Oct 8, 2026Jiaming Zhang, Xinyu Wang, Huafeng Shi +2Streaming Video Generation Memory-Augmented Video Generation
State Key Laboratory for Novel Software Technology, Nanjing University · Tsinghua University · Kling Team, Kuaishou Technology +1
Anchor Forcing: Anchor Memory and Tri-Region RoPE for Interactive Streaming Video Diffusion
Mar 12, 2026Yang Yang, Tianyi Zhang, Wei Huang +6Video Generation Streaming Video Generation
Zhejiang University · vivo BlueImage Lab · University of Hong Kong

Jun 3, 2026J/K move · Enter open · S save

Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation

Yuxuan Bian, Zeyue Xue, Songchun Zhang, Shiyi Zhang+8

1The Chinese University of Hong Kong · 2Joy Future Academy, JD · 3The Hong Kong University of Science and Technology+4

Long Video Generation Memory-Augmented Video Generation Streaming Video Generation Autoregressive Video Generation Efficient Video Generation

Open paper AskPDF arXiv

We present Echo Infinity, an autoregressive (AR) framework towards real-time infinite video generation that employs a learnable evolving memory to dynamically filter, abstract, and compress any-length history at constant cost. Existing methods mainly curate memory with predefined KV-cache schedules, fixed-ratio heuristic compression, or inference-time RoPE adaptation. These designs inevitably lose historical information and amplify compounding errors due to their limited cache window and ignorance of autoregressive generation noise. Inspired by human memory consolidation, Echo-Infinity replaces handcrafted memory curation with learnable Memory Query, which are updated by attention and a gating mechanism when past frames are evicted from the local window. The queries are optimized end-to-end with the video diffusion transformers (DiTs), forming an evolving memory that supports arbitrary compression ratios with constant computation independent of video length. They also act as a generalizable generation prior, improving quality even when only the optimized initial state is used. We further introduce Unified Relative RoPE Recipe, which anchors the sink frames to start from id 0 and lets the newest frame id grow at most to the DiTs' pretrained maximum temporal RoPE id throughout training and inference, freeing the model from the finite RoPE constraint and closing the train-test RoPE extrapolation gap. In long and short video generation, Echo-Infinity achieves state-of-the-art performance, and, to our knowledge, demonstrates promising 24-hour (>1.3 M frames) real-time rollouts for the first time, suggesting a practical path toward infinite video generation.

Abstract

Long-Horizon Video Generation

Explore similar work

Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation

Memory Forcing: Attendable Mid-Horizon History for Streaming Video Generation

Anchor Forcing: Anchor Memory and Tri-Region RoPE for Interactive Streaming Video Diffusion