Autoregressive video generation relies on memory retrieval to model long-range dependencies. This capability becomes particularly critical for multi-shot generation, where models must reconcile substantial visual changes across shot transitions with persistent semantic continuity. Existing methods tend to perform memory retrieval at the frame level and share the same historical context across all query tokens, overlooking the heterogeneous information needs and leading to redundant retrieval and inefficient use of historical memory. To bridge this gap, we introduce CineMem, an attention-aligned fine-grained memory retrieval framework that combines online memory organization with query-adaptive retrieval for autoregressive multi-shot video generation. Specifically, guided by intrinsic memory-access patterns revealed through attention analysis, we propose an online-clustered KV cache as a memory organization strategy over all historical tokens, enabling semantic-aware memory grouping and more precise retrieval. In addition, we introduce a query-level retrieval mechanism that provides broader access to historical information under a fixed retrieval budget, thereby improving long-term memory utilization. Together, these designs enable the multi-shot video generator to preserve long-range consistency across shot transitions, particularly in revisitation scenarios, while approximating full-memory access under a constrained memory budget. To facilitate evaluation, we further introduce a suite of task-specific, VLM-assisted metrics for faithful inter-shot consistency assessment. Extensive experiments demonstrate the effectiveness of our framework.
Figures & tables
Figure 1: 15-second multi-shot videos autoregressively generated by CineMem . By combining clustered memory with fine-grained query-adaptive retrieval, CineMem maintains coherent semantic flow across shot transitions, particularly in revisitation scenarios. Project page: https://uknowsth.github.io/CineMem/
Figure 2: Overview of CineMem. Existing autoregressive video generation methods tend to perform memory retrieval at the frame level, which may result in redundant retrieval, inefficient memory utilization, and overlooked heterogeneous information needs across query tokens. CineMem combines an online-clustered KV cache with query-adaptive retrieval to enable a finer-grained memory retrieval strategy. As illustrated by the attention receptive fields, CineMem provides broader (2/3 vs. 12/12) access to historical information under the same retrieval budget.
Figure 3: Our memory retrieval framework. Through multi-stage initial adaptation, we convert a bidirectional model into an autoregressive multi-shot model with memory retrieval. During inference, queries access memory from the first chunk, adjacent chunk, and retrieved candidates via token-wise and head-wise retrieval. Previous adjacent chunks are progressively integrated into retrieval candidates through online KV-cache clustering with cluster balancing. During training, resampling ground truth frames provides memory inputs to alleviate error accumulation.
Figure 4: Intrinsic memory-selection patterns under the full-memory setting. Memory tokens exhibit semantically meaningful clustering structures, while different query-head pairs show distinct preferences over memory tokens. Although each query-head pair selects a subset of the most relevant tokens, their union covers nearly the entire memory. See Appendix A for more details.
Figure 5: Qualitative Results. We compare CineMem with representative causal and bidirectional baselines on 15s multi-shot video generation. Representative examples illustrate the qualitative improvements of our method.
Figure 6: Implementation of PTCS. PTCS assigns low scores when either unchanged semantic elements are not preserved or intended changes are not correctly realized.
Method
Retrieval Overhead
Attention
Total FLOPs
(TFLOPs)
(TFLOPs)
(TFLOPs)
Full history
0
932.56
932.56
Frame retrieval ( K=20 )
0.0453
649.97
650.02
Cluster retrieval ( K=28 , S=1024 )
15.0180
623.80
638.82
Table 1: Theoretical efficiency comparison. FLOPs are calculated for self-attention and retrieval-related operations. Collected K/V denotes the logical amount of retrieved memory elements.
Method
Params.
Video-level
Intra-shot
Inter-shot
Trans. Ctrl. ↑
Text-Video Sim. ↑
Aesthetic ↑
Subject Cons. ↑
Background Cons. ↑
Inter-shot Sim. ↑
PCP ↑
PTA ↑
PTCS ↑
Causal multi-shot models
LongLive2
5B
0.3856
0.2118
0.5571
0.9496
0.9454
0.9420
0.9808
0.3764
0.3905
ShotStream
1.3B
0.9080
0.2207
0.5776
0.9571
0.9616
0.7409
0.9501
0.6756
0.6917
Causal multi-prompt models
Infinity-RoPE
1.3B
0.0124
0.2178
0.5747
0.9159
0.9167
0.9396
0.9801
0.3619
0.3881
Table 2: Quantitative comparison on streaming multi-shot video generation. The best and second-best results are highlighted in bold and underlined , respectively.
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
Training Strategy
Retrieval Strategy (Plug-in)
Teacher Forcing
0.9789
0.2306
0.5696
0.9615
0.7120
Full-memory
0.9983
0.2337
0.5776
0.9640
0.7922
Resampled Context
0.9983
0.2337
0.5776
0.9640
0.7922
F + Shared
0.9949
0.2379
0.5746
0.9643
0.7280
Query-level Retrieval Variant
F + Q-wise
0.9946
0.2368
0.5755
0.9645
0.7525
Query-group
0.9953
0.2362
0.5748
0.9656
0.8233
C + Shared
0.9950
0.2378
0.5732
0.9626
0.7442
Query-wise ( Ours )
0.9960
0.2336
0.5779
0.9673
0.8242
C + Q-wise ( Ours )
0.9953
0.2382
0.5768
0.9654
0.8007
Table 3: Ablation study. The best and runner-up are in bold and underlined . C: clustered KV cache; F: frame grouping; Q-wise: query- and head-wise retrieval; Shared: shared across queries.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Visualization of query-wise memory selection diversity. For the selected memory frames, different queries exhibit distinct spatial preferences within the same frame, revealing heterogeneous memory selection behaviors. Each query corresponds to the first latent of the current chunk, and each memory unit is formed by grouping neighboring 3×3 spatial tokens.
Figure 8: Visualization of head-wise memory selection diversity. For the selected memory frames, different attention heads exhibit distinct memory preferences. We visualize multiple query regions for generality. The variations among highlighted regions across rows in each heatmap illustrate the diversity of head-wise memory selection.
Figure 9: Comparison of memory-token coverage and selection overlap under three selection granularities: query-head-wise, query-wise with head averaging, and head-wise with query averaging. The horizontal axis denotes the fraction of memory tokens selected by each selection unit. The left panel reports the percentage of distinct memory-token positions covered by the union of all selections, while the right panel shows the percentage of selected positions with multiplicity at least two.
Figure 10: Visualization of token clustering before and after RoPE. Compared with Pre-RoPE representations, Post-RoPE similarities exhibit stronger positional dependency and weaker semantic clustering structures.
Figure 11: Precision (left) and recall (right) of different retrieval strategies under varying full-memory attention thresholds. For each query-head pair, full-memory attention serves as the oracle, and retrieved tokens are considered matched if they fall within the top- x% oracle-ranked memory tokens. Online-clustered KV cache retrieves tokens from query-specific selected clusters, while Latent-group, Frame-group, and Sliding Window perform frame-level or fixed-context retrieval. All curves are averaged over attention blocks. Online-clustered KV cache achieves a better precision-recall trade-off, providing a selective approximation to full-memory access.
Lk=∥vktrain−vk∗∥22
Appendix
Algorithm 2 Training with sequential memory retrieval and resampled context.
Figure 12: Practical efficiency comparison during autoregressive rollout. The left figure shows the per-chunk denoising time, and the right figure shows the peak allocated GPU memory during 42-chunk generation.
Figure 13: Qualitative comparison between PTCS and similarity-based inter-shot consistency metrics. The examples illustrate cases where PTCS better reflects transition correctness by jointly considering semantic preservation and prompt-specified changes.
Method
Overall Video Quality
Intra-shot Consistency
Inter-shot Consistency
ShotStream
4.00 ± 0.63
4.03 ± 0.77
4.03 ± 0.42
Self-Forcing
2.40 ± 0.44
3.90 ± 0.69
1.23 ± 0.45
Infinity-RoPE
2.50 ± 0.32
3.93 ± 0.66
1.41 ± 0.42
LongLive2
3.73 ± 0.50
3.98 ± 0.80
3.03 ± 0.50
MemFlow
2.71 ± 0.34
3.93 ± 0.80
1.97 ± 0.43
HoloCine
4.13 ± 0.65
4.13 ± 0.45
4.13 ± 0.52
Appendix
Table 4: User study results. Participants rate generated videos from three perspectives on a scale from 1 to 5. The best results are highlighted in bold .
Figure 14: Comparison between query-wise retrieval and the query-group retrieval variant. Query-wise retrieval performs independent memory retrieval for each query token, while the query-group variant recursively partitions query tokens based on intra-group content diversity. Queries with similar content preferences are assigned to the same group and share the retrieved memory.
Figure 15: A case of qualitative result of ablation study.
Figure 16: Distribution of evaluation prompts across different content categories, shot counts, and revisit distances.
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
Landscapes and Architecture
Objects and Still Life
LongLive2
0.5653
0.2743
0.6086
0.9422
0.3741
LongLive2
0.7803
0.2444
0.5695
0.9539
0.5017
ShotStream
0.9952
0.2313
0.5639
0.9597
0.6500
ShotStream
1.0000
0.2428
0.5579
0.9637
0.7791
Infinity-RoPE
0.0000
0.2674
0.5966
0.8736
0.3743
Infinity-RoPE
0.1248
0.2518
0.5692
0.9175
0.4577
Self-Forcing
0.5740
0.2606
0.5949
0.8773
0.5071
Self-Forcing
0.3119
0.2551
0.5708
0.9206
0.3354
MemFlow
0.2837
0.2669
0.5846
0.8762
0.3923
MemFlow
0.0147
0.2424
0.5750
0.9453
0.3291
Appendix
Table 5: Fine-grained results across different content categories. The best and runner-up within each category are in bold and underlined , respectively.
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
3 Shots
4 Shots
LongLive2
0.3609
0.2260
0.5578
0.9520
0.3433
LongLive2
0.4123
0.2000
0.5584
0.9482
0.3866
ShotStream
0.9000
0.2327
0.5669
0.9592
0.6788
ShotStream
0.9027
0.2170
0.5721
0.9594
0.6750
Infinity-RoPE
0.0340
0.2379
0.5918
0.9251
0.4320
Infinity-RoPE
0.0120
0.2081
0.5654
0.9231
0.3781
Self-Forcing
0.2502
0.2368
0.5803
0.9346
0.2648
Self-Forcing
0.1966
0.2150
0.5751
0.9253
0.3226
MemFlow
0.0680
0.2345
0.5869
0.9517
0.1980
MemFlow
0.0120
0.2114
0.5699
0.9464
0.2208
Appendix
Table 6: Fine-grained results across different shot counts. The best and runner-up within each category are in bold and underlined , respectively.
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
Method
T-Ctrl ↑
T-V Sim. ↑
Aes. ↑
Intra. ↑
PTCS ↑
No Revisit
4–8 Chunks
LongLive2
0.3696
0.2010
0.5404
0.9472
0.3659
LongLive2
0.3049
0.2085
0.5411
0.9512
0.3127
ShotStream
0.9609
0.2252
0.5451
0.9603
0.7442
ShotStream
0.7995
0.2178
0.5856
0.9584
0.5135
Infinity-RoPE
0.0000
0.2124
0.5542
0.9255
0.4194
Infinity-RoPE
0.0000
0.2150
0.5755
0.9232
0.3231
Self-Forcing
0.1896
0.2242
0.5574
0.9258
0.3223
Self-Forcing
0.1548
0.2176
0.5794
0.9381
0.1502
MemFlow
0.0295
0.2166
0.5511
0.9487
0.1811
MemFlow
0.0309
0.2138
0.5904
0.9547
0.1013
Appendix
Table 7: Fine-grained results across different revisit distances. The best and runner-up within each category are in bold and underlined , respectively.
Figure 17: Cases of longer video generated by CineMem.
Figure 18: A failure case in complex multi-entity scenarios. Although the generated video preserves revisitation consistency across shots, inconsistencies such as character disappearance and detail changes occur during the transition from Shot 1 to Shot 2. This example highlights the difficulty of handling multi-entity reasoning and spatial relationship preservation in autoregressive video generation.
Memory-based long-video generation aims to combine user control with long-range visual consistency. We organize this process into three functional components: a planner, an adapter, and a generator. Representative multi-shot systems couple memory to generator-specific interfaces or use predefined agent workflows, rather than independently training a memory policy for cross-shot objectives. We introduce REEL, a general-purpose visual-memory planner. Through a unified tool interface, REEL learns to retrieve references, create missing visual evidence, and update persistent memory. Backend-specific adapters translate its evidence plans into supported conditioning inputs, allowing the same policy to guide different black-box video generators. To train REEL, we construct two high-quality datasets, REEL-SFT-4k and REEL-RL-1k, covering diverse interaction patterns and prompt types. Training proceeds in three stages: supervised fine-tuning, single-shot Group Relative Policy Optimization (GRPO), and multi-shot GRPO. Multi-shot GRPO carries policy-generated memory across consecutive shots, so later outcomes guide earlier reference selection and memory updates. REEL-Bench tests identity preservation and intended state changes under text-only requests and timed reference-image uploads. On JoyAI-Echo~1.5 and MiniMax-H3, REEL improves cross-shot consistency on ST-Bench by 15.1% and 20.7%, and Core-100 on our proposed REEL-Bench by 2.41 and 8.32 points, respectively, over the no-memory baseline.
Hang Xu, Guohui Zhang, Xiaoxiao Ma +7
University of Science and Technology of China · JD Explore Academy
Multi-shot video generation requires maintaining a consistent appearance of recurring entities across shots while remaining faithful to shot-specific text prompts. Recent autoregressive methods reuse previously generated frames as memory. However, full-frame storage entangles persistent entity information with transient scene context, leading to irrelevant information leakage and high computational cost. We propose an entity-centric memory in the form of an entity-indexed bank of latent patches. We introduce sparse token conditioning compatible with pretrained models, restricting self-attention to entity-relevant tokens and reducing computational cost. To support this, we introduce a structured multi-shot script format. We additionally propose a budgeted memory update strategy to maintain a compact, evolving memory. Finally, we equip the entity representation with a noise-injection mechanism that enables fine-grained appearance control, preventing leakage of irrelevant information. Our method improves prompt adherence and efficiency while preserving subject consistency.
Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang +2
Max Planck Institute for Informatics, Germany and Adobe Research, UK · Adobe Research, UK · Adobe Research, USA
Autoregressive (AR) video generation extends videos by producing latent chunks sequentially, but scaling to long videos requires repeated access to a growing historical KV cache. Existing methods reduce this cost by truncating the KV cache or compressing it into implicit memory, but both lose explicit access to query-relevant historical details. We propose OmniMem, an explicit full-range memory retrieval framework that performs sparse KV retrieval over the historical cache. To make this practical for chunk-based AR video generation, OmniMem addresses two issues: (i) local bias in sparse KV selection and (ii) Union Explosion in memory access. Adaptive Window Exclusion removes local-window blocks from the selection candidates when sufficient long-range history is available, preserving the sparse budget for informative long-range retrieval. Query-Shared KV Selection reduces cross-query diversity, while Per-Head Scattered KV Access avoids expanding head-specific selections into a large selected KV buffer. This allows each attention head to retrieve non-contiguous KV blocks according to its own selection pattern. Experiments on long-video generation show that OmniMem improves Dynamic Degree by 52.3% and preserves strong consistency over strong baselines, while maintaining comparable memory usage.