CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation

Xiaoxue Wu; Xinyu Gao; Fang Yu; Yu Qiao; Xinyuan Chen

cs.CVOct 13, 2026

CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation

Authors: Xiaoxue Wu, Xinyu Gao, Fang Yu, Yu Qiao, Xinyuan Chen

Organizations: Shanghai Artificial Intelligence Laboratory · Fudan University · Shanghai Jiao Tong University · East China Normal University

Abstract

Autoregressive video generation relies on memory retrieval to model long-range dependencies. This capability becomes particularly critical for multi-shot generation, where models must reconcile substantial visual changes across shot transitions with persistent semantic continuity. Existing methods tend to perform memory retrieval at the frame level and share the same historical context across all query tokens, overlooking the heterogeneous information needs and leading to redundant retrieval and inefficient use of historical memory. To bridge this gap, we introduce CineMem, an attention-aligned fine-grained memory retrieval framework that combines online memory organization with query-adaptive retrieval for autoregressive multi-shot video generation. Specifically, guided by intrinsic memory-access patterns revealed through attention analysis, we propose an online-clustered KV cache as a memory organization strategy over all historical tokens, enabling semantic-aware memory grouping and more precise retrieval. In addition, we introduce a query-level retrieval mechanism that provides broader access to historical information under a fixed retrieval budget, thereby improving long-term memory utilization. Together, these designs enable the multi-shot video generator to preserve long-range consistency across shot transitions, particularly in revisitation scenarios, while approximating full-memory access under a constrained memory budget. To facilitate evaluation, we further introduce a suite of task-specific, VLM-assisted metrics for faithful inter-shot consistency assessment. Extensive experiments demonstrate the effectiveness of our framework.

Figures & tables

Figure 1: 15-second multi-shot videos autoregressively generated by CineMem . By combining clustered memory with fine-grained query-adaptive retrieval, CineMem maintains coherent semantic flow across shot transitions, particularly in revisitation scenarios. Project page: https://uknowsth.github.io/CineMem/

Figure 2: Overview of CineMem. Existing autoregressive video generation methods tend to perform memory retrieval at the frame level, which may result in redundant retrieval, inefficient memory utilization, and overlooked heterogeneous information needs across query tokens. CineMem combines an online-clustered KV cache with query-adaptive retrieval to enable a finer-grained memory retrieval strategy. As illustrated by the attention receptive fields, CineMem provides broader (2/3 vs. 12/12) access to historical information under the same retrieval budget.

Figure 3: Our memory retrieval framework. Through multi-stage initial adaptation, we convert a bidirectional model into an autoregressive multi-shot model with memory retrieval. During inference, queries access memory from the first chunk, adjacent chunk, and retrieved candidates via token-wise and head-wise retrieval. Previous adjacent chunks are progressively integrated into retrieval candidates through online KV-cache clustering with cluster balancing. During training, resampling ground truth frames provides memory inputs to alleviate error accumulation.

Panel of Figure 4 — Figure 4: Intrinsic memory-selection patterns under the full-memory setting. Memory tokens exhibit semantically meaningful clustering structures, while different query-head pairs show distinct preferences over memory tokens. Although each query-head pair selects a subset of the most relevant tokens, their union covers nearly the entire memory. See Appendix A for more details.

Panel of Figure 5 — Figure 5: Qualitative Results. We compare CineMem with representative causal and bidirectional baselines on 15s multi-shot video generation. Representative examples illustrate the qualitative improvements of our method.

Figure 6: Implementation of PTCS. PTCS assigns low scores when either unchanged semantic elements are not preserved or intended changes are not correctly realized.

Method	Retrieval Overhead	Attention	Total FLOPs
	(TFLOPs)	(TFLOPs)	(TFLOPs)
Full history	0	932.56	932.56
Frame retrieval ( $K=20$ )	0.0453	649.97	650.02
Cluster retrieval ( $K=28$ , $S=1024$ )	15.0180	623.80	638.82

Table 1: Theoretical efficiency comparison. FLOPs are calculated for self-attention and retrieval-related operations. Collected K/V denotes the logical amount of retrieved memory elements.

Method	Params.	Video-level			Intra-shot		Inter-shot
Method	Params.	Trans. Ctrl. $\uparrow$	Text-Video Sim. $\uparrow$	Aesthetic $\uparrow$	Subject Cons. $\uparrow$	Background Cons. $\uparrow$	Inter-shot Sim. $\uparrow$	PCP $\uparrow$	PTA $\uparrow$	PTCS $\uparrow$
Causal multi-shot models
LongLive2	5B	0.3856	0.2118	0.5571	0.9496	0.9454	0.9420	0.9808	0.3764	0.3905
ShotStream	1.3B	0.9080	0.2207	0.5776	0.9571	0.9616	0.7409	0.9501	0.6756	0.6917
Causal multi-prompt models
Infinity-RoPE	1.3B	0.0124	0.2178	0.5747	0.9159	0.9167	0.9396	0.9801	0.3619	0.3881

Table 2: Quantitative comparison on streaming multi-shot video generation. The best and second-best results are highlighted in bold and underlined , respectively.

Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$	Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$
Training Strategy						Retrieval Strategy (Plug-in)
Teacher Forcing	0.9789	0.2306	0.5696	0.9615	0.7120	Full-memory	0.9983	0.2337	0.5776	0.9640	0.7922
Resampled Context	0.9983	0.2337	0.5776	0.9640	0.7922	F + Shared	0.9949	0.2379	0.5746	0.9643	0.7280
Query-level Retrieval Variant						F + Q-wise	0.9946	0.2368	0.5755	0.9645	0.7525
Query-group	0.9953	0.2362	0.5748	0.9656	0.8233	C + Shared	0.9950	0.2378	0.5732	0.9626	0.7442
Query-wise ( Ours )	0.9960	0.2336	0.5779	0.9673	0.8242	C + Q-wise ( Ours )	0.9953	0.2382	0.5768	0.9654	0.8007

Table 3: Ablation study. The best and runner-up are in bold and underlined . C: clustered KV cache; F: frame grouping; Q-wise: query- and head-wise retrieval; Shared: shared across queries.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of query-wise memory selection diversity. For the selected memory frames, different queries exhibit distinct spatial preferences within the same frame, revealing heterogeneous memory selection behaviors. Each query corresponds to the first latent of the current chunk, and each memory unit is formed by grouping neighboring $3\times 3$ spatial tokens.

Figure 8: Visualization of head-wise memory selection diversity. For the selected memory frames, different attention heads exhibit distinct memory preferences. We visualize multiple query regions for generality. The variations among highlighted regions across rows in each heatmap illustrate the diversity of head-wise memory selection.

Figure 9: Comparison of memory-token coverage and selection overlap under three selection granularities: query-head-wise, query-wise with head averaging, and head-wise with query averaging. The horizontal axis denotes the fraction of memory tokens selected by each selection unit. The left panel reports the percentage of distinct memory-token positions covered by the union of all selections, while the right panel shows the percentage of selected positions with multiplicity at least two.

Figure 10: Visualization of token clustering before and after RoPE. Compared with Pre-RoPE representations, Post-RoPE similarities exhibit stronger positional dependency and weaker semantic clustering structures.

Figure 11: Precision (left) and recall (right) of different retrieval strategies under varying full-memory attention thresholds. For each query-head pair, full-memory attention serves as the oracle, and retrieved tokens are considered matched if they fall within the top-

x\%

oracle-ranked memory tokens. Online-clustered KV cache retrieves tokens from query-specific selected clusters, while Latent-group, Frame-group, and Sliding Window perform frame-level or fixed-context retrieval. All curves are averaged over attention blocks. Online-clustered KV cache achieves a better precision-recall trade-off, providing a selective approximation to full-memory access.

\mathcal{L}_{k}=\|\widehat{\mathbf{v}}^{\mathrm{train}}_{k}-\mathbf{v}^{*}_{k}\|_{2}^{2}

Appendix

Algorithm 2 Training with sequential memory retrieval and resampled context.

Figure 12: Practical efficiency comparison during autoregressive rollout. The left figure shows the per-chunk denoising time, and the right figure shows the peak allocated GPU memory during 42-chunk generation.

Figure 13: Qualitative comparison between PTCS and similarity-based inter-shot consistency metrics. The examples illustrate cases where PTCS better reflects transition correctness by jointly considering semantic preservation and prompt-specified changes.

Method	Overall Video Quality	Intra-shot Consistency	Inter-shot Consistency
ShotStream	4.00 $\pm$ 0.63	4.03 $\pm$ 0.77	4.03 $\pm$ 0.42
Self-Forcing	2.40 $\pm$ 0.44	3.90 $\pm$ 0.69	1.23 $\pm$ 0.45
Infinity-RoPE	2.50 $\pm$ 0.32	3.93 $\pm$ 0.66	1.41 $\pm$ 0.42
LongLive2	3.73 $\pm$ 0.50	3.98 $\pm$ 0.80	3.03 $\pm$ 0.50
MemFlow	2.71 $\pm$ 0.34	3.93 $\pm$ 0.80	1.97 $\pm$ 0.43
HoloCine	4.13 $\pm$ 0.65	4.13 $\pm$ 0.45	4.13 $\pm$ 0.52

Appendix

Table 4: User study results. Participants rate generated videos from three perspectives on a scale from 1 to 5. The best results are highlighted in bold .

Figure 14: Comparison between query-wise retrieval and the query-group retrieval variant. Query-wise retrieval performs independent memory retrieval for each query token, while the query-group variant recursively partitions query tokens based on intra-group content diversity. Queries with similar content preferences are assigned to the same group and share the retrieved memory.

Figure 15: A case of qualitative result of ablation study.

Figure 16: Distribution of evaluation prompts across different content categories, shot counts, and revisit distances.

Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$	Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$
Landscapes and Architecture						Objects and Still Life
LongLive2	0.5653	0.2743	0.6086	0.9422	0.3741	LongLive2	0.7803	0.2444	0.5695	0.9539	0.5017
ShotStream	0.9952	0.2313	0.5639	0.9597	0.6500	ShotStream	1.0000	0.2428	0.5579	0.9637	0.7791
Infinity-RoPE	0.0000	0.2674	0.5966	0.8736	0.3743	Infinity-RoPE	0.1248	0.2518	0.5692	0.9175	0.4577
Self-Forcing	0.5740	0.2606	0.5949	0.8773	0.5071	Self-Forcing	0.3119	0.2551	0.5708	0.9206	0.3354
MemFlow	0.2837	0.2669	0.5846	0.8762	0.3923	MemFlow	0.0147	0.2424	0.5750	0.9453	0.3291

Appendix

Table 5: Fine-grained results across different content categories. The best and runner-up within each category are in bold and underlined , respectively.

Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$	Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$
3 Shots						4 Shots
LongLive2	0.3609	0.2260	0.5578	0.9520	0.3433	LongLive2	0.4123	0.2000	0.5584	0.9482	0.3866
ShotStream	0.9000	0.2327	0.5669	0.9592	0.6788	ShotStream	0.9027	0.2170	0.5721	0.9594	0.6750
Infinity-RoPE	0.0340	0.2379	0.5918	0.9251	0.4320	Infinity-RoPE	0.0120	0.2081	0.5654	0.9231	0.3781
Self-Forcing	0.2502	0.2368	0.5803	0.9346	0.2648	Self-Forcing	0.1966	0.2150	0.5751	0.9253	0.3226
MemFlow	0.0680	0.2345	0.5869	0.9517	0.1980	MemFlow	0.0120	0.2114	0.5699	0.9464	0.2208

Appendix

Table 6: Fine-grained results across different shot counts. The best and runner-up within each category are in bold and underlined , respectively.

Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$	Method	T-Ctrl $\uparrow$	T-V Sim. $\uparrow$	Aes. $\uparrow$	Intra. $\uparrow$	PTCS $\uparrow$
No Revisit						4–8 Chunks
LongLive2	0.3696	0.2010	0.5404	0.9472	0.3659	LongLive2	0.3049	0.2085	0.5411	0.9512	0.3127
ShotStream	0.9609	0.2252	0.5451	0.9603	0.7442	ShotStream	0.7995	0.2178	0.5856	0.9584	0.5135
Infinity-RoPE	0.0000	0.2124	0.5542	0.9255	0.4194	Infinity-RoPE	0.0000	0.2150	0.5755	0.9232	0.3231
Self-Forcing	0.1896	0.2242	0.5574	0.9258	0.3223	Self-Forcing	0.1548	0.2176	0.5794	0.9381	0.1502
MemFlow	0.0295	0.2166	0.5511	0.9487	0.1811	MemFlow	0.0309	0.2138	0.5904	0.9547	0.1013

Appendix

Table 7: Fine-grained results across different revisit distances. The best and runner-up within each category are in bold and underlined , respectively.

Panel of Figure 17 — Figure 17: Cases of longer video generated by CineMem.

Figure 18: A failure case in complex multi-entity scenarios. Although the generated video preserves revisitation consistency across shots, inconsistencies such as character disappearance and detail changes occur during the transition from Shot 1 to Shot 2. This example highlights the difficulty of handling multi-entity reasoning and spatial relationship preservation in autoregressive video generation.

Memory-Augmented Video Generation

Memory-augmented video generation gives video models persistent context to preserve identities, scenes, and events across long sequences.

See all 110 papers

REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation
Oct 12, 2026Hang Xu, Guohui Zhang, Xiaoxiao Ma +7Memory-Augmented Video Generation Long-Horizon Video Generation
University of Science and Technology of China · JD Explore Academy
EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation
May 22, 2026Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang +2Memory-Augmented Video Generation Identity-Preserving Video Generation
Max Planck Institute for Informatics, Germany and Adobe Research, UK · Adobe Research, UK · Adobe Research, USA
OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation
May 28, 2026Lin Zhao, Yushu Wu, Yifan Gong +2KV Caching Long Video Generation
Northeastern University · Adobe Research

Oct 12, 2026J/K move · Enter open · S save

REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation

Hang Xu, Guohui Zhang, Xiaoxiao Ma, Hu Yu+6

University of Science and Technology of China · JD Explore Academy

Memory-Augmented Video Generation Long-Horizon Video Generation Multi-Shot Video Generation RL for Video Generation

Open paper AskPDF arXiv

Memory-based long-video generation aims to combine user control with long-range visual consistency. We organize this process into three functional components: a planner, an adapter, and a generator. Representative multi-shot systems couple memory to generator-specific interfaces or use predefined agent workflows, rather than independently training a memory policy for cross-shot objectives. We introduce REEL, a general-purpose visual-memory planner. Through a unified tool interface, REEL learns to retrieve references, create missing visual evidence, and update persistent memory. Backend-specific adapters translate its evidence plans into supported conditioning inputs, allowing the same policy to guide different black-box video generators. To train REEL, we construct two high-quality datasets, REEL-SFT-4k and REEL-RL-1k, covering diverse interaction patterns and prompt types. Training proceeds in three stages: supervised fine-tuning, single-shot Group Relative Policy Optimization (GRPO), and multi-shot GRPO. Multi-shot GRPO carries policy-generated memory across consecutive shots, so later outcomes guide earlier reference selection and memory updates. REEL-Bench tests identity preservation and intended state changes under text-only requests and timed reference-image uploads. On JoyAI-Echo~1.5 and MiniMax-H3, REEL improves cross-shot consistency on ST-Bench by 15.1% and 20.7%, and Core-100 on our proposed REEL-Bench by 2.41 and 8.32 points, respectively, over the no-memory baseline.

Abstract

Memory-Augmented Video Generation

Explore similar work

REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation

EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation

OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation