REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation

Hang Xu; Guohui Zhang; Xiaoxiao Ma; Hu Yu; Jie Huang; Jiayin Xiao; Lin Song; Haoyang Huang

cs.CVOct 12, 2026

REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation

Authors: Hang Xu, Guohui Zhang, Xiaoxiao Ma, Hu Yu, Jie Huang, Jiayin Xiao, Lin Song, Haoyang Huang,

Organizations: University of Science and Technology of China · JD Explore Academy

Abstract

Memory-based long-video generation aims to combine user control with long-range visual consistency. We organize this process into three functional components: a planner, an adapter, and a generator. Representative multi-shot systems couple memory to generator-specific interfaces or use predefined agent workflows, rather than independently training a memory policy for cross-shot objectives. We introduce REEL, a general-purpose visual-memory planner. Through a unified tool interface, REEL learns to retrieve references, create missing visual evidence, and update persistent memory. Backend-specific adapters translate its evidence plans into supported conditioning inputs, allowing the same policy to guide different black-box video generators. To train REEL, we construct two high-quality datasets, REEL-SFT-4k and REEL-RL-1k, covering diverse interaction patterns and prompt types. Training proceeds in three stages: supervised fine-tuning, single-shot Group Relative Policy Optimization (GRPO), and multi-shot GRPO. Multi-shot GRPO carries policy-generated memory across consecutive shots, so later outcomes guide earlier reference selection and memory updates. REEL-Bench tests identity preservation and intended state changes under text-only requests and timed reference-image uploads. On JoyAI-Echo~1.5 and MiniMax-H3, REEL improves cross-shot consistency on ST-Bench by 15.1% and 20.7%, and Core-100 on our proposed REEL-Bench by 2.41 and 8.32 points, respectively, over the no-memory baseline.

Figures & tables

Figure 2: Multi-shot inference with shared local resources. Each shot follows the planner–adapter–generator interaction in Figure 3 . Local resources include the previous state, memory bank, and registry. Shot 5 stores a generated neighbor reference for retrieval and reuse in Shot 8. The solid arrow denotes adjacent-shot context; dashed arrows denote writeback and retrieval. Shot 7 is omitted.

Figure 3: Single-shot inference with REEL. From a shot request and optional user uploads, the planner (1) identifies requirements and (2) queries local resources (previous state, memory bank, and registry) to retrieve and inspect evidence. Tool calls (3) create references if needed and (4) submit an evidence plan with a backend-agnostic enhanced prompt and reference bindings. The adapter performs backend-specific prompt enhancement and condition allocation for video generation. The planner (5) reviews the candidate against requirements and applied conditions, triggering revisions when needed and optionally updating memory and registry. Here, the saved robot reference is reused in Shots 8 and 10.

Figure 4: Data engine and REEL-Bench. (a) Prompt expansion, teacher rollouts, and filtering produce REEL-SFT-4k; requests and initial resources define REEL-RL-1k tasks. Badges identify the data-construction models. (b) REEL-Bench contains 300 stories and 2,700 shots across six input/interaction scenarios (left) and eight visual styles (right), with overlapping scenarios allowed. Images show input assets or recorded frames, and the waveform comes from a paired voice reference. Shot- and story-level visual scores form Core-100; audio is excluded (Appendix D ).

	Overall	Shot level				Story level
Method	Core $\uparrow$	PF $\uparrow$	Ref. $\uparrow$	VQ $\uparrow$	Motion $\uparrow$	Cons. $\uparrow$	State $\uparrow$	Narr. $\uparrow$
JoyAI-Echo 1.5
No Generated Memory	50.35	47.73	44.00	74.67	58.56	45.83	35.00	39.17
Recent Memory (FIFO)	48.55	43.57	37.22	69.35	57.52	50.00	35.83	39.17
CLIP-based Retrieval	51.16	45.69	39.34	72.18	58.81	55.00	37.50	40.83
StoryMem-style Memory	50.53	45.68	38.30	70.89	57.83	55.00	35.83	40.00

Table 1: REEL-Bench results (0–100, higher is better; Appendix D ). Core: Core-100; Ref./State: applicable cases. Per backend, bold/underlined: best/second best; gray: REEL.

	Quality	Prompt following		Cross-shot consistency
Method	Aesthetic $\uparrow$	Global PF $\uparrow$	Shot PF $\uparrow$	Cons. $\uparrow$	Top-10 $\uparrow$
JoyAI-Echo 1.5
No Generated Memory	0.598	0.226	0.245	0.457	0.489
Recent Memory (FIFO)	0.593	0.225	0.237	0.497	0.525
CLIP-based Retrieval	0.596	0.230	0.239	0.515	0.541
StoryMem-style Memory	0.595	0.225	0.239	0.511	0.534

Table 2: ST-Bench results. PF/Cons.: prompt following/cross-shot consistency; Top-10: text-selected pairs. Higher is better; bold/underlined: best/second best per backend; gray: REEL.

Table 3: REEL-Bench ablations on JoyAI-Echo 1.5 (a,b) and frozen full REEL transfer to unseen backends (c). Shot/Story: shot-/story-level quality (0–100); Avg. shots: mean successful shots per story (Appendix D ). Higher is better; bold/underlined: best/second best.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Execution contract
view_material_registry	Returns the request, subject and asset metadata, pagination state, supported conditions, and remaining budgets.
search_memory_bank	Retrieves committed evidence with a request-grounded query; a hit is not yet a visual observation.
view_memories	Exposes the selected representation and returns its current-session authorization.
generate_image	Creates a missing reference using an explicit brief and authorized input images.
submit_evidence_plan	Submits the complete enhanced prompt, immutable requirements, ranked bindings, and uncovered requirements.
inspect_candidate	Returns candidate observations and a runtime-defined checklist for the current assessment purpose.

Appendix

Table 4: Planner tools and their observable effects. Backend endpoints, random seeds, and physical condition-slot assignments are controlled by the host and adapter, not tool arguments chosen by the planner.

Split	Stories	Families	JoyAI-Echo 1.5 records	H3 records	Total records
Training	4,142	3,018	30,187	2,897	33,084
Validation	226	168	1,691	205	1,896
Test	232	169	1,693	157	1,850

Appendix

Table 5: Accepted trajectory records in the frozen SFT snapshot, before four training-only padding records. These are construction-split statistics, not counts of evaluated benchmark stories.

Figure 6: Input configurations and recorded story context. (a) A text-only story has no uploaded reference; both displayed images are source-run frames. (b) One uploaded reference is shown alongside source-run frames from Shots 1 and 8. (c) Separate character and object references accompany a source-run frame. (d) Three original reference images arrive at different shots. All images retain their full source content and aspect ratio. Source-run frames are illustrations, not benchmark ground truth or newly measured method comparisons.

Figure 7: Persistent memory reuse in an executed rollout. The full REEL checkpoint on JoyAI-Echo 1.5 stores an object reference at Shot 3 and retrieves it at Shot 10 alongside the tail frame of Shot 9. Both references match the backend’s recorded inputs. The right panel shows the selected video’s midpoint.

Property	REEL-Bench
Stories / shots	300 / 2,700
Text-only stories	60
Image-only stories	210
Image-plus-voice stories	30
Eight-shot / nine-shot / ten-shot stories	100 / 100 / 100
Initial-only uploads	119

Appendix

Table 6: Frozen REEL-Bench input counts. Table 9 distinguishes full-benchmark evaluations from the matched-adapter subset control.

Dimension	Evaluation target
Prompt following	Requested subjects, actions, setting, composition, style, and exclusions.
Reference fidelity	Stable identity, object detail, layout, or style from relevant images already uploaded. New requested states override obsolete reference states.
Visual quality	Composition, anatomy or geometry, lighting, materials, and visible artifacts, respecting the intended style.
Motion/temporal quality	Requested action progression and within-shot stability across the sampled frames.
Cross-shot consistency	Recurring identity, object design, scene geometry, and intended visual treatment across shots.
State compliance	Requested appearance, object, or environment changes and their persistence until superseded.

Appendix

Table 7: The seven visual dimensions used by REEL-Bench. The first four are judged per shot; the last three are judged over the story.

Applicable dimensions	PF	Ref.	VQ	Motion	Cons.	State	Narr.
Ref. and State	20	15	15	10	20	10	10
State, without Ref.	25	0	20	15	20	10	10
Ref., without State	20	15	15	10	25	0	15
Neither Ref. nor State	25	0	20	15	25	0	15

Appendix

Table 8: Fixed Core-100 weights (%). Zero indicates a dimension that is not applicable. The same profile-selection rule and weights apply to every method.

Comparison	Backend(s)	Stories	Shots
Main results	JoyAI-Echo 1.5; MiniMax-H3	300	2,700
Training stages	JoyAI-Echo 1.5	300	2,700
Memory mechanisms	JoyAI-Echo 1.5	300	2,700
Training-budget control	JoyAI-Echo 1.5	300	2,700
Full REEL transfer	JoyAI-Echo 1.0; Seedance 2.0; Seedance 2.5; Kling 3.0	300	2,700
Matched-adapter control	Seedance 2.0; Kling 3.0	30	263

Appendix

Table 9: REEL-Bench evaluation scope. Counts apply to each method or variant on each listed backend, not to a pooled total across backends. Shots are requested shots; successful generation is counted separately.

Setting	JoyAI-Echo 1.5	MiniMax-H3
ST-Bench video	$1280\times 736$ , 241 frames, 25 fps	5 s; 16:9; target short edge 768
REEL-Bench video	$1280\times 736$ , 241 frames, 25 fps	5 s; 16:9; target short edge 768
H3 sampling	Not applicable	30 inference steps
Video candidate cap	Two per requested shot	Two per requested shot
Condition-plan revision cap	One per requested shot	One per requested shot
First-attempt seed	Paired within backend	Paired within backend

Appendix

Table 10: Generation settings shared by ST-Bench and REEL-Bench within each backend. Reference-image creation, planning, and inspection are counted separately from video generation.

Parameter	Value
Base model	Qwen3.5-9B
Effective examples / epochs / optimizer steps	33,088 / 1 / 1,034
Learning rate	$10^{-5}$
Global batch / micro-batch size	32 / 1
Maximum sequence length	131,072 tokens
Maximum image pixels	262,144

Appendix

Table 11: Configuration of the SFT stage.

Variant	Updates single / multi	Training video calls	Completion (%) $\uparrow$	Matched stories / shots	Matched Shot $\uparrow$	Matched Story $\uparrow$
Single-shot	400 / 0	13145	86.67	213/1898	55.48	45.59
Single-shot, extended	1,000 / 0	33871	82.59	213/1898	55.43	45.71
REEL	400 / 600	20923	100.00	213/1898	57.69	46.02

Appendix

Table 12: Training-budget controls on JoyAI-Echo 1.5 using the full REEL-Bench (300 stories, 2,700 requested shots). All variants start from the same SFT checkpoint. Completion covers all requests; matched quality uses 213 common successful stories (1,898 shots). Training video calls are separate from optimizer updates.

Planner	Shot completion (0–1) $\uparrow$	Matched stories / shots	Matched Shot $\uparrow$	Matched Story $\uparrow$	$\Delta$ Story vs. SFT [95% CI]
Seedance 2.0
SFT	0.54	16/132	55.47	42.18	N/A
+ Sig. GRPO	0.77	16/132	58.43	46.31	4.13[+1.21,+7.05]
REEL	1.00	16/132	58.76	47.03	4.85[+1.75,+7.95]
Kling 3.0
SFT	0.59	18/150	54.11	43.98	N/A

Appendix

Table 13: Matched-adapter controls on a REEL-Bench subset of 30 stories and 263 requested shots. Variants share tools and the adapter within each backend. Matched quality uses 16 stories (132 shots) for Seedance 2.0 and 18 (150 shots) for Kling 3.0; completion is the fraction of the 263 requests completed. Story differences and 95% intervals are paired against SFT; gray rows denote full REEL.

Source-run method	Core-100 $\uparrow$	95% CI
No Generated Memory	50.35	[48.53, 52.44]
FIFO	48.55	[46.49, 50.79]
StoryMem-style	50.53	[48.38, 52.83]
CLIP Retrieval	51.16	[49.06, 53.29]
REEL	52.76	[50.86, 54.61]

Appendix

Table 14: Source-run uncertainty on the REEL-Bench. Intervals use 2,000 story-bootstrap resamples with seed 20260922 and estimate each method’s mean, not a paired difference.

Figure 9: An error propagated through a created reference. The selected image contains decorative suspension elements, reinforced by the compiled prompt and exact-opening condition. Two frames from the resulting MiniMax-H3 clip are shown at the same fixed 10% and 50% sample positions used in the recorded review of the full REEL checkpoint.

Memory-Augmented Video Generation

Memory-augmented video generation gives video models persistent context to preserve identities, scenes, and events across long sequences.

See all 109 papers

CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation
Oct 13, 2026Xiaoxue Wu, Xinyu Gao, Fang Yu +2Memory-Augmented Video Generation Multi-Shot Video Generation
Shanghai Artificial Intelligence Laboratory · Fudan University · Shanghai Jiao Tong University +1
Closed-Loop Triplet Synergistic Generation for Long-Form Video
Jun 15, 2026Xinlei Yin, Xiulian Peng, Xiao Li +2Long Video Generation Memory-Augmented Video Generation
Microsoft Research Asia · University of Science and Technology of China
Memento: Reconstruct to Remember for Consistent Long Video Generation
Jun 12, 2026Xuan Wei, Longbin Ji, Guan Wang +5Long Video Generation Memory-Augmented Video Generation
Xiamen University · ERNIE Team, Baidu Inc.

Oct 13, 2026J/K move · Enter open · S save

CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation

Xiaoxue Wu, Xinyu Gao, Fang Yu, Yu Qiao+1

Shanghai Artificial Intelligence Laboratory · Fudan University · Shanghai Jiao Tong University+1

Memory-Augmented Video Generation Multi-Shot Video Generation Autoregressive Video Generation Long-Horizon Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Autoregressive video generation relies on memory retrieval to model long-range dependencies. This capability becomes particularly critical for multi-shot generation, where models must reconcile substantial visual changes across shot transitions with persistent semantic continuity. Existing methods tend to perform memory retrieval at the frame level and share the same historical context across all query tokens, overlooking the heterogeneous information needs and leading to redundant retrieval and inefficient use of historical memory. To bridge this gap, we introduce CineMem, an attention-aligned fine-grained memory retrieval framework that combines online memory organization with query-adaptive retrieval for autoregressive multi-shot video generation. Specifically, guided by intrinsic memory-access patterns revealed through attention analysis, we propose an online-clustered KV cache as a memory organization strategy over all historical tokens, enabling semantic-aware memory grouping and more precise retrieval. In addition, we introduce a query-level retrieval mechanism that provides broader access to historical information under a fixed retrieval budget, thereby improving long-term memory utilization. Together, these designs enable the multi-shot video generator to preserve long-range consistency across shot transitions, particularly in revisitation scenarios, while approximating full-memory access under a constrained memory budget. To facilitate evaluation, we further introduce a suite of task-specific, VLM-assisted metrics for faithful inter-shot consistency assessment. Extensive experiments demonstrate the effectiveness of our framework.

Abstract

Memory-Augmented Video Generation

Explore similar work

CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation

Closed-Loop Triplet Synergistic Generation for Long-Form Video

Memento: Reconstruct to Remember for Consistent Long Video Generation