cs.CVAug 28, 2026
SaveLayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation
Organizations: Zhejiang University · The University of Hong Kong
Abstract
Autoregressive video diffusion enables scalable long-video generation by producing chunks from a bounded recent context. While recency-based caching preserves local continuity, it evicts historical cues needed when subjects, objects, scenes, or attributes reappear. Existing memory mechanisms expose models to nonlocal history, but access alone does not ensure effective use. Our analysis reveals that video DiT layers exhibit distinct preferences for current, recent, and distant context, suggesting that long-range memory requires deciding both what to retrieve and where to use it. We introduce LayerRecall, a current-conditioned, layer-selective memory router that retrieves relevant historical K/V states and injects them only into backbone-specific memory-sensitive layers while preserving local attention elsewhere. To reduce reliance on scarce high-quality long-horizon videos and explicit memory-allocation labels, we further propose Cross-Horizon Prediction Matching (CHPM), which uses a privileged long-context reference to supervise the bounded-memory router in prediction space. Across 100 multi-shot evaluation prompts, LayerRecall achieves the best overall results on MemoBench and MovieBench while matching its backbone on VBench-Long, demonstrating stronger long-range recovery without sacrificing local continuity. Qualitative analyses further reveal memory-guided self-correction, whereby initially mismatched local attributes return to their historical appearance without resetting ongoing motion or scene structure. Additional analyses show cross-backbone portability and negligible inference overhead.
Figures & tables
Figure 2: Attention profiling on the LongLive-2.0 backbone. (a) Memory-attention mass by prompt type and DiT layer; memory denotes far-history and recent-history buckets before the current chunk. Final memory-sensitive layers are outlined. (b) Prompt-level mean and min–max memory mass across 100 prompts, showing that the layer-preference pattern remains relatively stable over the evaluated prompts.
Figure 3: Overview of LayerRecall. Given the current chunk, the router retrieves finite historical K/V candidates from layer-indexed memory banks and scores which memories are relevant to the next generation step. Retrieved states are injected only into profiled memory-sensitive layers, while the remaining layers keep the original sink, sliding-window, and current-chunk attention context. K/V denotes key-value states.
Figure 4: Training framework of LayerRecall. A frozen long-horizon teacher predicts from an expanded historical context, while a frozen short-horizon student predicts with bounded local context augmented by LayerRecall memory routing. The routing module is optimized by matching student and teacher predictions; snowflakes indicate frozen backbones.
Figure 5: LayerRecall showcases across diverse three-shot reappearance scenarios. Each triplet samples one frame from Shots 1–3: a subject, object, or visual attribute is established in Shot 1, leaves the visible scene in Shot 2, and reappears in Shot 3. Across people, animals, apparel, instruments, and scene-specific objects, LayerRecall preserves the corresponding identity and fine-grained appearance over the intervening temporal gap, demonstrating stable long-term recall.
| MemoBench | MovieBench | |||||||||
| Method | Venue | VBench-Long Avg. | Overall | Obj. Reapp. | Layout Rec. | State Upd. | Overall | Shot Coh. | Scene Layout | Narrative |
| LongLive | ICLR26 | 0.977 | 0.499 | 0.516 | 0.534 | 0.464 | 0.544 | 0.491 | 0.624 | 0.509 |
| LongLive-2.0 | arXiv26 | 0.978 | 0.513 | 0.519 | 0.524 | 0.499 | 0.546 | 0.531 | 0.596 | 0.517 |
| LongLive-RAG | arXiv26 | 0.958 | 0.373 | 0.358 | 0.360 | 0.297 | 0.394 | 0.352 | 0.436 | 0.355 |
| CausVid | CVPR25 | 0.975 | 0.426 | 0.418 | 0.414 | 0.400 | 0.483 | 0.489 | 0.495 | 0.450 |
| Deep Forcing | ICML26 | 0.964 | 0.408 | 0.371 | 0.441 | 0.405 | 0.511 | 0.473 | 0.583 | 0.427 |
Table 1: Main results on the videos generated from 100 evaluation prompts. VBench-Long Avg. is the mean of VBench-Long subject consistency, background consistency, and motion smoothness. MemoBench and MovieBench provide task-aligned memory diagnostics. Bold and underline denote the best and second-best scores.
Figure 6: Temporal stability analysis on the 100 evaluation prompts. (a) Abrupt-change energy over time, with fewer frame-change spikes under LayerRecall. (b) Temporal spectrum of frame-change energy, with the HF ratio reduced from 0.60 to 0.38.
| Metric | All-Layer Routing | LayerRecall | Diff. |
| VBench Motion Smooth. | 0.9910 | 0.9916 | +0.0006 |
| Helios Smooth. | 0.9917 | 0.9923 | +0.0006 |
| CLIP Adjacent | 0.9397 | 0.9574 | +0.0177 |
| DINO Adjacent | 0.8795 | 0.9212 | +0.0417 |
Table 2: Temporal consistency ablation between LayerRecall and all-layer memory routing.
Figure 7: MemoBench evaluation of zero-training cross-backbone portability on 100 prompts. “L2 profile” applies LayerRecall with the LongLive-2.0 layer policy, while “Self profile” uses a backbone-specific layer policy obtained by profiling each target backbone.
| Backbone | Variant | End-to-end time (s/video) | FPS |
| LongLive-2.0 | w/o LayerRecall | 305.9 | 5.22 |
| w/ LayerRecall | 309.4 | 5.16 |
Table 3: Inference efficiency under matched H100 settings. FPS is computed from decoded frames divided by full end-to-end generation time.
Figure 8: MemoBench comparison of random and profile-guided historical-memory insertion in the dedicated matched ablation over 100 evaluation prompts. Under matched retrieval settings, LayerRecall (Profiled-10) improves the overall score and four of five reported dimensions, supporting the importance of selecting where retrieved memory enters the DiT.
| Method | Variant | Memo Overall | Memo ID | Memo State | Memo Camera |
| LayerRecall | Random init | 0.519 | 0.484 | 0.487 | 0.508 |
| Trained | 0.548 | 0.530 | 0.525 | 0.573 |
Table 4: Ablation of CHPM validity on the MemoBench. Both variants use the same ten memory-sensitive layers.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Layer-preference profiling on LongLive (left) and Self-Forcing (right), two backbones distinct from the LongLive-2.0 model used in the main experiments. For each backbone, the upper panel reports historical attention mass across DiT layers and prompt categories, while the lower panel shows the mean and min–max range over 100 held-out prompts. Both models exhibit heterogeneous and relatively prompt-stable layer-wise preferences, although their specific memory-sensitive layers differ. Red outlines mark the profiled insertion policies used in the cross-backbone experiments.
Figure 10: Memory-guided self-correction in a three shot sequence. The subject wears a solid blue inner garment in Shot 1, leaves the scene in Shot 2, and initially reappears with a mismatched pattern in Shot 3. LayerRecall later recalls the earlier blue garment while preserving the subject’s identity, gray cardigan, ongoing action, and scene structure, illustrating localized attribute recovery without a global visual reset.
Figure 11: Complete MemoBench results for zero-training LayerRecall transfer to LongLive and Self-Forcing over 100 evaluation prompts. Each group compares the backbone without LayerRecall, direct transfer using the LongLive-2.0 policy (L2 profile), and the same frozen router using the target backbone’s own profiled insertion policy (Self profile). “Self profile” denotes self-profiled layers for each target backbone, not Self-Forcing exclusively. Absolute scores and zoomed views are shown for object reappearance, identity, state, layout, camera, and overall performance. Thin black lines connect value labels to their bars and do not denote uncertainty intervals.
| Split | Size | Purpose | CHPM train | Layer selection | Final eval |
| Training set | 1,600 | Multi-shot prompts used to train CHPM. The prompt summary records 480 3-shot, 480 4-shot, 320 5-shot, and 320 6-shot cases across subject, color, position, and scene axes. | Yes | No | No |
| Profiling set | 100 | Held-out prompts used for prompt-stability analysis and layer-policy validation across subject, color, object, and spatial memory stresses. | No | Yes | No |
| Main evaluation set | 100 | Held-out 3-shot prompts used to generate the main evaluation videos. The summary records 25 cases for each of subject, color, position, and scene axes. | No | No | Yes |
Table 5: Data split usage for training, layer-policy analysis, and evaluation.
| Item | Setting |
| Backbone | LongLive-2.0 5B on Wan2.2-TI2V-5B. |
| Trainable parameters | LayerRecall router only; 1,648,416 parameters. |
| Training prompts | 1,600 multi-shot prompts. |
| Latent rollout | 384 latent frames, 48 chunks, 8 latent frames per chunk. |
| Prediction schedule | One sampled diffusion timestep and one denoising prediction-matching forward per supervised anchor. |
| Teacher/student history | Teacher retains the complete causal prefix at each anchor, with a maximum attention horizon of 384 latent frames; student has a 32-frame total attention-visible budget comprising an 8-frame sink, up to two retrieved history chunks (16 frames), and the current 8-frame chunk. |
Table 6: CHPM training and reproducibility settings.
| Method | Native multi-shot support | Input format | Adaptation rule |
| LongLive | Yes/partial | Block prompt list | Use the 48-block prompt script directly when supported. |
| LongLive-2.0 | Yes/partial | Block prompt list | Use the 48-block prompt script directly. |
| LongLive-RAG | Partial | Retrieved prompt/context | Preserve the same shot script and use its retrieval interface. |
| CausVid | No/partial | Single text prompt | Concatenate shots into a coherent long-video prompt with explicit reappearance constraints. |
| Deep Forcing | No/partial | Single text prompt | Concatenate shots; preserve subject, layout, and return constraints. |
| Dummy Forcing | No/partial | Single text prompt | Concatenate shots; preserve the memory target and final return. |
Table 7: Baseline prompt adaptation policy. “No/partial” means the local evaluation used a single-prompt adaptation or a method-specific wrapper rather than assuming a native 48-block prompt interface.
More in this topic
Long-Horizon Video Generation
Long-horizon video generation creates coherent, controllable videos over extended durations despite limited context and accumulating errors.
See all 164 papers