What Remains in Sight? Autoregressive Video Decoding as Representation-Guided Context Rewriting
Organizations: Stanford University
Abstract
Long-horizon video generation is increasingly moving toward autoregressive rollout, where each newly generated segment becomes part of the evidence used for future prediction. Recent causal decoders, long-context training, and KV-cache mechanisms have extended feasible duration, but most inference pipelines still decide the visible past implicitly through recent windows, compressed caches, or external memory. This raises a direct question: can the choice of which past frames or segments to show to the decoder be treated as an explicit test-time decision? We answer this question with ReCR, a training-free method that formulates autoregressive video decoding as representation-guided context rewriting. At each step, ReCR builds a selected visible context from the generated history by using internal decoder representations to favor context units that match the global rollout state, avoid redundancy with the recent continuation, preserve temporal boundary coverage, and widen the range of spatial layouts in view. The selected units are then mapped onto a compact causal axis before decoding the next block. Across long-horizon text-to-video benchmarks, ReCR consistently improves multiple autoregressive backbones under matched visible-token budgets. Empirical analyses such as representation-space diagnostics and transition-count prompt switching further show that improving what remains visible is an effective path toward more stable long-video decoding.
Figures & tables
| Method | Quality block: 30s / 60s | |||||||
|---|---|---|---|---|---|---|---|---|
| Total Score | Dynamic Degree | Subject Consistency | Background Consistency | Temporal Flickering | Motion Smoothness | Aesthetic Quality | Imaging Quality | |
| Training-based Baselines | ||||||||
| Self-Forcing [ 26 ] | 79.09 / 78.19 | 40.12 / 33.45 | 96.75 / 96.67 | 95.63 / 95.82 | 95.93 / 96.02 | 96.31 / 97.17 | 56.01 / 54.21 | 70.49 / 66.44 |
| Self-Forcing++ [ 11 ] | 79.56 / 79.59 | 39.45 / 38.67 | 96.71 / 96.43 | 95.76 / 95.96 | 95.10 / 96.63 | 97.83 / 97.25 | 56.64 / 56.13 | 70.15 / 69.20 |
| LongLive [ 67 ] | 81.32 / 80.72 | 36.74 / 35.43 | 97.63 / 97.42 | 96.15 / 96.08 | 97.77 / 97.21 | 98.25 / 98.17 | 58.82 / 58.21 | 70.57 / 70.02 |
| Causal-Forcing [ 84 ] | 80.77 / 77.41 | 62.64 / 56.82 | 95.78 / 94.86 | 95.30 / 95.27 | 96.10 / 95.20 | 97.64 / 96.44 | 54.43 / 48.22 | 68.35 / 63.26 |
| Method | Semantic stability | Appearance / low-level drift | Temporal drift | |||||
|---|---|---|---|---|---|---|---|---|
| TA overall | TA tail | Aes. drift | AQ signed | SAT shift | SHARP tail | LPIPS tail | DINO tail | |
| Training-based Baselines | ||||||||
| Self-Forcing baseline | 24.14 | +2.47 | 5.81 | -3.61 | +5.76 | -335.23 | -4.74 | -0.31 |
| LongLive baseline | 26.45 | +0.37 | 3.05 | -1.01 | -2.11 | +42.55 | +1.30 | +0.45 |
| Causal-Forcing baseline | 23.59 | +4.11 | 9.70 | -8.66 | +7.02 | -316.66 | -3.18 | +1.64 |
| HiAR baseline | 26.27 | +0.45 | 3.67 | -1.78 | +1.72 | -259.35 | +1.52 | +0.09 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Block Latency |
|---|---|
| Native Backbones | |
| Self-Forcing [ 26 ] | 728 ms |
| LongLive [ 67 ] | 685 ms |
| Causal-Forcing [ 84 ] | 737 ms |
| Infinity-RoPE [ 70 ] | 667 ms |
| Backbone + ReCR | |
Video Generation
Video generation studies models that create and edit moving images from text, images, audio, and structured controls.
See all 383 papers