Tether the Subject, Release the Scene: Query-Aware Memory Routing for Long-Horizon Autoregressive Video Generation

Chen Li; Peng Zhang; Hanyu Zhou; Jialong Zuo; Fei Wang; Daiguo Zhou; Nong Sang; Changxin Gao

cs.CVAug 27, 2026

Tether the Subject, Release the Scene: Query-Aware Memory Routing for Long-Horizon Autoregressive Video Generation

Authors: Chen Li, Peng Zhang, Hanyu Zhou, Jialong Zuo, Fei Wang, Daiguo Zhou, Nong Sang, Changxin Gao

Organizations: Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc.

Abstract

Streaming autoregressive video models generate long videos chunk by chunk, using historical memory to maintain consistency. Existing methods typically expose subject and scene queries to history through similar policies. This stabilizes the subject, but can also lock backgrounds, viewpoints, and scene structure to previously generated states even when local motion continues. We call this failure memory-anchored scene under-progression; consistency and motion metrics alone can miss it. We introduce TetherMem, a training-free, query-aware spatiotemporal memory router for frozen video generators. TetherMem separates subject and scene queries and modulates historical access with region- and age-conditioned priors: subject queries retain identity-bearing history, while scene queries reduce reliance on subject history and stale backgrounds. Across 2,400 blinded pairwise judgments from 10 annotators, TetherMem achieves the highest estimated expected preference among eight streaming long-video baselines for overall quality (0.780) and scene progression (0.769). On complete 30-second videos, it sustains changes in background, viewpoint, and scene state while preserving subject recognizability and temporal continuity.

Figures & tables

Figure 1: Memory-anchored scene under-progression. This matched prompt–seed trajectory illustrates distinct long-horizon scene dynamics across four generation methods, with TetherMem continuing to reveal new scene states through 120 seconds. Human Preference reports the aggregate 30-second scene-progression expected preference from Table 1 . VBench Subject Consistency ( SC ) measures subject preservation, Dynamic Degree ( DD ) captures overall motion, and nT is an auxiliary scene-progression diagnostic.

Figure 2: Overview of TetherMem. (a) End-to-end routing preserves identity memory for subject queries and favors recent backgrounds for scene queries. (b) Prior methods select memory frames, whereas TetherMem routes query roles within the retrieved history.

Method	5s	30s human EP $\uparrow$				Diag.	VBench $\uparrow$
	Img5 $\uparrow$	Ovr.	Prog.	ID	L1	nT $\uparrow$	Subj.	Back.	Smooth.	Dyn.
Base model (non-autoregressive)
Wan2.1-T2V-1.3B	65.6	–	–	–	–	0.322	–	–	–	–
Long-memory / retrieval
LongLive-RAG	71.0	0.615	0.608	0.507	0.587	0.868	0.861	0.897	0.973	0.867
Forcing-based generation

Table 1: Main comparison with human preference and VBench diagnostics. Human values are tie-aware Davidson EP on complete 30-second videos. VBench uses 270 evaluation videos: Subj./Back. use complete clips, while Smooth./Dyn. use a uniform 8-second trim. Bold and underlined values mark best and second-best; ID excludes the subject-free P05 prompt.

Criterion	TetherMem EP (95% CI)	Strongest baseline	Baseline EP	Difference (95% CI)
Ovr.	0.780 [0.684, 0.868]	MemRoPE	0.640	0.140 [0.004, 0.273]
Prog.	0.769 [0.666, 0.861]	LongLive-RAG	0.608	0.161 [0.076, 0.240]
ID	0.600 [0.504, 0.699]	Rolling Forcing	0.633	$-0.033$ [ $-0.182$ , 0.111]
L1	0.708 [0.625, 0.794]	MemRoPE	0.666	0.042 [ $-0.086$ , 0.182]

Table 2: Statistical reliability across evaluation units and prompts. (a) Percentile intervals from 2,000 crossed annotator–prompt-seed cluster bootstrap refits. (b) A stricter analysis clusters the three seeds of each prompt before resampling; 1,995 of 2,000 refits are finite.

(a) Routing components: Full-Memory-centered comparison
Variant	Ovr.	Prog.	ID	L1	nT $\uparrow$	Tail subj. $\uparrow$
Full Memory	0.426	0.391	0.472	0.457	0.980	–
Routing w/o Region	0.461	0.472	0.489	0.505	1.232	–
Routing w/o Age	0.476	0.428	0.505	0.490	1.083	–
TetherMem	0.638	0.709	0.534	0.548	1.458	–
(b) Post-attention design baseline

Table 3: Routing-component and design ablations. Human columns are regularized, tie-aware Davidson EP fitted separately within each panel. (a) Four-way routing comparison over 14 prompt–seed units. (b) Three-way design comparison over seven prompts and two seeds; nT and final-five-second subject consistency are averaged over 14 cells.

Figure 3: Compact memory-control and cross-method comparison. Top four rows: matched P01 trajectories at 0/5/10/20/27/30 seconds for Full Memory, the two single-factor routing controls, and TetherMem. Bottom four rows: paired P10 and P05 trajectories at 0/12/27 seconds, followed by two independent P09 Painter Coast comparisons at 32/42/52 seconds. The extended 80-frame gallery with denser temporal sampling appears in Appendix Figure 7 .

Figure 4: The evaluation landscape exposes two distinct gaps. (a) Identity EP versus scene-progression EP shows that strong identity can coexist with weak progression. (b) nT versus human progression shows that coherent translation is informative but incomplete: Causal Forcing has high nT and low human progression. Values are the method-level aggregates from Table 1 .

Figure 5: Realized routing behavior. A Regional routing reduces cross-region historical access; the two routed variants have closely matched regional profiles. B Within background memory, TetherMem redistributes attention toward the recent half relative to Routing w/o Age. Points and bars show means with video-level bootstrap 95% CIs over seven videos.

Figure 6: Long-horizon trajectories and cross-stack transfer. A Matched LongLive-RAG and TetherMem trajectories for P03, P07, and P10 at 2–28 seconds. B Deep Forcing versus Deep Forcing+TetherMem for P04, P08, and P09 at 0–29 seconds; the Wan2.1 foundation family is held fixed while the autoregressive host stack changes.

	30s human EP $\uparrow$
Method	Ovr.	Prog.	ID	L1
Deep Forcing	0.404	0.360	0.470	0.439
Deep Forcing + TetherMem	0.596	0.640	0.530	0.561

Table 4: Blinded transfer to the Deep Forcing host. Thirty matched prompt–seed pairs receive three judgments each. The progression difference is 0.281 with crossed-cluster 95% CI [0.033, 0.522].

Prior	nT $\uparrow$	Artifact veto $\downarrow$	Tail subject $\uparrow$
Original SAM 2	1.490	0.158	0.934
Low-res coarse	1.463	0.107	0.932
Bounding box	1.664	0.207	0.878

Table 5: Sensitivity to subject-prior approximation. Automatic diagnostics over seven matched prompt-videos (one seed). Artifact veto is the fraction flagged by the artifact classifier; Tail subject is subject consistency in the final five seconds.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Extended memory-control and cross-method gallery. The top five rows show eight sampled times for Full Memory, Value Reweighting, Routing w/o Region, Routing w/o Age, and TetherMem on P01. The bottom rows compare CausVid/TetherMem on P10, LongLive-RAG/TetherMem on P05, Rolling Forcing/TetherMem on P07, and two P09 Painter Coast trajectories.

Figure 8: Automatic subject priors at the routing grid. Prompt blocks: P10, P04, and P03. Rows: original SAM 2, coarse, and bounding-box priors. Columns: six frames from 3.7 to 25.9 seconds. Cyan overlays show the binary $30\times 52$ routing prior.

Baseline	W	T	L	U	Half	Dec.
CausVid	254	8	27	11	0.893	0.904
Causal Forcing	259	22	12	7	0.922	0.956
Deep Forcing	207	25	59	9	0.754	0.778
LongLive-RAG	157	92	49	2	0.681	0.762
MemRoPE	173	41	80	6	0.658	0.684
Reward Forcing	193	29	67	11	0.718	0.742

Appendix

Table 6: Direct observed outcomes from TetherMem’s perspective. (a) Overall preference, including the half-tie score (Half) and decided-only rate (Dec.). (b) Spatial and scene progression (L3) and prompt-directed progression (L4). W/T/L/U denote win/tie/loss/unsure. Prog. in the main text averages the separately fitted L3 and L4 EP scores rather than pooling these counts.

Method	Checkpoint	Sampling	Chunk/context/memory	Method-specific settings
LongLive-RAG	causal_forcing.pt + ae_latent_mem.pt	4 warped steps: 1000/750/500/250	block 3; local 12; sink 1; memory 6; exclude-recent 5	AE compression; top- $k$ retrieval
Self Forcing	self_forcing_dmd.pt	4 warped steps: 1000/750/500/250; EMA	block 3; full KV attention	timestep shift 5.0
Rolling Forcing	rolling_forcing_ dmd.pt	5 warped steps: 1000/800/600/400/200; EMA	block 3; rolling-window length 5	window advances one block
Deep Forcing	self_forcing_dmd.pt	4 warped steps: 1000/750/500/250; EMA	block 3; local 21; sink 14	Budget 16 / Recent 4 supplied by the launch command
Causal Forcing	causal_forcing.pt	4 warped steps: 1000/750/500/250	block 3; local 12; sink 1; retrieval disabled	executed through the LongLive-RAG code path without latent memory
Reward Forcing	rewardforcing.pt	4 warped steps: 1000/750/500/250; EMA	block 3; local 9; sink 3	timestep shift 5.0

Appendix

Table 7: Executed inference configurations. Settings correspond to the runs reported in Table 1 .

Method group	Frames	Duration	30-second adaptation
LongLive-RAG, Causal Forcing, TetherMem	474	29.625 s	120 latent frames; native single-pass causal rollout
Self Forcing, Rolling Forcing, Deep Forcing, Reward Forcing; MemRoPE	477	29.813 s	120 latent frames; native single-pass or rolling KV rollout
CausVid	480	30.000 s	seven overlapping rollouts yield 504 frames, then trim and H.264 re-encode (CRF 18)

Appendix

Table 8: Final output adaptation. No method uses looping, frame-rate conversion, or spatial rescaling. CausVid alone applies the listed trim and re-encode.

Query	Historical key	Prior
Subject	Subject	$1$
Subject	Background	$\gamma_{n}$
Background	Subject	$\gamma_{n}$
Background	Background	$\rho_{i}$
Any	Local/sink	$1$

Appendix

Table 9: Routing priors. Priors are converted to additive logit biases before softmax.

Figure 9: Reference-prior drift across the 30 evaluation videos. (a) Median and IQR over time. (b) Late IoU by prompt and seed. (c) Correlations with human scores. (d) Low, median, and high Late-IoU examples at 5, 17, and 28 seconds; cyan is the reference prior and orange is the controlled-output re-extraction.

Figure 10: Human mask comparison for the reference prior and controlled output. (a) Agreement at 5, 17, and 28 seconds. (b) Paired comparison on 21 subject-present frames at 28 seconds. (c) False-positive area on six target-absent frames. (d) Representative low, median, and high agreement cases. Contours: human mask (purple), reference prior (blue), and controlled re-extraction (orange).

28-s metric (subject present, $n=21$ )	Ref. prior	Ctrl.-auto
Subject coverage $\uparrow$	0.574	1.000
Prior precision $\uparrow$	0.238	0.550
Raw semantic IoU $\uparrow$	0.202	0.520
Route-envelope IoU $\uparrow$	0.387	0.956
Normalized centroid distance $\downarrow$	0.138	0.038

Appendix

Table 10: Mask alignment at 28 seconds. Entries are medians over the 21 subject-present frames. “Ref. prior” is the consumed

M^{\mathrm{ref}}_{\mathrm{used}}

; “Ctrl.-auto” is the controlled-output re-extraction.

Stage	Time (s)	Share	Peak memory
Reference rollout	$72.6\pm 7.5$	24.3%	30.5 GB
SAM 2 extraction	$24.9\pm 1.0$	8.3%	shared
Controlled rollout	$201.9\pm 3.8$	67.4%	30.5 GB
End-to-end	$\mathbf{299.4\pm 10.0}$	100%	30.5 GB
Full Memory, one pass	$72.6\pm 7.5$	–	30.5 GB

Appendix

Table 11: Runtime of the offline two-pass pipeline. Mean

\pm

standard deviation over seven approximately 30-second videos.

Figure 11: Output sensitivity to subject-prior approximation. Prompt blocks: P10, P04, and P03. Rows: original, coarse, and bounding-box priors. Columns: frames at 4, 10, 16, 22, and 28 seconds. All videos share the same seed and settings.

Variant	Ovr.	Prog.	ID	L1
Routing w/o Region	14/16/12	15/17/10	12/18/11	14/17/11
Routing w/o Age	15/15/12	12/20/10	12/19/10	14/16/12
TetherMem	25/6/11	29/5/8	18/9/14	19/10/13

Appendix

Table 12: Full-memory-centered routing ablation. (a) Direct win/tie/loss counts from each row variant’s perspective. (b) Regularized Davidson EP with nT averaged over two seeds.

Figure 12: Participant-facing blind-evaluation form. The anonymized view shows the prompt, video pair, L1–L4 and Overall questions, and optional diagnostics for one Table 1 task.

Long-Horizon Video Generation

Long-horizon video generation creates coherent, controllable videos over extended durations despite limited context and accumulating errors.

See all 164 papers

TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment
Jun 11, 2026Yu Meng, Xiangyang Luo, Letian Li +5Long-Horizon Video Generation Diffusion Model Caching
Tsinghua University · D-INFK, ETH Zürich
Recast the Past: Stable Long Video Generation with Full Context Memory
Oct 20, 2026Yukuo Ma, Cong Liu, Junke Wang +4Long-Horizon Video Generation Memory-Augmented Video Generation
Fudan University · Institute of Artificial Intelligence (TeleAI), China Telecom
MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation
Oct 1, 2026Yiwen Zhang, Haocheng Xi, Michael Tian-Yue Liu +4Image-to-Video Generation Long-Horizon Video Generation
Cornell University · Impossible, Inc. · University of California, Berkeley +1

Jun 11, 2026J/K move · Enter open · S save

TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment

Yu Meng, Xiangyang Luo, Letian Li, Wenyuan Jiang+4

Tsinghua University · D-INFK, ETH Zürich

Long-Horizon Video Generation Diffusion Model Caching Autoregressive Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Autoregressive video diffusion models provide a natural formulation for streaming and variable-length video generation by conditioning newly generated frames on previously generated content. However, extending these models to minute-level generation remains challenging: the limited KV-cache budget prevents the model from retaining the full history, while repeatedly conditioning on self-generated frames induces a context distribution shift that accumulates over time, leading to visual artifacts, quality degradation, and temporal drift. In this paper, we propose TetherCache, a training-free and plug-and-play cache management strategy for drift-resistant long video generation. TetherCache organizes the cache into sink, memory, and recent regions, and introduces two complementary mechanisms. First, GRAB (Gated Recall with Attention-Diversity Balancing) selects long-range memory frames using a gated score that combines attention-based relevance with temporal diversity, preserving informative yet diverse historical context under a fixed cache budget. Second, TAME (Trusted Alignment via Memory Editing) lightly edits newly recalled memory tokens by aligning their statistics to a trusted context distribution, reducing the pollution caused by drifted historical features. Built on Self-Forcing, TetherCache consistently improves long-video generation quality on VBench-Long across 30s, 60s, and 240s settings. In particular, for 240s generation, it substantially improves overall and semantic scores while reducing quality drift from 7.84 to 1.33, demonstrating its effectiveness for stable long-horizon autoregressive video diffusion.

Abstract

Long-Horizon Video Generation

Explore similar work

TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment

Recast the Past: Stable Long Video Generation with Full Context Memory

MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation