Execution Before Memory: Diagnosing Memory Failures in Video World Models

Weiyang Jin; Jiwen Yu; Sihan Xu; Yuwei Niu; Xihui Liu

cs.CVOct 16, 2026

Execution Before Memory: Diagnosing Memory Failures in Video World Models

Authors: Weiyang Jin, Jiwen Yu, Sihan Xu, Yuwei Niu, Xihui Liu

Organizations: University of Hong Kong (hku.hk) · University of Michigan - Ann Arbor (umich.edu) · Peking University (pku.edu.cn)

Abstract

A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.

Figures & tables

Figure 1: Execution before memory. Left: the three settings, each a clip in which the camera looks away before the change is due. Right: MIST scores the gates of the reveal; M3-Agent reads the last frame, names the failed move, and revises only that move on the same frozen generator. Bottom: what each setting shows once the gates are scored.

Figure 2: The last frame does not show the asked-for result. Three settings, not one run. (Top) WRBench: the child walks off before sitting, so the chair is empty when the camera comes back. (Middle) MBench-T: the camera returns to candy that is still whole. (Bottom) MIST: two baseline clips that stay closed, each next to the same case after the instruction is revised, where the box opens onto the hidden objects.

Figure 3: MIST scores each gate of the reveal; M3-Agent revises the failed move until the change is shown. One Internal illustration. Generators in this paper include Wan TI2V-5B, Wan T2V-A14B, and MiniMax-H3 ( Sections 6 and C ). (a) Setup and leak check the first two shots; reveal and inside split the last shot into the factors of Eq. ( 1 ). The raw take opens the tin onto another object. (b) Diagnose the last frame, revise the move rather than the answer, and generate again with the same model. The rewrite puts in the orange marble and opens to it; the original stays a candidate.

	Target state	Required visible operation
R0	Presence, absence, count, final state	Open, reveal, or hold the right objects
R1	Object–container or left–right binding	Place the object in the named slot or box
R2	History update: swap, remove, transform	Execute and reveal the updated state

Table 1: MIST varies both the target state and the operation needed to reveal it.

	Base	Rewrite	$\boldsymbol{\Delta}$
Setup	0.852	0.851	$-0.001$
Reveal	0.147	0.492	+0.344
Inside match	0.047	0.442	+0.395
Never opens	41	20	$-21$
Empty / blocked	1	0	$-1$
Open, wrong	0	10	+10

Table 2: The rewrite changes the reveal gate, not the setup. One kept clip per case, not best-of-five. Decimals are mean judge scores; counts use one ordered label per clip (closed

\succ

empty

\succ

wrong

\succ

correct). Inside match need not reveal; Correct must.

Arm	Ans.?	Reveal	Inside	Closed	Ok
Look, then rewrite	yes	0.763	0.721	5	29
Answer only	yes	0.684	0.581	3	24
Same sentence	yes	0.635	0.512	4	21
Open and hold	no	0.268	0.093	6	3
Rewrite, no answer	no	0.174	0.047	28	2

Table 3: With five tries, the rewrite helps only when the answer is also in the prompt. Same judge; each arm keeps the try with the highest last-frame score. Closed / Ok count never-open and fully correct kept clips. Look-then-rewrite beats answer-only; the same rewrite with the target withheld falls below open-and-hold.

Figure 4: Rewriting the failed move opens more lids, but only when the target is named. (a) Failure classes for one kept clip per case (Table 2 ). (b) Best-of-five reveal (dark) and inside (light) for the five arms of Table 3 ; hatched arms have no target in the prompt. (c) Kept-clip reveal by operation family, with the lids that stay closed after the rewrite. (d) Best reveal so far as the five-try budget grows; dashed lines have no target; the arrow marks the same rewrite without the target.

Clip	$\boldsymbol{U}$	State	Corr.	Trig.
Baseline	0.292	0.350	0.233	0.93 / 0.83
Force leave–return	0.251	0.242	0.260	0.93 / 0.87
Force last-frame lock	0.228	0.208	0.247	0.97 / 0.97
Keep-best	0.446	0.492	0.400	1.00 / 1.00

Table 4: Official MBench-T: selection can refuse a harmful rewrite. Keep-best raises

U

by 52.7%; one clip per item gives

0.443

. Trig. is the state/correctness fire rate. State here is not MIST reveal.

Clip	Vis.	D5	D6	$\boldsymbol{T}$
Baseline	0.648	16/40	18/40	0.452
One leave–return	0.690	23/40	14/40	0.496
Best-looking	0.755	20/40	13/40	0.514
Search, first set	0.681	29/40	23/40	0.550
+ second set	0.662	28/40	27/40	0.547
+ last step (kept)	0.654	31/40	31/40	0.570

Table 5: Official WRBench: the search opens more return gates than visual quality. Kept clip raises

T

by 26.1%.

T

zeros a missing return. Best-looking differs from the

T

-kept clip on 24 of 40 items.

Figure 5: The same diagnosis has different consequences on the two official suites. (a) Share of the WRBench

T

rise from visual quality, the departure check

D5^{\prime}

, and the return check

D6^{\prime}

. (b) WRBench departure and return gates that stay open, newly open, newly shut, or stay shut between the baseline and the kept clip. (c) MBench-T items whose kept clip is the baseline, the leave–return instruction, or the last-frame lock. Panels keep separate denominators.

Input	122B	8B
Gold text only	n/a	0.91
Setup caption only	n/a	0.40
Shuffled gold only	n/a	0.07
Last frame, baseline	0.023	0.09
Last frame, rewrite	0.442	0.56

Table 6: A judge can name the target from text before the video shows it. YES rates. The top block is text only; the bottom block is the locked last frame. The 122B text-only cells were not run.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Generator	Clip	$\boldsymbol{n}$	Scorer	Reported
MIST	Wan TI2V-5B	$3{\times}5$ s	43	Internal	setup / leak / reveal / inside
H3 pilot	MiniMax-H3	packed 15s	43	Internal	failure classes (qual.)
WRBench	Wan T2V-A14B	10s, yaw_LR	40	Official	$T=(D3{+}D4{+}D5^{\prime}{+}D6^{\prime})/4$
MBench-T	Wan T2V-A14B	10s, 5 captions	30	Official	$U=(\mathrm{state}{+}\mathrm{corr})/2$

Appendix

Table 7: Three settings, three scorers.

T

averages WRBench

D3

D4

D5^{\prime}

D6^{\prime}

(leave/return);

U

averages MBench-T state and correctness. A missing return or untriggered clip scores zero.

	MIST (Internal)	WRBench unique	MBench-T unique
Backbone	TI2V-5B, 3 $\times$ 5s	T2V-A14B, 161f/10s	T2V-A14B, 161f/10s
Prompt	hide $s_{T}^{\star}$ in the gap	official yaw_LR	native 5 captions
Gold in caption?	baseline no; rewrite arm yes	task-context none	no; a lock is an intervention
Judge	Qwen3.5-122B	official scorer	official mbench
Score	reveal / inside	$T=(D3{+}D4{+}D5^{\prime}{+}D6^{\prime})/4$	$U=(\mathrm{state}{+}\mathrm{corr})/2$
N/A	never-open is not forgetting	missing return scores 0 in $T$	untriggered scores 0 in $U$

Appendix

Table 8: What each track actually runs. Do not add a number from one column into another.

$k$	Pass mean	Reveal best	Inside of chosen	Closed	Wrong	Correct	$\boldsymbol{\geq 0.6}$
1	0.371	0.371	0.256	19	14	10	10/43
2	0.399	0.546	0.442	10	17	16	16/43
3	0.484	0.700	0.628	8	10	25	25/43
4	0.444	0.720	0.651	7	10	26	26/43
5	0.560	0.763	0.721	5	9	29	28/43

Appendix

Table 9: Complete MIST running-best budget. Per-pass means are not monotone. “Inside of chosen” evaluates the reveal-argmax clip and is not the kept-clip inside

0.442

What we do	Vis.	D5	D6	$\boldsymbol{T}$
Camera only	0.648	16/40	18/40	0.452
Search over first skill	0.681	29/40	23/40	0.550
${+}$ second skill	0.662	28/40	27/40	0.547
${+}$ last step (kept)	0.654	31/40	31/40	0.570
One skill, no search	0.690	23/40	14/40	0.496
Peek gate, then vis.	0.710	35/40	21/40	0.586

Appendix

Table 10: WRBench search budget. Same clips as Table 5 . Adding a second skill is not monotone.

Metric	Baseline	Fixed leave–return	Keep-best
Prompt score	31.528	31.382	32.303
State trigger	0.933	0.933	0.967
State conditional	0.375	0.259	0.474
Correctness trigger	0.833	0.867	0.967
Correctness conditional	0.280	0.300	0.352

Appendix

Table 11: Official MBench-T five-segment details. Trigger and conditional scores are separate. They are not MIST metrics.

Selector	WRBench $T$ ( $n{=}40$ )	95% CI $\Delta$	MBench-T $U$ ( $n{=}30$ )	95% CI $\Delta$
First clip (baseline)	0.452	n/a	0.292	n/a
First rewrite only	0.496	$[-0.027,+0.114]$	0.251	$[-0.172,+0.094]$
Random, 20 draws	$0.487\pm 0.019$	$[-0.008,+0.076]$	$0.265\pm 0.030$	$[-0.101,+0.027]$
Blind 8B, last frame	0.466	$[-0.044,+0.070]$	0.248	$[-0.107,+0.007]$
Keep-best (sees scorer)	0.570	$[+0.070,+0.166]$	0.443	$[+0.074,+0.243]$

Appendix

Table 12: Official

T

and

U

when the selector cannot see the scorer. Same candidate clips as the main tables. Intervals are paired differences from the baseline. Do not add across columns.

Figure 6: Official-score component plots. (a–b) WRBench

T

against

D5^{\prime}

and visual quality. (c) MBench

U

against returned state. Because these totals contain the plotted components, the panels describe score composition. They do not establish an independent causal correlation.

Observation	Supported reading	Not established
Valid setup, clean gap, closed reveal	The visible reveal path is missing	The hidden state was forgotten
$41/43$ never-open, TI2V-5B $3{\times}5$ s	Reveal path missing in this Internal stitch	Every generator or packed 15s take fails to open
Bo5 rewrite $0.763$ vs answer-only $0.684$ ; target withheld $0.174$	The rewrite helps only with the answer in the prompt; keep-best is an oracle	Diagnosis separable from the target in the prompt
Blind 8B selector $T$ $0.466$ , $U$ $0.248$	Scorer-free selection does not beat the baseline	A deployable selector for either suite
Official keep-best $T$ / $U$ rises	Oracle coverage improves for that protocol	A deployable critic, or one intervention that transfers
H3 $n{=}6$ Internal 8B reveal $1.000$	The same cards can be packed as 15s	Official $T$ / $U$ , stored $s_{T}^{\star}$ , or a scale claim

Appendix

Table 13: Evidence boundaries. Each row is an observation already reported in the main text. The middle column is the reading used in this paper. The right column is not established.

Cell	Generator	$n$	Judge	What it is
MIST kept reveal	TI2V-5B $3{\times}5$ s	43	122B	$0.147\!\rightarrow\!0.492$ ; Internal diagnosis
MIST best-of-five	same	43	122B	$0.763$ ; oracle, answer in the prompt
Official $T$	A14B, 10s	40	official	$0.452\!\rightarrow\!0.570$ ; leave–return
Official $U$	A14B, 10s	30	official	$0.292\!\rightarrow\!0.446$ ; mostly keep baseline
H3 $n{=}6$ reveal	H3, 15s packed	6	Internal 8B	$0.435\!\rightarrow\!1.000$ ; a six-case pilot
H3-43 continuity	H3, 15s packed	43	MAE $\geq 22$	$0.605\!\rightarrow\!0.721$ ; not memory

Appendix

Table 14: Same cards, different experiments. Do not add across rows. Continuity is not reveal.

Arm	Reveal	Inside	Never-open	Empty	Wrong	Ok
H3 raw	0.387	0.256	3	6	24	10
H3 + M3	0.729	0.674	6	1	8	28

Appendix

Table 15: Internal 43-case H3 reveal. Locked 122B, same positions as MIST. Inside is the match rate of the exposed state. Not the stitch population, and not Official.

Arm	$n$	Harness pass	Mean max MAE
H3 raw	43	0.605	19.8
H3 + M3	43	0.721	17.93

Appendix

Table 16: Internal 43-case H3 continuity. Adjacent-frame harness. A hard cut is gray MAE

\geq 22

. Not a memory score.

Family	$n$	Raw pass	M3 pass	Both pass
R0	16	0.625	0.688	9
R1	11	0.545	0.818	5
R2	16	0.625	0.688	5
All	43	0.605	0.721	19

Appendix

Table 17: Internal 43-case continuity by family. Adjacent-frame harness only. Not a memory score, and not Table 18 .

Pipeline / arm	Overall	Establish	Middle shot	Reveal
H3 raw	0.500	0.682	0.785	0.435
H3 + prompt expansion	0.937	0.842	0.503	1.000
H3 + M3	0.937	0.747	0.685	1.000
H3 + M3 + expansion	0.855	0.829	0.665	0.879
Lingbot raw	0.035	0.008	0.025	0.042
Lingbot + expansion	0.311	0.790	0.569	0.190

Appendix

Table 18: Six-case Internal transfer pilots. Small

n

, a changed generator, and an Internal judge. Not a model-scale result and not Official

T

U

Figure 7: H3 raw versus M3 on the same cards. Each panel is put-in $\rightarrow$ closed gap $\rightarrow$ open/hold; top row raw, bottom row M3. (a) R0-03 (also Fig. 3 ). (b–c) other leftover high-pass takes. (d) leftover fail: both arms break continuity and show the wrong object. (e–f) seed-61001 43-case pairs, not the leftover pack. Internal qualitative evidence only—not Official $T$ / $U$ , not Table 18 , and not reveal scores.

Suite	Cell	Base	M3	$\boldsymbol{\Delta}$	Track
WRBench/40	Official $T$	0.452	0.570	+0.118	Official
WRBench/40	Official vis. (other clip)	0.648	0.755	+0.107	Official
MBench-T/30	Official $U$	0.292	0.446	+0.154	Official
WRBench/40	Internal reveal	0.372	0.668	+0.296	Internal
MBench-T/12	Internal unique	0.230	0.426	+0.196	Internal

Appendix

Table 19: Official and Internal scores on the same suite names. Do not add across rows. WRBench

T

scores a missing return as zero. The visual-quality row uses a different clip on 24 of 40 items.

Set	8B YES	Reference rate	Agree
MIST last, base (vs 122B inside)	0.09	0.023	0.93
MIST last, M3 (vs 122B inside)	0.56	0.442	0.88
MIST last, base (vs 122B act)	0.09	0.070	0.84
MIST last, M3 (vs 122B act)	0.56	0.535	0.51
WRBench unique last (vs D5 gate)	0.33	0.78	0.50
WRBench unique last (vs D6 gate)	0.33	0.78	0.35

Appendix

Table 20: Held-out 8B against the locked 122B judge and the official gates. MIST inside agreement is high under class imbalance. An 8B YES is not official

D5

D6

, or MBench state.

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 212 papers

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
Oct 13, 2026Nan Zhou, Jianxiong Gao, Qiuyu Wang +10World Model Evaluation Video World Models
Zhejiang University · Ant Group · Fudan University +1
MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
May 30, 2026Shengjun Zhang, Zhang Zhang, Simin Huang +11Long-Horizon Video Generation World Model Evaluation
1Tsinghua University · 2WeChat Vision, Tecent Inc. · 3Peking University
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Jun 25, 2026Haoyu Chen, Kaichen Zhou, Hang Hua +11Physical Consistency in Video Generation World Model Evaluation
Harvard University · MIT · MIT-IBM Watson AI Lab +5

Oct 13, 2026J/K move · Enter open · S save

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

Nan Zhou, Jianxiong Gao, Qiuyu Wang, Wen Wang+9

Zhejiang University · Ant Group · Fudan University+1

World Model Evaluation Video World Models Temporal Consistency in Video Generation

Open paper AskPDF arXiv

As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.

Abstract

World Model Evaluation

Explore similar work

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

MemoBench: Benchmarking World Modeling in Dynamically Changing Environments