A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.
Figures & tables
Figure 1: Execution before memory. Left: the three settings, each a clip in which the camera looks away before the change is due. Right: MIST scores the gates of the reveal; M3-Agent reads the last frame, names the failed move, and revises only that move on the same frozen generator. Bottom: what each setting shows once the gates are scored.
Figure 2: The last frame does not show the asked-for result. Three settings, not one run. (Top) WRBench: the child walks off before sitting, so the chair is empty when the camera comes back. (Middle) MBench-T: the camera returns to candy that is still whole. (Bottom) MIST: two baseline clips that stay closed, each next to the same case after the instruction is revised, where the box opens onto the hidden objects.
Figure 3: MIST scores each gate of the reveal; M3-Agent revises the failed move until the change is shown. One Internal illustration. Generators in this paper include Wan TI2V-5B, Wan T2V-A14B, and MiniMax-H3 ( Sections 6 and C ). (a) Setup and leak check the first two shots; reveal and inside split the last shot into the factors of Eq. ( 1 ). The raw take opens the tin onto another object. (b) Diagnose the last frame, revise the move rather than the answer, and generate again with the same model. The rewrite puts in the orange marble and opens to it; the original stays a candidate.
Target state
Required visible operation
R0
Presence, absence, count, final state
Open, reveal, or hold the right objects
R1
Object–container or left–right binding
Place the object in the named slot or box
R2
History update: swap, remove, transform
Execute and reveal the updated state
Table 1: MIST varies both the target state and the operation needed to reveal it.
Base
Rewrite
Δ
Setup
0.852
0.851
−0.001
Reveal
0.147
0.492
+0.344
Inside match
0.047
0.442
+0.395
Never opens
41
20
−21
Empty / blocked
1
0
−1
Open, wrong
0
10
+10
Table 2: The rewrite changes the reveal gate, not the setup. One kept clip per case, not best-of-five. Decimals are mean judge scores; counts use one ordered label per clip (closed ≻ empty ≻ wrong ≻ correct). Inside match need not reveal; Correct must.
Arm
Ans.?
Reveal
Inside
Closed
Ok
Look, then rewrite
yes
0.763
0.721
5
29
Answer only
yes
0.684
0.581
3
24
Same sentence
yes
0.635
0.512
4
21
Open and hold
no
0.268
0.093
6
3
Rewrite, no answer
no
0.174
0.047
28
2
Table 3: With five tries, the rewrite helps only when the answer is also in the prompt. Same judge; each arm keeps the try with the highest last-frame score. Closed / Ok count never-open and fully correct kept clips. Look-then-rewrite beats answer-only; the same rewrite with the target withheld falls below open-and-hold.
Figure 4: Rewriting the failed move opens more lids, but only when the target is named. (a) Failure classes for one kept clip per case (Table 2 ). (b) Best-of-five reveal (dark) and inside (light) for the five arms of Table 3 ; hatched arms have no target in the prompt. (c) Kept-clip reveal by operation family, with the lids that stay closed after the rewrite. (d) Best reveal so far as the five-try budget grows; dashed lines have no target; the arrow marks the same rewrite without the target.
Clip
U
State
Corr.
Trig.
Baseline
0.292
0.350
0.233
0.93 / 0.83
Force leave–return
0.251
0.242
0.260
0.93 / 0.87
Force last-frame lock
0.228
0.208
0.247
0.97 / 0.97
Keep-best
0.446
0.492
0.400
1.00 / 1.00
Table 4: Official MBench-T: selection can refuse a harmful rewrite. Keep-best raises U by 52.7%; one clip per item gives 0.443 . Trig. is the state/correctness fire rate. State here is not MIST reveal.
Clip
Vis.
D5
D6
T
Baseline
0.648
16/40
18/40
0.452
One leave–return
0.690
23/40
14/40
0.496
Best-looking
0.755
20/40
13/40
0.514
Search, first set
0.681
29/40
23/40
0.550
+ second set
0.662
28/40
27/40
0.547
+ last step (kept)
0.654
31/40
31/40
0.570
Table 5: Official WRBench: the search opens more return gates than visual quality. Kept clip raises T by 26.1%. T zeros a missing return. Best-looking differs from the T -kept clip on 24 of 40 items.
Figure 5: The same diagnosis has different consequences on the two official suites. (a) Share of the WRBench T rise from visual quality, the departure check D5′ , and the return check D6′ . (b) WRBench departure and return gates that stay open, newly open, newly shut, or stay shut between the baseline and the kept clip. (c) MBench-T items whose kept clip is the baseline, the leave–return instruction, or the last-frame lock. Panels keep separate denominators.
Input
122B
8B
Gold text only
n/a
0.91
Setup caption only
n/a
0.40
Shuffled gold only
n/a
0.07
Last frame, baseline
0.023
0.09
Last frame, rewrite
0.442
0.56
Table 6: A judge can name the target from text before the video shows it. YES rates. The top block is text only; the bottom block is the locked last frame. The 122B text-only cells were not run.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Generator
Clip
n
Scorer
Reported
MIST
Wan TI2V-5B
3×5 s
43
Internal
setup / leak / reveal / inside
H3 pilot
MiniMax-H3
packed 15s
43
Internal
failure classes (qual.)
WRBench
Wan T2V-A14B
10s, yaw_LR
40
Official
T=(D3+D4+D5′+D6′)/4
MBench-T
Wan T2V-A14B
10s, 5 captions
30
Official
U=(state+corr)/2
Appendix
Table 7: Three settings, three scorers. T averages WRBench D3 , D4 , D5′ , D6′ (leave/return); U averages MBench-T state and correctness. A missing return or untriggered clip scores zero.
MIST (Internal)
WRBench unique
MBench-T unique
Backbone
TI2V-5B, 3 × 5s
T2V-A14B, 161f/10s
T2V-A14B, 161f/10s
Prompt
hide sT⋆ in the gap
official yaw_LR
native 5 captions
Gold in caption?
baseline no; rewrite arm yes
task-context none
no; a lock is an intervention
Judge
Qwen3.5-122B
official scorer
official mbench
Score
reveal / inside
T=(D3+D4+D5′+D6′)/4
U=(state+corr)/2
N/A
never-open is not forgetting
missing return scores 0 in T
untriggered scores 0 in U
Appendix
Table 8: What each track actually runs. Do not add a number from one column into another.
k
Pass mean
Reveal best
Inside of chosen
Closed
Wrong
Correct
≥0.6
1
0.371
0.371
0.256
19
14
10
10/43
2
0.399
0.546
0.442
10
17
16
16/43
3
0.484
0.700
0.628
8
10
25
25/43
4
0.444
0.720
0.651
7
10
26
26/43
5
0.560
0.763
0.721
5
9
29
28/43
Appendix
Table 9: Complete MIST running-best budget. Per-pass means are not monotone. “Inside of chosen” evaluates the reveal-argmax clip and is not the kept-clip inside 0.442 .
What we do
Vis.
D5
D6
T
Camera only
0.648
16/40
18/40
0.452
Search over first skill
0.681
29/40
23/40
0.550
+ second skill
0.662
28/40
27/40
0.547
+ last step (kept)
0.654
31/40
31/40
0.570
One skill, no search
0.690
23/40
14/40
0.496
Peek gate, then vis.
0.710
35/40
21/40
0.586
Appendix
Table 10: WRBench search budget. Same clips as Table 5 . Adding a second skill is not monotone.
Metric
Baseline
Fixed leave–return
Keep-best
Prompt score
31.528
31.382
32.303
State trigger
0.933
0.933
0.967
State conditional
0.375
0.259
0.474
Correctness trigger
0.833
0.867
0.967
Correctness conditional
0.280
0.300
0.352
Appendix
Table 11: Official MBench-T five-segment details. Trigger and conditional scores are separate. They are not MIST metrics.
Selector
WRBench T ( n=40 )
95% CI Δ
MBench-T U ( n=30 )
95% CI Δ
First clip (baseline)
0.452
n/a
0.292
n/a
First rewrite only
0.496
[−0.027,+0.114]
0.251
[−0.172,+0.094]
Random, 20 draws
0.487±0.019
[−0.008,+0.076]
0.265±0.030
[−0.101,+0.027]
Blind 8B, last frame
0.466
[−0.044,+0.070]
0.248
[−0.107,+0.007]
Keep-best (sees scorer)
0.570
[+0.070,+0.166]
0.443
[+0.074,+0.243]
Appendix
Table 12: Official T and U when the selector cannot see the scorer. Same candidate clips as the main tables. Intervals are paired differences from the baseline. Do not add across columns.
Figure 6: Official-score component plots. (a–b) WRBench T against D5′ and visual quality. (c) MBench U against returned state. Because these totals contain the plotted components, the panels describe score composition. They do not establish an independent causal correlation.
Observation
Supported reading
Not established
Valid setup, clean gap, closed reveal
The visible reveal path is missing
The hidden state was forgotten
41/43 never-open, TI2V-5B 3×5 s
Reveal path missing in this Internal stitch
Every generator or packed 15s take fails to open
Bo5 rewrite 0.763 vs answer-only 0.684 ; target withheld 0.174
The rewrite helps only with the answer in the prompt; keep-best is an oracle
Diagnosis separable from the target in the prompt
Blind 8B selector T0.466 , U0.248
Scorer-free selection does not beat the baseline
A deployable selector for either suite
Official keep-best T / U rises
Oracle coverage improves for that protocol
A deployable critic, or one intervention that transfers
H3 n=6 Internal 8B reveal 1.000
The same cards can be packed as 15s
Official T / U , stored sT⋆ , or a scale claim
Appendix
Table 13: Evidence boundaries. Each row is an observation already reported in the main text. The middle column is the reading used in this paper. The right column is not established.
Cell
Generator
n
Judge
What it is
MIST kept reveal
TI2V-5B 3×5 s
43
122B
0.147→0.492 ; Internal diagnosis
MIST best-of-five
same
43
122B
0.763 ; oracle, answer in the prompt
Official T
A14B, 10s
40
official
0.452→0.570 ; leave–return
Official U
A14B, 10s
30
official
0.292→0.446 ; mostly keep baseline
H3 n=6 reveal
H3, 15s packed
6
Internal 8B
0.435→1.000 ; a six-case pilot
H3-43 continuity
H3, 15s packed
43
MAE ≥22
0.605→0.721 ; not memory
Appendix
Table 14: Same cards, different experiments. Do not add across rows. Continuity is not reveal.
Arm
Reveal
Inside
Never-open
Empty
Wrong
Ok
H3 raw
0.387
0.256
3
6
24
10
H3 + M3
0.729
0.674
6
1
8
28
Appendix
Table 15: Internal 43-case H3 reveal. Locked 122B, same positions as MIST. Inside is the match rate of the exposed state. Not the stitch population, and not Official.
Arm
n
Harness pass
Mean max MAE
H3 raw
43
0.605
19.8
H3 + M3
43
0.721
17.93
Appendix
Table 16: Internal 43-case H3 continuity. Adjacent-frame harness. A hard cut is gray MAE ≥22 . Not a memory score.
Family
n
Raw pass
M3 pass
Both pass
R0
16
0.625
0.688
9
R1
11
0.545
0.818
5
R2
16
0.625
0.688
5
All
43
0.605
0.721
19
Appendix
Table 17: Internal 43-case continuity by family. Adjacent-frame harness only. Not a memory score, and not Table 18 .
Pipeline / arm
Overall
Establish
Middle shot
Reveal
H3 raw
0.500
0.682
0.785
0.435
H3 + prompt expansion
0.937
0.842
0.503
1.000
H3 + M3
0.937
0.747
0.685
1.000
H3 + M3 + expansion
0.855
0.829
0.665
0.879
Lingbot raw
0.035
0.008
0.025
0.042
Lingbot + expansion
0.311
0.790
0.569
0.190
Appendix
Table 18: Six-case Internal transfer pilots. Small n , a changed generator, and an Internal judge. Not a model-scale result and not Official T or U .
Figure 7: H3 raw versus M3 on the same cards. Each panel is put-in → closed gap → open/hold; top row raw, bottom row M3. (a) R0-03 (also Fig. 3 ). (b–c) other leftover high-pass takes. (d) leftover fail: both arms break continuity and show the wrong object. (e–f) seed-61001 43-case pairs, not the leftover pack. Internal qualitative evidence only—not Official T / U , not Table 18 , and not reveal scores.
Suite
Cell
Base
M3
Δ
Track
WRBench/40
Official T
0.452
0.570
+0.118
Official
WRBench/40
Official vis. (other clip)
0.648
0.755
+0.107
Official
MBench-T/30
Official U
0.292
0.446
+0.154
Official
WRBench/40
Internal reveal
0.372
0.668
+0.296
Internal
MBench-T/12
Internal unique
0.230
0.426
+0.196
Internal
Appendix
Table 19: Official and Internal scores on the same suite names. Do not add across rows. WRBench T scores a missing return as zero. The visual-quality row uses a different clip on 24 of 40 items.
Set
8B YES
Reference rate
Agree
MIST last, base (vs 122B inside)
0.09
0.023
0.93
MIST last, M3 (vs 122B inside)
0.56
0.442
0.88
MIST last, base (vs 122B act)
0.09
0.070
0.84
MIST last, M3 (vs 122B act)
0.56
0.535
0.51
WRBench unique last (vs D5 gate)
0.33
0.78
0.50
WRBench unique last (vs D6 gate)
0.33
0.78
0.35
Appendix
Table 20: Held-out 8B against the locked 122B judge and the official gates. MIST inside agreement is high under class imbalance. An 8B YES is not official D5 , D6 , or MBench state.
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.
Nan Zhou, Jianxiong Gao, Qiuyu Wang +10
Zhejiang University · Ant Group · Fudan University +1
Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primarily emphasize visual quality, motion coherence, and text-video alignment, they largely overlook memory, the core capability of a world model to preserve consistency across long-term horizons and complex interactions. To address this gap, we present MBench, a comprehensive benchmark dedicated to quantifying and evaluating the memory capability of video world models. We systematically decompose the memory capability of video world models into three hierarchical and complementary core dimensions: entity consistency, environment consistency, and causal consistency, which are further refined into 12 quantifiable sub-dimensions for comprehensive characterization of long-term memory. Our benchmark is built upon rigorously curated real-captured long videos, and evaluated by rule-based quantitative matrices and VLM to enable objective and comprehensive consistency assessment. Extensive evaluations of mainstream state-of-the-art video world models reveal critical systemic limitations of existing methods in long-term state retention, providing a standardized benchmark and clear research direction to advance the field.
Shengjun Zhang, Zhang Zhang, Simin Huang +11
1Tsinghua University · 2WeChat Vision, Tecent Inc. · 3Peking University
Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.
Haoyu Chen, Kaichen Zhou, Hang Hua +11
Harvard University · MIT · MIT-IBM Watson AI Lab +5