WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

Nan Zhou; Jianxiong Gao; Qiuyu Wang; Wen Wang; Jiahao Wang; Yufeng Yuan; Shuaiting Li; Hao Ouyang

cs.CVOct 13, 2026

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

Authors: Nan Zhou, Jianxiong Gao, Qiuyu Wang, Wen Wang, Jiahao Wang, Yufeng Yuan, Shuaiting Li, Hao Ouyang,

Organizations: Zhejiang University · Ant Group · Fudan University · Nanjing University

Abstract

As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.

Figures & tables

Figure 1: Overview of WorldMemBench . Top: case coverage across humans, objects, environments, and dynamic events. Bottom left: controlled departure-and-revisit evaluation compares each return against a fixed initial reference. Bottom right: capability profiles of nine video world models.

Panel of Table 1 — Table 1: Comparison with world-model evaluation benchmarks. Input denotes model paradigm: C (camera-driven), W A S D A (action-driven), and T L (language-driven). Gray icons indicate unevaluated paradigms. $\checkmark$ : explicit coverage; $\triangle$ : partial coverage; $\times$ : not reported.

Figure 2: Composition and diversity of WorldMemBench. (a) Case taxonomy, with Human, Object, Environment, and Dynamic categories in the inner ring and subcategories in the outer ring. (b) Case counts for static memory, dynamic memory, general video quality, and camera-following evaluation. (c) Representative cases showing target boxes and masks, scene depth, and event states before and after revisiting.

Figure 3: Evaluation camera trajectories in WorldMemBench . Translation preserves the initial viewing direction and completes a square loop ( green ), while Rotation keeps the camera position fixed and applies left–right yaw ( orange ) or up–down pitch ( blue ).

Video Quality
Metric	HY	Matrix	Matrix	SANA-WM	YUME	LingBot	LingBot	Evoke	DreamX
Metric	WorldPlay	Game-3	Game-3.5	SANA-WM	YUME	World	World-v2	Evoke	World
Imaging $\uparrow$	0.659	0.680	0.603	0.542	0.713	0.671	0.683	0.670	0.686
Aesthetic $\uparrow$	0.527	0.510	0.429	0.434	0.533	0.545	0.534	0.533	0.537
Static Memory
Human ID $\uparrow$	0.828	0.483	0.019	0.134	0.298	0.393	0.534	0.550	0.286

Table 2: Quantitative comparison of video world models on WorldMemBench.

\uparrow

\downarrow

indicate higher / lower is better. The best and second-best results for each metric are shown in bold and underlined , respectively. App., Env., Recon., Trans., Rot., Prog., Persist., and Phys. denote Appearance, Environment, Reconstruction, Translation, Rotation, Progress, Persistence, and Physics, respectively.

Figure 4: Qualitative comparison of video world models on an environment case.

Figure 5: Per-model changes of nine models across the three revisits of LRLRLR, four metrics per row.

\uparrow

\downarrow

indicate higher/lower is better; all panels keep each metric’s original value direction.

Config	Human		Object		Environment			Dynamic Memory				Inference Speed
Config	ID $\uparrow$	App. $\uparrow$	App. $\uparrow$	Geometry $\uparrow$	Visible $\uparrow$	Depth $\uparrow$	Recon. $\uparrow$	Return $\uparrow$	Persist. $\uparrow$	Final $\uparrow$	Causal $\uparrow$	Latency (s) $\downarrow$	Speedup $\uparrow$
FullKV	0.534	0.645	0.564	0.506	0.694	0.517	0.253	0.667	0.733	0.592	0.792	322.16	1.00 $\times$
FoV	0.547	0.663	0.531	0.485	0.687	0.550	0.287	0.725	0.792	0.675	0.800	262.69	1.23 $\times$
Sliding Windows	0.392	0.540	0.426	0.392	0.610	0.501	0.235	0.667	0.717	0.600	0.775	260.42	1.24 $\times$

Table 3: Controlled comparison of context-access policies on LingBot World v2. Best and second-best results are bold and underlined .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Resolution	FPS	Frames / Duration	Seed	Sampling	Inference Setting
HY-WorldPlay	$848{\times}480$	24	493 / 20.5 s	1	4-step AR	Built-in
Matrix-Game 3	$1280{\times}704$	17	497 / 29.2 s	42	3-step AR $\times$ 12	Built-in
Matrix-Game 3.5	$1280{\times}704$	16	505 / 31.6 s	3407	3-step, six-block	Built-in
SANA-WM	$1280{\times}704$	16	481 / 30.1 s	42	4-step self-forcing	FullKV
YUME	$1280{\times}704$	16	493–551 / 30.8–34.4 s	43	4-step Euler	Built-in
LingBot World	$832{\times}464$	16	493 / 30.8 s	1	causal AR	FullKV

Appendix

Table 4: Video specifications and sampling configurations of the formal inference artifacts. Resolution and FPS are the models’ native outputs; Frames / Duration reports the actual frame count of each output video together with its duration; Seed and Sampling give the sampling configuration of formal inference; Inference Setting only distinguishes explicit FullKV from the models’ built-in memory modes.

Model	Started	Start rate (%)	Gated Overall	Conditional
HY-WorldPlay	19/60	31.7	0.294	0.928
Matrix-Game 3	3/60	5.0	0.046	0.917
Matrix-Game 3.5	23/60	38.3	0.240	0.625
SANA-WM	35/60	58.3	0.415	0.711
YUME	54/60	90.0	0.775	0.861
LingBot World	56/60	93.3	0.808	0.866

Appendix

Table 5: Dynamic Memory decomposition from the frozen 60-case evaluation. Started is the number of judge-positive event-start decisions. Conditional averages only the Started subset.

Figure 6: Qualitative comparison of video world models on a human case.

Figure 7: Qualitative comparison of video world models on an object case.

Metric	Agree/Decisive	Tie/Split	Agreement (%)	Spearman $\rho$
Human Identity	65/76	1/7	85.5	0.929
Human Appearance	71/80	2/2	88.8	0.952
Object Geometry	62/79	0/5	78.5	0.606
Object Appearance	67/80	2/2	83.8	0.699
Environment Visible	74/80	1/3	92.5	0.988
Environment Depth	63/77	4/3	81.8	0.301

Appendix

Table 6: Agreement between automatic Static Memory metrics and blinded human A/B preferences. Pairwise Agreement measures the per-pair directional consistency between an automatic metric and the human majority vote; model-level Spearman

\rho

is computed over the human and automatic win rates of the eight models.

Figure 8: Blinded A/B preference annotation interface for Static Memory.

Task	Checklist Items	Judgements	Spearman $\rho\uparrow$	PLCC $\uparrow$
Return Progress	100	300	0.684	0.683
Persistence / Reset	100	300	0.582	0.612
Final State	100	300	0.624	0.641
Causal / Physics	100	300	0.401	0.491
Overall	400	1200	0.732	0.808

Appendix

Table 7: Human alignment of the Dynamic Memory checklist evaluation. The per-item majority vote of three annotators forms the human reference labels; Spearman

\rho

and PLCC are computed over the checklist satisfaction scores of the 50 instances.

Figure 9: Event-gated checklist annotation interface for Dynamic Memory.

(a) LRLR
Metric	HY-WorldPlay	Matrix-Game 3	Matrix-Game 3.5	SANA-WM	YUME	LingBot World	LingBot World v2	Evoke	DreamX World
Imaging $\uparrow$	0.668	0.686	0.613	0.554	0.712	0.677	0.698	0.673	0.683
Aesthetic $\uparrow$	0.538	0.522	0.446	0.451	0.539	0.556	0.553	0.541	0.530
Human ID $\uparrow$	0.863	0.577	0.015	0.150	0.295	0.389	0.559	0.546	0.267
Human App. $\uparrow$	0.889	0.597	0.041	0.282	0.529	0.643	0.702	0.783	0.522
Object App. $\uparrow$	0.925	0.647	0.510	0.311	0.498	0.442	0.626	0.744	0.486

Appendix

Table 8: Per-model results on the three formal Rotation trajectories. Best and second-best results within each trajectory are bold and underlined.

Metric	LRLR			LRUD			$\Delta_{\mathrm{axis}}$
Metric	$R_{1}$	$R_{2}$	$d$	$R_{1}$	$R_{2}$	$d$	$\Delta_{\mathrm{axis}}$
Human ID $\uparrow$	0.490	0.324	0.166	0.435	0.249	0.186	+0.020
Human App. $\uparrow$	0.623	0.486	0.137	0.556	0.371	0.185	+0.049
Object App. $\uparrow$	0.657	0.496	0.162	0.634	0.345	0.288	+0.127
Object Geometry $\uparrow$	0.588	0.457	0.132	0.547	0.296	0.251	+0.120
Env. Visible $\uparrow$	0.781	0.655	0.126	0.770	0.492	0.278	+0.152

Appendix

Table 9: Nine-model mean Static Memory scores at the two revisits of LRLR and LRUD.

d=R_{1}-R_{2}

and

\Delta_{\mathrm{axis}}=d_{\mathrm{LRUD}}-d_{\mathrm{LRLR}}

; positive

\Delta_{\mathrm{axis}}

indicates greater degradation after switching from yaw to pitch.

(a) LRLR: two yaw revisits
Metric	Slot	HY	Matrix	Matrix	SANA-WM	YUME	LingBot	LingBot	Evoke	DreamX	Mean
Metric	Slot	WorldPlay	Game-3	Game-3.5	SANA-WM	YUME	World	World-v2	Evoke	World	Mean
Human ID $\uparrow$	$R1$	0.873	0.738	0.030	0.229	0.431	0.517	0.663	0.579	0.349	0.490
Human ID $\uparrow$	$R2$	0.853	0.416	0.000	0.072	0.159	0.261	0.456	0.514	0.186	0.324
Human App. $\uparrow$	$R1$	0.894	0.719	0.076	0.395	0.620	0.734	0.785	0.799	0.582	0.623
Human App. $\uparrow$	$R2$	0.884	0.476	0.006	0.168	0.438	0.553	0.618	0.767	0.463	0.486

Appendix

Table 10: Full per-model LRLR–LRUD cross-axis comparison. Panels (a) and (b) report the two revisit scores; panel (c) reports

\Delta_{\mathrm{axis}}

. All memory metrics are higher-is-better.

Model	H.ID	H.App.	Object	$D$	$R_{30}$	$N_{\mathrm{multi}}$	$N_{\mathrm{cross}}$
	Tracking support (%)			Environment pose (%)		$Q_{30}$ paired cases
HY-WorldPlay	80.8	58.6	65.2	99.2	99.4	68	67
Matrix-Game 3	56.7	45.1	31.7	99.4	91.2	69	25
Matrix-Game 3.5	3.5	6.9	61.2	100.0	91.0	62	47
SANA-WM	26.6	34.7	43.9	99.6	80.8	54	25
YUME	30.0	19.5	22.0	92.9	94.5	52	41

Appendix

Table 11: Tracking-supported target revisits and pose-qualified Environment returns. Entity columns are percentages averaged over targets, prescribed revisits, and videos; Object uses the shared appearance/geometry tracking evidence. Environment percentages use 490 slots per model.

D

: departure

\geq 30^{\circ}

;

R_{30}

: return error

\leq 30^{\circ}

;

Q_{30}

: departure

\geq 30^{\circ}

and return error

\leq 30^{\circ}

. The last columns count retained Environment model–case units for multi-revisit (

N_{\mathrm{multi}}

) and cross-axis (

N_{\mathrm{cross}}

) comparisons.

Metric	$N$	$M$	All cases	Qualified	Positive	Criterion
(a) Multi-revisit decrease: $R_{1}-R_{3}$
Human Identity	125	8	0.237	0.110	8/8	Tracking
Human Appearance	93	7	0.162	0.058	5/7	Tracking
Object Appearance	155	9	0.229	0.158	9/9	Tracking
Object Geometry	155	9	0.205	0.101	9/9	Tracking
Environment Visible	431	9	0.161	0.152	9/9	Pose $30^{\circ}$

Appendix

Table 12: Paired revisit diagnostics.

N

counts retained model–case units (a cross-axis unit contains two videos);

M

counts contributing models. All cases uses the same

M

models as Qualified, without case filtering. Positive reports the number of models with a positive conditional effect. Tracking requires support for every target at every compared revisit; Pose uses the

30^{\circ}

criterion. Values are descriptive means, not significance tests or conditional model rankings.

	Multi-revisit decrease				Cross-axis excess degradation
Tolerance	$N/M$	Vis.	Dep.	Rec.	$N/M$	Vis.	Dep.	Rec.
$5^{\circ}$	195/8	0.092	0.057	0.116	30/4	0.094	0.158	0.050
$15^{\circ}$	345/8	0.116	0.085	0.141	134/7	0.090	0.010	0.029
$30^{\circ}$	431/9	0.152	0.135	0.139	281/9	0.128	0.076	0.056

Appendix

Table 13: Environment return-tolerance sensitivity, always requiring departure

\geq 30^{\circ}

and fixed cases across compared revisits.

N/M

denotes retained model–case units / contributing models. Vis., Dep., and Rec. denote Visible, Depth, and Reconstruction. The retained case and model sets change with the tolerance.

Figure 10: World memory under different context-access policies in LingBot World v2. The examples depict (a) slicing a pepper, (b) performing a deadlift, and (c) erasing a drawing. Within each example, rows show Sliding Windows, FoV, and FullKV from top to bottom. Columns present selected frames along the prescribed LR-Hold trajectory, spanning the initial observation, departure, and revisit. Camera phases and frame indices are indicated above.

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 203 papers

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
May 30, 2026Shengjun Zhang, Zhang Zhang, Simin Huang +11Long-Horizon Video Generation World Model Evaluation
1Tsinghua University · 2WeChat Vision, Tecent Inc. · 3Peking University
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Jun 25, 2026Haoyu Chen, Kaichen Zhou, Hang Hua +11Physical Consistency in Video Generation World Model Evaluation
Harvard University · MIT · MIT-IBM Watson AI Lab +5
R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models
Aug 27, 2026Qiwen Gu, Bingjie Gao, Rui Chen +7World Model Evaluation Video World Models
DreamX Team, Alibaba Group · Tongji University · Shanghai Jiao Tong University

May 30, 2026J/K move · Enter open · S save

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

Shengjun Zhang, Zhang Zhang, Simin Huang, Zhenyu Tang+10

1Tsinghua University · 2WeChat Vision, Tecent Inc. · 3Peking University

Long-Horizon Video Generation World Model Evaluation Temporal Consistency in Video Generation Video World Models

Open paper AskPDF arXiv

Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primarily emphasize visual quality, motion coherence, and text-video alignment, they largely overlook memory, the core capability of a world model to preserve consistency across long-term horizons and complex interactions. To address this gap, we present MBench, a comprehensive benchmark dedicated to quantifying and evaluating the memory capability of video world models. We systematically decompose the memory capability of video world models into three hierarchical and complementary core dimensions: entity consistency, environment consistency, and causal consistency, which are further refined into 12 quantifiable sub-dimensions for comprehensive characterization of long-term memory. Our benchmark is built upon rigorously curated real-captured long videos, and evaluated by rule-based quantitative matrices and VLM to enable objective and comprehensive consistency assessment. Extensive evaluations of mainstream state-of-the-art video world models reveal critical systemic limitations of existing methods in long-term state retention, providing a standardized benchmark and clear research direction to advance the field.

Abstract

World Model Evaluation

Explore similar work

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models