High similarity between first-visit and return frames does not necessarily show that a video world model remembered the scene; the intervening rollout may simply have changed very little. This ambiguity makes absolute revisit scores sensitive to rendering stability, repetitive content, and failed motion. We introduce R2M-Bench (Relative Revisit Memory Benchmark), a benchmark of observable revisit-selective consistency. For every detected return, R2M-Bench compares the revisit pair with two controls from the same rollout: a gap-matched non-revisit pair that measures generic temporal stability and a short-range pair that estimates short-horizon consistency. These comparisons produce MemoryGain (MG), the revisit advantage over the temporal baseline, and the Normalized Memory Ratio (NMR), which normalizes this advantage by the short-to-baseline dynamic range. R2M-Bench combines 100 reference scenes with three leave-and-return trajectories to form 300 instances and evaluates appearance fidelity, scene and object identity, local geometry, and persistent state. Across seven action-conditioned video world models, Overall NMR correlates with human consistency judgments at Spearman's ρ=0.547 (95% CI [0.45,0.63]). Its within-model correlation magnitude with generated motion is 0.072, compared with 0.207 for raw revisit similarity, indicating that relative calibration substantially reduces the slow-motion shortcut. DreamX-World-Memo achieves the highest Overall NMR among the evaluated video models. Together, these results support same-rollout relative calibration as a practical way to distinguish revisit-specific consistency from generic temporal stability.
Figures & tables
Figure 2: Overview of R2M-Bench. (a) Absolute first-visit/revisit similarity is not sufficient evidence of memory because motion magnitude and rendering stability can inflate the raw score. (b) R2M-Bench evaluates navigation-style generated rollouts by detecting leave-and-return revisit pairs from commanded trajectories, then compares each revisit pair with a gap-matched baseline and a short-range reference and reports MemoryGain and NMR across the metric families.
Benchmark
Long
Mem.
Rev.
View-Rob.
Rel. Score
Multi-axis
VBench ( Huang et al., 2023 )
–
–
–
–
–
✓
EvalCrafter ( Liu et al., 2023b )
–
–
–
–
–
✓
WorldModelBench ( Li et al., 2025a )
–
–
–
–
–
✓
WorldScore ( Duan et al., 2025 )
✓
△
–
–
–
✓
WBench ( Ying et al., 2026 )
✓
△
–
–
–
✓
Omni-WorldBench ( Wu et al., 2026a )
△
–
–
–
–
✓
Table 1: Comparison with representative video-generation, world-model, and memory-oriented benchmarks from the perspective of evaluation design. Long : long-horizon or multi-turn evaluation; Mem. : explicit memory or persistent-state evaluation; Rev. : constructed revisit or loop-closure test cases; View-Rob. : robustness to moderate viewpoint deviations; Rel. Score : relative scoring against within-video temporal controls rather than absolute similarity; Multi-axis : evaluation across multiple consistency levels. ✓denotes an explicit benchmark design goal; △ denotes partial or indirect coverage.
Figure 3: Bird’s-eye view of the three trajectory templates used in R2M-Bench. Color indicates temporal progression from start ( blue ) to end ( red ), and arrows indicate camera motion. (a) Out-and-back : the agent moves away and reverses to the starting pose. (b) Translation–rotation : the agent revisits the starting region after heading changes. (c) Closed-loop : the agent traverses a loop and returns to the starting pose. The ⋆ marker denotes the revisit point.
MemoryGain (MG; higher is better)
Appearance Fidelity
Scene Identity
Object Identity
Local Geometry
Persistent State
Method
Params
Mem. Type
PSNR
SSIM
LPIPS
DINO-sim
BoQ-sim
MVPR-sim
App. Persist
Sem. Persist
Feat. Corr.
Geo. Inlier
State Cons.
WonderWorld
–
Explicit 3D
12.828
0.398
0.548
0.369
0.633
0.327
0.254
0.067
0.535
0.147
2.676
LingBot-World-Fast
14B
RC
0.283
0.018
0.007
0.029
0.041
0.027
0.022
0.012
0.040
0.008
0.186
LingBot-World-2-Fast
14B
RC
0.271
0.009
0.011
0.083
0.013
0.057
0.031
0.014
0.015
0.023
0.262
Matrix-Game 3.0
5B
FoV-R
1.039
0.024
0.060
0.145
0.176
0.104
0.098
0.045
0.145
0.037
0.476
Table 2: Template-balanced MemoryGain (MG; top) and Normalized Memory Ratio (NMR; bottom), each averaged equally over Out-and-back, Translation–rotation, and Closed-loop. Memory types: RC = raw context, FoV-R = field-of-view retrieval, G-R = geometric retrieval, and SSM = state-space recurrence. WonderWorld (gray) is a separate 3D rendering reference. For video models, bold and underlined values mark the best and second-best results. Overall NMR first averages metrics within each family and then the five families; MG is not aggregated because its units are metric-specific.
Figure 4: Trajectory-wise metric-level NMR profiles for the video world models. Larger values indicate a stronger normalized revisit advantage. Radial limits are adapted per panel for legibility; compare model shapes and rankings within a panel rather than polygon area across panels.
(a) Video-level alignment and motion dependence
Score
ρ(Hcons)
95% CI
Partial ρ
95% CI
Action-valid ρ
ρ(Hmotion)
Raw
0.594
[0.54,0.64]
0.221
[0.12,0.30]
0.466
−0.336
MG
0.398
[0.33,0.46]
0.096
[0.00,0.19]
0.280
−0.053
NMR
0.480
[0.41,0.54]
0.149
[0.05,0.25]
0.369
−0.119
Overall NMR
0.547
[0.45,0.63]
0.106
[0.02,0.26]
0.409
−0.151
Table 3: Human validation on 210 sampled rollouts. (a) Video-level correlations, including the absolute Raw similarity reference. ρ is Spearman rank correlation; Hcons and Hmotion are human consistency and motion ratings. Partial ρ controls for camera control, motion, and trajectory. Action-valid requires mean camera-control and motion ratings both ≥1.5 . Raw, MG, and NMR use family-balanced Fisher- z aggregation, whereas Overall NMR is the per-video family-first score. The 95% intervals use trajectory–scene cluster bootstrap. (b) Correlation between each family’s NMR and human consistency.
Within-model Spearman ρ with motion
Family
Raw rev.
Temp. base
MG
NMR
Appearance Fidelity
−0.299
−0.304
0.008
0.089
Scene Identity
−0.274
−0.345
0.185
0.063
Local Geometry
−0.216
−0.337
0.156
0.129
Object Identity
−0.061
−0.088
0.090
0.023
Persistent State
−0.186
−0.283
0.116
0.056
Table 4: Family-level correlations with generated motion. Motion is the mean adjacent-frame optical-flow magnitude computed with RAFT. Correlations are first computed within each model; family cells average their constituent metrics, and Overall weights the five families equally.
SSIM ↑
LPIPS ↓
DINO-sim ↑
Geo. Inlier ↑
Setting
Rev.
Base.
Gain ↑
NMR
Rev.
Base.
Gain ↑
NMR
Rev.
Base.
Gain ↑
NMR
Rev.
Base.
Gain ↑
NMR
Motion speed split
Low-motion
0.507
0.445
+0.062
0.39
0.318
0.444
+0.126
0.47
0.916
0.861
+0.055
0.48
0.829
0.815
+0.014
0.12
Mid-motion
0.468
0.404
+0.064
0.43
0.437
0.509
+0.072
0.39
0.872
0.803
+0.069
0.38
0.770
0.746
+0.025
0.17
High-motion
0.452
0.408
+0.044
0.33
0.458
0.549
+0.091
0.31
0.691
0.620
+0.071
0.42
0.722
0.693
+0.029
0.16
Quality perturbation on the same High-motion split
Table 5: Diagnostic analysis of confounds in raw revisit similarity using DreamX-World-Memo generations. Videos are grouped by motion speed, and visual-quality perturbations are applied to the same High-motion split. Raw Rev. scores vary with speed and image quality, while Gain and NMR measure revisit-selective consistency after calibration against the same rollout. Bold denotes the best score within each diagnostic block.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Qualitative examples of the three pair types used by the relative protocol. Each row shows one model; columns show a revisit pair, a gap-matched non-revisit baseline, and a short-range reference from the same rollout. The protocol asks whether the revisit recovers more shared scene evidence than the long-gap baseline, calibrated by the consistency observed at short range.
Figure 6: Human annotation interface used in our study. Annotators view the generated video together with the target camera trajectory and motion-magnitude references, then independently rate temporal/revisit consistency, motion magnitude, and camera control as Bad, Normal, or Good. The separate controls make it possible to distinguish perceived revisit inconsistency from insufficient motion or failure to follow the commanded trajectory.
(a) Per-family ρ(Hcons)
Family
Raw
MG
NMR
Appearance Fidelity
0.454
0.487
0.499
Scene Identity
0.733
0.457
0.618
Local Geometry
0.651
0.364
0.435
Persistent State
0.598
0.332
0.386
Object Identity
0.482
0.339
0.437
Appendix
Table 7: Expanded human-validation diagnostics. (a) Family-level Spearman correlations; each entry aggregates only the benchmark metrics in that family. (b) Percentile bootstrap intervals under the primary trajectory–scene clustering and a conservative seven-method clustering.
Within-model Spearman ρ with objective motion
Metric
Raw revisit
Temporal baseline
MG
NMR
Appearance Fidelity
PSNR
−0.242∗
−0.235∗
−0.063
0.034
SSIM
−0.282∗
−0.293∗
0.045
0.109
LPIPS
−0.373∗
−0.384∗
0.041
0.124
Scene Identity
Appendix
Table 8: Full metric-level motion correlations. Motion is the mean adjacent-frame optical-flow magnitude computed with RAFT. Correlations are computed within each model and aggregated across models with equal weighting using Fisher’s z transformation. LPIPS raw and baseline signs are oriented so that higher always indicates better similarity; ∗ denotes a 95% confidence interval excluding zero.
Family-wise mean rank (Raw / MG / NMR)
Method
Appearance Fidelity
Scene Identity
Object Identity
Local Geometry
Persistent State
Overall
RAFT px/frame
LingBot-World-Fast
6.33 / 6.00 / 6.33
6.67 / 5.00 / 5.00
5.50 / 5.00 / 5.00
6.00 / 5.50 / 5.00
6.00 / 5.00 / 5.00
6.10 / 5.30 / 5.27
2.699
LingBot-World-2-Fast
6.33 / 6.33 / 6.33
5.00 / 6.00 / 6.00
5.50 / 6.50 / 6.50
5.00 / 6.50 / 6.50
5.00 / 6.00 / 6.00
5.37 / 6.27 / 6.27
2.482
Matrix-Game 3.0
4.00 / 4.00 / 4.00
4.00 / 2.33 / 3.33
1.50 / 1.00 / 2.00
3.50 / 2.50 / 3.00
4.00 / 3.00 / 3.00
3.40 / 2.57 / 3.07
2.199
HY-WorldPlay
2.00 / 2.00 / 2.00
1.00 / 2.67 / 1.67
1.50 / 2.50 / 2.50
1.00 / 3.00 / 2.00
1.00 / 2.00 / 2.00
1.30 / 2.43 / 2.03
1.038
SANA-WM
5.00 / 5.67 / 5.33
6.33 / 7.00 / 7.00
7.00 / 6.50 / 6.50
7.00 / 6.00 / 6.50
7.00 / 7.00 / 7.00
6.47 / 6.43 / 6.47
1.309
Appendix
Table 9: Family-wise mean ranks among the seven video world models. Each cell reports Raw / MG / NMR ; lower ranks are better. The Overall cell averages the corresponding ranks across the five families. RAFT is mean optical-flow magnitude in pixels per frame and is not ranked.
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.
Nan Zhou, Jianxiong Gao, Qiuyu Wang +10
Zhejiang University · Ant Group · Fudan University +1
Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primarily emphasize visual quality, motion coherence, and text-video alignment, they largely overlook memory, the core capability of a world model to preserve consistency across long-term horizons and complex interactions. To address this gap, we present MBench, a comprehensive benchmark dedicated to quantifying and evaluating the memory capability of video world models. We systematically decompose the memory capability of video world models into three hierarchical and complementary core dimensions: entity consistency, environment consistency, and causal consistency, which are further refined into 12 quantifiable sub-dimensions for comprehensive characterization of long-term memory. Our benchmark is built upon rigorously curated real-captured long videos, and evaluated by rule-based quantitative matrices and VLM to enable objective and comprehensive consistency assessment. Extensive evaluations of mainstream state-of-the-art video world models reveal critical systemic limitations of existing methods in long-term state retention, providing a standardized benchmark and clear research direction to advance the field.
Shengjun Zhang, Zhang Zhang, Simin Huang +11
1Tsinghua University · 2WeChat Vision, Tecent Inc. · 3Peking University
Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.
Haoyu Chen, Kaichen Zhou, Hang Hua +11
Harvard University · MIT · MIT-IBM Watson AI Lab +5