R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models

Qiwen Gu; Bingjie Gao; Rui Chen; Geng Li; Jifan Li; Qishuai Wen; Li Niu; Jing Tang

cs.CVAug 27, 2026

R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models

Authors: Qiwen Gu, Bingjie Gao, Rui Chen, Geng Li, Jifan Li, Qishuai Wen, Li Niu, Jing Tang,

Organizations: DreamX Team, Alibaba Group · Tongji University · Shanghai Jiao Tong University

Abstract

High similarity between first-visit and return frames does not necessarily show that a video world model remembered the scene; the intervening rollout may simply have changed very little. This ambiguity makes absolute revisit scores sensitive to rendering stability, repetitive content, and failed motion. We introduce R2M-Bench (Relative Revisit Memory Benchmark), a benchmark of observable revisit-selective consistency. For every detected return, R2M-Bench compares the revisit pair with two controls from the same rollout: a gap-matched non-revisit pair that measures generic temporal stability and a short-range pair that estimates short-horizon consistency. These comparisons produce MemoryGain (MG), the revisit advantage over the temporal baseline, and the Normalized Memory Ratio (NMR), which normalizes this advantage by the short-to-baseline dynamic range. R2M-Bench combines 100 reference scenes with three leave-and-return trajectories to form 300 instances and evaluates appearance fidelity, scene and object identity, local geometry, and persistent state. Across seven action-conditioned video world models, Overall NMR correlates with human consistency judgments at Spearman's $ρ=0.547$ (95% CI $[0.45,0.63]$ ). Its within-model correlation magnitude with generated motion is $0.072$ , compared with $0.207$ for raw revisit similarity, indicating that relative calibration substantially reduces the slow-motion shortcut. DreamX-World-Memo achieves the highest Overall NMR among the evaluated video models. Together, these results support same-rollout relative calibration as a practical way to distinguish revisit-specific consistency from generic temporal stability.

Figures & tables

Figure 2: Overview of R2M-Bench. (a) Absolute first-visit/revisit similarity is not sufficient evidence of memory because motion magnitude and rendering stability can inflate the raw score. (b) R2M-Bench evaluates navigation-style generated rollouts by detecting leave-and-return revisit pairs from commanded trajectories, then compares each revisit pair with a gap-matched baseline and a short-range reference and reports MemoryGain and NMR across the metric families.

Benchmark	Long	Mem.	Rev.	View-Rob.	Rel. Score	Multi-axis
VBench ( Huang et al., 2023 )	–	–	–	–	–	✓
EvalCrafter ( Liu et al., 2023b )	–	–	–	–	–	✓
WorldModelBench ( Li et al., 2025a )	–	–	–	–	–	✓
WorldScore ( Duan et al., 2025 )	✓	$\triangle$	–	–	–	✓
WBench ( Ying et al., 2026 )	✓	$\triangle$	–	–	–	✓
Omni-WorldBench ( Wu et al., 2026a )	$\triangle$	–	–	–	–	✓

Table 1: Comparison with representative video-generation, world-model, and memory-oriented benchmarks from the perspective of evaluation design. Long : long-horizon or multi-turn evaluation; Mem. : explicit memory or persistent-state evaluation; Rev. : constructed revisit or loop-closure test cases; View-Rob. : robustness to moderate viewpoint deviations; Rel. Score : relative scoring against within-video temporal controls rather than absolute similarity; Multi-axis : evaluation across multiple consistency levels. ✓denotes an explicit benchmark design goal;

\triangle

denotes partial or indirect coverage.

Figure 3: Bird’s-eye view of the three trajectory templates used in R2M-Bench. Color indicates temporal progression from start ( blue ) to end ( red ), and arrows indicate camera motion. (a) Out-and-back : the agent moves away and reverses to the starting pose. (b) Translation–rotation : the agent revisits the starting region after heading changes. (c) Closed-loop : the agent traverses a loop and returns to the starting pose. The $\star$ marker denotes the revisit point.

MemoryGain (MG; higher is better)
			Appearance Fidelity			Scene Identity			Object Identity		Local Geometry		Persistent State
Method	Params	Mem. Type	PSNR	SSIM	LPIPS	DINO-sim	BoQ-sim	MVPR-sim	App. Persist	Sem. Persist	Feat. Corr.	Geo. Inlier	State Cons.
WonderWorld	–	Explicit 3D	12.828	0.398	0.548	0.369	0.633	0.327	0.254	0.067	0.535	0.147	2.676
LingBot-World-Fast	14B	RC	0.283	0.018	0.007	0.029	0.041	0.027	0.022	0.012	0.040	0.008	0.186
LingBot-World-2-Fast	14B	RC	0.271	0.009	0.011	0.083	0.013	0.057	0.031	0.014	0.015	0.023	0.262
Matrix-Game 3.0	5B	FoV-R	1.039	0.024	0.060	0.145	0.176	0.104	0.098	0.045	0.145	0.037	0.476

Table 2: Template-balanced MemoryGain (MG; top) and Normalized Memory Ratio (NMR; bottom), each averaged equally over Out-and-back, Translation–rotation, and Closed-loop. Memory types: RC = raw context, FoV-R = field-of-view retrieval, G-R = geometric retrieval, and SSM = state-space recurrence. WonderWorld (gray) is a separate 3D rendering reference. For video models, bold and underlined values mark the best and second-best results. Overall NMR first averages metrics within each family and then the five families; MG is not aggregated because its units are metric-specific.

Panel of Figure 4 — Figure 4: Trajectory-wise metric-level NMR profiles for the video world models. Larger values indicate a stronger normalized revisit advantage. Radial limits are adapted per panel for legibility; compare model shapes and rankings within a panel rather than polygon area across panels.

(a) Video-level alignment and motion dependence
Score	$\rho(H_{\mathrm{cons}})$	95% CI	Partial $\rho$	95% CI	Action-valid $\rho$	$\rho(H_{\mathrm{motion}})$
Raw	0.594	$[0.54,0.64]$	0.221	$[0.12,0.30]$	0.466	$-0.336$
MG	0.398	$[0.33,0.46]$	0.096	$[0.00,0.19]$	0.280	$-0.053$
NMR	0.480	$[0.41,0.54]$	0.149	$[0.05,0.25]$	0.369	$-0.119$
Overall NMR	0.547	$[0.45,0.63]$	0.106	$[0.02,0.26]$	0.409	$-0.151$

Table 3: Human validation on 210 sampled rollouts. (a) Video-level correlations, including the absolute Raw similarity reference.

\rho

is Spearman rank correlation;

H_{\mathrm{cons}}

and

H_{\mathrm{motion}}

are human consistency and motion ratings. Partial

\rho

controls for camera control, motion, and trajectory. Action-valid requires mean camera-control and motion ratings both

\geq 1.5

. Raw, MG, and NMR use family-balanced Fisher-

z

aggregation, whereas Overall NMR is the per-video family-first score. The 95% intervals use trajectory–scene cluster bootstrap. (b) Correlation between each family’s NMR and human consistency.

Within-model Spearman $\rho$ with motion
Family	Raw rev.	Temp. base	MG	NMR
Appearance Fidelity	$-0.299$	$-0.304$	0.008	0.089
Scene Identity	$-0.274$	$-0.345$	0.185	0.063
Local Geometry	$-0.216$	$-0.337$	0.156	0.129
Object Identity	$-0.061$	$-0.088$	0.090	0.023
Persistent State	$-0.186$	$-0.283$	0.116	0.056

Table 4: Family-level correlations with generated motion. Motion is the mean adjacent-frame optical-flow magnitude computed with RAFT. Correlations are first computed within each model; family cells average their constituent metrics, and Overall weights the five families equally.

Motion speed split
	SSIM $\uparrow$				LPIPS $\downarrow$				DINO-sim $\uparrow$				Geo. Inlier $\uparrow$
Setting	Rev.	Base.	Gain $\uparrow$	NMR	Rev.	Base.	Gain $\uparrow$	NMR	Rev.	Base.	Gain $\uparrow$	NMR	Rev.	Base.	Gain $\uparrow$	NMR
Low-motion	0.507	0.445	+0.062	0.39	0.318	0.444	+0.126	0.47	0.916	0.861	+0.055	0.48	0.829	0.815	+0.014	0.12
Mid-motion	0.468	0.404	+0.064	0.43	0.437	0.509	+0.072	0.39	0.872	0.803	+0.069	0.38	0.770	0.746	+0.025	0.17
High-motion	0.452	0.408	+0.044	0.33	0.458	0.549	+0.091	0.31	0.691	0.620	+0.071	0.42	0.722	0.693	+0.029	0.16
Quality perturbation on the same High-motion split

Table 5: Diagnostic analysis of confounds in raw revisit similarity using DreamX-World-Memo generations. Videos are grouped by motion speed, and visual-quality perturbations are applied to the same High-motion split. Raw Rev. scores vary with speed and image quality, while Gain and NMR measure revisit-selective consistency after calibration against the same rollout. Bold denotes the best score within each diagnostic block.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative examples of the three pair types used by the relative protocol. Each row shows one model; columns show a revisit pair, a gap-matched non-revisit baseline, and a short-range reference from the same rollout. The protocol asks whether the revisit recovers more shared scene evidence than the long-gap baseline, calibrated by the consistency observed at short range.

Figure 6: Human annotation interface used in our study. Annotators view the generated video together with the target camera trajectory and motion-magnitude references, then independently rate temporal/revisit consistency, motion magnitude, and camera control as Bad, Normal, or Good. The separate controls make it possible to distinguish perceived revisit inconsistency from insufficient motion or failure to follow the commanded trajectory.

(a) Per-family $\rho(H_{\mathrm{cons}})$
Family	Raw	MG	NMR
Appearance Fidelity	0.454	0.487	0.499
Scene Identity	0.733	0.457	0.618
Local Geometry	0.651	0.364	0.435
Persistent State	0.598	0.332	0.386
Object Identity	0.482	0.339	0.437

Appendix

Table 7: Expanded human-validation diagnostics. (a) Family-level Spearman correlations; each entry aggregates only the benchmark metrics in that family. (b) Percentile bootstrap intervals under the primary trajectory–scene clustering and a conservative seven-method clustering.

Within-model Spearman $\rho$ with objective motion
Metric	Raw revisit	Temporal baseline	MG	NMR
Appearance Fidelity
PSNR	$-0.242^{*}$	$-0.235^{*}$	$-0.063$	0.034
SSIM	$-0.282^{*}$	$-0.293^{*}$	0.045	0.109
LPIPS	$-0.373^{*}$	$-0.384^{*}$	0.041	0.124
Scene Identity

Appendix

Table 8: Full metric-level motion correlations. Motion is the mean adjacent-frame optical-flow magnitude computed with RAFT. Correlations are computed within each model and aggregated across models with equal weighting using Fisher’s

z

transformation. LPIPS raw and baseline signs are oriented so that higher always indicates better similarity; ∗ denotes a 95% confidence interval excluding zero.

Family-wise mean rank (Raw / MG / NMR)
Method	Appearance Fidelity	Scene Identity	Object Identity	Local Geometry	Persistent State	Overall	RAFT px/frame
LingBot-World-Fast	6.33 / 6.00 / 6.33	6.67 / 5.00 / 5.00	5.50 / 5.00 / 5.00	6.00 / 5.50 / 5.00	6.00 / 5.00 / 5.00	6.10 / 5.30 / 5.27	2.699
LingBot-World-2-Fast	6.33 / 6.33 / 6.33	5.00 / 6.00 / 6.00	5.50 / 6.50 / 6.50	5.00 / 6.50 / 6.50	5.00 / 6.00 / 6.00	5.37 / 6.27 / 6.27	2.482
Matrix-Game 3.0	4.00 / 4.00 / 4.00	4.00 / 2.33 / 3.33	1.50 / 1.00 / 2.00	3.50 / 2.50 / 3.00	4.00 / 3.00 / 3.00	3.40 / 2.57 / 3.07	2.199
HY-WorldPlay	2.00 / 2.00 / 2.00	1.00 / 2.67 / 1.67	1.50 / 2.50 / 2.50	1.00 / 3.00 / 2.00	1.00 / 2.00 / 2.00	1.30 / 2.43 / 2.03	1.038
SANA-WM	5.00 / 5.67 / 5.33	6.33 / 7.00 / 7.00	7.00 / 6.50 / 6.50	7.00 / 6.00 / 6.50	7.00 / 7.00 / 7.00	6.47 / 6.43 / 6.47	1.309

Appendix

Table 9: Family-wise mean ranks among the seven video world models. Each cell reports Raw / MG / NMR ; lower ranks are better. The Overall cell averages the corresponding ranks across the five families. RAFT is mean optical-flow magnitude in pixels per frame and is not ranked.

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 203 papers

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
Oct 13, 2026Nan Zhou, Jianxiong Gao, Qiuyu Wang +10World Model Evaluation Video World Models
Zhejiang University · Ant Group · Fudan University +1
MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
May 30, 2026Shengjun Zhang, Zhang Zhang, Simin Huang +11Long-Horizon Video Generation World Model Evaluation
1Tsinghua University · 2WeChat Vision, Tecent Inc. · 3Peking University
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Jun 25, 2026Haoyu Chen, Kaichen Zhou, Hang Hua +11Physical Consistency in Video Generation World Model Evaluation
Harvard University · MIT · MIT-IBM Watson AI Lab +5

Oct 13, 2026J/K move · Enter open · S save

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

Nan Zhou, Jianxiong Gao, Qiuyu Wang, Wen Wang+9

Zhejiang University · Ant Group · Fudan University+1

World Model Evaluation Video World Models Temporal Consistency in Video Generation

Open paper AskPDF arXiv

As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.

Abstract

World Model Evaluation

Explore similar work

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

MemoBench: Benchmarking World Modeling in Dynamically Changing Environments