Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context. We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. On a long-horizon revisit benchmark, World-T3 raises revisit PSNR from 13.52 to 18.31 dB and reduces absolute rotation error from 3.344 to 1.075 relative to the strongest baseline on both metrics, while sustaining 20.6 FPS at 1280 x 704 on a single NVIDIA B200 GPU.
Figures & tables
Figure 1: Left: Revisit consistency. When returning to a previously visited viewpoint, our World-T 3 enables faithful scene recall, whereas existing autoregressive baselines forget earlier observations beyond the context window and resynthesize new content. Right: Speed vs. quality on one NVIDIA B200 GPU. x -axis: wall-clock seconds required per second of generated 16 FPS 1280×704 video (points left of the dashed line are real-time). y -axis: PSNR on the long-horizon revisit benchmark.
Figure 2: Camera Conditioning. Plücker raymaps modulate each DiT block via camera-dependent scale and shift.
Figure 3: Stage 2: Autoregressive Teacher with TTT Memory. (a) Overview of the context window modeling, which consists of a sink block, K recent clean blocks, and the current block. The revisit viewpoint may be evicted and thus fall outside the active context window. (b) Write encodes an evicted block’s raymap Πej and clean hidden state hej into a key-value pair and updates TTT memory state Mb by recursive least squares (Eq. 4 ). Read uses the target pose key kb to retrieve memory feature mb from the fast weights (Eq. 5 ), without distorting the memory state with Jb writes before block b . (c) The retrieved memory is adaptively injected into each DiT block through a gated residual (Eq. 6 ) for long-horizon consistency during viewpoint revisits.
Figure 4: Scene Consistency along Out-and-Back Camera Trajectories. Qualitative comparisons on an indoor scene (top) and an outdoor scene (bottom). Each 10-second camera input follows the forward trajectory for 5 seconds and then retraces it in reverse for 5 seconds. Columns show selected timestamps shared across methods, and the first and last columns compare the initial and return viewpoints.
Visual Quality ↑
Revisit Fidelity ↑
Camera Following ↓
Efficiency
Method
Steps
VBench
Semantic Sim.
PSNR
RotErr ∘
CamMC
RPE Rot. ∘
RPE Trans.
FPS ↑
Mem ↓
GT (VAE-decoded)
–
75.69
0.9982
–
0.194
0.0055
0.073
0.00102
–
–
Matrix-Game 3.0
3
74.72
0.8225
11.15
11.179
0.2958
0.357
0.00285
7.1
76
HY-World 1.5
4
75.32
0.8563
10.84
12.864
0.3484
0.315
0.00395
4.5
110
DreamX-AR
4
74.83
0.8768
11.66
8.011
0.2276
0.448
0.00508
11.0
76
LingBot-Fast
4
74.98
0.8580
10.59
9.817
0.2639
0.317
0.00290
1.6
144
Table 1: Comparison on the Revisit Set. We average across seven VBench visual quality dimensions ( 0 - 100 scale). Revisit fidelity evaluates scene consistency via DINOv2 similarity (“Semantic Sim.”) between paired views and via PSNR against ground truth (GT). “Mem” is peak memory during generation on a single NVIDIA B200 GPU in GB. Bold marks the best generated result, and underline indicates the second best. VAE-decoded GT provides a reference for the evaluation.
Method
T1
T2
T3
T3 − T1
GT (VAE-decoded)
0.9981
0.9983
0.9981
0.000
Matrix-Game 3.0
0.8743
0.8196
0.7737
−0.101
HY-World 1.5
0.8612
0.8581
0.8497
−0.012
DreamX-AR
0.9074
0.8941
0.8289
−0.079
LingBot-Fast
0.8532
0.8701
0.8506
−0.003
SANA-WM
0.9201
0.9311
0.9090
−0.011
Table 2: Revisit Consistency under Increasing Camera Sweep. T1–T3 are equal-sized terciles ( 40 clips each) of yaw sweep normalized by horizontal field of view. Higher tiers indicate larger sweeps. Bold and underline mark the best and second-best generated similarities in each tier, respectively.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Scene Consistency across Approximate Revisits. We identify three approximate revisits within a 30-second real camera motion sequence. Unlike the prescribed out-and-back trajectory used for quantitative evaluation, these revisits observe the same scene regions from similar, rather than identical, camera poses. Visit 0 denotes the initial observation, and Visits 1, 2, and 3 mark the selected revisits in temporal order. Colored boxes link full views to enlarged crops comparing local details in the baselines and World-T 3 . All enlarged crops compare Visits 0 and 3.
Stage 1
Stage 2
Stage 3
SFT
AR
Memory
CD
DMD
Training iterations
6000
10,000
15,000
2500
800
Latent frames
21
21
63
63
63
GPUs
512
256
128
128
128
Batch per GPU
2
2
1
1
1
Global batch
1024
512
128
128
128
Appendix
Table 3: Training Configuration. “AR” denotes block-causal autoregressive adaptation, and “Memory” represents the subsequent memory-augmented fine-tuning stage. Learning rates correspond to peak values, and gradient clipping thresholds are specified as gradient norms. DMD iteration counts represent critic updates, with generator updates executed every five iterations.
Method
Imaging
Aesthetic
Subject
Background
Motion
Flicker
Overall
Avg.
GT (VAE-decoded)
74.00
53.84
91.27
93.44
98.27
94.37
24.65
75.69
Matrix-Game 3.0
74.03
50.66
89.53
91.81
98.42
94.59
24.00
74.72
HY-World 1.5
76.29
52.44
90.72
92.34
97.79
93.82
23.87
75.32
DreamX-AR
73.59
52.49
88.53
91.99
97.67
94.48
25.05
74.83
LingBot-Fast
74.13
52.84
88.97
91.97
97.08
94.62
25.22
74.98
SANA-WM
65.21
47.04
88.40
91.46
98.86
95.40
23.75
72.87
Appendix
Table 4: Visual Quality on the Revisit Benchmark. All seven VBench dimensions are scored on a 0 – 100 scale (higher is better). “Avg.” is their unweighted mean and equals the VBench column of Table 1 . Bold indicates the best generated result and underline indicates the second best.
Method
dˉ↓
dfirst↓
dreturn↓
Δ↓
Self-SSIM ↑
GT-SSIM ↑
LM ↑
GT (VAE-decoded)
0.0018
–
–
–
0.9470
–
0.9756
Matrix-Game 3.0
0.1775
0.0411
0.2185
0.1774
0.3759
0.3748
0.8875
HY-World 1.5
0.1437
0.0274
0.1867
0.1593
0.3576
0.3526
0.9874
DreamX-AR
0.1232
0.0327
0.1610
0.1282
0.3805
0.3781
0.9838
LingBot-Fast
0.1420
0.0279
0.1832
0.1553
0.3431
0.3409
0.9703
SANA-WM
0.0800
0.0587
0.1218
0.0631
0.4704
0.4360
0.9006
Appendix
Table 5: Complementary Revisit Consistency Measures. dˉ denotes the DINOv2 self-distance across four outbound-return pairs. dfirst and dreturn evaluate feature distance against GT for outbound and return frames, respectively ( Δ=dreturn−dfirst ). Self-SSIM compares generated return views to outbound views, and GT-SSIM compares return views to GT references. Bold marks the best generated result and underline denotes the second best.
Outbound-Return Consistency
GT-Referenced Fidelity
Method
DINO dˉ↓
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
GT (VAE-decoded)
0.0018
0.9470
0.0221
–
–
–
Matrix-Game 3.0
0.1775
0.3759
0.4478
11.15
0.3748
0.4629
HY-World 1.5
0.1437
0.3576
0.4687
10.84
0.3526
0.4784
DreamX-AR
0.1232
0.3805
0.4619
11.66
0.3781
0.4699
LingBot-Fast
0.1420
0.3431
0.4775
10.59
0.3409
0.4911
Appendix
Table 6: Comprehensive Revisit Fidelity Metrics. Scores are averaged over the four outbound-return pairs per clip across all 120 benchmark clips. Outbound-return consistency compares each generated return view with its generated outbound view using DINOv2 feature distance ( dˉ ), SSIM, and LPIPS. GT-referenced fidelity compares generated return views with the GT references using PSNR, SSIM, and LPIPS. The VAE-decoded GT row gives the reference level for self-consistency. Bold marks the best generated result and underline denotes the second best.
Table 12
Test
Measure
Reference bf16
Accelerated
Denoising forward
rel. RMS error to FP32 ↓
1.54×10−2
8.46×10−3
Context encoding
rel. RMS error to FP32 ↓
1.28×10−2
8.25×10−3
VAE decoder
rel. RMS error to FP32 ↓
7.17×10−3
5.95×10−3
PSNR to FP32 (dB) ↑
53.82
55.44
Rollout
PSNR to FP32 rollout (dB) ↑
27.40
29.61
SSIM to FP32 rollout ↑
0.9034
0.9257
Appendix
Table 8: Numerical Fidelity of the Accelerated Implementation. Operator-level errors use the captured inputs of block 5 of a held-out clip and an FP32 copy of the model with TF32 disabled, and the decoder decodes the same 21 latent frames. Rollout rows compare full 81 -frame bf16 rollouts with FP32 rollouts from the same inputs and seed.
E2E latency (s)
L30
Method
Steps
GPUs
5s
30s
s / video s
FPS
Peak Mem (GB)
One GPU
Matrix-Game 3.0
3
1
12.23
69.73
2.24
7.1
76
HY-World 1.5
4
1
7.96
107.04
3.59
4.5
110
DreamX-AR
4
1
7.23
42.22
1.45
11.0
76
LingBot-Fast
4
1
44.56
292.54
10.07
1.6
144
Appendix
Table 9: Inference Cost Comparison on NVIDIA B200 GPUs. Metrics are the median over three generations, run one at a time. L30 is the 30 s video setting. Mem. is peak memory in GB per GPU. For multi-GPU rows, we use maximum peak memory over GPUs.
Video world models aim to generate explorable, 3D-consistent scene videos conditioned on camera trajectories. Existing approaches often rely on external memory systems that explicitly retrieve previously observed content to mitigate scene drift during long-horizon generation. However, these auxiliary memory pathways operate outside the model's internal generative dynamics, preventing the model from intrinsically learning when and what historical information should be retrieved. We propose to internalize memory retrieval into the generation process, allowing retrieval to emerge as an intrinsic behavior of the video world model rather than relying on an external memory system. Based on this principle, we introduce Learning-to-Retrieve (L2R), which repurposes the model's persistent internal state as a memory for historical context. A camera-conditioned retrieval gate selectively accesses relevant historical information from this state, determining what to retrieve, while a retrieval trigger determines when to retrieve. We further supervise the trigger with a 3D re-visibility signal, activating retrieval when previously observed content re-enters the current view while otherwise preserving the existing context. Together, these components enable the model to intrinsically acquire memory retrieval behavior and incorporate relevant historical observations into generation without a separate retrieval pathway. Across multiple base models and camera-revisit benchmarks, L2R improves long-term scene consistency while eliminating the need for an external memory bank or 3D conditions. https://jkhu29.github.io/l2r
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.
With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models.