Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen. Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure. Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time. We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall. SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache. Consequently, recall is conditioned on inter-view geometry rather than on recency. Each read additionally provides a confidence estimate that conditions a chunk-causal diffusion transformer, allowing it to selectively trust retrieved content while relying on its generative prior elsewhere. On long trajectories in Memory Maze and Minecraft, SMART maintains high-fidelity recall far beyond its training horizon while using constant memory and per-step cost.
Figures & tables
Figure 1: Long-horizon video generation. Top: on Memory Maze , Smart and DFoT ( Song et al., 2025 ) observe a forward trajectory and generate the return leg, revisiting places seen long before. Bottom: on Minecraft , Smart and MilliVid ( Chandratreya et al., 2026 ) generate a continuation trajectory from 256 context frames written into the memory. Smart retains observed content more faithfully than the baselines at constant per-step cost.
Figure 2: Smart inference overview.
Figure 3: Tiled tokenization. Write token j queries only the patches of tile j .
Figure 4: Slot read and write. Address routes the write; Read is a cross-attention over slot states.
Figure 5: Smart memory module. Write and read paths of the slot memory.
Figure 6: Qualitative comparison on Minecraft . Generated frames for each method in Table 2 , on the same trajectory, at the 2048 frame horizon with 256 initial context frames.
Method
(201+21)+201=423 frames
(401+21)+401=823 frames
PFLOPs ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
FVD ↓
PFLOPs ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
FVD ↓
Causal (100F)
0.096
25.0
0.809
0.131
6.39
4.06
0.193
21.8
0.739
0.198
8.85
7.19
Causal (Full)
0.117
30.0
0.891
0.034
5.36
2.13
0.299
24.3
0.789
0.100
7.96
4.61
GeoCache (GT-D)
0.454
24.6
0.811
0.098
6.25
3.57
0.936
24.6
0.810
0.095
6.03
2.52
DFoT
0.566
32.0
0.919
0.030
5.74
1.89
1.130
28.4
0.865
0.079
5.54
1.89
Smart
0.108
31.1
0.919
0.040
5.22
2.25
0.213
29.9
0.900
0.046
5.08
1.83
Table 1: 3D consistent video generation on Memory Maze . Methods generate the reverse leg from the observed forward trajectory in parentheses. Column groups are total trajectory length. FID pools a band’s frames, FVD its 16 -frame clips.
Method
1–64 frames
65–256 frames
PFLOPs ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
FVD ↓
PFLOPs ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
FVD ↓
Causal (7F)
0.72
17.18
0.583
0.325
43.36
64.37
2.62
11.67
0.437
0.551
47.78
182.03
FramePack
1.71
19.13
0.612
0.260
44.66
56.67
6.62
14.04
0.498
0.448
48.67
112.84
MilliVid
2.40
21.16
0.615
0.196
44.68
47.83
8.95
18.43
0.562
0.284
43.34
44.28
Smart
0.91
21.96
0.647
0.187
44.60
45.86
3.46
18.54
0.571
0.294
44.01
56.00
Table 2: 3D consistent video generation on Minecraft . Methods generate frames 257 – 2048 from 256 ground-truth context frames. Column groups are distance past the context. FID pools a band’s frames, FVD its 16 -frame clips.
Figure 7: What the slot memory stores, and how a read recovers it. Smart -SM on 401 -frame maze episodes: the forward leg is written, the reverse leg read from the state. Write. (a) Euclidean distance between two forward frame positions and (b) the probability that their region tokens land in the same slot. Read. (d) Mean read mass per slot against the distance from the query to the content that slot currently holds where content position is the decayed average of all content written to respective slot. (e, f) Attribution of each reverse read to the forward frames in memory slots where the white trace is the pose-nearest forward frame.
Variant
401 frames
1601 frames
3201 frames
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Smart-Slot Memory
32.7
0.940
0.024
32.4
0.936
0.024
31.9
0.931
0.026
No SE(3) Value Warping
31.5
0.924
0.031
30.0
0.906
0.041
29.6
0.900
0.045
Smart-Dense Memory
32.9
0.939
0.020
26.5
0.852
0.074
21.5
0.756
0.191
Scaling Down Heads (/2)
32.2
0.935
0.028
24.5
0.810
0.146
22.9
0.772
0.192
Table 3: Memory Only ablation results. Reconstruction quality on Memory Maze , changing one component of Smart at a time. Smart applies the SE(3) value transform to the 64 value channels of the head. All models are trained on 401 frame episodes.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Evaluation trajectories. Top-down view of one episode per dataset: (a) a Memory Maze forward–turn–return rollout, and (b) a Minecraft continuation rollout.
Figure 9: Attention masks. Allowed attention between query and key chunks for the generators in this work: (a) the bidirectional pretraining of Section A.4 , (b) the full-history causal DiT of the Causal (Full) arm, (c) the windowed chunk-causal attention of Smart , in which a generation chunk reads the W clean frames before it and the memory’s draft. Anchor tokens are the clean frame 0 .
Memory Maze
Minecraft
Backbone
Wan2.1-style DiT ( Wan et al., 2025 )
DiT-B ( Peebles and Xie, 2023 ; Chandratreya et al., 2026 )
Parameters
250 M
130 M
Blocks × width × heads
14×1,024×8
12×768×12
Control signal
camera pose (PRoPE)
discrete action for modulation
Positional code
temporal and 2D rotary, camera PRoPE
sinusoidal over frame, row and column
Autoencoder
Wan2.1 VAE, finetuned
MilliVid hierarchical autoencoder, pretrained
Appendix
Table 4: Instantiation of Smart on the two datasets. The memory interface, chunk size and context window are shared; everything else follows the backbone each dataset inherits.
Variant
401 frames
1601 frames
3201 frames
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Smart-Slot Memory
32.7
0.940
0.024
32.4
0.936
0.024
31.9
0.931
0.026
Scaling Up Memory Slots (×2)
33.0
0.943
0.023
33.3
0.947
0.020
33.2
0.946
0.020
Appendix
Table 5: Memory capacity. Reconstruction quality of the memory alone on Memory Maze , trained on 401 -frame episodes. Doubling the memory parameters gains 0.3 dB at 401 frames and 1.3 dB at 3,201 .
Method
(25+21)+25=71 frames
(51+21)+51=123 frames
(101+21)+101=223 frames
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
StateSpaceDiffuser
28.5
0.874
0.145
24.8
0.806
0.177
21.0
0.717
0.290
Smart
30.2
0.906
0.108
32.3
0.931
0.073
33.3
0.939
0.039
Appendix
Table 6: Comparison against StateSpaceDiffuser ( Savov et al., 2026 ) reimplementation on Memory Maze . Both generators share the DiT architecture and draft conditioning differing in how the memory is conditioned and read. Column groups are total trajectory length. PSNR and SSIM are computed over each target frame that were visible during the forward leg and averaged per frame.
Figure 10: Qualitative comparison on Memory Maze. Generated frames for each method in Table 1 , on the same trajectory with 823 frame context frame and 801 target frame generation.
Figure 11: Additional qualitative comparisons on Minecraft . Generated frames for each method in Table 2 , on the same trajectory, at the 2048 frame horizon with 256 initial context frames. Note that only Smart faithfully retrieves the position of the sun.
Figure 12: Additional qualitative comparisons on Minecraft .
Figure 13: Additional qualitative comparisons on Minecraft .