Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory (FSM), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck.
Figures & tables
Figure 2: (Left) Overview of FSM . The model takes a sequence of posed images captured at different times and learns to infer novel view-time combinations. Camera information is converted into Plücker ray maps as geometric augmentation for visual tokens. The model directly predict the target view with decoders. (Right) The LaCET Block. It maintains two sets of parameters, anchor weights and fast weights . During adaptation, the fast weights are updated using information from the current chunk (queries, keys, and values), while the anchor weights act as a stable reference. The model tracks parameter importance online and softly restores critical weights toward their anchors to prevent drift. This stabilizes rapid updates while preserving the adaptability of TTT, addressing the plasticity issue.
Figure 3: FSM -LVSM and FSM -LRM architectural designs. (a) LVSM-style rendering predicts target image patches directly from query tokens and does not build an explicit scene representation. (b) LRM-style rendering first predicts an explicit 4D scene representation with Gaussian primitives and then renders target views from that representation.
Figure 4: Qualitative illustration of the ablation studies, obtained after the same training steps (16K) with the same training and inference random seed on the same Stereo4D test set example.
EWC
Train
Test
Test
Anchor
Fisher
Train
Test
#Chunk
#Chunk
Batch Size
Update
Estimate
ℓ2 Loss ( ×103 ) ↓
PSNR ↑
LPIPS ↓
SSIM ↑
✗
1
1
1
-
-
1.80
26.021
0.1179
0.792
✗
4
4
1
-
-
2.04
26.908
0.0988
0.814
✓
4
4
1
streaming-ema
SI
2.36
29.989
0.0517
0.903
✓
4
4
1
streaming-ema
EWC
2.36
29.781
0.0537
0.897
✓
4
4
1
streaming-ema
MAS
2.28
29.922
0.0519
0.899
Table 1: Ablation Studies. The training ℓ2 loss is reported from the exponential moving average (EMA) model ( α=0.1 ) to ensure robustness against noise. When the number of chunks is 1 , it corresponds to the original full-sequence setup in LaCT. With 4 chunks, each chunk contains 2048 input tokens. We find that EWC effectively mitigates the overfitting issue observed in LaCT due to full plasticity. The streaming-ema anchor update policy proves critical for achieving stable performance.
Dataset
Source
Dyn.
#Frames
#Scenes
Ratio
RealEstate10K [ 77 ]
Real
✗
10M
80K
1
DL3DV [ 30 ]
Real
✗
51M
10K
1
PointOdyssey [ 75 ]
Syn.
✓
6K
131
200
Spring [ 35 ]
Syn.
✓
200K
37
500
Multi-Cam Video [ 2 ]
Syn.
✓
11M
13.6K
1
DynamicReplica [ 20 ]
Real
✓
145K
484
100
Table 2: Summary of datasets. Source indicates whether the dataset is captured from the real world or synthesized. Dyn amic specifies whether the scenes are dynamic. #Frames and #Scenes denote the total number of image frames and unique scenes, respectively. Ratio represents the per-scene sampling multiplier used during training for data balancing.
Figure 5: Test-time scaling curves. Shown are PSNR/SSIM/LPIPS of LaCT (1/4 chunks) and LaCET (4 chunks; streaming-ema), trained with 32 images (vertical line) and evaluated with varying numbers of input images. Each point uses a 136-frame Stereo4D clip. For sparse views , input and target frames are randomly sampled across the long full span. For continuous views , we select a contiguous sub-sequence (e.g., 40 frames for 32-in/8-out) and randomly mask the target frames inside it for the model to predict, reducing to frame interpolation.
Figure 6: Qualitative comparison on Steoro4D test set. Note that for MoVieS we use a higher default resolution (504 × 504).
Model
Stereo4D [ 18 ]
NVIDIA [ 65 ]
Resolution
PSNR ↑
LPIPS ↓
SSIM ↑
Resolution
PSNR ↑
LPIPS ↓
SSIM ↑
Optimization-based
SoM [ 53 ]
—— OOT ⋆ ——
379 × 672
15.30
0.509
0.317
MoSca [ 26 ]
—— OOT ⋆ ——
379 × 672
21.45
0.265
0.712
Rendering-based
L4GM [ 39 ]
—— OOT † ——
256 × 256
10.07
0.587
0.235
Table 3: 4D NVS Results. Metrics are resolution-dependent (e.g., higher resolutions typically produce higher PSNR). We adopt the lowest resolution for meaningful comparison with baselines. Steoro4D test set contains 7109 scenes, which is out of time (OOT) for some methods.
Figure 7: Qualitative comparison on DL3DV benchmark.
Model
DL3DV [ 30 ]
Resolution
PSNR ↑
LPIPS ↓
SSIM ↑
Static Models
DepthSplat [ 59 ]
512 × 448
17.81
0.356
0.596
GS-LRM [ 70 ]
256 × 256
23.02
0.266
0.705
LVSM [ 17 ]
256 × 256
23.10
0.257
0.703
RayZer † [ 16 ]
256 × 256
23.72
0.222
0.733
Table 4: 3D NVS Results. Metrics are resolution-dependent (e.g., higher resolutions typically produce higher PSNR). We adopt the lowest resolution for meaningful comparison with baselines.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Ablation
Base
Resolution
Multi-Length
Parameters
Training
Training
Scaling
Fine-tuning
#layers
12
24
24
24
#input frames
32
32
32
12-64
#target frames
32
32
32
32
resolution
128
128
256
256
temporal window
128
128
256
256
Appendix
Table 5: Summary of configurations across ablation studies, base training, resolution scaling, and variable-length fine-tuneing.
Model
DL3DV [ 30 ]
Stereo4D [ 18 ]
Res.
PSNR ↑
LPIPS ↓
SSIM ↑
Res.
PSNR ↑
LPIPS ↓
SSIM ↑
FSM-LRM
128 × 128
20.99
0.243
0.683
128 × 128
28.19
0.097
0.897
FSM-LVSM
128 × 128
21.25
0.169
0.655
128 × 128
31.06
0.041
0.931
FSM-LVSM (w/ RoPE)
128 × 128
20.75
0.237
0.680
128 × 128
30.54
0.059
0.922
Appendix
Table 6: Additional ablation study results on (i) side-by-side comparison of LVSM-style decoder vs. LRM-style decoder and (ii) using explicit temporal channel vs. using RoPE.
Figure 8: Additional comparison on Steoro4D test set. Note that for MoVieS we use a higher default resolution (504 × 504).
Figure 9: Qualitative examples on Steoro4D test set.
Figure 10: Qualitative failure example.
Figure 11: Qualitative results on NVIDIA benchmark.
Figure 12: Qualitative results on DL3DV-140 benchmark.
Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.