Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory (FSM), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck.
Figures & tables
Figure 2: (Left) Overview of FSM . The model takes a sequence of posed images captured at different times and learns to infer novel view-time combinations. Camera information is converted into Plücker ray maps as geometric augmentation for visual tokens. The model directly predict the target view with decoders. (Right) The LaCET Block. It maintains two sets of parameters, anchor weights and fast weights . During adaptation, the fast weights are updated using information from the current chunk (queries, keys, and values), while the anchor weights act as a stable reference. The model tracks parameter importance online and softly restores critical weights toward their anchors to prevent drift. This stabilizes rapid updates while preserving the adaptability of TTT, addressing the plasticity issue.
Figure 3: FSM -LVSM and FSM -LRM architectural designs. (a) LVSM-style rendering predicts target image patches directly from query tokens and does not build an explicit scene representation. (b) LRM-style rendering first predicts an explicit 4D scene representation with Gaussian primitives and then renders target views from that representation.
Figure 4: Qualitative illustration of the ablation studies, obtained after the same training steps (16K) with the same training and inference random seed on the same Stereo4D test set example.
EWC
Train
Test
Test
Anchor
Fisher
Train
Test
#Chunk
#Chunk
Batch Size
Update
Estimate
ℓ2 Loss ( ×103 ) ↓
PSNR ↑
LPIPS ↓
SSIM ↑
✗
1
1
1
-
-
1.80
26.021
0.1179
0.792
✗
4
4
1
-
-
2.04
26.908
0.0988
0.814
✓
4
4
1
streaming-ema
SI
2.36
29.989
0.0517
0.903
✓
4
4
1
streaming-ema
EWC
2.36
29.781
0.0537
0.897
✓
4
4
1
streaming-ema
MAS
2.28
29.922
0.0519
0.899
Table 1: Ablation Studies. The training ℓ2 loss is reported from the exponential moving average (EMA) model ( α=0.1 ) to ensure robustness against noise. When the number of chunks is 1 , it corresponds to the original full-sequence setup in LaCT. With 4 chunks, each chunk contains 2048 input tokens. We find that EWC effectively mitigates the overfitting issue observed in LaCT due to full plasticity. The streaming-ema anchor update policy proves critical for achieving stable performance.
Dataset
Source
Dyn.
#Frames
#Scenes
Ratio
RealEstate10K [ 77 ]
Real
✗
10M
80K
1
DL3DV [ 30 ]
Real
✗
51M
10K
1
PointOdyssey [ 75 ]
Syn.
✓
6K
131
200
Spring [ 35 ]
Syn.
✓
200K
37
500
Multi-Cam Video [ 2 ]
Syn.
✓
11M
13.6K
1
DynamicReplica [ 20 ]
Real
✓
145K
484
100
Table 2: Summary of datasets. Source indicates whether the dataset is captured from the real world or synthesized. Dyn amic specifies whether the scenes are dynamic. #Frames and #Scenes denote the total number of image frames and unique scenes, respectively. Ratio represents the per-scene sampling multiplier used during training for data balancing.
Figure 5: Test-time scaling curves. Shown are PSNR/SSIM/LPIPS of LaCT (1/4 chunks) and LaCET (4 chunks; streaming-ema), trained with 32 images (vertical line) and evaluated with varying numbers of input images. Each point uses a 136-frame Stereo4D clip. For sparse views , input and target frames are randomly sampled across the long full span. For continuous views , we select a contiguous sub-sequence (e.g., 40 frames for 32-in/8-out) and randomly mask the target frames inside it for the model to predict, reducing to frame interpolation.
Figure 6: Qualitative comparison on Steoro4D test set. Note that for MoVieS we use a higher default resolution (504 × 504).
Model
Stereo4D [ 18 ]
NVIDIA [ 65 ]
Resolution
PSNR ↑
LPIPS ↓
SSIM ↑
Resolution
PSNR ↑
LPIPS ↓
SSIM ↑
Optimization-based
SoM [ 53 ]
—— OOT ⋆ ——
379 × 672
15.30
0.509
0.317
MoSca [ 26 ]
—— OOT ⋆ ——
379 × 672
21.45
0.265
0.712
Rendering-based
L4GM [ 39 ]
—— OOT † ——
256 × 256
10.07
0.587
0.235
Table 3: 4D NVS Results. Metrics are resolution-dependent (e.g., higher resolutions typically produce higher PSNR). We adopt the lowest resolution for meaningful comparison with baselines. Steoro4D test set contains 7109 scenes, which is out of time (OOT) for some methods.
Figure 7: Qualitative comparison on DL3DV benchmark.
Model
DL3DV [ 30 ]
Resolution
PSNR ↑
LPIPS ↓
SSIM ↑
Static Models
DepthSplat [ 59 ]
512 × 448
17.81
0.356
0.596
GS-LRM [ 70 ]
256 × 256
23.02
0.266
0.705
LVSM [ 17 ]
256 × 256
23.10
0.257
0.703
RayZer † [ 16 ]
256 × 256
23.72
0.222
0.733
Table 4: 3D NVS Results. Metrics are resolution-dependent (e.g., higher resolutions typically produce higher PSNR). We adopt the lowest resolution for meaningful comparison with baselines.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Ablation
Base
Resolution
Multi-Length
Parameters
Training
Training
Scaling
Fine-tuning
#layers
12
24
24
24
#input frames
32
32
32
12-64
#target frames
32
32
32
32
resolution
128
128
256
256
temporal window
128
128
256
256
Appendix
Table 5: Summary of configurations across ablation studies, base training, resolution scaling, and variable-length fine-tuneing.
Model
DL3DV [ 30 ]
Stereo4D [ 18 ]
Res.
PSNR ↑
LPIPS ↓
SSIM ↑
Res.
PSNR ↑
LPIPS ↓
SSIM ↑
FSM-LRM
128 × 128
20.99
0.243
0.683
128 × 128
28.19
0.097
0.897
FSM-LVSM
128 × 128
21.25
0.169
0.655
128 × 128
31.06
0.041
0.931
FSM-LVSM (w/ RoPE)
128 × 128
20.75
0.237
0.680
128 × 128
30.54
0.059
0.922
Appendix
Table 6: Additional ablation study results on (i) side-by-side comparison of LVSM-style decoder vs. LRM-style decoder and (ii) using explicit temporal channel vs. using RoPE.
Figure 8: Additional comparison on Steoro4D test set. Note that for MoVieS we use a higher default resolution (504 × 504).
Figure 9: Qualitative examples on Steoro4D test set.
Figure 10: Qualitative failure example.
Figure 11: Qualitative results on NVIDIA benchmark.
Figure 12: Qualitative results on DL3DV-140 benchmark.
Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.
We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.
Streaming recurrent models enable efficient 3D reconstruction by maintaining persistent state representations. However, they suffer from catastrophic forgetting over long sequences due to balancing historical information with new observations. Recent methods alleviate this by deriving adaptive signals from the attention perspective, but they operate on single dimensions without considering temporal and spatial consistency. To this end, we propose a training-free framework termed TTSA3R that leverages both temporal state evolution and spatial observation quality for adaptive state updates in 3D reconstruction. In particular, we devise a Temporal Adaptive Update Module that regulates update magnitude by analyzing temporal state evolution patterns. Then, a Spatial Context Update Module is introduced to localize spatial regions that require updates through observation-state alignment and observation changes. These complementary signals are finally fused to determine the state updating strategies. Extensive experiments show that TTSA3R achieves competitive performance on standard short-sequence benchmarks and provides substantially stronger robustness on extended sequences. On NRGBD, as sequences extend from 50 to 250 frames, TTSA3R exhibits only a 1.33× error increase, compared with 3.8× degradation for CUT3R. This highlights the practical value of temporal-spatial adaptive updates for long-term reconstruction stability. Our code is available at https://github.com/anonus2357/ttsa3r.
Feedforward geometric foundation models achieve strong short-window reconstruction, yet scaling them to minutes-long videos is bottlenecked by quadratic attention complexity or limited effective memory in recurrent designs. We present LoGeR (Long-context Geometric Reconstruction), a novel architecture that scales dense 3D reconstruction to extremely long sequences without post-optimization. LoGeR processes video streams in chunks, leveraging strong bidirectional priors for high-fidelity intra-chunk reasoning. To manage the critical challenge of coherence across chunk boundaries, we propose a learning-based hybrid memory module. This dual-component system combines a parametric Test-Time Training (TTT) memory to anchor the global coordinate frame and prevent scale drift, alongside a non-parametric Sliding Window Attention (SWA) mechanism to preserve uncompressed context for high-precision adjacent alignment. Remarkably, this memory architecture enables LoGeR to be trained on sequences of 128 frames, and generalize up to thousands of frames during inference. Evaluated across standard benchmarks and a newly repurposed VBR dataset with sequences of up to 19k frames, LoGeR substantially outperforms prior state-of-the-art feedforward methods--reducing ATE on KITTI by over 74%--and achieves robust, globally consistent reconstruction over unprecedented horizons.