MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

Jiacheng Dong; Huan Li; Sicheng Zhou; Wenhao Hu; Weili Xu; Yan Wang

cs.CVMar 16, 2026

MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

Authors: Jiacheng Dong, Huan Li, Sicheng Zhou, Wenhao Hu, Weili Xu, Yan Wang

Organizations: Zhejiang University · Institute for AI Industry Research, Tsinghua University

Abstract

Reconstruction is a fundamental task in 3D vision and a fundamental capability for spatial intelligence. Particularly, streaming 3D reconstruction is central to real-time spatial perception, yet existing recurrent online models often suffer from progressive degradation on long sequences due to state drift and forgetting, motivating inference-time remedies. We present MeMix, a training-free, plug-and-play module that improves streaming reconstruction by recasting the recurrent state into a Memory Mixture. MeMix partitions the state into multiple independent memory patches and updates only the least-aligned memory patches while exactly preserving others. This selective update mitigates catastrophic forgetting while retaining $O(1)$ inference memory, and requires no fine-tuning or additional learnable parameters, making it directly applicable to existing recurrent reconstruction models. Across standard benchmarks (ScanNet, 7-Scenes, KITTI, etc.), under identical backbones and inference settings, MeMix reduces reconstruction completeness error by 15.3% on average (up to 40.0%) across 300--500 frame streams on 7-Scenes. The code is available at https://dongjiacheng06.github.io/MeMix/

Figures & tables

Figure 1: MeMix. A training-free, plug-and-play state-update module for recurrent streaming 3D reconstruction. MeMix recasts the recurrent state as a mixture of memory patches, updates Bottom-k patches and preserves the rest. This reduces interference and improves long-horizon stability with $O(1)$ inference memory.

Figure 2: Where to write: Mixture Memory Updates. (a) CUT3R overwrites all state tokens at every timestep. (b) TTT3R applies a dense per-token gate to modulate how much to write, but still updates every token. (c–d) MeMix enables where-to-write updates via Mixture Memory: only a subset of memory patches/tokens are written while the rest are exactly preserved, and it can be plugged into CUT3R (c) or combined with TTT-style gating (d). Colored token squares $\blacksquare\!\blacksquare\!\blacksquare$ indicate tokens that are progressively reinforced over time.

Figure 3: Overview of MeMix. A ViT encoder encodes each frame to tokens $\mathbf{X}_{t}$ , which interact with state tokens $\mathbf{S}_{t-1}$ through a dual-stream cross-attention decoder to produce predictions $\mathbf{Y}_{t}$ and candidate state $\hat{\mathbf{S}}_{t}$ . MeMix computes dot scores between $\hat{\mathbf{S}}_{t}$ and ${\mathbf{X}_{t}}$ , selects Bottom-k patches to construct a binary mask $\mathbf{M}_{t}$ , updating only Bottom-K patches. Decoded image tokens $\mathbf{Y}_{t}$ are fed to the prediction head for output.

Method	Rule
Unified	$S_{t}=G_{t}\odot\hat{S}_{t}$ $+(1-G_{t})\odot S_{t-1}$
CUT3R	$G_{t}=1$
TTT3R/TTSA3R	$G_{t}=\beta_{t}$
CUT3R + MeMix	$G_{t}=M_{t}$
TTT3R/TTSA3R + MeMix	$G_{t}=M_{t}\odot\beta_{t}$

Table 1: Unified memory update rules. The methods in Section 3 can be expressed under a shared gated state update framework. CUT3R corresponds to full-state overwrite, TTT3R/TTSA3R to dense token-wise gating, MeMix to sparse binary routing, and their combination to sparse routed soft gating.

Model

MeMix

Input

7-Scenes-S

NRGBD-S

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

VGGT (Offline) [ 2 ]

–

300

OOM

–

400

OOM

–

500

OOM

StreamVGGT [ 25 ]

–

300

OOM

Table 2: 3D Reconstruction Results on 7-Scenes [ 48 ] and NRGBD [ 49 ] . We test MeMix on 7-Scenes and NRGBD, with one frame sampled every two frames (Sparse Sampling, -S ). Green boxes indicate improved or unchanged performance over the base model (w/o MeMix) under the same input length.

Figure 4: Qualitative results of 3D reconstruction. We compare CUT3R, TTT3R, and TTSA3R with their MeMix variants on long input streams. MeMix consistently improves reconstruction quality by reducing drift and recovering more complete, sharper surfaces. Red boxes highlight representative regions where MeMix corrects failures such as surface tearing, missing geometry, and ghosting.

Figure 5: Long-sequence pose estimation on TUM and ScanNet. We compare CUT3R, TTT3R, and TTSA3R with their MeMix variants on long input streams, and report the absolute trajectory error (ATE) as the number of input views increases.

Figure 6: Evaluation on Video Depth Estimation. We compare CUT3R, TTT3R and TTSA3R with their w/o MeMix version under long input streams. MeMix generally improves depth estimation quality for input lengths ranging from 50 to 1000 frames. Notably, the outcomes largely depend on the capacity of the original model.

Variant	Depth@KITTI		Camera@TUM			3R@NRGBD
Variant	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	ATE $\downarrow$	RPE ${}_{\text{trans}}$ $\downarrow$	RPE ${}_{\text{rot}}$ $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
Default (TTT3R with MeMix)	0.103	92.1	0.028	0.013	0.376	0.099	0.020	0.616
Patch selection
Top-k	0.102	91.8	0.068	0.021	0.595	0.178	0.049	0.587
Random-k	0.108	91.4	0.028	0.013	0.382	0.102	0.023	0.616
Scoring function

Table 3: Ablations on routing policy and score design. The Default row corresponds to Bottom-k + Dot +

\text{score}(\hat{\mathbf{S}}_{t},{\mathbf{X}}_{t})

; other rows change one component at a time.

Figure 7: Sensitivity to k. We present a representative sweep of k on KITTI video depth estimation and TUM camera pose estimation for CUT3R and TTT3R with/without MeMix.

Figure 8: Visualization of distinct strategies. We employ Top-k and Bottom-k strategies separately, and tally the state tokens that are updated in each input frame. Representative examples show that Bottom-k not only achieves a higher update frequency, but also yields a more balanced update distribution across all state tokens.

Model

MeMix

Input

7-Scenes-D

NRGBD-D

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

VGGT (Offline) [ 2 ]

–

300

OOM

–

400

OOM

–

500

OOM

StreamVGGT [ 25 ]

–

300

OOM

Table 5: 3D Reconstruction Results on 7-Scenes [ 48 ] and NRGBD [ 49 ] . We test MeMix on 7-Scenes and NRGBD, with every frame sampled (Dense Sampling, -D) . Green boxes indicate improved or unchanged performance over the base model (w/o MeMix) under the same input length.

		TUM-dynamics (90 frames)			ScanNet (90 frames)			Sintel (50 frames)
Method	Online	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
Robust-CVD [ 57 ]	$\times$	0.153	0.026	3.528	0.227	0.064	7.374	0.360	0.154	3.443
CasualSAM [ 58 ]	$\times$	0.071	0.010	1.712	0.158	0.034	1.618	0.141	0.035	0.615
DUSt3R [ 1 ]	$\times$	0.083	0.017	3.567	0.081	0.028	0.784	0.417	0.250	5.796
MASt3R [ 18 ]	$\times$	0.038	0.012	0.448	0.078	0.020	0.475	0.185	0.060	1.496
MonST3R [ 44 ]	$\times$	0.098	0.019	0.935	0.077	0.018	0.529	0.111	0.044	0.869

Table 6: Evaluation on Short-Sequence Pose Estimation. To show that MeMix does not undermine performance under short-sequence evaluation, we evaluate on three datasets using input clips shorter than 100 frames. Green boxes indicate improved or unchanged performance over the corresponding base model (w/o MeMix).

Alignment	Method	Online	KITTI (110 frames)		Sintel (50 frames)		Bonn (110 frames)
Alignment	Method	Online	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
Per-sequence scale	DUSt3R-GA [ 1 ]	$\times$	0.144	81.3	0.656	45.2	0.155	83.3
	MASt3R-GA [ 18 ]	$\times$	0.183	74.5	0.641	43.9	0.252	70.1
	MonST3R-GA [ 44 ]	$\times$	0.168	74.4	0.378	55.8	0.067	96.3
	Easi3R [ 45 ]	$\times$	0.102	91.2	0.377	55.9	0.059	97.0
	VGGT [ 2 ]	$\times$	0.070	96.5	0.287	66.1	0.055	97.1

Table 7: Video Depth Estimation. We evaluate scale-invariant and metric depth accuracy on KITTI [ 50 ] , Sintel [ 52 ] , and Bonn [ 51 ] datasets. Methods that require global alignment are denoted as "GA" and green boxes indicate metrics that improve compared to corresponding base model (w/o MeMix).

Figure 9: Visualization of Estimated Camera Trajectories – Long Sequence. We compare $\bullet$ GT , $\bullet$ baseline trajectories , and $\bullet$ their corresponding MeMix variants across three backbones: CUT3R, TTT3R, and TTSA3R. Across different backbones, the MeMix variants generally stay closer to the ground-truth trajectories and exhibit reduced drift over long sequences.

Streaming 3D Reconstruction

Streaming 3D reconstruction incrementally builds geometrically consistent scene maps from incoming images or video under tight memory and latency limits.

See all 99 papers

Apr 8, 2026

Mem3R: Streaming 3D Reconstruction with Hybrid Memory

Streaming 3D perception is well suited to robotics and augmented reality, where long visual streams must be processed efficiently and consistently. Recent recurrent models offer a promising solution by maintaining fixed-size states and enabling linear-time inference, but they often suffer from temporal forgetting over long sequences due to the limited capacity of compressed memories. We propose Mem3R, a streaming 3D reconstruction model with a hybrid memory design that decouples camera tracking from geometric mapping to improve temporal consistency over long sequences. For camera tracking, Mem3R employs an implicit fast-weight memory implemented as a lightweight Multi-Layer Perceptron updated via Test-Time Training. For geometric mapping, Mem3R maintains an explicit token-based fixed-size state. Compared with CUT3R, this design not only significantly improves long-sequence performance but also reduces the model size from 793M to 644M parameters. Mem3R supports existing improved plug-and-play state update strategies developed for CUT3R. Specifically, Mem3R combined with TTT3R reduces absolute trajectory error by up to 39% relative to CUT3R combined with TTT3R on 500- to 1000-frame sequences from the TUM Dynamics dataset. The resulting improvements also extend to other downstream tasks, including video depth estimation and 3D reconstruction. Project page: https://lck666666.github.io/Mem3R/.

Changkun Liu, Jiezhi Yang, Zeman Li +4

Google · The Hong Kong University of Science and Technology · University of Southern California

Streaming 3D Reconstruction Memory-Augmented Neural Networks Visual Odometry

Ask PDF

Jul 6, 2026

ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7

\times

reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.

Xinze Li, Yiyuan Wang, Pengxu Chen +4

Beijing Normal-Hong Kong Baptist University · Hong Kong Baptist University · Jilin University +2

Streaming 3D Reconstruction 3D Scene Representation Adaptive Learning Rate

Ask PDF

Jan 30, 2026

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

Streaming recurrent models enable efficient 3D reconstruction by maintaining persistent state representations. However, they suffer from catastrophic forgetting over long sequences due to balancing historical information with new observations. Recent methods alleviate this by deriving adaptive signals from the attention perspective, but they operate on single dimensions without considering temporal and spatial consistency. To this end, we propose a training-free framework termed TTSA3R that leverages both temporal state evolution and spatial observation quality for adaptive state updates in 3D reconstruction. In particular, we devise a Temporal Adaptive Update Module that regulates update magnitude by analyzing temporal state evolution patterns. Then, a Spatial Context Update Module is introduced to localize spatial regions that require updates through observation-state alignment and observation changes. These complementary signals are finally fused to determine the state updating strategies. Extensive experiments show that TTSA3R achieves competitive performance on standard short-sequence benchmarks and provides substantially stronger robustness on extended sequences. On NRGBD, as sequences extend from 50 to 250 frames, TTSA3R exhibits only a 1.33

\times

error increase, compared with 3.8

\times

degradation for CUT3R. This highlights the practical value of temporal-spatial adaptive updates for long-term reconstruction stability. Our code is available at https://github.com/anonus2357/ttsa3r.

Zhijie Zheng, Xinhao Xiang, Jiawei Zhang

University of California, Davis CA 95616, USA

Streaming 3D Reconstruction 3D Reconstruction Catastrophic Forgetting

Ask PDF

Abstract

Streaming 3D Reconstruction

Explore similar work

Mem3R: Streaming 3D Reconstruction with Hybrid Memory

ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction