MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

Jiacheng Dong; Huan Li; Sicheng Zhou; Wenhao Hu; Weili Xu; Yan Wang

cs.CVMar 16, 2026

MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

Authors: Jiacheng Dong, Huan Li, Sicheng Zhou, Wenhao Hu, Weili Xu, Yan Wang

Organizations: Zhejiang University · Institute for AI Industry Research, Tsinghua University

Abstract

Reconstruction is a fundamental task in 3D vision and a fundamental capability for spatial intelligence. Particularly, streaming 3D reconstruction is central to real-time spatial perception, yet existing recurrent online models often suffer from progressive degradation on long sequences due to state drift and forgetting, motivating inference-time remedies. We present MeMix, a training-free, plug-and-play module that improves streaming reconstruction by recasting the recurrent state into a Memory Mixture. MeMix partitions the state into multiple independent memory patches and updates only the least-aligned memory patches while exactly preserving others. This selective update mitigates catastrophic forgetting while retaining $O(1)$ inference memory, and requires no fine-tuning or additional learnable parameters, making it directly applicable to existing recurrent reconstruction models. Across standard benchmarks (ScanNet, 7-Scenes, KITTI, etc.), under identical backbones and inference settings, MeMix reduces reconstruction completeness error by 15.3% on average (up to 40.0%) across 300--500 frame streams on 7-Scenes. The code is available at https://dongjiacheng06.github.io/MeMix/

Figures & tables

Figure 1: MeMix. A training-free, plug-and-play state-update module for recurrent streaming 3D reconstruction. MeMix recasts the recurrent state as a mixture of memory patches, updates Bottom-k patches and preserves the rest. This reduces interference and improves long-horizon stability with $O(1)$ inference memory.

Figure 2: Where to write: Mixture Memory Updates. (a) CUT3R overwrites all state tokens at every timestep. (b) TTT3R applies a dense per-token gate to modulate how much to write, but still updates every token. (c–d) MeMix enables where-to-write updates via Mixture Memory: only a subset of memory patches/tokens are written while the rest are exactly preserved, and it can be plugged into CUT3R (c) or combined with TTT-style gating (d). Colored token squares $\blacksquare\!\blacksquare\!\blacksquare$ indicate tokens that are progressively reinforced over time.

Figure 3: Overview of MeMix. A ViT encoder encodes each frame to tokens $\mathbf{X}_{t}$ , which interact with state tokens $\mathbf{S}_{t-1}$ through a dual-stream cross-attention decoder to produce predictions $\mathbf{Y}_{t}$ and candidate state $\hat{\mathbf{S}}_{t}$ . MeMix computes dot scores between $\hat{\mathbf{S}}_{t}$ and ${\mathbf{X}_{t}}$ , selects Bottom-k patches to construct a binary mask $\mathbf{M}_{t}$ , updating only Bottom-K patches. Decoded image tokens $\mathbf{Y}_{t}$ are fed to the prediction head for output.

Method	Rule
Unified	$S_{t}=G_{t}\odot\hat{S}_{t}$ $+(1-G_{t})\odot S_{t-1}$
CUT3R	$G_{t}=1$
TTT3R/TTSA3R	$G_{t}=\beta_{t}$
CUT3R + MeMix	$G_{t}=M_{t}$
TTT3R/TTSA3R + MeMix	$G_{t}=M_{t}\odot\beta_{t}$

Table 1: Unified memory update rules. The methods in Section 3 can be expressed under a shared gated state update framework. CUT3R corresponds to full-state overwrite, TTT3R/TTSA3R to dense token-wise gating, MeMix to sparse binary routing, and their combination to sparse routed soft gating.

Model

MeMix

Input

7-Scenes-S

NRGBD-S

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

VGGT (Offline) [ 2 ]

–

300

OOM

–

400

OOM

–

500

OOM

StreamVGGT [ 25 ]

–

300

OOM

Table 2: 3D Reconstruction Results on 7-Scenes [ 48 ] and NRGBD [ 49 ] . We test MeMix on 7-Scenes and NRGBD, with one frame sampled every two frames (Sparse Sampling, -S ). Green boxes indicate improved or unchanged performance over the base model (w/o MeMix) under the same input length.

Figure 4: Qualitative results of 3D reconstruction. We compare CUT3R, TTT3R, and TTSA3R with their MeMix variants on long input streams. MeMix consistently improves reconstruction quality by reducing drift and recovering more complete, sharper surfaces. Red boxes highlight representative regions where MeMix corrects failures such as surface tearing, missing geometry, and ghosting.

Figure 5: Long-sequence pose estimation on TUM and ScanNet. We compare CUT3R, TTT3R, and TTSA3R with their MeMix variants on long input streams, and report the absolute trajectory error (ATE) as the number of input views increases.

Figure 6: Evaluation on Video Depth Estimation. We compare CUT3R, TTT3R and TTSA3R with their w/o MeMix version under long input streams. MeMix generally improves depth estimation quality for input lengths ranging from 50 to 1000 frames. Notably, the outcomes largely depend on the capacity of the original model.

Variant	Depth@KITTI		Camera@TUM			3R@NRGBD
Variant	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	ATE $\downarrow$	RPE ${}_{\text{trans}}$ $\downarrow$	RPE ${}_{\text{rot}}$ $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
Default (TTT3R with MeMix)	0.103	92.1	0.028	0.013	0.376	0.099	0.020	0.616
Patch selection
Top-k	0.102	91.8	0.068	0.021	0.595	0.178	0.049	0.587
Random-k	0.108	91.4	0.028	0.013	0.382	0.102	0.023	0.616
Scoring function

Table 3: Ablations on routing policy and score design. The Default row corresponds to Bottom-k + Dot +

\text{score}(\hat{\mathbf{S}}_{t},{\mathbf{X}}_{t})

; other rows change one component at a time.

Figure 7: Sensitivity to k. We present a representative sweep of k on KITTI video depth estimation and TUM camera pose estimation for CUT3R and TTT3R with/without MeMix.

Figure 8: Visualization of distinct strategies. We employ Top-k and Bottom-k strategies separately, and tally the state tokens that are updated in each input frame. Representative examples show that Bottom-k not only achieves a higher update frequency, but also yields a more balanced update distribution across all state tokens.

Model

MeMix

Input

7-Scenes-D

NRGBD-D

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Acc.

\downarrow

Comp.

\downarrow

\uparrow

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

Mean

Med.

VGGT (Offline) [ 2 ]

–

300

OOM

–

400

OOM

–

500

OOM

StreamVGGT [ 25 ]

–

300

OOM

Table 5: 3D Reconstruction Results on 7-Scenes [ 48 ] and NRGBD [ 49 ] . We test MeMix on 7-Scenes and NRGBD, with every frame sampled (Dense Sampling, -D) . Green boxes indicate improved or unchanged performance over the base model (w/o MeMix) under the same input length.

		TUM-dynamics (90 frames)			ScanNet (90 frames)			Sintel (50 frames)
Method	Online	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
Robust-CVD [ 57 ]	$\times$	0.153	0.026	3.528	0.227	0.064	7.374	0.360	0.154	3.443
CasualSAM [ 58 ]	$\times$	0.071	0.010	1.712	0.158	0.034	1.618	0.141	0.035	0.615
DUSt3R [ 1 ]	$\times$	0.083	0.017	3.567	0.081	0.028	0.784	0.417	0.250	5.796
MASt3R [ 18 ]	$\times$	0.038	0.012	0.448	0.078	0.020	0.475	0.185	0.060	1.496
MonST3R [ 44 ]	$\times$	0.098	0.019	0.935	0.077	0.018	0.529	0.111	0.044	0.869

Table 6: Evaluation on Short-Sequence Pose Estimation. To show that MeMix does not undermine performance under short-sequence evaluation, we evaluate on three datasets using input clips shorter than 100 frames. Green boxes indicate improved or unchanged performance over the corresponding base model (w/o MeMix).

Alignment	Method	Online	KITTI (110 frames)		Sintel (50 frames)		Bonn (110 frames)
Alignment	Method	Online	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
Per-sequence scale	DUSt3R-GA [ 1 ]	$\times$	0.144	81.3	0.656	45.2	0.155	83.3
	MASt3R-GA [ 18 ]	$\times$	0.183	74.5	0.641	43.9	0.252	70.1
	MonST3R-GA [ 44 ]	$\times$	0.168	74.4	0.378	55.8	0.067	96.3
	Easi3R [ 45 ]	$\times$	0.102	91.2	0.377	55.9	0.059	97.0
	VGGT [ 2 ]	$\times$	0.070	96.5	0.287	66.1	0.055	97.1

Table 7: Video Depth Estimation. We evaluate scale-invariant and metric depth accuracy on KITTI [ 50 ] , Sintel [ 52 ] , and Bonn [ 51 ] datasets. Methods that require global alignment are denoted as "GA" and green boxes indicate metrics that improve compared to corresponding base model (w/o MeMix).

Figure 9: Visualization of Estimated Camera Trajectories – Long Sequence. We compare $\bullet$ GT , $\bullet$ baseline trajectories , and $\bullet$ their corresponding MeMix variants across three backbones: CUT3R, TTT3R, and TTSA3R. Across different backbones, the MeMix variants generally stay closer to the ground-truth trajectories and exhibit reduced drift over long sequences.

Streaming 3D Reconstruction

Streaming 3D reconstruction incrementally builds geometrically consistent scene maps from incoming images or video under tight memory and latency limits.

See all 99 papers

Mem3R: Streaming 3D Reconstruction with Hybrid Memory
Apr 8, 2026Changkun Liu, Jiezhi Yang, Zeman Li +4Streaming 3D Reconstruction Memory-Augmented Neural Networks
Google · The Hong Kong University of Science and Technology · University of Southern California
ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction
Jul 6, 2026Xinze Li, Yiyuan Wang, Pengxu Chen +4Streaming 3D Reconstruction 3D Scene Representation
Beijing Normal-Hong Kong Baptist University · Hong Kong Baptist University · Jilin University +2
TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction
Jan 30, 2026Zhijie Zheng, Xinhao Xiang, Jiawei ZhangStreaming 3D Reconstruction 3D Reconstruction
University of California, Davis CA 95616, USA

Apr 8, 2026J/K move · Enter open · S save

Mem3R: Streaming 3D Reconstruction with Hybrid Memory

Changkun Liu, Jiezhi Yang, Zeman Li, Yuan Deng+3

Google · The Hong Kong University of Science and Technology · University of Southern California

Streaming 3D Reconstruction Memory-Augmented Neural Networks Visual Odometry

Open paper AskPDF arXiv

Streaming 3D perception is well suited to robotics and augmented reality, where long visual streams must be processed efficiently and consistently. Recent recurrent models offer a promising solution by maintaining fixed-size states and enabling linear-time inference, but they often suffer from temporal forgetting over long sequences due to the limited capacity of compressed memories. We propose Mem3R, a streaming 3D reconstruction model with a hybrid memory design that decouples camera tracking from geometric mapping to improve temporal consistency over long sequences. For camera tracking, Mem3R employs an implicit fast-weight memory implemented as a lightweight Multi-Layer Perceptron updated via Test-Time Training. For geometric mapping, Mem3R maintains an explicit token-based fixed-size state. Compared with CUT3R, this design not only significantly improves long-sequence performance but also reduces the model size from 793M to 644M parameters. Mem3R supports existing improved plug-and-play state update strategies developed for CUT3R. Specifically, Mem3R combined with TTT3R reduces absolute trajectory error by up to 39% relative to CUT3R combined with TTT3R on 500- to 1000-frame sequences from the TUM Dynamics dataset. The resulting improvements also extend to other downstream tasks, including video depth estimation and 3D reconstruction. Project page: https://lck666666.github.io/Mem3R/.

Abstract

Streaming 3D Reconstruction

Explore similar work

Mem3R: Streaming 3D Reconstruction with Hybrid Memory

ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction