Mem3R: Streaming 3D Reconstruction with Hybrid Memory

Changkun Liu; Jiezhi Yang; Zeman Li; Yuan Deng; Jiancong Guo; Binbin Xu; Luca Ballan

cs.CVApr 8, 2026

Mem3R: Streaming 3D Reconstruction with Hybrid Memory

Authors: Changkun Liu, Jiezhi Yang, Zeman Li, Yuan Deng, Jiancong Guo, Binbin Xu, Luca Ballan

Organizations: Google · The Hong Kong University of Science and Technology · University of Southern California

Abstract

Streaming 3D perception is well suited to robotics and augmented reality, where long visual streams must be processed efficiently and consistently. Recent recurrent models offer a promising solution by maintaining fixed-size states and enabling linear-time inference, but they often suffer from temporal forgetting over long sequences due to the limited capacity of compressed memories. We propose Mem3R, a streaming 3D reconstruction model with a hybrid memory design that decouples camera tracking from geometric mapping to improve temporal consistency over long sequences. For camera tracking, Mem3R employs an implicit fast-weight memory implemented as a lightweight Multi-Layer Perceptron updated via Test-Time Training. For geometric mapping, Mem3R maintains an explicit token-based fixed-size state. Compared with CUT3R, this design not only significantly improves long-sequence performance but also reduces the model size from 793M to 644M parameters. Mem3R supports existing improved plug-and-play state update strategies developed for CUT3R. Specifically, Mem3R combined with TTT3R reduces absolute trajectory error by up to 39% relative to CUT3R combined with TTT3R on 500- to 1000-frame sequences from the TUM Dynamics dataset. The resulting improvements also extend to other downstream tasks, including video depth estimation and 3D reconstruction. Project page: https://lck666666.github.io/Mem3R/.

Figures & tables

Figure 2: Overview of Mem3R. Top: For each frame $I_{t}$ in the streaming image sequence, a ViT encoder extracts image features $F_{t}$ . The MLP-based fast-weight module $\mathcal{W}$ performs camera tracking with decoder under Test-Time Training (TTT), while the fixed-size state $\mathcal{S}$ and decoder preserve and update geometric information, producing an intermediate state $S_{t}^{\prime}$ . $S_{t}^{\prime}$ is then fused with the previous state $S_{t-1}$ through a channel-wise update gate module (see Sec. 3.3 ) to obtain the final updated state $S_{t}$ . By replacing pose-related state token with $\mathcal{W}$ , Mem3R maintains a constant-size recurrent state while reducing long-term temporal forgetting in streaming 3D reconstruction. Bottom: Illustration of the fast-weight and decoder module with TTT (see Sec. 3.2 ).

Figure 3: Quantitative evaluation of camera pose estimation from the ScanNet dataset (left) and the TUM Dynamics dataset (right). Mem3R achieves significant improvement compared with CUT3R. Moreover, when combined with the training-free, plug-and-play state update strategies TTT3R or TTSA3R, Mem3R further yields better results compared to applying these strategies to CUT3R.

Figure 4: Qualitative visualization of predicted camera trajectories on the TUM Dynamics dataset (top row) and the ScanNet dataset (bottom row). Each row is evaluated on the same sequence. Our Mem3R achieves significant improvement compared with CUT3R. Moreover, when combined with TTT3R or TTSA3R, Mem3R further reduces drift compared to applying these strategies to CUT3R.

Metric Depth
Method	300 frames		350 frames		400 frames		450 frames		500 frames
Method	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$
CUT3R	0.134	83.8	0.140	82.7	0.144	82.1	0.149	81.1	0.152	80.4
Ours	0.119	88.4	0.123	87.4	0.126	87.1	0.129	86.8	0.132	86.0
TTT3R	0.117	88.5	0.122	87.7	0.125	87.3	0.130	86.6	0.132	86.5
Ours + TTT3R	0.112	89.5	0.115	88.9	0.119	88.6	0.122	88.2	0.125	87.5

Table 1: Quantitative evaluation of video depth estimation on long sequences from the KITTI dataset. Bold indicates the best result per column within each depth category, and green indicates improved or maintained performance of our model over its respective base counterpart.

Metric Depth
Method	300 frames		350 frames		400 frames		450 frames		500 frames
Method	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$
CUT3R	0.106	88.8	0.107	88.8	0.106	89.5	0.101	90.2	0.10	90.6
Ours	0.105	89.7	0.107	89.6	0.105	90.2	0.101	90.8	0.10	91.2
TTT3R	0.108	90.2	0.105	90.8	0.103	91.3	0.102	91.8	0.10	92.1
Ours + TTT3R	0.105	91.4	0.101	91.9	0.100	92.3	0.099	92.6	0.098	92.9

Table 2: Quantitative evaluation on the Bonn dataset. Bold indicates the absolute best result per column within each depth category; green indicates improved or maintained performance over its direct baseline.

Figure 5: Quantitative evaluation of 3D Reconstruction on 7-Scenes. OOM denotes out-of-memory.

Figure 6: Qualitative 3D reconstruction results on long sequences. Red circles mark some representative regions where the difference is particularly clear.

Method	Runtime (fps) $\uparrow$	Memory (MiB) $\downarrow$	Params $\downarrow$
CUT3R	26	7930	793M
Ours	26	7340	644M
TTT3R	25	8364	793M
Ours + TTT3R	25	7774	644M
TTSA3R	25	8786	793M
Ours + TTSA3R	25	8208	644M

Table 3: Efficiency comparison of runtime (fps), GPU memory usage (MiB), and parameter count. Green indicates that our model matches or outperforms its corresponding base model.

TUM Dynamics: ATE (m) $\downarrow$
Method	#frames
	200	400	600	800	1000
TTT3R	0.041	0.051	0.074	0.091	0.110
Ours + TTT3R (w/o. gate)	0.031	0.044	0.059	0.090	0.100
Ours + TTT3R (w/o. fast weight)	0.027	0.039	0.054	0.067	0.078
Ours + TTT3R	0.025	0.037	0.046	0.060	0.073

Table 4: Ablation study of Mem3R. Both variants are independently retrained after removing the corresponding module. 1. w/o. channel-wise gate replaces CUT3R’s original 152M-parameter Transformer-based pose LocalMemory with our implicit fast-weight pose memory, while removing the proposed channel-wise state update mechanism. 2. w/o. fast weight uses CUT3R’s original 152M-parameter Transformer-based pose LocalMemory while retaining the proposed channel-wise update mechanism.

Figure 7: Qualitative 3D reconstruction results on long sequences. Red circles mark some representative regions where the difference is particularly clear.

Figure 8: Qualitative 3D reconstruction results on long sequences. Red circles mark some representative regions where the difference is particularly clear.

Figure 9: Qualitative visualization of predicted camera trajectories on the TUM Dynamics dataset (top row) and the ScanNet dataset (bottom row). Each row is evaluated on the same sequence. Our Mem3R achieves significant improvement compared with CUT3R. Moreover, when combined with TTT3R or TTSA3R, Mem3R further reduces drift compared to applying these strategies to CUT3R.

Dataset	Memory Update Interval
Dataset	1	2	4	6	8	10
ScanNet	0.29	0.25	0.29	0.37	0.43	0.47
TUM Dynamics	0.073	0.056	0.071	0.077	0.095	0.10

Table 5: ATE (m) (

\downarrow

) on 1,000-image sequences for ScanNet and TUM Dynamics. We evaluate Mem3R equipped with TTT3R across various memory update strides.

Training length	50	100	150	200	400	600	800	1000
Ours, 16 views	0.042	0.085	0.141	0.216	0.518	0.685	0.740	0.782
Ours, 32 views	0.039	0.083	0.136	0.196	0.466	0.600	0.654	0.690
Ours, 64 views	0.040	0.083	0.138	0.200	0.432	0.584	0.650	0.706
Ours, 96 views	0.039	0.082	0.142	0.207	0.421	0.569	0.635	0.688

Table 6: Effect of the maximum training sequence length on long-horizon relative camera pose estimation on ScanNet. We report ATE

\downarrow

at different evaluation sequence lengths. Best results are shown in bold .

TUM Dynamics
#frames	50	100	150	200	300	400	500	600	700	800	900	1000
CUT3R	0.023	0.033	0.043	0.061	0.093	0.12	0.14	0.14	0.15	0.17	0.17	0.17
Ours	0.012	0.020	0.031	0.042	0.064	0.078	0.096	0.11	0.13	0.14	0.15	0.15
TTT3R	0.014	0.020	0.031	0.041	0.045	0.051	0.067	0.074	0.085	0.091	0.102	0.11
Ours + TTT3R	0.011	0.015	0.019	0.025	0.031	0.037	0.041	0.046	0.053	0.060	0.066	0.073
TTSA3R	0.012	0.018	0.025	0.030	0.038	0.043	0.056	0.064	0.074	0.079	0.086	0.091

Table 7: ATE (m) (

\downarrow

) of relative camera pose estimation on long sequences on TUM Dynamics and ScanNet. Bold denotes the best result in each column within each dataset, and green indicates that our model matches or outperforms its corresponding base model.

		Sintel (50 frames)
Method	Online	Camera pose			Depth (Per-seq.)		Depth (Metric)
		ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$
DUSt3R-GA [ 63 ]	✗	0.417	0.250	5.796	0.656	45.2	–	–
MASt3R-GA [ 33 ]	✗	0.185	0.060	1.496	0.641	43.9	1.02	14.3
MonST3R-GA [ 78 ]	✗	0.111	0.044	0.869	0.378	55.8	–	–
Easi3R [ 13 ]	✗	0.110	0.042	0.758	0.377	55.9	–	–

Table 8: Quantitative evaluation of camera pose and depth estimation on short sequences from the Sintel [ 10 ] dataset. For depth estimation, we report results under both per-sequence scale ( Per-seq. ) and metric-scale settings, when available. GA denotes global alignment. ‘–‘ indicates that metric-scale depth is unavailable for methods that do not support metric-scale estimation. Best results are shown in bold .

Method	Online	KITTI (110 frames)		Bonn (110 frames)
		Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$	Abs Rel $\downarrow$	$\delta_{1.25}\uparrow$
DUSt3R-GA [ 63 ]	✗	0.144	81.3	0.155	83.3
MASt3R-GA [ 33 ]	✗	0.183	74.5	0.252	70.1
MonST3R-GA [ 78 ]	✗	0.168	74.4	0.067	96.3
Easi3R [ 13 ]	✗	0.102	91.2	0.059	97.0
VGGT [ 61 ]	✗	0.070	96.5	0.055	97.1

Table 9: Video Depth Estimation on short sequence. We evaluate scale-invariant depth accuracy on KITTI and Bonn datasets. Methods that require global alignment are denoted as “GA”.

NRGBD Dataset
Method	200 frames		250 frames		300 frames		350 frames		400 frames
Method	CD $\downarrow$	NC $\uparrow$	CD $\downarrow$	NC $\uparrow$	CD $\downarrow$	NC $\uparrow$	CD $\downarrow$	NC $\uparrow$	CD $\downarrow$	NC $\uparrow$
VGGT (offline)	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
Stream-VGGT	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
CUT3R	0.081	0.602	0.130	0.594	0.162	0.576	0.188	0.566	0.220	0.552
Ours	0.060	0.612	0.085	0.609	0.101	0.592	0.114	0.585	0.135	0.577

Table 10: Quantitative evaluation of multi-view reconstruction. OOM denotes out-of-memory. Bold denotes the best result in each column within each dataset, and green indicates that our model matches or outperforms its corresponding base model. For the NRGBD dataset, one test sequence contains fewer than 400 frames.

Streaming 3D Reconstruction

Streaming 3D reconstruction incrementally builds geometrically consistent scene maps from incoming images or video under tight memory and latency limits.

See all 99 papers

MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction
Mar 16, 2026Jiacheng Dong, Huan Li, Sicheng Zhou +3Streaming 3D Reconstruction 3D Reconstruction
Zhejiang University · Institute for AI Industry Research, Tsinghua University
TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction
Jan 30, 2026Zhijie Zheng, Xinhao Xiang, Jiawei ZhangStreaming 3D Reconstruction 3D Reconstruction
University of California, Davis CA 95616, USA
Stream3D: Sequential Multi-View 3D Generation via Evidential Memory
May 20, 2026Kaichen Zhou, Zeyang Bai, Xinhai Chang +3Streaming 3D Reconstruction 3D Reconstruction
World Mind Lab, HKUST · Media Lab and EECS, MIT · Kempner

Mar 16, 2026J/K move · Enter open · S save

MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

Jiacheng Dong, Huan Li, Sicheng Zhou, Wenhao Hu+2

Zhejiang University · Institute for AI Industry Research, Tsinghua University

Streaming 3D Reconstruction 3D Reconstruction Catastrophic Forgetting

Open paper AskPDF arXiv

Reconstruction is a fundamental task in 3D vision and a fundamental capability for spatial intelligence. Particularly, streaming 3D reconstruction is central to real-time spatial perception, yet existing recurrent online models often suffer from progressive degradation on long sequences due to state drift and forgetting, motivating inference-time remedies. We present MeMix, a training-free, plug-and-play module that improves streaming reconstruction by recasting the recurrent state into a Memory Mixture. MeMix partitions the state into multiple independent memory patches and updates only the least-aligned memory patches while exactly preserving others. This selective update mitigates catastrophic forgetting while retaining

O(1)

inference memory, and requires no fine-tuning or additional learnable parameters, making it directly applicable to existing recurrent reconstruction models. Across standard benchmarks (ScanNet, 7-Scenes, KITTI, etc.), under identical backbones and inference settings, MeMix reduces reconstruction completeness error by 15.3% on average (up to 40.0%) across 300--500 frame streams on 7-Scenes. The code is available at https://dongjiacheng06.github.io/MeMix/

Abstract

Streaming 3D Reconstruction

Explore similar work

MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

Stream3D: Sequential Multi-View 3D Generation via Evidential Memory