Organizations: The authors are with the Institute of Medical Robotics and the School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China, and also with the Key Laboratory of System Control and Information Processing, Ministry of Education, Shanghai 200240, China
Streaming 3D reconstruction must maintain geometric quality over long inputs within a fixed GPU budget. A pretrained recurrent model provides bounded state, but freezing its backbone also fixes its learned state-observation interactions. We introduce Mamba3R, a jointly designed state-memory adapter that provides a trainable adaptation path without updating the backbone or task heads. A selective state-space mixer jointly transforms native state features and a fixed-capacity auxiliary memory, explicit read and write paths carry information across frames, and a decomposed feedback path regulates what the added memory feeds back. Only 1.28 million adapter parameters are trained. Under a shared periodic-reset protocol, it lowers trajectory error and improves accumulated surface and depth accuracy over recurrent baselines on public indoor and outdoor datasets and robot-recorded traversals, gaining up to 40 percent on one metric. Matched comparisons support both the choice of the mixing operator and the utility of carrying the auxiliary memory across frames. Allocated memory remains unchanged and throughput moves by at most 1.4 percent from one to two thousand frames. Replaying its recorded traversals offline, the platform's RTX 4060 averages 11.7 frames per second within 5.1 GiB with the same checkpoint, a mean throughput covering the 10 Hz sensor rate.
Figures & tables
Fig. 1: Overview of Mamba3R. A frozen encoder and recurrent decoder are augmented by the state–memory adapter: it mixes the auxiliary memory with the state carried into the call, its read residual corrects the current frame’s decoder feature before the heads run, and its memory half supplies the write read by the next frame (Section III ; execution order in supplementary Algorithm S1).
Method
CD ↓
ATE ↓
AbsRel ↓
ScanNet [ 29 ] (5 seq.)
CUT3R
0.1696
0.6540
0.05165
TTT3R
0.1142
0.4951
0.05670
TTSA3R
0.8900
2.071
0.04978
Point3R
0.3830
1.068
0.03164
Spann3R †
0.9037
1.972
0.2188
TABLE I: 500-frame geometry on ScanNet and KITTI-360 under matched reset100. CD and ATE in meters; depth uses one sequence-wide scale. † Spann3R camera recovery; TTSA3R protocol notes are in the supplement.
Fig. 2: 500-frame point clouds for the reference (GT) and every evaluated method on ScanNet and KITTI-360; Mamba3R stays registered where the cached and spatial-memory comparators fragment.
Input tier
Mamba3R
TTT3R
CUT3R + reset
1,000
25.54
32.81
57.23
2,000
94.88
105.6
125.0
3,000 (2,848 / 3,000)
100.6
114.4
127.4
TABLE II: Long-sequence KITTI-360 ATE (m ↓ ), averaged over test_2 and test_3 at reset100; tiers are prefixes of the same inputs (the nominal 3,000-frame tier holds 2,848 and 3,000 frames).
500 frames
Method
FPS ↑
Alloc.
Proc.
CUT3R
15.52
5.387
6.522
TTT3R
14.02
5.388
6.849
Point3R
4.608
3.608
8.050
Spann3R †
1.776
2.805
3.757
STream3R- β
7.036
15.70
80.57
TABLE III: Inference cost on H200 at reset100 under the accuracy-table kernel configuration. The 500-frame block averages one ScanNet and one KITTI-360 input. FPS is the synchronized inference-loop rate (stage list in the supplement); Alloc./Proc.: allocated-tensor and sampled process peaks (GiB).
Configuration
CD ↓
ATE ↓
AbsRel ↓
Memory continuation
Zero added memory
0.1021
0.4790
0.04838
Full added memory
0.1009
0.4465
0.04653
New mixer from CUT3R
Pointwise MLP
0.1327
0.4982
0.05864
Cross-slot convolution
0.1190
0.4955
0.05176
TABLE IV: Matched ablations on the five 500-frame ScanNet sequences at reset100: a memory-continuation pair and four mixers at matched step-1000 checkpoints, plus inference-time feedback and decay interventions (Section V-D ). CD/ATE in meters.
Streaming front-end only
Method
FPS ↑
ms/frame ↓
Alloc.
Resv.
Mamba3R
11.734
85.2
4.670
5.131
TTT3R
11.399
87.7
4.650
5.092
CUT3R
11.262
88.8
4.652
5.035
Complete coupled system
Method
Front ↑
p50 (s)
p95 (s)
drain (s)
GPU (GiB)
TABLE V: Onboard RTX 4060 execution at 320×240 , reset100, on a recorded 1,699-frame traversal under the deployment kernel: the streaming front-end alone (top) and the complete coupled system (bottom). Bold: fastest front-end in the top block; the coupled block is unranked, since its columns trade against one another.
ATE ↓
CD ↓
NC ↑
AbsRel ↓
Seq.
Frames
CUT3R
TTT3R
Mamba3R
CUT3R
TTT3R
Mamba3R
CUT3R
TTT3R
Mamba3R
CUT3R
TTT3R
Mamba3R
Indoor
I1
1,989
7.017
4.141
2.919
1.100
0.6690
0.3497
0.5160
0.5092
0.5204
0.2872
0.2608
0.2817
I2
1,591
3.626
2.796
1.423
0.7902
0.5225
0.4271
0.5400
0.5489
0.5574
0.1722
0.1765
0.2064
I3
1,508
2.897
5.269
2.310
0.3512
0.8982
0.2967
0.5101
0.5062
0.5172
0.3762
0.3862
0.3632
I4
1,791
4.453
2.158
1.867
0.6296
0.4884
0.3872
0.5129
0.5204
0.5290
0.2717
0.2758
0.2386
TABLE VI: Streaming front-end on the robot-recorded cohort ( 384×512 , reset100, one full-sequence Sim(3), no back-end). CD/ATE in meters; AbsRel at sequence scale; outdoor rows carry trajectory error only (Section IV-C ). Means cover I1–I10.
Fig. 3: Front-end trajectories on I2, I3, and I8: one full-sequence Sim(3) per method, no ICP or segment-wise fit.
Fig. 4: Local map detail in two fixed metric crops (top-down slab views): a revisited wall on I2 (top) and ScanNet scene0499_00 (bottom); each row is one axis-aligned box in the shared reference frame, so position inside a panel is comparable across panels. Crop CDs are in the supplementary material.
ATE ↓
CD ↓
AbsRel ↓
Method
front
+ back
front
+ back
front
+ back
Single indoor traversal with an internal revisit (I8, 1,640 frames)
TABLE VII: Front-end substitution with the back-end fixed within each block: equal-weight means before and after optimization. ATE/CD in meters; AbsRel at per-session scale; the outdoor block carries trajectory error only. Bold: best post-optimization.
Per-session ATE ↓
Overall
Fused
Method
I4
I5
I6
ATE ↓
CD ↓
TTT3R
2.634
3.871
5.015
3.705
0.419
Mamba3R
1.798
3.041
3.951
2.839
0.345
TABLE VIII: Indoor three-session fusion under one shared global alignment: per-session and overall ATE and fused-map CD (meters) under a single Sim(3) to the registered union reference (Section VII-B ).
Streaming 3D reconstruction incrementally builds geometrically consistent scene maps from incoming images or video under tight memory and latency limits.
University of Macau, Macau SAR, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, Beijing, China