RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction

Feiran Wang; Zezhou Shang; Gaowen Liu; Yan Yan

cs.CVMar 21, 2026

RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction

Authors: Feiran Wang, Zezhou Shang, Gaowen Liu, Yan Yan

Organizations: University of Illinois Chicago · Cisco Research

Abstract

Streaming feed-forward 3D reconstruction enables real-time joint estimation of scene geometry and camera poses from RGB images. However, without explicit dynamic reasoning, streaming models can be affected by moving objects, causing artifacts and drift. In this work, we propose RayMap3R, a training-free streaming framework for dynamic scene reconstruction. We observe that RayMap-based predictions exhibit a static-scene bias, providing an internal cue for dynamic identification. Based on this observation, we construct a dual-branch inference scheme that identifies dynamic regions by contrasting RayMap and image predictions, suppressing their interference during memory updates. We further introduce reset metric alignment and state-aware smoothing to preserve metric consistency and stabilize predicted trajectories. Our method achieves state-of-the-art performance among streaming approaches on dynamic scene reconstruction across multiple benchmarks.

Figures & tables

Figure 1: Streaming 3D Reconstruction for Dynamic Scenes. RayMap3R leverages the static bias of RayMap-only predictions to identify and suppress dynamic regions at inference time, reducing camera drift and improving geometry fidelity.

Figure 2: RayMap Static Bias. Left: The main prediction and RayMap prediction produce differing depth estimates in dynamic regions, suggesting a static bias in RayMap predictions. Right: The depth discrepancy correlates with ground-truth dynamic ratio across multiple datasets, suggesting a consistent signal for dynamic identification.

Figure 3: Dynamic Map Visualization. The dynamic map is the per-pixel depth difference between the main branch and RayMap-only predictions, closely aligning with ground-truth dynamic masks across both synthetic and real scenes.

Figure 4: Method Overview. Our method performs streaming 3D reconstruction via dual-branch inference. At time step $t$ , the main branch predicts depth and pose from state $s_{t-1}$ using both image and RayMap features. The predicted pose $\hat{\mathbf{T}}_{t}$ is remapped into RayMap and encoded into tokens $r^{\prime}_{t}$ , then queried against the frozen state $s_{t-1}$ by the RayMap branch to obtain a static-biased prediction. The depth difference between branches is projected via image-state attention to form staticness weights $\alpha_{t}$ that suppress dynamic regions during state update ( $s_{t}=s_{t-1}+\alpha_{t}\odot\Delta s_{t}$ ). This dual-branch scheme identifies and suppresses dynamic regions at inference time.

			KITTI [ 13 ]		BONN [ 23 ]		Sintel [ 2 ]
Alignment	Method	Onl.	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$
Per-sequence scale	DUSt3R [ 39 ]	✗	0.144	81.3	0.155	83.3	0.656	45.2
	MonST3R [ 46 ]	✗	0.168	74.4	0.067	96.3	0.378	55.8
	VGGT [ 37 ]	✗	0.070	96.5	0.055	97.1	0.287	66.1
	Spann3R [ 36 ]	✓	0.198	73.7	0.144	81.3	0.622	42.6
	Point3R [ 42 ]	✓	0.135	84.0	0.061	96.2	0.451	48.7

Table 1: Video Depth Estimation. We report scale-invariant relative depth (per-sequence scale alignment) and metric-scale absolute depth accuracy. RayMap3R achieves leading depth accuracy among streaming methods under both settings. ✗ denotes offline methods; ✓ denotes streaming methods.

		Sintel [ 2 ]			TUM-dyn [ 30 ]			ScanNet [ 7 ]
Method	Onl.	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$
DUSt3R [ 39 ]	✗	0.417	0.250	5.796	0.083	0.017	3.567	0.081	0.028	0.784
MASt3R [ 11 ]	✗	0.185	0.060	1.496	0.038	0.012	0.448	0.078	0.020	0.475
MonST3R [ 46 ]	✗	0.111	0.044	0.869	0.098	0.019	0.935	0.077	0.018	0.529
VGGT [ 37 ]	✗	0.172	0.062	0.471	0.012	0.010	0.310	0.035	0.015	0.377
Spann3R [ 36 ]	✓	0.329	0.110	4.471	0.056	0.021	0.591	0.096	0.023	0.661

Table 2: Camera Pose Estimation. RayMap3R achieves leading trajectory accuracy among streaming methods across all datasets, with particularly strong performance on dynamic sequences, while remaining competitive on static scenes.

		Acc $\downarrow$			Comp $\downarrow$			NC $\uparrow$			Chamfer $\downarrow$
Method	Onl.	Mean	Med.	Min	Mean	Med.	Min	Mean	Med.	Min	Mean	Med.	Min
DUSt3R-GA [ 39 ]	✗	0.146	0.077	0.052	0.181	0.067	0.042	0.736	0.839	0.865	0.327	0.144	0.094
MASt3R-GA [ 11 ]	✗	0.185	0.081	0.056	0.180	0.069	0.047	0.701	0.792	0.821	0.365	0.150	0.103
MonST3R-GA [ 46 ]	✗	0.248	0.185	0.142	0.266	0.167	0.121	0.672	0.759	0.783	0.514	0.352	0.263
Spann3R [ 36 ]	✓	0.298	0.226	0.170	0.205	0.112	0.078	0.650	0.730	0.754	0.503	0.338	0.248
CUT3R [ 38 ]	✓	0.043	0.026	0.015	0.031	0.018	0.009	0.621	0.618	0.523	0.027	0.025	0.013

Table 3: 3D Reconstruction on 7-Scenes [ 29 ] . Each metric reports Mean, Median, and Minimum across scenes. RayMap3R achieves leading reconstruction quality among streaming methods across the majority of metrics.

Figure 5: Qualitative Results on DAVIS [ 24 ] Videos. We compare our method with CUT3R [ 38 ] and TTT3R [ 6 ] . Our method achieves more stable camera pose estimation and produces clearer reconstructions.

Table 9

Dataset	Seq	Frames	Mean disc $\uparrow$	Mean AUC $\uparrow$	Mean IoU $\uparrow$	$\rho$ $\uparrow$
MPI Sintel [ 2 ]	19	746	1.61	0.464	0.298	0.900
DAVIS 2017 [ 24 ]	81	5205	1.68	0.538	0.189	0.713
TUM RGB-D [ 30 ]	8	680	1.88	0.560	0.206	0.643
All	108	6631	1.69	0.532	0.203	0.771

Table 6: Dynamic Map Evaluation. Evaluation across 108 sequences from three datasets. The depth discrepancy signal shows consistent discriminability and positive correlation with ground-truth dynamics across diverse scenes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Dynamic Map Visualization. We visualize the dynamic map inferred by RayMap3R across DAVIS (rows 1–2), MPI Sintel (rows 3–4), and TUM Dynamic (row 5). The RayMap-only branch suppresses dynamic objects relative to the main branch, and the resulting dynamic map aligns with ground-truth masks.

Figure 7: Temporal Consistency of Dynamic Maps on DAVIS longboard . Top: input frames. Bottom: dynamic maps. The dynamic map consistently produces elevated responses at moving subjects, including partially visible persons at the frame boundary.

Dynamic Ratio	Seq	Frames	Mean $\delta$	Mean disc $\uparrow$	Mean AUC $\uparrow$	Mean IoU $\uparrow$
Low ( $\leq$ 10%)	66	4168	0.065	2.24	0.591	0.145
Med (10–30%)	45	2862	0.077	1.93	0.551	0.240
High ( $>$ 30%)	20	985	0.153	1.36	0.532	0.409

Appendix

Table 7: Dynamic Map Evaluation by Scene Dynamic Ratio. We extend the evaluation in Tab. 6 by stratifying sequences into three groups based on their mean ground-truth dynamic ratio: Low (

\leq

10%), Medium (10–30%), and High (

>

30%). The disc metric exceeds 1 across all groups, indicating that the depth discrepancy signal remains discriminative even when dynamic content is sparse.

Figure 8: Effect of Reset Metric Alignment. Baseline ( left ) produces fragmented point clouds across reset boundaries. Our reset metric alignment restores segment consistency ( middle ), yielding coherent full-sequence reconstruction ( right ).

	Baseline	Fix $\beta{=}0.3$	Fix $\beta{=}0.5$	Fix $\beta{=}0.8$	Only $a_{t}$	Only $\mathrm{sc}_{t}$	Full $a_{t}\!\times\!\mathrm{sc}_{t}$
Mean ATE $\downarrow$	0.112	0.091	0.092	0.095	0.091	0.120	0.080
Mean $RPE_{t}$ $\downarrow$	0.036	0.022	0.026	0.033	0.028	0.021	0.021

Appendix

Table 8: Smoothing Signal Ablation. We compare the full adaptive smoothing (

a_{t}\times\mathrm{sc}_{t}

) against fixed-coefficient baselines and individual signal components. The full scheme achieves the lowest ATE, as jointly conditioning on both signals avoids under-suppression from fixed filtering and drift from single-signal over-smoothing.

	Camera Pose (TUM-dyn)		Video Depth (KITTI)		3D Recon (7-Scenes)
Setting	ATE $\downarrow$	RPE t $\downarrow$	AbsRel $\downarrow$	$\delta{<}1.25$ $\uparrow$	Acc $\downarrow$	Chamfer $\downarrow$
Base	0.033	0.011	0.120	88.3	0.041	0.029
+R	0.032	0.010	0.102	92.2	0.032	0.026
+R+M	0.031	0.008	0.100	92.5	0.028	0.025
+R+S	0.020	0.006	0.101	92.3	0.026	0.025
Full	0.019	0.005	0.099	92.7	0.023	0.024

Appendix

Table 9: Cross-Dataset Component Ablation. Each task is evaluated on a representative dataset. R: dual-branch RayMap identification; M: reset metric alignment; S: state-aware smoothing. Each component contributes gains across tasks, and the full model achieves the best results across metrics.

Explore similar work

SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes
Mar 24, 2026Zhicheng Qiu, Jiarui Meng, Tong-an Luo +4Dynamic Scene Reconstruction Streaming 3D Reconstruction
Huawei Technologies Ltd.
PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences
Mar 22, 2026Lanbo Xu, Liang Guo, Caigui Jiang +1Streaming 3D Reconstruction Monocular 3D Reconstruction
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · University of East Anglia, Norwich, NR47TJ, United Kingdom
Ray-Aware Pointer Memory with Adaptive Updates for Streaming 3D Reconstruction
May 7, 2026Feifei Li, Qi Song, Chi Zhang +1Streaming 3D Reconstruction 3D Reconstruction
The Chinese University of HongKong, Shenzhen · Tsinghua University

Mar 24, 2026J/K move · Enter open · S save

SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes

Zhicheng Qiu, Jiarui Meng, Tong-an Luo, Yican Huang+3

Huawei Technologies Ltd.

Dynamic Scene Reconstruction Streaming 3D Reconstruction 3D Scene Representation Feed-Forward 3D Reconstruction 3D Semantic Segmentation

Open paper AskPDF arXiv

We propose SLARM, a feed-forward model that unifies dynamic scene reconstruction, semantic understanding, and real-time streaming inference. SLARM captures complex, non-uniform motion through higher-order motion modeling, trained solely on differentiable renderings without any flow supervision. Besides, SLARM distills semantic features from LSeg to obtain language-aligned representations. This design enables semantic querying via natural language, and the tight coupling between semantics and geometry further enhances the accuracy and robustness of dynamic reconstruction. Moreover, SLARM processes image sequences using window-based causal attention, achieving stable, low-latency streaming inference without accumulating memory cost. Within this unified framework, SLARM achieves state-of-the-art results in dynamic estimation, rendering quality, and scene parsing, improving motion accuracy by 21%, reconstruction PSNR by 1.6 dB, and segmentation mIoU by 20% over existing methods.

Abstract

Explore similar work

SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes

PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences

Ray-Aware Pointer Memory with Adaptive Updates for Streaming 3D Reconstruction