FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction

Seonghyun Jin; Jong Chul Ye

cs.CVMar 19, 2026

FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction

Authors: Seonghyun Jin, Jong Chul Ye

Organizations: Kim Jaechul Graduate School of AI, KAIST, Seoul, South Korea

Abstract

Streaming 3D reconstruction maintains a persistent latent state that is updated online from incoming frames, enabling constant-memory inference. A key failure mode is the state update rule: aggressive overwrites forget useful history, while conservative updates fail to track new evidence, and both behaviors become unstable beyond the training horizon. To address this challenge, we propose FILT3R, a training-free latent filtering layer that casts recurrent state updates as stochastic state estimation in token space. FILT3R maintains a per-token variance and computes a Kalman-style gain that adaptively balances memory retention against new observations. Process noise -- governing how much the latent state is expected to change between frames -- is estimated online from EMA-normalized temporal drift of candidate tokens. Using extensive experiments, we demonstrate that FILT3R yields an interpretable, plug-in update rule that generalizes common overwrite and gating policies as special cases. Specifically, we show that gains shrink in stable regimes as uncertainty contracts with accumulated evidence, and rise when genuine scene change increases process uncertainty, improving long-horizon stability for depth, pose, and 3D reconstruction, compared to the existing methods.

Figures & tables

Figure 1: FILT3R enables stable long-horizon streaming 3D reconstruction without resets. Top: dense input frames from a long video stream. Middle: final reconstructions from the same frozen streaming backbone under different latent-state update rules. Uniform overwrite (CUT3R) and heuristic gating (TTT3R) accumulate drift and/or forget previously integrated evidence, leading to long-horizon instability and degraded geometry. Bottom: a zoom-in around frames beyond 600 highlights that FILT3R preserves stable geometry over long rollouts. Trajectory colors indicate time.

Figure 2: FILT3R overview. The persistent memory $\mathbf{s}_{t-1}$ is fused with the decoder’s candidate $\tilde{\mathbf{s}}_{t}$ via a token-wise Kalman-style gain $\mathbf{k}_{t}$ . Process noise $\mathbf{q}_{t}$ is estimated adaptively from EMA-normalized temporal drift, while measurement noise $r$ is a scalar hyperparameter shared across tokens. Variance $\mathbf{p}_{t}$ is propagated across time steps, enabling gains that naturally shrink in stable regimes and increase during scene change.

Figure 3: Qualitative long-horizon streaming 3D reconstruction. CUT3R often suffers from catastrophic forgetting and drift, while TTT3R reduces but does not eliminate long-horizon instability, leading to fragmented surfaces and inconsistent revisited regions (red boxes). FILT3R produces more coherent geometry over long rollouts. Ground-truth trajectory is shown in orange; the predicted trajectory is color-coded by time.

Method	Length 300						Length 500						Length 1000
	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
CUT3R	0.134	0.090	0.076	0.041	0.544	0.565	0.183	0.133	0.093	0.036	0.530	0.544	0.233	0.170	0.101	0.017	0.511	0.515
Point3R	0.045	0.025	0.032	0.013	0.564	0.598	0.057	0.026	0.025	0.008	0.556	0.584	OOM
TTT3R	0.040	0.025	0.024	0.005	0.566	0.602	0.066	0.038	0.032	0.006	0.551	0.576	0.145	0.092	0.051	0.010	0.525	0.536
FILT3R (ours)	0.020	0.009	0.022	0.004	0.568	0.603	0.024	0.011	0.023	0.004	0.559	0.589	0.054	0.028	0.028	0.005	0.535	0.550

Table 1: 3D reconstruction on 7-Scenes and NRGBD. Accuracy (Acc), Completeness (Comp), and Normal Consistency (NC); Mean and Median (Med.) are reported. Point3R runs out of memory (OOM) at length 1000.

Method	TUM-400				TUM-600				TUM-800
Method	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\!\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\!\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\!\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$
CUT3R	0.109	0.302	0.010	0.381	0.145	0.425	0.008	0.430	0.173	0.493	0.008	0.486
TTT3R	0.055	0.128	0.010	0.328	0.084	0.176	0.009	0.365	0.097	0.214	0.009	0.387
FILT3R (ours)	0.033	0.074	0.009	0.305	0.042	0.082	0.009	0.332	0.057	0.107	0.009	0.362

Table 2: Long-horizon camera pose on TUM-RGBD. We report standard ATE (with global Procrustes alignment) and origin-aligned ATE (

\text{ATE}_{\text{orig}}

), which aligns only the first frame and therefore exposes progressive drift. FILT3R provides clear long-horizon stability gains.

Method	Bonn-300			Bonn-400			Bonn-500
Method	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$
CUT3R	0.107	88.7	0.162	0.107	89.6	0.161	0.101	90.6	0.156
TTT3R	0.108	90.1	0.163	0.104	91.2	0.158	0.100	92.1	0.154
FILT3R (ours)	0.093	93.8	0.152	0.090	94.1	0.149	0.089	94.4	0.148

Table 3: Long-horizon video depth on Bonn (metric scale). Sequences are truncated to the indicated length. FILT3R maintains stable metric depth predictions as sequence length grows, whereas competing methods suffer from accumulating drift.

Panel of Figure 4 — Figure 4: Long-horizon metrics vs. sequence length. Top (a, b): Camera pose on TUM-RGBD (50–1000 frames). FILT3R’s drift grows more slowly. Bottom (c, d): Metric scale video depth on Bonn (50–500 frames). FILT3R consistently mitigates drift and improves over the baselines as the sequence progresses.

Figure 5: FILT3R on tttLRM fast-weight memory (DL3DV-140). Novel-view renders from GT, tttLRM, and tttLRM+FILT3R (top to bottom). Inserting FILT3R into the fast-weight update sharpens fine structure and suppresses the blur and ghosting that the unfiltered model accumulates over long view sequences; highlighted regions mark representative cases.

Variant	TUM-800 (pose)				Bonn-500 (metric depth)
Variant	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$	logRMSE $\downarrow$
FILT3R (full)	0.057	0.107	0.009	0.362	0.089	94.4	0.148
Fixed- $\mathbf{Q}$ (fixed $r$ )	0.100	0.229	0.009	0.393	0.099	91.9	0.155
No variance propagation (reset- $\mathbf{P}$ )	0.149	0.489	0.009	0.446	0.099	91.1	0.154
Fixed- $\beta$ EMA ( $\beta{=}0.05$ , tuned)	0.049	0.078	0.010	0.658	0.093	93.5	0.152
No EMA normalization for drift	0.058	0.109	0.010	0.367	0.091	94.1	0.150

Table 5: Ablation study of FILT3R components. All variants use identical frozen backbone weights and identical inference code; only the online update policy is changed. We report long-horizon pose on TUM-800 and long-horizon metric depth on Bonn-500. Lower is better (

\downarrow

) except

\delta{<}1.25

(

\uparrow

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: FILT3R slows long-horizon error growth. Across both camera pose and metric-scale depth, FILT3R’s errors grow more slowly than those of overwrite (CUT3R) and heuristic gating (TTT3R) as the evaluated prefix length increases. This is the central empirical pattern that the remainder of the appendix explains.

Figure 8: How different update rules relate to the gain curve. With fixed

r

, the steady-state Kalman-style gain is determined by the noise ratio (

q/r

). Overwrite (CUT3R) corresponds to the limit (

k\to 1

). A fixed gate chooses one constant operating point. FILT3R adapts

q_{t}

and therefore moves along the curve, becoming conservative in stable regimes and reopening at transitions. TTT3R is also adaptive, but its gate is heuristic and based on current cues rather than propagated covariance, so it is shown schematically off the steady-state curve.

Figure 9: The deployed filter settles near a finite gain floor rather than freezing completely. The dashed line marks the implementation-faithful prediction

k_{\mathrm{floor}}^{\star}\approx 0.132

from Eq. ( 22 ). Here, the early and late windows denote the first and last 20% of frames of each analyzed sequence. Across both TUM-800 and Bonn-500, FILT3R’s late-window mean gain remains only

0.013

–

0.015

above this floor, while TTT3R stays substantially more reactive. This is consistent with FILT3R entering a conservative stable regime and reopening only when transition frames push

q_{t}

upward.

Variant	TUM-800 (pose)				Bonn-500 (metric depth)
Variant	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$	logRMSE $\downarrow$
FILT3R (full)	0.057	0.107	0.009	0.362	0.089	94.4	0.148
Fixed- $\mathbf{Q}$ (fixed $r$ )	0.100	0.229	0.009	0.393	0.099	91.9	0.155
No variance propagation (reset- $\mathbf{P}$ )	0.149	0.489	0.009	0.446	0.099	91.1	0.154
Fixed- $\beta$ EMA ( $\beta{=}0.05$ , tuned)	0.049	0.078	0.010	0.658	0.093	93.5	0.152
No EMA normalization for drift	0.058	0.109	0.010	0.367	0.091	94.1	0.150

Appendix

Table 6: FILT3R versus fixed smoothing: ablations on TUM-800 and Bonn-500. Removing adaptive process noise or variance propagation weakens long-horizon stability. A tuned fixed-

\beta

EMA can suppress certain global drift metrics, but does so by uniformly damping all updates, which substantially degrades local rotation accuracy. No single constant coefficient matches FILT3R’s joint performance across long-horizon drift, local motion accuracy, and depth quality.

Panel of Figure 10 — Figure 10: FILT3R contracts updates in stable regimes and reopens at transitions. Panel A plots the per-frame mean temporal score $\bar{g}_{t}$ together with its MA-11 smoothing, the 90th-percentile transition threshold, and the detected above-threshold windows. Panels B–D report the corresponding mean effective gain, mean posterior covariance, and update ratio $\rho_{t}$ . Detected transition windows raise process noise, while propagated covariance keeps FILT3R in a substantially lower-update regime than both TTT3R and the no-propagation ablation. This behavior provides the mechanistic explanation for the slower error growth observed in Figure 4 .

Figure 11: Extended long-horizon fixed-

\beta

sweep. Very small fixed EMA coefficients can outperform FILT3R on individual long-horizon metrics, but only by globally over-smoothing the stream: the same settings that reduce

\text{ATE}_{\text{orig}}

or improve depth also produce substantially elevated TUM-800 RPE-r. FILT3R occupies a stronger operating region because it does not commit to a single forgetting rate for the entire rollout.

Figure 12: Development-set fixed-

\beta

sweep. This sweep provides context for representative fixed-EMA operating points and shows the development-set optimum at

\beta=0.05

under the explicit objective above. The more informative comparison is the long-horizon sweep above: a fixed gate can be tuned toward one regime, yet it remains globally conservative rather than transition-aware.

Method	TUM-400				TUM-600				TUM-800
Method	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\!\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\!\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\!\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$
CUT3R	0.109	0.302	0.010	0.381	0.145	0.425	0.008	0.430	0.173	0.493	0.008	0.486
Point3R	0.119	0.179	0.028	0.640	0.133	0.205	0.026	0.912	0.157	0.241	0.024	1.127
TTT3R	0.055	0.128	0.010	0.328	0.084	0.176	0.009	0.365	0.097	0.214	0.009	0.387
TTT3R+Reset	0.047	0.140	0.008	0.308	0.059	0.181	0.007	0.331	0.077	0.221	0.007	0.338
FILT3R (ours)	0.033	0.074	0.009	0.305	0.042	0.082	0.009	0.332	0.057	0.107	0.009	0.362

Appendix

Table 7: Resets improve within-segment metrics but fail to reduce cumulative drift. On TUM-RGBD, TTT3R+Reset improves certain locally aligned metrics, yet FILT3R remains substantially better on origin-aligned trajectory error, which directly exposes accumulated drift.

Method	Bonn-300			Bonn-400			Bonn-500
Method	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$
CUT3R	0.090	93.8	0.150	0.091	93.5	0.150	0.086	94.0	0.145
Point3R	0.083	94.7	0.149	0.083	94.6	0.147	0.081	94.8	0.145
TTT3R	0.079	95.0	0.142	0.078	95.1	0.139	0.075	95.3	0.136
TTT3R+Reset	0.085	94.3	0.147	0.082	94.5	0.142	0.081	94.8	0.141
FILT3R (ours)	0.080	95.1	0.143	0.078	95.3	0.140	0.076	95.4	0.138

Appendix

Table 8: Reset depth comparison on Bonn. (a) Per-sequence scale alignment at 300, 400, and 500 frames shows that resets can help transiently. (b) Under metric-scale evaluation, however, FILT3R remains consistently stronger as the horizon grows, because it avoids the scale inconsistencies introduced at reset boundaries.

Panel of Figure 13 — Figure 13: Resets improve short segments but not the persistent state. The reset baseline reduces locally aligned drift at moderate lengths, but its benefit diminishes as scale discontinuities accumulate across reset boundaries. FILT3R instead maintains a single continuously filtered state throughout the rollout.

Figure 14: Representative reset failure cases. Resetting discards previously accumulated scene context, causing scale jumps near reset boundaries and inconsistent behavior on revisited regions. FILT3R avoids these artifacts by maintaining a continuously filtered latent state.

Benchmark	Training ref.	Prefix lengths	Prefix / 64	Evaluator details
TUM-RGBD pose	CUT3R 4–64 views	50–1000	$0.78\times$ – $15.63\times$	ATE uses align=True , correct_scale=True ; $\text{ATE}_{\text{orig}}$ uses align_origin=True ; RPE is evaluated at $\Delta{=}1$ frame with similarity alignment.
Bonn depth	CUT3R 4–64 views	50–500	$0.78\times$ – $7.81\times$	Main long-horizon table uses metric scale; the reset comparison additionally reports per-sequence scale alignment. Valid pixels satisfy GT depth $>0$ and $<70$ m; predictions are bicubically resized and averaged with valid-pixel weighting.
KITTI depth	CUT3R 4–64 views	300 / 400 / 500	$4.69\times$ – $7.81\times$	Both metric-scale and per-sequence-scale metrics are reported. The evaluator keeps the dataset’s native depth range ( max_depth=None ) and uses the same valid-pixel-weighted aggregation.
7-Scenes recon	CUT3R 4–64 views	100 / 200 / 300 / 500 / 1000	$1.56\times$ – $15.63\times$	Input size 512 with a 224-pixel center crop for point cloud optimization. With ICP applied to report Accuracy, Completeness, and Normal Consistency.
NRGBD recon	CUT3R 4–64 views	300 / 400 / 500	$4.69\times$ – $7.81\times$	The same reconstruction evaluator is used on the nine prepared NRGBD scenes.

Appendix

Table 9: Exact evaluator settings and relation to the released checkpoint’s training bound. The checkpoint tag and README specify a 4–64-view model; we therefore state how far each benchmark extends beyond the 64-view upper bound.

Variant	ATE $\downarrow$	$\text{ATE}_{\text{orig}}\downarrow$	RPE-r $\downarrow$
Ours	0.057	0.107	0.362
Gate variant ( $q_{\max}{=}0.8,\tau_{q}{=}4.0$ )	0.057	0.107	0.364

Appendix

Table 10: Targeted perturbation check around the main configuration. These runs come from retained evaluation logs and are intended as a local sensitivity check rather than a dense robustness study. The nearby setting stays in the same qualitative regime across both pose and reconstruction.

Method	$\text{ATE}_{\text{orig}}$ mean $\pm$ std $\downarrow$	worst $\downarrow$	wins $\uparrow$	RPE-r mean $\pm$ std $\downarrow$	worst $\downarrow$	wins $\uparrow$
CUT3R	0.493 $\pm$ 0.318	1.137	0	0.486 $\pm$ 0.240	0.862	0
TTT3R	0.214 $\pm$ 0.097	0.427	0	0.387 $\pm$ 0.158	0.588	0
TTT3R+Reset	0.221 $\pm$ 0.060	0.290	0	0.338 $\pm$ 0.113	0.483	5
FILT3R (ours)	0.107 $\pm$ 0.029	0.136	8	0.362 $\pm$ 0.148	0.581	3

Appendix

Table 11: Per-sequence robustness at long horizons. TUM-800 pose uses all eight prepared TUM-dynamics sequences. Bonn-400 metric depth uses the five long Bonn sequences, which is the longest horizon for which per-sequence outputs are available for all compared methods.

Method	KITTI-300			KITTI-400			KITTI-500
Method	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	log RMSE $\downarrow$
CUT3R	0.122	87.5	0.165	0.127	86.9	0.170	0.130	86.5	0.175
TTT3R	0.111	90.3	0.152	0.115	89.7	0.159	0.118	89.2	0.165
FILT3R (ours)	0.108	91.1	0.149	0.112	90.2	0.156	0.116	89.6	0.162

Appendix

Table 12: Long-horizon video depth on KITTI. (a) Per-sequence scale alignment and (b) metric-scale evaluation are both reported on prepared KITTI validation clips truncated to 300, 400, and 500 frames.

Alignment	Method	Type	Sintel		Bonn		KITTI
			Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$
Per-seq. scale	DUSt3R-GA $\dagger$ [ 34 ]	Optim	0.656	45.2	0.155	83.3	0.144	81.3
	MASt3R-GA $\dagger$ [ 11 ]	Optim	0.641	43.9	0.252	70.1	0.183	74.5
	MonST3R-GA $\dagger$ [ 39 ]	Optim	0.378	55.8	0.067	96.3	0.168	74.4
	Easi3R $\dagger$ [ 3 ]	FA	0.377	55.9	0.059	97.0	0.102	91.2
	VGGT $\dagger$	FA	0.287	66.1	0.055	97.1	0.070	96.5

Appendix

Table 13: Video depth estimation on short sequences (full comparison). Both per-sequence scale and metric-scale results are shown. Rows marked with

\dagger

are reproduced from prior work.

Method	Length 100						Length 200
	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
CUT3R	0.033	0.018	0.022	0.005	0.592	0.644	0.089	0.053	0.048	0.017	0.566	0.600
TTT3R	0.020	0.008	0.016	0.004	0.596	0.649	0.028	0.015	0.023	0.005	0.582	0.627
FILT3R (ours)	0.017	0.006	0.017	0.004	0.595	0.648	0.019	0.008	0.022	0.004	0.584	0.629

Appendix

Table 14: 3D reconstruction on 7-Scenes at short sequences (100 and 200 frames). Even near the training horizon, FILT3R is already competitive with or slightly better than prior streaming updates, though the gap is considerably smaller than in the long-horizon settings reported above.

Symbol	Description	Value
$p_{0}$	Initial per-token variance	1.5
$k_{\min},k_{\max}$	Kalman-style gain clamp bounds	0.01, 0.99
$q_{\min}$ , $q_{\max}$	Process noise bounds	0.02, 0.5
$\alpha_{q}$ , $\tau_{q}$	Process-noise sigmoid parameters	20, 3.0
$r$	Fixed measurement-noise scalar	1.0
$\lambda_{\Delta}$	Drift EMA rate	0.05

Appendix

Table 15: FILT3R hyperparameters.

Figure 15: TUM-RGBD trajectory visualizations. FILT3R stays closer to the ground-truth trajectory over long horizons than both the overwrite and heuristic-gating baselines, particularly after the rollout extends well beyond the training horizon.

Figure 16: Bonn depth prediction visualizations. FILT3R stays closer to the ground-truth depth over long horizons than both the overwrite and heuristic-gating baselines, producing more metric-aligned depth predictions with sharper detail.

Len.	Method	PSNR $\uparrow$	$\Delta$	SSIM $\uparrow$	LPIPS $\downarrow$	wins
16	Original-AR	23.0966	–	0.7638	0.2681	–
	Fixed-gain EMA	23.1423	+0.0457	0.7648	0.2677	127/140
	+ FILT3R-FW	23.1457	+0.0491	0.7648	0.2677	127/140
64	Original-AR	24.8116	–	0.8140	0.2253	–
	Fixed-gain EMA	24.9444	+0.1328	0.8173	0.2230	–
	+ FILT3R-FW	24.9647	+0.1531	0.8192	0.2220	–

Appendix

Table 16: tttLRM length scaling at the standard LaCT step (DL3DV-140). Original-AR is the unmodified autoregressive update; Fixed-gain EMA is the scalar floor-binding control; + FILT3R-FW is the row-wise adaptive variant. The wins column counts scenes (out of 140) beating Original-AR. The gain

\Delta

grows with sequence length.

Len.	chunks (filtered)	Original-AR	Fixed-gain EMA $\Delta$	+ FILT3R-FW $\Delta$	wins
16	4 (3)	23.097	+0.046	+0.049	127/140
	8 (7)	21.290	+0.480	+0.764	140/140
	16 (15)	17.255	+0.702	+1.618	140/140
20	5 (4)	23.597	+0.050	+0.052	130/140
	10 (9)	21.588	+0.530	+0.923	140/140
	20 (19)	17.724	+0.605	+1.542	140/140

Appendix

Table 17: Chunk-axis stress test on DL3DV-140. Shrinking the LaCT update step (more chunks) collapses the unfiltered baseline by

{\approx}\,5.8

dB; FILT3R-FW (row-wise) recovers up to

+1.6

dB on all 140 scenes. Baseline is absolute PSNR; the two FILT3R columns are

\Delta

PSNR over the baseline.

chunk	$k_{\mathrm{mean}}$	$k_{\min}$	$k_{\max}$	per-row $k_{\mathrm{std}}$
1	0.900	0.900	0.900	0.000
2	0.827	0.731	0.830	0.010
3	0.798	0.674	0.829	0.025
4	0.769	0.652	0.826	0.027
5	0.738	0.590	0.826	0.040
6	0.712	0.551	0.826	0.048

Appendix

Table 18: Row-wise gain trajectory at the finest update step (15 filtered chunks, logged on a DL3DV sample). The mean gain anneals downward and the per-row spread grows as consistent evidence accumulates.

Setting	FILT3R-FW (row-wise)	Fixed-gain EMA (scalar)
granularity	row-wise	scalar
$q_{\mathrm{base}}$	0.01	— (gain floor binds)
drift clip $(q_{\min},q_{\max})$	$(0.25,\ 4.0)$	—
measurement noise $r$	0.01	1.0
initial variance $p_{0}$	0.05	(floor-binding)
gain clamp $(k_{\min},k_{\max})$	$(0.5,\ 0.995)$	$(0.88,\ 0.995)$

Appendix

Table 19: tttLRM FILT3R hyperparameters. The drift clip

(q_{\min},q_{\max})

bounds the normalized ratio

d_{t}/\hat{d}_{t}

before scaling by

q_{\mathrm{base}}

Method	Memory form	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
Point3R	explicit pointer memory	0.049	0.029	0.562
Point3R + FILT3R	filtered pointer fusion	0.042	0.024	0.563
InfiniteVGGT	attention/KV-cache memory	0.040	0.025	0.570
FILT3R (ours) (main)	compact latent memory	0.020	0.022	0.568

Appendix

Table 20: 7-Scenes-300 comparison across memory designs. Point3R validates the pointer-memory fusion variant of FILT3R, while InfiniteVGGT provides an attention/KV-cache memory comparison. The main CUT3R+FILT3R model retains a compact latent memory and achieves the best Acc/Comp, with NC within

0.002

of the heavier attention-memory baseline.

Streaming 3D Reconstruction

Streaming 3D reconstruction incrementally builds geometrically consistent scene maps from incoming images or video under tight memory and latency limits.

See all 99 papers

Jul 6, 2026

ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7

\times

reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.

Xinze Li, Yiyuan Wang, Pengxu Chen +4

Beijing Normal-Hong Kong Baptist University · Hong Kong Baptist University · Jilin University +2

Streaming 3D Reconstruction 3D Scene Representation Adaptive Learning Rate

Ask PDF

Jan 30, 2026

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

Streaming recurrent models enable efficient 3D reconstruction by maintaining persistent state representations. However, they suffer from catastrophic forgetting over long sequences due to balancing historical information with new observations. Recent methods alleviate this by deriving adaptive signals from the attention perspective, but they operate on single dimensions without considering temporal and spatial consistency. To this end, we propose a training-free framework termed TTSA3R that leverages both temporal state evolution and spatial observation quality for adaptive state updates in 3D reconstruction. In particular, we devise a Temporal Adaptive Update Module that regulates update magnitude by analyzing temporal state evolution patterns. Then, a Spatial Context Update Module is introduced to localize spatial regions that require updates through observation-state alignment and observation changes. These complementary signals are finally fused to determine the state updating strategies. Extensive experiments show that TTSA3R achieves competitive performance on standard short-sequence benchmarks and provides substantially stronger robustness on extended sequences. On NRGBD, as sequences extend from 50 to 250 frames, TTSA3R exhibits only a 1.33

\times

error increase, compared with 3.8

\times

degradation for CUT3R. This highlights the practical value of temporal-spatial adaptive updates for long-term reconstruction stability. Our code is available at https://github.com/anonus2357/ttsa3r.

Zhijie Zheng, Xinhao Xiang, Jiawei Zhang

University of California, Davis CA 95616, USA

Streaming 3D Reconstruction 3D Reconstruction Catastrophic Forgetting

Ask PDF

Oct 12, 2026

BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates

Feed-forward transformer models have driven remarkable progress in 3D vision,yet their quadratic complexity renders them impractical for long video sequences. Streaming 3D reconstruction addresses this bottleneck by processing frames sequen?tially with constant memory. However, existing recurrent architectures inevitably suffer from progressive degradation over long sequences, as they fail to account for the intrinsic reliability of observations and the accumulated confidence of the historical state, leading to severe error accumulation. To address these issues, we introduce BAT3R, a training-free Bayesian adaptation framework comprising two synergistic modules: State Innovation Gating (SIG) and Bayesian-guided Adaptive State Updating (BASU). SIG acts as a distributional regularizer that anchors state evolution to its geometric initialization, effectively arresting representational drift during long-term inference. Building on this, BASU reformulates state update as a recursive Bayesian inference process within a Kalman filtering framework. By jointly modeling the temporal reliability of observations via information-theoretic gain and spatial relevance via cross-attention maps, BASU achieves an asymmetric spatio-temporal state fusion: well-consolidated historical regions are protected from redundant updates, while genuinely novel scene content is aggressively incor?porated. Extensive experiments on standard benchmarks (7-Scenes, TUM, ScanNet, KITTI, etc.) demonstrate that BAT3R consistently outperforms state-of-the-art online 3D reconstruction methods, with the most significant gains observed on long sequences and complex scenes.

Lijian Li, Yuanpeng He, Yudian Zheng +6

University of Macau, Macau SAR, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, Beijing, China

Streaming 3D Reconstruction Bayesian Filtering 3D ViTs

Ask PDF

Abstract

Streaming 3D Reconstruction

Explore similar work

ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates