cs.CVOct 31, 2026

SceneBound: How Much Should a Streaming 3D Reconstruction Model Remember?

Authors: Ziming Wang, Chuyue Gong, Chutian Wang, Jingmin Zhu, Hongliang Cao, Yuheng He, Xiuyu Li

Organizations: The Hong Kong University of Science and Technology · Sun Yat-sen University · Monash University · Beihang University · 

Abstract

Streaming 3D reconstruction models keep past video frames in a recurrent state, fast weights from test-time training, or a key-value (KV) cache. Each written frame shifts the coordinate frame of later cameras or occupies GPU memory. Training-free rules decide what to write from a signal of the current frame. How much they write accounts for their mean accuracy: spending the update budget of a gate at uniformly spaced frames, with no signal, matches nine published gates, and the budget bounds the drift of the state wherever the writes fall. We propose SceneBound, a training-free rule that writes a frame only when its predicted camera pose lies at least a fixed radius from every stored keyframe. Its keyframe count stays within a constant factor of the covering number of the visited poses, the fewest views that keep a keyframe near every camera, so the count grows with the places visited and stays flat when the camera returns. In large scenes, SceneBound closes a submap once decoding its first frame again reveals drift. SceneBound uses one radius across four released models and three memory types. In the recurrent state of CUT3R and the KV cache of StreamVGGT, it achieves the lowest mean trajectory error among the published training-free rules we evaluate, in one pass and over nine passes of a scene, where its keyframes grow by 3% to 6%.

Explore similar work

May 16, 2026

Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction

Streaming 3D reconstruction under a strict constant-memory budget hinges on how the recurrent state is updated as the stream evolves. We profile TTT3R-style per-token gates across five benchmarks and discover a structural bottleneck: the gate is intrinsically bounded in magnitude (median 0.310.31; never exceeding 0.60.6) and nearly frame-invariant, yielding an effective memory horizon of only ∼\sim3 frames per state token, which serves as the structural origin of long-sequence drift. We trace this to a missing axis: existing inference-time methods modulate updates only at the per-token, intra-frame level, while the orthogonal frame-level question of how strongly each frame should contribute to the state has been treated as content-independent. We close this gap with a scalar frame-level gate αt∈(0,1]α_t \in (0, 1] derived in closed form from frame-to-frame changes of internal features---a graded write weight, inspired by classical Simultaneous Localization and Mapping (SLAM) keyframe selection, that never discards a frame and requires no parameters, no training, and no extra forward pass. Across six benchmarks spanning camera pose, video depth, and 3D reconstruction at sequence lengths up to 4,6614,661 frames, our gate cuts ATE by 51%51\% on long TUM-RGBD pose sequences, reduces AbsRel by 13.0%13.0\% on Bonn video depth, and on KITTI long-sequence pose estimation surpasses both LongStream and Keyframe-VO in average ATE, while retaining strictly constant memory at zero training cost. Code is available at https://github.com/rkj12345/AFG.
Mar 16, 2026

SSR: A Training-Free Approach for Streaming 3D Reconstruction

Streaming 3D reconstruction demands long-horizon state updates under strict latency constraints, yet stateful recurrent models often suffer from geometric drift as errors accumulate over time. We revisit this problem from a Grassmannian manifold perspective: the latent persistent state can be viewed as a subspace representation, i.e., a point evolving on a Grassmannian manifold, where temporal coherence implies the state trajectory should remain on (or near) this manifold.Based on this view, we propose Self-expressive Sequence Regularization (SSR), a plug-and-play, training-free operator that enforces Grassmannian sequence regularity during inference.Given a window of historical states, SSR computes an analytical affinity matrix via the self-expressive property and uses it to regularize the current update, effectively pulling noisy predictions back toward the manifold-consistent trajectory with minimal overhead. Experiments on long-sequence benchmarks demonstrate that SSR consistently reduces drift and improves reconstruction quality across multiple streaming 3D reconstruction tasks.
Jul 6, 2026

ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7×\times reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.