OnlineX: Unified Online 3D Reconstruction and Understanding with Active-to-Stable State Evolution

Chong Xia; Fangfu Liu; Yule Wang; Yize Pang; Yueqi Duan

cs.CVMar 2, 2026

OnlineX: Unified Online 3D Reconstruction and Understanding with Active-to-Stable State Evolution

Authors: Chong Xia, Fangfu Liu, Yule Wang, Yize Pang, Yueqi Duan

Organizations: Tsinghua University

Abstract

Recent advances in generalizable 3D Gaussian Splatting (3DGS) have enabled rapid 3D scene reconstruction within seconds, eliminating the need for per-scene optimization. However, existing methods primarily follow an offline reconstruction paradigm, lacking the capacity for continuous reconstruction, which limits their applicability to online scenarios such as robotics and VR/AR. In this paper, we introduce OnlineX, a feed-forward framework that reconstructs both 3D visual appearance and language fields in an online manner using only streaming images. A key challenge in online formulation is the cumulative drift issue, which is rooted in the fundamental conflict between two opposing roles of the memory state: an active role that constantly refreshes to capture high-frequency local geometry, and a stable role that conservatively accumulates and preserves the long-term global structure. To address this, we introduce a decoupled active-to-stable state evolution paradigm. Our framework decouples the memory state into a dedicated active state and a persistent stable state, and then cohesively fuses the information from the former into the latter to achieve both fidelity and stability. Moreover, we jointly model visual appearance and language fields and incorporate an implicit Gaussian fusion module to enhance reconstruction quality. Experiments on mainstream datasets demonstrate that our method consistently outperforms prior work in novel view synthesis and semantic understanding, showcasing robust performance across input sequences of varying lengths with real-time inference speed.

Figures & tables

Figure 1: We introduce OnlineX , a framework for continuous and progressive 3D scene reconstruction from streaming images. Our core contribution is a active-to-stable state evolution paradigm, which effectively mitigates long-term drift by decoupling the processing of high-fidelity active local details from the maintenance of a stable global structure.

Figure 2: Overall architecture of OnlineX. Our framework features a two-stage, active-to-stable pipeline. First, the Relative Geometry Extractor processes consecutive frames to capture high-fidelity active relative information. The Anchor State Director then uses this local information to recurrently update its stable global state, yielding a globally consistent representation for the final output. The diagram illustrates this process for a single time step, which would be sequentially repeated for each frame in the input stream. Dashed lines represent information passed from the previous time step or carried over to the next.

Method	2 views			4 views			8 views
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
MVSplat [ 3 ]	24.73	0.829	0.171	21.91	0.753	0.358	20.41	0.696	0.434
NoPoSplat [ 42 ]	25.59	0.849	0.140	22.63	0.771	0.302	21.08	0.671	0.412
FLARE [ 48 ]	24.83	0.834	0.165	23.16	0.794	0.218	22.41	0.765	0.314
Spann3R [ 34 ] +GS	22.43	0.761	0.317	22.07	0.732	0.374	21.86	0.701	0.396
CUT3R [ 35 ] +GS	23.12	0.798	0.208	22.91	0.775	0.241	22.78	0.774	0.236

Table 1: Quantitative comparison of novel view synthesis on RE10K [ 52 ] . Our method achieves comparable performance with previous SOTA methods on few-view settings, and outperforms them with more views input.

Method	10 views			20 views			30 views
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
MVSplat [ 3 ]	20.03	0.686	0.414	18.72	0.561	0.511	16.02	0.505	0.591
NoPoSplat [ 42 ]	21.34	0.729	0.360	19.42	0.635	0.469	18.73	0.574	0.502
FLARE [ 48 ]	21.75	0.751	0.330	19.91	0.652	0.447	19.15	0.619	0.485
Spann3R [ 34 ] +GS	21.54	0.732	0.357	20.91	0.712	0.367	20.42	0.689	0.387
CUT3R [ 35 ] +GS	21.97	0.762	0.323	20.79	0.709	0.382	20.01	0.687	0.401

Table 2: Quantitative comparison of novel view synthesis on ScanNet [ 6 ] . Our method outperforms previous SOTA methods on all input view settings, even larger gains on 30-view setting.

Figure 3: Qualitative comparison for novel view synthesis on RE10K (top two rows) and ScanNet (bottom two rows). We adopt the 4-view setting for RE10K and 15-view setting for ScanNet.

Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
Spann3R [ 34 ]	0.096	0.023	0.661
CUT3R [ 35 ]	0.099	0.022	0.600
Ours	0.085	0.019	0.550

Table 3: Camera pose estimation on ScanNet. We report Absolute Translation Error (ATE), Relative Translation Error (RPE trans), and Relative Rotation Error (RPE rot).

Method	5 views		15 views
Method	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
LangSplat [ 24 ]	54.63	71.15	56.91	73.41
GS-Group [ 43 ]	51.61	68.32	53.26	67.63
Ours	58.83	77.12	57.79	76.36

Table 4: Quantitative comparison of semantic segmentation on ScanNet [ 6 ] . We report the average IoU scores (

\%

) and average accuracy (

\%

). We denote Gaussian-Grouping [ 43 ] as GS-Group.

Figure 4: Qualitative comparison for semantic segmentation on ScanNet. Here we showcase one scene with 15 input views. The masks predicted by ours contain more complete regions than other methods, such as the ”Wall” prompt, which also surpasses the GT masks.

Figure 5: Qualitative results for zero-shot generalization on DL3DV. Our model can easily transfer to out-of-distribution data.

Method	Spann3R [ 34 ]	CUT3R [ 35 ]	Ours
FPS	13.35	25.78	23.12
Memory(GB)	32.73	19.76	21.64

Table 5: Runtime and memory analysis. We report the FPS and memory usage metric compared with other online methods.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o Relative Extractor	20.85	0.712	0.368
w/o Anchor State	19.92	0.685	0.441
w/o Implicit Transform	16.52	0.533	0.552
w/o Implicit GS Fusion	22.61	0.778	0.284
Ours (Full Model)	24.13	0.811	0.174

Table 6: Quantitative results of the ablation study. We report novel view synthesis metrics on ScanNet with 10 views.

Figure 6: Qualitative results of the ablation study. From left to right, we visualize the results of our full model and four variants: without the Relative Geometry Extractor, without the Anchor State, without the Implicit Pose Transformation, and without the Implicit GS Fusion.

Explore similar work

S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction
Mar 15, 2026Renhe Zhang, Yuyang Tan, Jingyu Gong +4Streaming 3D Reconstruction 3D Scene Understanding
East China Normal University · Shanghai Artificial Intelligence Laboratory
MoonSplat: Monocular Online Gaussian Splatting with Sim(3) Global Optimization
Jun 16, 2026Guo Pu, Yixuan Han, Haofeng Li +33D Gaussian Splatting Language-Embedded 3DGS
Wangxuan Institute of Computer Technology, Peking University, China · Beijing Hydrogen Intelligent Tech. Co., Ltd., China
BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates
Oct 12, 2026Lijian Li, Yuanpeng He, Yudian Zheng +6Streaming 3D Reconstruction Bayesian Filtering
University of Macau, Macau SAR, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, Beijing, China

Mar 15, 2026J/K move · Enter open · S save

S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

Renhe Zhang, Yuyang Tan, Jingyu Gong, Zhizhong Zhang+3

East China Normal University · Shanghai Artificial Intelligence Laboratory

Streaming 3D Reconstruction 3D Scene Understanding 3D Gaussian Splatting 3D Instance Segmentation

Open paper AskPDF arXiv

Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.

Abstract

Explore similar work

S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

MoonSplat: Monocular Online Gaussian Splatting with Sim(3) Global Optimization

BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates