A robotic microscope watching living cells cannot afford to look as often as it would like. Every volume it acquires costs photons the specimen does not get back, and time owed to other wells. What such a platform exists to produce is a record of individual cells through time: which cell is which from one volume to the next, and which cell divided into which two. Sampling sparsely breaks that record exactly where it matters, and the fault lies in the acquisition schedule rather than in the analysis software. We fill the gaps by reconstructing them, fitting a 4D Gaussian model to whatever volumes the hardware could afford. The model is a cloud of light-emitting blobs, each carrying a position, a shape and a lifetime. Being continuous in time, it renders any missing volume on demand, decoupling how often the robot analyses from how often it can afford to look. The microscope's point-spread function is measured from the data rather than inherited from acquisition metadata or left to the optimizer, because metadata inflates it and the optimizer cannot recover it at all: a wider blur around a smaller blob fits the images equally well. Each blob's lifetime is stored in frames rather than as a fraction of the recording, so that it denotes a fixed duration on any sequence. Unmeasured timesteps are supervised at coarse scale by a 3D U-Net that predicts the intermediate volume directly and estimates no motion field, since a dividing cell becomes two and no motion describes that. On two Cell Tracking Challenge sequences, a C. elegans embryo and a Chinese Hamster Ovarian (CHO), with fidelity scored per cell nucleus, our reconstruction holds the highest nucleus fidelity at every distance from an acquired frame, has the flattest decay across the gap, and best recovers focal planes it was never shown with graceful degradation across the gap.
Figures & tables
Fig. 1: We turn a sparsely sampled acquisition into a continuous-time volumetric state. Because the state is queried by rendering, the rate at which the robot’s downstream modules run is decoupled from the rate at which its hardware can afford to acquire.
Held-out nucleus flux ↑
Training window
σt stored as
5 frames
100 frames
window fraction (upstream)
0.934
0.478
frames (ours)
0.970
0.853
Gain (ours − upstream)
+0.036
+0.375
TABLE I: The temporal gate stored as a fraction of the training window ( upstream ) and in frames ( ours ), on a short recording and a long one. Read down a column to compare the two parameterizations at a fixed window length; the bottom row is their difference. The gain scales with window length , the signature of a parameter defined proportionally to that length.
Fig. 2: Qualitative comparison across the acquisition gap at k=8 (CE02). Fidelity next to an acquired frame is not what separates the methods; what separates them is how much structure survives to the middle of the gap.
Method
d= all ↑
d=1↑
d=2↑
d=3↑
d=4↑
Decay d=1→4↑
Division window ↑
Primed ↑
Ours
0.481
0.694
0.476
0.326
0.370
−46.7%
0.417
0.664
Deformable 3DGS [ 13 ]
0.194
0.439
0.167
0.156
0.101
−77.0%
0.263
0.468
4DRGS [ 20 ]
0.233
0.576
0.290
0.152
0.080
−86.1%
0.303
0.432
TABLE II: Held-out nucleus flux at k=8 (CE02), by distance d to the nearest acquired frame. Clean role, quiescent phase, frozen cross-arm panel, median of three seeds. The primary statistic is the relative decay from d=1 to d=4 .
Method
Train flux ↑
Sharpness ↑
Retention ↑
Centroid ↓
Ours
0.878
0.225
0.547
14.13
Deformable 3DGS
0.908
0.101
0.214
22.81
4DRGS
0.906
0.088
0.258
12.08
TABLE III: Gating and morphology at k=8 . Sharpness is measured on held-out frames; centroid offset is in absolute pixels.
Masked z -PSNR (dB) ↑
CE02, k=8
CHO, k=4
Method
sup.
held
sup.
held
Ours
12.00
12.04
15.94
12.23
Deformable 3DGS
11.12
11.16
14.58
0 9.01
4DRGS
11.49
11.40
16.81
11.70
TABLE IV: Axial generalization on two sequences, as masked z -PSNR (dB). All share the PSF-aware forward model, so these columns compare temporal representations. Two columns come from same volumes under same foreground mask and differ only in which z -planes they pool: sup. covers the planes the model was fitted to, held the planes removed by the axial holdout and never seen in training.
Fig. 3: Second cell line (CHO, k=4 ). Held-out z -planes rendered by each method against the acquired volume, at the top and bottom of the stack where a plane has neighbours on one side only. The sequence transfers without retuning: no hyper-parameter is changed from the CE02 setting except the gate initialization, which follows the stride.
Fig. 4: Volume renderings of one held-out timestep in the middle of the gap ( d=1 , k=8 , CE02): the acquired volume and the three reconstructions. The slice views of Fig. 2 show what survives in plane; rendered as volumes, what separates the methods is how many nuclei stay resolvable as separate bodies instead of merging into their neighbours or disappearing.
Variant
Flux ↑
Centroid ↓
Div. win. ↑
Train ↑
Axial →0
(clean)
(px)
flux
flux
gap
off
0.353± .068
10.49± .87
0.439± .111
0.892± .033
0.171± .025
T
0.480± .042
11.10± .84
0.449± .012
0.890± .023
0.097± .021
TABLE V: Completion-signal ablation, k=8 , CE02, as mean ± s.d. over three seeds. off supervises only the acquired frames; T adds a low-frequency pseudo-target at the unobserved timesteps.
Recent progress in 4D Gaussian Splatting (4DGS) has achieved impressive dynamic scene reconstruction results. While these methods demonstrate remarkable performance, the specific factors behind their gains remain underexplored, making a systematic understanding of the underlying principles challenging. In this paper, we perform a comprehensive analysis of these hidden factors to provide a clearer perspective on the 4DGS framework. We first establish a controlled baseline, FreeTimeGS_ours, by formalizing and reproducing the heuristics of the state-of-the-art FreeTimeGS. Using this framework, we examine 4DGS along its fundamental axes and identify practical secrets, including the emergent temporal partitioning driven by Gaussian durations and the decoupling between photometric fidelity and motion behavior. Based on these insights, we propose FreeTimeGS++, a principled method that employs gated marginalization, UFM-guided initialization, and color correction to improve stability and reproducibility. Our approach yields reproducible results with reduced run-to-run variance.
4D Gaussian Splatting (4DGS) can render dynamic scenes photorealistically. However, with limited viewpoint coverage, some spatiotemporal regions remain sparsely observed, leading to artifacts, particularly in scenes with large motion. Existing approaches leveraging generative models rely on heuristic virtual-viewpoint selection before refining rendered views. As a result, they cannot actively explore such sparsely observed regions. To address this issue, we propose a pipeline that actively selects spatiotemporal virtual viewpoints to improve 4DGS reconstruction. Our method selects virtual viewpoints for generative enhancement based on the rendering sensitivity and motion-aware observation density of 4D Gaussians, prioritizing views that alleviate observation sparsity. In the refined images, we filter out regions that conflict with captured observations or are likely to contain generative artifacts and then fine-tune 4DGS using only the reliable regions. We evaluate our method on multi-view video benchmarks using new train/test splits designed to induce observation gaps. Results show consistent improvements over prior viewpoint selection strategies and fine-tuning methods in both qualitative and quantitative evaluations, while reducing artifacts.
We present a 4D Gaussian Splatting framework for the Sparse-View Track of the SIGGRAPH Asia 2026 Volumetric Video Challenge, which requires dynamic scene reconstruction from only six cameras with wide baselines. To achieve robust dynamic reconstruction under such sparse views, our framework integrates three components. (1) Region-adaptive spatial priors: We use foreground masks to guide Gaussian initialization and mask voting to control densification separately for the dynamic foreground and static background. Background geometry is regularized using monocular depth aligned to metric scale. (2) Motion-consistent temporal priors: We provide supervision at intermediate times through frame interpolation and constrain projected Gaussian motion with estimated optical flow. (3) Generative assistance: We place virtual cameras in the widest angular gaps and restore their rendered images using a diffusion-based model conditioned on camera poses. The restored images are iteratively incorporated into training as pseudo-supervision. On the validation set, our framework improves full-frame PSNR from 25.60 dB for the baseline to 29.75 dB. On the official test benchmark, it achieves 30.04 dB full-frame PSNR and 27.88 dB foreground PSNR, ranking first overall in the Sparse-View Track.
Shengqi Wang, Zhengxian Yang, Kaiwen Tian +6
Department of Automation, Tsinghua University, Beijing, China · JD.com, Beijing, China · Beijing MEET YUAN Co., Ltd, Beijing, China