We present ReFlow, a unified framework for monocular dynamic scene reconstruction that learns 3D motion in a novel self-correction manner from raw video. Existing methods often suffer from incomplete scene initialization for dynamic regions, leading to unstable reconstruction and motion estimation, which often resorts to external dense motion guidance such as pre-computed optical flow to further stabilize and constrain the reconstruction of dynamic components. However, this introduces additional complexity and potential error propagation. To address these issues, ReFlow integrates a Complete Canonical Space Construction module for enhanced initialization of both static and dynamic regions, and a Separation-Based Dynamic Scene Modeling module that decouples static and dynamic components for targeted motion supervision. The core of ReFlow is a novel self-correction flow matching mechanism, consisting of Full Flow Matching to align 3D scene flow with time-varying 2D observations, and Camera Flow Matching to enforce multi-view consistency for static objects. Together, these modules enable robust and accurate dynamic scene reconstruction. Extensive experiments across diverse scenarios demonstrate that ReFlow achieves superior reconstruction quality and robustness, establishing a novel self-correction paradigm for monocular 4D reconstruction.
Figures & tables
Figure 1: Typical Challenges in monocular dynamic scene reconstruction. Top: Incomplete initialization for dynamic regions: the initial 3D structure from SfM often misses dynamic components and initializes Gaussians without separating static points (green) from dynamic points (red) , leading to an entangled and incomplete representation. Bottom: To compensate, existing methods frequently resort to external dense motion guidance to constrain and stabilize the reconstruction of dynamic regions.
Figure 2: Motivation of Self-correction Flow Matching. (a) We start with a simple observation: 2D observations, such as the shifting balloon, are caused by 3D motion. Accurate reconstructed 3D Motion should naturally align with these visible changes. (b) Unlike previous methods that use external motion priors to supervise 3D motion, we instead uses raw video as motion supervision through a self-correction flow matching mechanism to directly align predicted 3D motion projections with 2D frame differences.
Figure 3: Overview of ReFlow . We start by constructing a complete canonical space(Sec. 3.2.1 ), which includes both static and dynamic components, ensuring a reliable 3D scene initialization. Next, we disentangle these elements using spatial and spatiotemporal feature planes(Sec. 3.2.2 ), providing a structured representation that separately handles static and dynamic regions. This preparation allows us to introduce targeted motion constraints(Sec. 3.3 ): Full Flow supervises motion across the entire scene, while Camera Flow enforces consistency in static regions, enabling the self-correction learning mechanism for accurate 3D motion reconstruction.
Figure 4: Self-correction flow matching mechanism. (a) Different Motion and Flow in the 4D Scene. Static areas move only due to camera motion (camera flow), while dynamic areas involve both camera and object motion (full flow). Accurate motion learning requires region-specific flow supervision. (b) Self-correction flow matching. We apply full flow to warp the entire image from state t1 to state t2 and compare with the real observation, validating overall motion. Camera flow is used similarly but only on static regions, ensuring their stability. Together, these provide a complementary self-correction signal for 3D motion learning.
Method
Balloon1
Balloon2
Jumping
Playground
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
3DGS [ 28 ]
17.71
0.407
0.473
20.62
0.500
0.431
14.45
0.394
0.385
15.75
0.341
0.493
STGS [ 34 ]
20.36
0.746
0.196
23.12
0.881
0.124
20.82
0.631
0.187
19.23
0.842
0.151
Ex4DGS [ 30 ]
14.69
0.351
0.503
16.29
0.471
0.457
18.93
0.661
0.321
14.16
0.442
0.437
Efficient-D3DGS [ 27 ]
23.32
0.864
0.140
26.20
0.906
0.099
21.54
0.784
0.253
24.08
0.892
0.098
Deformable-3DGS [ 63 ]
26.12
0.891
0.122
26.48
0.910
0.100
23.21
0.837
0.199
23.96
0.901
0.088
Table 1: Quantitative comparison on Nvidia Monocular dataset. We report PSNR/SSIM/LPIPS per scene; the last block shows the mean across all available scenes (including dynamicFace when available).
Figure 5: Qualitative comparison on Nvidia Monocular dataset [ 18 ] . Yellow boxes highlight zoomed-in regions for detail examination. Per-scene average PSNR values are provided.
Figure 6: Qualitative comparison on Nerfies-HyperNeRF dataset [ 45 , 46 ] . Yellow boxes highlight zoomed-in regions for detail examination. Per-scene average PSNR values are provided.
Methods
(a) Nerfies-HyperNeRF
PSNR ↑
SSIM ↑
T-NeRF [ 47 ]
20.43
0.508
NSFF [ 35 ]
19.41
0.423
Nerfies [ 45 ]
20.08
0.462
HyperNeRF [ 46 ]
20.40
0.467
Deformable-3DGS [ 63 ]
22.19
0.558
Table 2: Quantitative comparison on Nerfies-HyperNeRF dataset [ 45 , 46 ] . We highlight the best and second best results.
Figure 7: Visualization of self-correction flow matching effects. Top: Incorrect motion leads to erroneous warping results that do not align with the ground truth, highlighted by yellow boxes, which in turn drives motion optimization. Bottom: After flow matching, finer details are better reconstructed, demonstrating improved rendering quality.
#
CompInit.
Sep.
FullFM
CamFM
PSNR ↑
SSIM ↑
LPIPS ↓
Baseline
25.81
0.844
0.210
1
✓
26.45
0.865
0.172
2
w/o align
24.50
0.815
0.260
3
✓
26.60
0.870
0.167
4
✓
✓
27.00
0.880
0.159
5
✓
✓
✓
27.85
0.897
0.116
Table 3: Ablation study of key components in our method. CompInit.: Complete Canonical Space Initialization; Sep.: Separation-based Dynamic Scene Modeling; FullFM: Full Flow Matching; CamFM: Camera Flow Matching. We highlight best performance in bold .
Figure 8: Illustration of the computation of Full Flow (top-left) and Camera Flow (bottom-right).
Method
Balloon1
Balloon2
dynamicFace
Jumping
Baseline [ 55 ]
35m 32s / 6.30G
37m 49s / 16.88G
58m 23s / 6.24G
37m 56s / 3.20G
Ours
44m 52s / 7.53G
46m 4s / 18.33G
68m 28s / 7.34G
46m 41s / 4.53G
Method
Playground
Skating
Truck
Umbrella
Baseline [ 55 ]
51m 19s / 6.45G
37m 19s / 2.32G
34m 56s / 4.12G
44m 45s / 5.96G
Ours
61m 9s / 7.50G
47m 30s / 3.60G
42m 58s / 5.61G
54m 15s / 7.11G
Table 4: Training Time and Peak GPU Memory Usage. We compare our method with the baseline on the Nvidia Monocular dataset.
Method
Balloon1
Balloon2
dynamicFace
Jumping
Baseline [ 55 ]
20fps / 74M
21fps / 72M
15fps / 106M
26fps / 35M
Ours
18fps / 87M
17fps / 86M
13fps / 121M
23fps / 52M
Method
Playground
Skating
Truck
Umbrella
Baseline [ 55 ]
15fps / 93M
30fps / 33M
24fps / 45M
25fps / 54M
Ours
15fps / 105M
25fps / 47M
21fps / 65M
21fps / 72M
Table 5: Rendering Speed and Model Storage Size. We report the rendering frame rate (FPS) and the final model size (in MBs) for all scenes on the Nvidia Monocular dataset.
Figure 9: Failure case of the Utilized Geometry Foundation Model. Top Left: Aggregated point cloud exhibits noise and distortion from merging inconsistent frames. Top Right: Per-frame structure reconstructions show significant noise and fragmentation, demonstrating its difficulty in establishing reliable geometry.
Figure 10: Visualization of our self-correction flow matching progress across training iterations, using the DynamicFace sequence from the Nvidia Monocular dataset [ 18 ] . Each row shows results at different training iterations: 300 (top row), 1000 (second row), 5000 (third row), and 7000 (bottom row). The columns present: Left column ( I1 ): Ground truth image at time t1 . Middle column ( I1warped ): Ground truth image from t1 warped using our predicted flow field F , demonstrating how content transforms to match the target frame. Right column ( I2 ): Ground truth image at time t2 , serving as the reference for evaluating warping accuracy. As training progresses, I1warped increasingly aligns with I2 , demonstrating that our self-correction flow matching effectively learns to model dynamic scene motion without requiring external motion guidances.
Method
Balloon1
Balloon2
Jumping
dynamicFace
Ours
27.65 / 0.903 / 0.092
29.01 / 0.908 / 0.098
24.85 / 0.855 / 0.151
30.98 / 0.973 / 0.036
Ours (External Flow)
27.35 / 0.899 / 0.095
27.74 / 0.843 / 0.179
24.92 / 0.853 / 0.162
31.10 / 0.975 / 0.034
Method
Playground
Skating
Truck
Umbrella
Ours
25.58 / 0.889 / 0.086
30.17 / 0.946 / 0.076
30.18 / 0.917 / 0.115
27.21 / 0.831 / 0.170
Ours (External Flow)
20.97 / 0.820 / 0.099
30.09 / 0.976 / 0.032
29.30 / 0.905 / 0.133
27.35 / 0.831 / 0.171
Table 6: Quantitative comparison using PSNR( ↑ ), SSIM( ↑ ), and LPIPS( ↓ ) metrics. Best results are in bold .
Figure 11: Qualitative comparison on all scenes from the Nvidia Monocular dataset [ 18 ] .
Figure 12: More qualitative comparisons from the Nerfies-HyperNeRF dataset [ 45 , 46 ] .
Figure 13: Novel time synthesis results with trajectory visualization across different dynamic scenes. Each column shows a different dataset: Nvidia-Playground, Nvidia-Umbrella, Nvidia-Skating, Hypernerf-broom, and Hypernerf-banana. The yellow boxes highlight dynamic/static regions with our tracked trajectories visualized using the Deform-GS approach. Rows represent different timesteps with fixed camera positions, demonstrating how our method correctly models temporal scene evolution. Note how static background elements remain perfectly stable across frames while dynamic components exhibit physically plausible motion paths. The Playground scene (leftmost column) particularly demonstrates our method’s capability to preserve fine structures like blue ribbons during motion, which are typically challenging to reconstruct accurately.
Method
Broom
Tail
Toby-sit
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
T-NeRF [ 47 ]
20.17
0.257
22.11
0.385
18.53
0.330
NSFF [ 35 ]
20.46
0.247
21.72
0.388
18.65
0.329
Nerfies [ 45 ]
19.51
0.202
21.17
0.305
18.41
0.326
HyperNeRF [ 46 ]
19.23
0.197
21.13
0.301
18.33
0.324
Deformable-3DGS [ 63 ]
20.51
0.352
22.10
0.474
21.13
0.427
Table 8: Per-scene performance metrics on Nerfies-HyperNeRF [ 45 , 46 ] dataset. We highlight the best and second best results.
Method
3D Printer
Chicken
Broom
Banana
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
T-NeRF [ 47 ]
18.60
0.591
24.41
0.764
20.17
0.257
22.07
0.721
NSFF [ 35 ]
16.26
0.426
20.72
0.619
20.46
0.247
18.62
0.530
Nerfies [ 45 ]
18.81
0.588
22.71
0.742
19.51
0.202
19.85
0.609
HyperNeRF [ 46 ]
18.73
0.583
23.88
0.753
19.23
0.197
21.08
0.641
Deformable-3DGS [ 63 ]
20.53
0.641
22.82
0.618
20.51
0.352
26.05
0.833
Table 9: Per-scene performance metrics on HyperNeRF [ 46 ] dataset. We highlight the best and second best results.
Reconstructing dynamic 4D scenes remains challenging due to the presence of moving objects that corrupt camera pose estimation. Existing optimization methods alleviate this issue with additional supervision, but they are mostly computationally expensive and impractical in real-time applications. To address these limitations, we propose MoRe, a feedforward 4D reconstruction network that efficiently recovers dynamic 3D scenes from monocular videos. Built upon a strong static reconstruction backbone, MoRe employs an attention-forcing strategy to disentangle dynamic motion from static structure. To further enhance robustness, we fine-tune the model on large-scale, diverse datasets encompassing both dynamic and static scenes. Moreover, our grouped causal attention captures temporal dependencies and adapts to varying token lengths across frames, ensuring temporally coherent geometry reconstruction. Extensive experiments on multiple benchmarks demonstrate that MoRe achieves high-quality dynamic reconstructions with exceptional efficiency.
Juntong Fang, Zequn Chen, Weiqi Zhang +4
School of Software, Tsinghua University, Beijing, China · Li Auto
Reconstructing dynamic 4D scenes requires jointly estimating correspondence, geometry, object motion, and camera motion. Existing feed-forward methods typically predict dense task-specific maps or independently process source-target pairs, leading to unnecessary computation for sparse queries and limited feature reuse across different frame pairs. We present UniQuery4R, a query-conditioned framework that encodes a multi-frame clip once and selects the source view, target view, and continuous source-image coordinate only at decoding time via source-to-target cross-attention. Each query jointly predicts target correspondence, target-time 3D position, and scene flow, along with source depth, while camera parameters are estimated per view. This design allows the encoded clip to be reused across arbitrary source-target selections and supports both sparse inference and dense reconstruction through batched queries, without learned temporal embeddings tied to a fixed clip length. We further introduce a direction-magnitude parameterization of scene flow with separate supervision for moving and static points. Among the evaluated methods, UniQuery4R achieves the best macro-average results on WorldTrack for both scene-flow estimation and dynamic-point reconstruction.
Tiancheng Chen, Sheng Tang, Wenhua Jin +4
Kosmo Research · Automotive Engineering Department, Jilin University · School of Software, Tsinghua University
Streaming feed-forward 3D reconstruction enables real-time joint estimation of scene geometry and camera poses from RGB images. However, without explicit dynamic reasoning, streaming models can be affected by moving objects, causing artifacts and drift. In this work, we propose RayMap3R, a training-free streaming framework for dynamic scene reconstruction. We observe that RayMap-based predictions exhibit a static-scene bias, providing an internal cue for dynamic identification. Based on this observation, we construct a dual-branch inference scheme that identifies dynamic regions by contrasting RayMap and image predictions, suppressing their interference during memory updates. We further introduce reset metric alignment and state-aware smoothing to preserve metric consistency and stabilize predicted trajectories. Our method achieves state-of-the-art performance among streaming approaches on dynamic scene reconstruction across multiple benchmarks.