We present a feed-forward human performance capture method that renders novel views of a performer from a monocular RGB stream. A key challenge in this setting is the lack of sufficient observations, especially for unseen regions. Assuming the subject moves continuously over time, we take advantage of the fact that more body parts become observable by maintaining a canonical space that is progressively updated with each incoming frame. This canonical space accumulates appearance information over time and serves as a context bank when direct observations are missing in the current live frame. To effectively utilize this context while respecting the deformation of the live state, we formulate the rendering process as probabilistic regression. This resolves conflicts between past and current observations, producing sharper reconstructions than deterministic regression approaches. Furthermore, it enables plausible synthesis even in regions with no prior observations. Experiments on in-domain (4D-Dress) and out-of-distribution (MVHumanNet) datasets demonstrate the effectiveness of our approach.
Figures & tables
Figure 2: GenFusion renders novel views of live frames in a feed-forward manner from a monocular RGB stream. Given a live frame It , feature map Ft is extracted and aligned to the SMPL-X template mesh, resulting in vertex-aligned feature set St . St is fused into the canonical feature set Scan , updating temporal history. The canonical feature set Scan is then warped and densified into the live space, forming the input Gcontext,t for the denoising network Udenoiser . The denoising network denoises the noisy image Z into the final novel view live frame, conditioned on the live frame’s state Glive,t . The right side shows the progressive update of the canonical space. The first column shows the feature map; the second shows its RGB rendering. Our method synthesizes realistic details even without observations (top row) and refines the canonical space as more frames are incorporated.
Figure 3: Deterministic regression supervision (e.g., pixel-wise loss) penalizes deformation mismatch, leading to blurry outputs, while probabilistic regression supervision focuses on perceptually realistic synthesis rather than pixel-wise mismatch.
Figure 4: In-domain generalization results on the 4D-Dress dataset. Monocular input stream is shown in the white boxes. Our method effectively reconstructs novel views that align with past observations. Per-frame probabilistic methods (Champ, SIFU) fail to reconstruct the details observed in prior observations. Per-frame deterministic regression methods (SHERF, GHG) struggle with synthesizing unobserved details. Temporal deterministic method NHP can leverage temporal context but produces blurry outputs. GauHuman requires per-subject optimization but delivers lower visual quality and lacks generalizability.
Method
Generalizable
Temporal Context
Synthesis Objective
PSNR ( ↑ )
LPIPS-VGG ( ↓ )
FVD ( ↓ )
GauHuman Hu et al. (2024b)
✗ (Per-subject)
✗
Deterministic
23.19
83.34
500.8
Champ ( Zhu et al., 2024b )
✓
✗
Probabilistic
19.37
98.61
254.5
SHERF ( Hu et al., 2023 )
✓
✗
Deterministic
21.86
86.34
735.3
GHG ( Kwon et al., 2025 )
✓
✗
Deterministic
24.50
75.60
502.93
NHP ( Kwon et al., 2021 )
✓
✗
Deterministic
24.72
96.26
630.0
Ours
✓
✓
Probabilistic
25.07
62.97
176.7
Table 1: In-domain generalization results on the 4D-Dress dataset. Our method demonstrates superior performance on perceptual metrics, highlighting the effectiveness of leveraging temporal context and probabilistic rendering. For readability, LPIPS-VGG is scaled by a factor of 1000.
Figure 5: Cross-dataset generalization results on the MVHumanNet dataset. Left column presents the monocular input stream. Champ hallucinates irrelevant details when those regions are invisible in the current live input frame. In contrast, our method generates details relevant to past observations.
Method
Generalizable
Temporal Context
Synthesis Objective
PSNR ( ↑ )
LPIPS-VGG ( ↓ )
FVD ( ↓ )
Champ ( Zhu et al., 2024b )
✓
✗
Probabilistic
21.06
97.61
674.1
NHP ( Kwon et al., 2021 )
✓
✓
Deterministic
22.25
131.91
1321.4
Ours
✓
✓
Probabilistic
21.25
87.85
436.9
Table 2: Cross-dataset generalization on MVHumanNet. Our method effectively leverages temporal context to synthesize details coherent with frame history. Champ, while visually appealing, generates details that are inconsistent with past observations, as reflected in the FVD metric.
Figure 6: In-the-wild generalization on TikTok. Task: generate a frontal view from a back view. Champ copies the input, NHP is blurry, and ours produces a clear, observation-faithful frontal view.
Figure 7: Ablation results on the 4D-Dress dataset . (a) Removing temporal context leads to inconsistencies, while (b) using raw RGB values instead of encoded features limits detail recovery. (c) A deterministic regression struggles to capture fine-grained details and dynamic deformations. (d) Our full method, combining all components, achieves superior performance in terms of visual quality and coherence to temporal context.
Method
Temporal Context
Synthesis Objective
PSNR ( ↑ )
LPIPS-VGG ( ↓ )
FVD ( ↓ )
(a) No temporal context
None (Normal map of current frame)
Probabilistic
25.03
63.34
177.4
(b) No feature context
Raw RGB values
Probabilistic
24.37
64.51
191.9
(c) No probabilistic objective
Encoded features ( Scan )
Deterministic (Pixel MSE)
25.23
95.70
572.3
(d) Our full method
Encoded features ( Scan )
Probabilistic
25.07
62.97
176.7
Table 3: Ablation studies on the 4D-Dress dataset . (a) Without temporal context, frame-level quality is maintained but temporal coherence degrades. (b) Using raw RGB values as input leads to a decline in perceptual quality due to the loss of fine details. (c) The deterministic regression objective exhibits limitations in capturing high-frequency details. (d) Our full method effectively leverages rich temporal observations encoded in the feature representation and the probabilistic regression to achieve high-quality results align with past observations.
Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions. Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world datasets and demonstrate applications such as novel trajectory rendering and human replacement.
We present DuoMo, a generative method that recovers human motion in world-space coordinates from unconstrained videos with noisy or incomplete observations. Reconstructing such motion requires solving a fundamental trade-off: generalizing from diverse and noisy video inputs while maintaining global motion consistency. Our approach addresses this problem by factorizing motion learning into two diffusion models. The camera-space model first estimates motion from videos in camera coordinates. The world-space model then lifts this initial estimate into world coordinates and refines it to be globally consistent. Together, the two models can reconstruct motion across diverse scenes and trajectories, even from highly noisy or incomplete observations. Moreover, our formulation is general, generating the motion of mesh vertices directly and bypassing parametric models. DuoMo achieves state-of-the-art performance. On EMDB, our method obtains a 16% reduction in world-space reconstruction error while maintaining low foot skating. On RICH, it obtains a 30% reduction in world-space error. Project page: https://yufu-wang.github.io/duomo/
Yufu Wang, Evonne Ng, Soyong Shin +8
Meta Reality Labs · University of Pennsylvania · Carnegie Mellon University
Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering. Recent feed-forward methods can efficiently predict geometry, but they are often limited to non-photorealistic representations such as point clouds and meshes, or they fail to handle non-rigid objects, particularly dynamic humans. To fill this gap, we present GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction), a feed-forward framework for online dynamic human-scene reconstruction. From a monocular human-scene video, our method reconstructs dynamic humans (everyone) and static scenes (everywhere) in a single forward pass (all at once) as 3D Gaussian Splatting (3DGS) primitives (as gaussians), which are geometrically consistent and capable of novel view synthesis. Experiments on monocular human-scene datasets demonstrate that our approach achieves competitive novel view synthesis quality while significantly improving inference efficiency compared to optimization-based methods.