GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates

Youngjoong Kwon; Yao He; Heejung Choi; Chen Geng; Zhengmao Liu; Jiajun Wu; Ehsan Adeli

cs.CVMar 30, 2026

GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates

Authors: Youngjoong Kwon, Yao He, Heejung Choi, Chen Geng, Zhengmao Liu, Jiajun Wu, Ehsan Adeli

Organizations: Stanford University

Abstract

We present a feed-forward human performance capture method that renders novel views of a performer from a monocular RGB stream. A key challenge in this setting is the lack of sufficient observations, especially for unseen regions. Assuming the subject moves continuously over time, we take advantage of the fact that more body parts become observable by maintaining a canonical space that is progressively updated with each incoming frame. This canonical space accumulates appearance information over time and serves as a context bank when direct observations are missing in the current live frame. To effectively utilize this context while respecting the deformation of the live state, we formulate the rendering process as probabilistic regression. This resolves conflicts between past and current observations, producing sharper reconstructions than deterministic regression approaches. Furthermore, it enables plausible synthesis even in regions with no prior observations. Experiments on in-domain (4D-Dress) and out-of-distribution (MVHumanNet) datasets demonstrate the effectiveness of our approach.

Figures & tables

Figure 2: GenFusion renders novel views of live frames in a feed-forward manner from a monocular RGB stream. Given a live frame $I_{t}$ , feature map $F_{t}$ is extracted and aligned to the SMPL-X template mesh, resulting in vertex-aligned feature set $S_{t}$ . $S_{t}$ is fused into the canonical feature set $S_{\text{can}}$ , updating temporal history. The canonical feature set $S_{\text{can}}$ is then warped and densified into the live space, forming the input $G_{\text{context,t}}$ for the denoising network $\mathcal{U}_{\text{denoiser}}$ . The denoising network denoises the noisy image $Z$ into the final novel view live frame, conditioned on the live frame’s state $G_{\text{live,t}}$ . The right side shows the progressive update of the canonical space. The first column shows the feature map; the second shows its RGB rendering. Our method synthesizes realistic details even without observations (top row) and refines the canonical space as more frames are incorporated.

Figure 3: Deterministic regression supervision (e.g., pixel-wise loss) penalizes deformation mismatch, leading to blurry outputs, while probabilistic regression supervision focuses on perceptually realistic synthesis rather than pixel-wise mismatch.

Figure 4: In-domain generalization results on the 4D-Dress dataset. Monocular input stream is shown in the white boxes. Our method effectively reconstructs novel views that align with past observations. Per-frame probabilistic methods (Champ, SIFU) fail to reconstruct the details observed in prior observations. Per-frame deterministic regression methods (SHERF, GHG) struggle with synthesizing unobserved details. Temporal deterministic method NHP can leverage temporal context but produces blurry outputs. GauHuman requires per-subject optimization but delivers lower visual quality and lacks generalizability.

Method	Generalizable	Temporal Context	Synthesis Objective	PSNR ( $\uparrow$ )	LPIPS-VGG ( $\downarrow$ )	FVD ( $\downarrow$ )
GauHuman Hu et al. (2024b)	✗ (Per-subject)	✗	Deterministic	23.19	83.34	500.8
Champ ( Zhu et al., 2024b )	✓	✗	Probabilistic	19.37	98.61	254.5
SHERF ( Hu et al., 2023 )	✓	✗	Deterministic	21.86	86.34	735.3
GHG ( Kwon et al., 2025 )	✓	✗	Deterministic	24.50	75.60	502.93
NHP ( Kwon et al., 2021 )	✓	✗	Deterministic	24.72	96.26	630.0
Ours	✓	✓	Probabilistic	25.07	62.97	176.7

Table 1: In-domain generalization results on the 4D-Dress dataset. Our method demonstrates superior performance on perceptual metrics, highlighting the effectiveness of leveraging temporal context and probabilistic rendering. For readability, LPIPS-VGG is scaled by a factor of 1000.

Figure 5: Cross-dataset generalization results on the MVHumanNet dataset. Left column presents the monocular input stream. Champ hallucinates irrelevant details when those regions are invisible in the current live input frame. In contrast, our method generates details relevant to past observations.

Method	Generalizable	Temporal Context	Synthesis Objective	PSNR ( $\uparrow$ )	LPIPS-VGG ( $\downarrow$ )	FVD ( $\downarrow$ )
Champ ( Zhu et al., 2024b )	✓	✗	Probabilistic	21.06	97.61	674.1
NHP ( Kwon et al., 2021 )	✓	✓	Deterministic	22.25	131.91	1321.4
Ours	✓	✓	Probabilistic	21.25	87.85	436.9

Table 2: Cross-dataset generalization on MVHumanNet. Our method effectively leverages temporal context to synthesize details coherent with frame history. Champ, while visually appealing, generates details that are inconsistent with past observations, as reflected in the FVD metric.

Figure 6: In-the-wild generalization on TikTok. Task: generate a frontal view from a back view. Champ copies the input, NHP is blurry, and ours produces a clear, observation-faithful frontal view.

Figure 7: Ablation results on the 4D-Dress dataset . (a) Removing temporal context leads to inconsistencies, while (b) using raw RGB values instead of encoded features limits detail recovery. (c) A deterministic regression struggles to capture fine-grained details and dynamic deformations. (d) Our full method, combining all components, achieves superior performance in terms of visual quality and coherence to temporal context.

Method	Temporal Context	Synthesis Objective	PSNR ( $\uparrow$ )	LPIPS-VGG ( $\downarrow$ )	FVD ( $\downarrow$ )
(a) No temporal context	None (Normal map of current frame)	Probabilistic	25.03	63.34	177.4
(b) No feature context	Raw RGB values	Probabilistic	24.37	64.51	191.9
(c) No probabilistic objective	Encoded features ( $S_{\text{can}}$ )	Deterministic (Pixel MSE)	25.23	95.70	572.3
(d) Our full method	Encoded features ( $S_{\text{can}}$ )	Probabilistic	25.07	62.97	176.7

Table 3: Ablation studies on the 4D-Dress dataset . (a) Without temporal context, frame-level quality is maintained but temporal coherence degrades. (b) Using raw RGB values as input leads to a decline in perceptual quality due to the loss of fine details. (c) The deterministic regression objective exhibits limitations in capturing high-frequency details. (d) Our full method effectively leverages rich temporal observations encoded in the feature representation and the probabilistic regression to achieve high-quality results align with past observations.

Explore similar work

4D Human-Scene Reconstruction from Low-Overlap Captures
Jul 10, 2026Minhyuk Hwang, Sangmin Kim, Seunguk Do +24D Reconstruction Dynamic Scene Reconstruction
Seoul National University, Republic of Korea
DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction
Mar 3, 2026Yufu Wang, Evonne Ng, Soyong Shin +83D Human Reconstruction Markerless Motion Capture
Meta Reality Labs · University of Pennsylvania · Carnegie Mellon University
GUSH3R: Everyone Everywhere All at Once as Gaussians
Jul 6, 2026Keito Abe, Kaede Shiohara, Takashi Otonari +13D Gaussian Splatting 3D Reconstruction
The University of Tokyo

Jul 10, 2026J/K move · Enter open · S save

4D Human-Scene Reconstruction from Low-Overlap Captures

Minhyuk Hwang, Sangmin Kim, Seunguk Do, Daneul Kim+1

Seoul National University, Republic of Korea

4D Reconstruction Dynamic Scene Reconstruction Dynamic Novel View Synthesis Sparse-View 3D Reconstruction

Open paper AskPDF arXiv

Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions. Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world datasets and demonstrate applications such as novel trajectory rendering and human replacement.

Abstract

Explore similar work

4D Human-Scene Reconstruction from Low-Overlap Captures

DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction

GUSH3R: Everyone Everywhere All at Once as Gaussians