Real-Time Human Reconstruction and Animation using Feed-Forward Gaussian Splatting

Devdoot Chatterjee; Zakaria Laskar; C. V. Jawahar

cs.CVApr 11, 2026

Real-Time Human Reconstruction and Animation using Feed-Forward Gaussian Splatting

Authors: Devdoot Chatterjee, Zakaria Laskar, C. V. Jawahar

Organizations: IIIT Hyderabad, India · IISER Thiruvananthapuram, India

Abstract

We present HumanGS, a generalizable feed-forward Gaussian splatting framework for human 3D reconstruction and real-time animation from sparse multi-view RGB images and their associated SMPL-X poses. Unlike prior methods that rely on depth supervision, fixed input views, UV maps, repeated feed-forward inference for each target pose or view, or computationally expensive vertex-to-image cross-attention, HumanGS employs a simple transformer architecture that explicitly associates SMPL-X vertices with multi-scale image features through geometric back-projection. This eliminates the need for large pre-trained human representation models while naturally aggregating complementary information from multiple views. The aggregated vertex features are mapped by a lightweight MLP decoder to a canonical set of 3D Gaussian primitives aligned with SMPL-X vertices. One Gaussian is regularized to remain close to the SMPL-X surface, providing a strong geometric prior and stable correspondence to the parametric body model, while a small set of unconstrained Gaussians per vertex captures geometric details beyond the body surface, such as clothing and hair. The resulting canonical representation is animated efficiently using linear blend skinning and Gaussian rasterization without further network inference. Trained entirely from scratch on only 10K frames from THuman2.1, HumanGS achieves reconstruction quality comparable to or better than state-of-the-art methods while reducing reconstruction time by over 15 times compared to recent transformer-based approaches, enabling real-time animation and interactive applications. Code and pre-trained models are available at https://github.com/Devdoot57/HumanGS .

Figures & tables

Figure 1: Overview of the HumanGS architecture and explicit inference pipeline. A Transformer Encoder processes sparse input images and pose maps to extract global 2D feature maps. Canonical SMPL-X [ 38 ] vertices project onto these maps to sample local geometry-aware features. A Vertex Query Decoder predicts 3D Gaussian attributes in the canonical T-pose space. During inference (green dashed box), these Gaussians are animated via Linear Blend Skinning (LBS) with target pose transforms and rasterized, bypassing neural network evaluation.

Figure 2: Feature Aggregation. Tokens from four uniform transformer depths are reshaped into 2D maps, projected via parallel $1\times 1$ convolutions, concatenated, and fused. Progressive upsampling quadruples the resolution to produce a dense feature map for vertex-aligned sampling.

Panel of Figure 3 — Figure 3: Qualitative Generalization on AvatarReX. We visualize the animation capabilities on unseen subjects. Left: Input reference view ( ∗ denotes the single input for LHM; other methods use 4 views). Right: Re-enactment results driven by a novel motion sequence.

Panel of Figure 4 — Figure 4: Qualitative comparison on THuman2.1 For each subject, the left block shows the sparse input views. The right block displays novel view synthesis results from different methods.

	TH2.1 $\rightarrow$ AvRex
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Modeling (s) $\downarrow$	Synthesis (s) $\downarrow$
LHM*	23.72	0.95	0.045	4.59	0.053
HumanRAM	24.58	0.96	0.042	-	0.287
Animatable Gaussians	25.51	0.96	0.041	12000	0.245
HumanGS	25.24	0.96	0.04	0.29	0.048

Table 1: Quantitative evaluation on TH2.1

\rightarrow

AvRex. HumanGS demonstrates robust cross-dataset generalization, matching the rendering fidelity of optimization-based methods while drastically reducing computational time. (*Note: LHM was evaluated using its official pre-trained model, which leverages a single-image input and was trained on a large-scale dataset of

\sim

300k videos.)

Table 6

Figure 5: Ablation of Intermediate Feature Aggregation. Left: Distribution of localized LPIPS differences on $64\times 64$ foreground patches (positive difference indicates superior quality). Right: Qualitative comparisons from the distribution’s high-performance tail. Intermediate features successfully recover high-frequency textures blurred by the baseline.

Figure S1 : Generalization Performance Comparison Across Varying View Counts on AvatarReX (Animation). We evaluate PSNR, SSIM, and LPIPS under 1, 2, 4, and 6 input views with varying poses on unseen subjects from the AvatarReX dataset. HumanGS consistently outperforms feed-forward baselines HumanRAM and GST. While the optimization-based Animatable Gaussians achieves comparable metrics given sufficient data (4 to 6 views), it suffers a catastrophic drop in performance under extreme sparsity (1 to 2 views), whereas our method remains robust. Note that the single-image baseline GST is reported only at $N=1$ .

Methods	1 View			2 Views			4 Views			6 Views
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
HumanRAM	25.44	0.946	0.052	27.39	0.950	0.044	29.10	0.960	0.039	29.90	0.966	0.036
LVSM	17.78	0.900	0.101	22.64	0.930	0.066	25.96	0.950	0.055	27.03	0.950	0.051
GHG	21.71	0.920	0.094	22.43	0.930	0.089	22.69	0.930	0.088	22.72	0.930	0.088
HumanGS (Ours)	24.07	0.936	0.068	25.71	0.946	0.059	27.19	0.950	0.054	27.84	0.950	0.051

Table S1 : Reconstruction Performance Across Varying View Counts. Quantitative comparison across varying numbers of input views (

N\in\{1,2,4,6\}

) on the modified THuman 2.1 test set, where subjects are positioned closer to the cameras to evaluate fine-grained detail synthesis.

Figure S2 : Effectiveness of the Global Token Under Varying Input Sparsity. We evaluate reconstruction performance with and without the Global Token across 1 to 8 input views. By encoding holistic subject context, the token provides a crucial fallback for severely occluded regions in extreme sparse-view scenarios ( $N\in\{1,2\}$ ), yielding significant improvements. As spatial coverage increases ( $N\geq 4$ ) and occlusions naturally decrease, the performance margins converge.

Dataset	1 Gaussian per Vertex			1 Tight + 4 Free Gaussians (Ours)
Dataset	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
THuman 2.1	30.66	0.970	0.028	30.81	0.980	0.024
THuman 2.1 (Near)	27.09	0.950	0.062	27.19	0.950	0.054
AvatarReX	25.20	0.960	0.044	25.24	0.960	0.040
THuman 4.0	25.42	0.960	0.034	25.53	0.960	0.029

Table S2: Ablation on Gaussian Primitives per Vertex. Comparing reconstruction and animation performance using a single Gaussian versus our proposed hybrid representation (1 tight + 4 free Gaussians). The hybrid approach consistently improves perceptual quality (lower LPIPS) across all datasets by better capturing loose clothing and structures that deviate from the parametric mesh.

Panel of Figure 12 — Figure S3 : Qualitative comparison on THuman2.1, AvatarReX, and THuman4.0. For each subject, the left block shows the sparse input views. The right block displays novel view synthesis results from different methods. Our method achieves comparable visual quality to HumanRAM and superior visual quality to other methods while enabling real-time animation capabilities. The first four rows are from THuman2.1 and the last two rows are from AvatarReX and THuman4.0 respectively.

Panel of Figure 13 — Figure S4 : Qualitative Generalization on AvatarReX. Left: Input reference view. Right: Re-enactment results driven by a novel motion sequence.

Panel of Figure 14 — Figure S5 : Novel Pose Synthesis Results. We demonstrate the explicit animation capabilities of HumanGS using subjects from THuman2.1 driven by motion sequences from THuman4.0. Left: The sparse set of 4 input views used to predict the canonical asset. Right: The predicted asset re-posed into various unseen configurations using Linear Blend Skinning and rasterization. This confirms that our method learns a consistent, animatable 3D representation from sparse supervision.

Explore similar work

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras
Sep 14, 2026Hanzhang Tu, Zhanfeng Liao, Wei Min +23D Gaussian Splatting Camera Pose Estimation
Tsinghua University, Beijing, China · Shadow AI, Beijing, China
PointSplat: Compact Gaussian Splatting via Human-Centric Prediction
Jun 30, 2026Yujie Guo, Yudong Jin, Lingteng Qiu +73DGS Compression 3D Gaussian Splatting
State Key Lab of CAD&CG, Zhejiang University, Hangzhou, China · The Chinese University of Hong Kong, Shenzhen, China · ByteDance, Beijing, China
HumanSplatHMR: Closing the Loop Between Human Mesh Recovery and Gaussian Splatting Avatar
May 4, 2026Yeheng Zong, Pou-Chun Kung, Yike Pan +43D Reconstruction Human Mesh Recovery
University of Michigan

Sep 14, 2026J/K move · Enter open · S save

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras

Hanzhang Tu, Zhanfeng Liao, Wei Min, Jiajun Zhang+1

Tsinghua University, Beijing, China · Shadow AI, Beijing, China

3D Gaussian Splatting Camera Pose Estimation Dynamic Scene Reconstruction Sparse-View 3D Reconstruction 3D Human Reconstruction Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Live free-viewpoint visualization of real humans is critical for immersive communication and interactive digital experiences. Existing methods either rely on computationally expensive optimization or require calibrated cameras and low-resolution inputs, making real-time high-resolution deployment impractical. In this work, we present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization from sparse, unposed RGB streams. Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance in a single forward pass. To achieve this, we start by designing a lightweight sparsity-aware multi-view transformer backbone that tokenizes foreground human regions while preserving global context through a shared scene token. We then employ a fully transformer-based Gaussian decoder to mitigate convolution-induced over-smoothing while keeping decoding sparse and efficient. In addition, we introduce a hybrid feature pyramid that injects multi-scale appearance cues into geometry prediction. We further introduce a lightweight differentiable Levenberg-Marquardt camera refinement layer to enhance multi-view consistency and geometric alignment. Moreover, to stabilize learning under sparse, unposed inputs, we transfer multi-view geometry priors from a large visual-geometry foundation model via teacher-student distillation. Finally, the predicted Gaussian maps are streamed with video codecs to remote devices for interactive free-viewpoint rendering. Extensive experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU. Additional captured sequences illustrate its performance across varied subjects, clothing, and motions under our multi-camera setup.

Abstract

Explore similar work

Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras

PointSplat: Compact Gaussian Splatting via Human-Centric Prediction

HumanSplatHMR: Closing the Loop Between Human Mesh Recovery and Gaussian Splatting Avatar