We present HumanGS, a generalizable feed-forward Gaussian splatting framework for human 3D reconstruction and real-time animation from sparse multi-view RGB images and their associated SMPL-X poses. Unlike prior methods that rely on depth supervision, fixed input views, UV maps, repeated feed-forward inference for each target pose or view, or computationally expensive vertex-to-image cross-attention, HumanGS employs a simple transformer architecture that explicitly associates SMPL-X vertices with multi-scale image features through geometric back-projection. This eliminates the need for large pre-trained human representation models while naturally aggregating complementary information from multiple views. The aggregated vertex features are mapped by a lightweight MLP decoder to a canonical set of 3D Gaussian primitives aligned with SMPL-X vertices. One Gaussian is regularized to remain close to the SMPL-X surface, providing a strong geometric prior and stable correspondence to the parametric body model, while a small set of unconstrained Gaussians per vertex captures geometric details beyond the body surface, such as clothing and hair. The resulting canonical representation is animated efficiently using linear blend skinning and Gaussian rasterization without further network inference. Trained entirely from scratch on only 10K frames from THuman2.1, HumanGS achieves reconstruction quality comparable to or better than state-of-the-art methods while reducing reconstruction time by over 15 times compared to recent transformer-based approaches, enabling real-time animation and interactive applications. Code and pre-trained models are available at https://github.com/Devdoot57/HumanGS .
Figures & tables
Figure 1: Overview of the HumanGS architecture and explicit inference pipeline. A Transformer Encoder processes sparse input images and pose maps to extract global 2D feature maps. Canonical SMPL-X [ 38 ] vertices project onto these maps to sample local geometry-aware features. A Vertex Query Decoder predicts 3D Gaussian attributes in the canonical T-pose space. During inference (green dashed box), these Gaussians are animated via Linear Blend Skinning (LBS) with target pose transforms and rasterized, bypassing neural network evaluation.
Figure 2: Feature Aggregation. Tokens from four uniform transformer depths are reshaped into 2D maps, projected via parallel 1×1 convolutions, concatenated, and fused. Progressive upsampling quadruples the resolution to produce a dense feature map for vertex-aligned sampling.
Figure 3: Qualitative Generalization on AvatarReX. We visualize the animation capabilities on unseen subjects. Left: Input reference view ( ∗ denotes the single input for LHM; other methods use 4 views). Right: Re-enactment results driven by a novel motion sequence.
Figure 4: Qualitative comparison on THuman2.1 For each subject, the left block shows the sparse input views. The right block displays novel view synthesis results from different methods.
TH2.1 → AvRex
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
Modeling (s) ↓
Synthesis (s) ↓
LHM*
23.72
0.95
0.045
4.59
0.053
HumanRAM
24.58
0.96
0.042
-
0.287
Animatable Gaussians
25.51
0.96
0.041
12000
0.245
HumanGS
25.24
0.96
0.04
0.29
0.048
Table 1: Quantitative evaluation on TH2.1 → AvRex. HumanGS demonstrates robust cross-dataset generalization, matching the rendering fidelity of optimization-based methods while drastically reducing computational time. (*Note: LHM was evaluated using its official pre-trained model, which leverages a single-image input and was trained on a large-scale dataset of ∼ 300k videos.)
Table 6
Figure 5: Ablation of Intermediate Feature Aggregation. Left: Distribution of localized LPIPS differences on 64×64 foreground patches (positive difference indicates superior quality). Right: Qualitative comparisons from the distribution’s high-performance tail. Intermediate features successfully recover high-frequency textures blurred by the baseline.
Figure S1 : Generalization Performance Comparison Across Varying View Counts on AvatarReX (Animation). We evaluate PSNR, SSIM, and LPIPS under 1, 2, 4, and 6 input views with varying poses on unseen subjects from the AvatarReX dataset. HumanGS consistently outperforms feed-forward baselines HumanRAM and GST. While the optimization-based Animatable Gaussians achieves comparable metrics given sufficient data (4 to 6 views), it suffers a catastrophic drop in performance under extreme sparsity (1 to 2 views), whereas our method remains robust. Note that the single-image baseline GST is reported only at N=1 .
Methods
1 View
2 Views
4 Views
6 Views
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
HumanRAM
25.44
0.946
0.052
27.39
0.950
0.044
29.10
0.960
0.039
29.90
0.966
0.036
LVSM
17.78
0.900
0.101
22.64
0.930
0.066
25.96
0.950
0.055
27.03
0.950
0.051
GHG
21.71
0.920
0.094
22.43
0.930
0.089
22.69
0.930
0.088
22.72
0.930
0.088
HumanGS (Ours)
24.07
0.936
0.068
25.71
0.946
0.059
27.19
0.950
0.054
27.84
0.950
0.051
Table S1 : Reconstruction Performance Across Varying View Counts. Quantitative comparison across varying numbers of input views ( N∈{1,2,4,6} ) on the modified THuman 2.1 test set, where subjects are positioned closer to the cameras to evaluate fine-grained detail synthesis.
Figure S2 : Effectiveness of the Global Token Under Varying Input Sparsity. We evaluate reconstruction performance with and without the Global Token across 1 to 8 input views. By encoding holistic subject context, the token provides a crucial fallback for severely occluded regions in extreme sparse-view scenarios ( N∈{1,2} ), yielding significant improvements. As spatial coverage increases ( N≥4 ) and occlusions naturally decrease, the performance margins converge.
Dataset
1 Gaussian per Vertex
1 Tight + 4 Free Gaussians (Ours)
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
THuman 2.1
30.66
0.970
0.028
30.81
0.980
0.024
THuman 2.1 (Near)
27.09
0.950
0.062
27.19
0.950
0.054
AvatarReX
25.20
0.960
0.044
25.24
0.960
0.040
THuman 4.0
25.42
0.960
0.034
25.53
0.960
0.029
Table S2: Ablation on Gaussian Primitives per Vertex. Comparing reconstruction and animation performance using a single Gaussian versus our proposed hybrid representation (1 tight + 4 free Gaussians). The hybrid approach consistently improves perceptual quality (lower LPIPS) across all datasets by better capturing loose clothing and structures that deviate from the parametric mesh.
Figure S3 : Qualitative comparison on THuman2.1, AvatarReX, and THuman4.0. For each subject, the left block shows the sparse input views. The right block displays novel view synthesis results from different methods. Our method achieves comparable visual quality to HumanRAM and superior visual quality to other methods while enabling real-time animation capabilities. The first four rows are from THuman2.1 and the last two rows are from AvatarReX and THuman4.0 respectively.
Figure S4 : Qualitative Generalization on AvatarReX. Left: Input reference view. Right: Re-enactment results driven by a novel motion sequence.
Figure S5 : Novel Pose Synthesis Results. We demonstrate the explicit animation capabilities of HumanGS using subjects from THuman2.1 driven by motion sequences from THuman4.0. Left: The sparse set of 4 input views used to predict the canonical asset. Right: The predicted asset re-posed into various unseen configurations using Linear Blend Skinning and rasterization. This confirms that our method learns a consistent, animatable 3D representation from sparse supervision.
Live free-viewpoint visualization of real humans is critical for immersive communication and interactive digital experiences. Existing methods either rely on computationally expensive optimization or require calibrated cameras and low-resolution inputs, making real-time high-resolution deployment impractical. In this work, we present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization from sparse, unposed RGB streams. Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance in a single forward pass. To achieve this, we start by designing a lightweight sparsity-aware multi-view transformer backbone that tokenizes foreground human regions while preserving global context through a shared scene token. We then employ a fully transformer-based Gaussian decoder to mitigate convolution-induced over-smoothing while keeping decoding sparse and efficient. In addition, we introduce a hybrid feature pyramid that injects multi-scale appearance cues into geometry prediction. We further introduce a lightweight differentiable Levenberg-Marquardt camera refinement layer to enhance multi-view consistency and geometric alignment. Moreover, to stabilize learning under sparse, unposed inputs, we transfer multi-view geometry priors from a large visual-geometry foundation model via teacher-student distillation. Finally, the predicted Gaussian maps are streamed with video codecs to remote devices for interactive free-viewpoint rendering. Extensive experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU. Additional captured sequences illustrate its performance across varied subjects, clothing, and motions under our multi-camera setup.
Hanzhang Tu, Zhanfeng Liao, Wei Min +2
Tsinghua University, Beijing, China · Shadow AI, Beijing, China
Producing 3D human representations from input views on the fly is essential for immersive live streaming systems, where representation compactness is as critical as high fidelity given limited computational power and transmission bandwidth. Although recent feed-forward reconstruction methods achieve impressive quality through the view-centric prediction of 3D representations, they repeatedly encode the same subject content across multiple views, leading to significant inter-view redundancy. Our key insight is to perform predictions directly in 3D space, enabling the network to learn and produce a highly compact representation. To this end, we propose PointSplat, a novel human-centric approach that directly infers Gaussian primitives from an input point set. The proposed method first estimates a coarse geometric proxy and performs ray casting to prune redundant points and establish explicit 2D--3D correspondences. Subsequently, it employs a Point-Image Transformer to fuse appearance and geometry features, predicting Gaussian attributes in a single forward pass. This design restricts predictions to foreground regions of interest, substantially reducing the total number of Gaussians while improving novel-view rendering quality. Extensive experiments demonstrate that PointSplat achieves higher efficiency and quality while exhibiting strong robustness to variations in view count and image resolution across multiple datasets.
Yujie Guo, Yudong Jin, Lingteng Qiu +7
State Key Lab of CAD&CG, Zhejiang University, Hangzhou, China · The Chinese University of Hong Kong, Shenzhen, China · ByteDance, Beijing, China
Accurately recovering human pose and appearance from video is an essential component of scene reconstruction, with applications to motion capture, motion prediction, virtual reality, and digital twinning. Despite significant interest in building realistic human avatars from video, this paper demonstrates that existing methods do not accurately recover the 3D geometry of humans. ViT-based approaches are not consistently reliable and can overfit to 2D views, while NeRF- and Gaussian Splatting-based avatars treat pose and appearance separately, limiting rendering generalization to new poses. To resolve these shortcomings, this paper proposes HumanSplatHMR, a joint optimization framework that refines 3D human poses while simultaneously learning a high-fidelity avatar for novel-view and novel-pose synthesis. Our key insight is to close the loop between geometric pose estimation and differentiable rendering. Unlike prior human avatar methods that rely on accurate human pose obtained through motion capture systems or offline refinement, which are impractical in in-the-wild scenarios, our approach uses only human mesh estimates from a state-of-the-art human pose estimator to better reflect real-world conditions. Therefore, instead of using the human pose only as a deformation prior, HumanSplatHMR backpropagates photometric, segmentation, and depth losses through a differentiable renderer to the pose parameters and global position. This coupling refines the global 3D pose over time, improving accuracy and alignment while producing better renderings from novel views. Experiments show consistent improvements over pose recovery baselines that omit image-level refinement and avatar baselines that decouple pose estimation from avatar reconstruction.