Autoregressive Appearance Prediction for 3D Gaussian Avatars

Michael Steiner; Zhang Chen; Alexander Richard; Vasu Agrawal; Markus Steinberger; Michael Zollhöfer

cs.CVApr 1, 2026

Autoregressive Appearance Prediction for 3D Gaussian Avatars

Authors: Michael Steiner, Zhang Chen, Alexander Richard, Vasu Agrawal, Markus Steinberger, Michael Zollhöfer

Organizations: Graz University of Technology, Austria · Meta Reality Labs, Pittsburgh, USA

Abstract

A photorealistic and immersive human avatar experience demands capturing fine, person-specific details such as cloth and hair dynamics, subtle facial expressions, and characteristic motion patterns. Achieving this requires large, high-quality datasets, which often introduce ambiguities and spurious correlations when very similar poses correspond to different appearances. Models that fit these details during training can overfit and produce unstable, abrupt appearance changes for novel poses. We propose a 3D Gaussian Splatting avatar model with a spatial MLP backbone that is conditioned on both pose and an appearance latent. The latent is learned during training by an encoder, yielding a compact representation that improves reconstruction quality and helps disambiguate pose-driven renderings. At driving time, our predictor autoregressively infers the latent, producing temporally smooth appearance evolution and improved stability. Overall, our method delivers a robust and practical path to high-fidelity, stable avatar driving.

Figures & tables

Figure 1: We represent the posed avatar with a hierarchical 3D Gaussian structure controlled by per-anchor spatial MLPs. Each anchor receives localized driving features via skinning-weight-based masking of pose parameters

\bm{\mathbf{\theta}}

(and face-region masking of

\bm{\mathbf{\phi}}

), together with an appearance latent

\bm{\mathbf{l}}_{i}

. During training,

\bm{\mathbf{l}}_{i}

is obtained by encoding a per-frame UV texture into a 2D feature map and sampling it at the anchor’s UV coordinates; at test time, a transformer autoregressively predicts temporally smooth latents from a short pose history for stable driving. The encoder reconstructs training poses extremely well (a) and generalizes to novel poses and unseen textures (b), while our appearance predictor yields realistic appearances and smooth transitions on test sequences when textures are not available (c).

Panel of Figure 2 — Figure 2: Following Zhan et al. [ 46 ] , we initialize a hierarchical point cloud on the template mesh, consisting of anchors/control points/Gaussians (300/10k/200k for our example, colored as orange/blue/red). Every anchor holds an MLP, whose outputs are interpolated by control points and Gaussians from the closest three anchors to calculate their positional displacement and Gaussian correctives.

Panel of Figure 3 — Figure 3: (a) Providing pose parameter to every anchor leads to spurious correlations between unrelated regions. (b) We locally mask out pose parameters per anchor based on the skinning weights of the template mesh to restrict them to their local region.

Panel of Figure 4 — Figure 4: (a) Similar poses can exhibit vastly different appearances, leading to an ambiguous one-to-many mapping. Therefore, we model appearance through a texture encoder that receives (b) a multi-view projected UV texture as input (face region masked).

Figure 5: Architecture of our appearance predictor. The transformer predicts the next latents from the previous $N_{b}$ poses—value, plus their velocity and acceleration calculated via finite differences—and the previous latents.

	Test			Train
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	#Params
Ours	32.99	0.939	0.063	36.21	0.950	0.053	192.1M
MMLPs $\dagger$	32.34	0.937	0.066	36.13	0.950	0.054	187.2M
nRFGCA $\dagger$	32.74	0.939	0.065	35.94	0.950	0.055	155.9M
Ours (re-init. 30)	33.85	0.942	0.060	–	–	–	192.1M
Ours (encoder)	34.78	0.946	0.056	–	–	–	189.8M

Table 1: Image metrics on all compared methods. As appearance is inherently ambiguous and drifts over time, we initialize the appearance predictor for Ours with the encoder output. Additionally, we show results for re-initializing every 30 frames, and for using the encoder during driving.

	Test			Train
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Ours	32.99	0.939	0.063	36.21	0.950	0.053
Ours w/o local pca	32.86	0.938	0.064	36.23	0.950	0.053
Ours w/o local pose $\downarrow$	32.87	0.938	0.064	36.29	0.951	0.052

Table 2: Ablation of our components.

Panel of Figure 6 — Figure 6: Qualitative comparison of global (MMLPs $\dagger$ ) versus localized (Ours) pose conditioning. We manually perturb a single pose parameter (top: neck twist; bottom: wrist rotate) between two frames: global conditioning induces large, non-local deformations and spurious appearance changes (visualized as $\ell_{1}$ error in the cutouts), whereas localized conditioning confines the effect to local, physically plausible motion.

Panel of Figure 7 — Figure 7: Two temporally adjacent frames from a test sequence and the corresponding cutout $\ell_{1}$ error against the previous frame. Our method remains temporally stable due to explicit pose–appearance disentanglement and the learned appearance predictor, while nRFGCA † and MMLPs † exhibit abrupt, implausible appearance changes.

Panel of Figure 8 — Figure 8: Test-sequence comparison of our method using the texture encoder (unseen texture input) versus our appearance predictor (no textures), alongside MMLPs † . The encoder provides an upper bound closest to the ground truth, while the predictor yields temporally smooth, plausible appearance evolution without texture inputs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Our dataset consists of six captures of varying difficulty.

Panel of Figure 10 — Figure 10: Multiple image pairs showing the generalization capabilities of our texture encoder on the test set, i.e . , novel poses and unseen UV textures. The left image shows ground truth and the right image is the appearance encoder result.

Panel of Figure 11 — Figure 11: Test-sequence comparison of our method using the texture encoder (unseen texture input) versus our appearance predictor (no textures), alongside MMLPs † .

PSNR ↑	Actor 1	Actor 2	Actor 3	Actor 4	Actor 5	Actor 6	Mean
Ours	31.52	31.90	28.82	35.03	34.38	36.31	32.99
MMLPs †	31.29	31.79	28.41	33.68	34.31	34.55	32.34
nRFGCA †	31.47	31.94	28.95	34.84	34.29	34.97	32.74
Ours (re-init. 30)	32.39	32.93	30.04	35.80	35.02	36.92	33.85
Ours (encoder)	33.33	33.85	31.48	36.71	35.63	37.67	34.78
SSIM ↑	Actor 1	Actor 2	Actor 3	Actor 4	Actor 5	Actor 6	Mean

Appendix

Table 3: Test set image metrics on all compared methods. We initialize the appearance predictor for Ours with the encoder output and show additional results for re-initializing every 30 frames, and for using the texture encoder on the test sequences.

PSNR ↑	Actor 1	Actor 2	Actor 3	Actor 4	Actor 5	Actor 6	Mean
Ours	34.97	35.40	34.58	37.14	36.52	38.63	36.21
MMLPs †	35.00	35.36	34.22	37.07	36.51	38.62	36.13
nRFGCA †	34.69	35.58	33.42	36.84	36.53	38.56	35.94
SSIM ↑	Actor 1	Actor 2	Actor 3	Actor 4	Actor 5	Actor 6	Mean
Ours	0.939	0.956	0.949	0.956	0.934	0.969	0.950
MMLPs †	0.939	0.956	0.947	0.955	0.934	0.969	0.950

Appendix

Table 4: Training set image metrics on all compared methods.

3D Avatar Generation

3D avatar generation creates realistic, controllable digital humans from images, video, text, or other inputs.

See all 90 papers

AvatarPointillist: AutoRegressive 4D Gaussian Avatarization
Apr 6, 2026Hongyu Liu, Xuan Wang, Zijian Wu +73D Avatar Generation Human Avatar Animation
HKUST · Ant Group · City University of Hong Kong
AGORA: Adversarial Generation Of Real-time Animatable 3D Gaussian Head Avatars
Dec 6, 2025Ramazan Fazylov, Sergey Zagoruyko, Aleksandr Parkin +23D Gaussian Splatting 3D Asset Generation
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Polynome AI, Dubai, UAE · MTS AI, Moscow, Russia
Splatshot: 3D Face Avatar Generation from a Single Unconstrained Photo
May 31, 2026Hao Liang, Zhixuan Ge, Soumendu Majee +33D Gaussian Splatting 3D Asset Generation
Rice University Houston, TX 77005 · Samsung Research America Plano, TX 75023

Apr 6, 2026J/K move · Enter open · S save

AvatarPointillist: AutoRegressive 4D Gaussian Avatarization

Hongyu Liu, Xuan Wang, Zijian Wu, Yating Wang+6

HKUST · Ant Group · City University of Hong Kong

3D Avatar Generation Human Avatar Animation 3D Gaussian Splatting Point Cloud Learning

Open paper AskPDF arXiv

We introduce AvatarPointillist, a novel framework for generating dynamic 4D Gaussian avatars from a single portrait image. At the core of our method is a decoder-only Transformer that autoregressively generates a point cloud for 3D Gaussian Splatting. This sequential approach allows for precise, adaptive construction, dynamically adjusting point density and the total number of points based on the subject's complexity. During point generation, the AR model also jointly predicts per-point binding information, enabling realistic animation. After generation, a dedicated Gaussian decoder converts the points into complete, renderable Gaussian attributes. We demonstrate that conditioning the decoder on the latent features from the AR generator enables effective interaction between stages and markedly improves fidelity. Extensive experiments validate that AvatarPointillist produces high-quality, photorealistic, and controllable avatars. We believe this autoregressive formulation represents a new paradigm for avatar generation, and we will release our code inspire future research.

Abstract

3D Avatar Generation

Explore similar work

AvatarPointillist: AutoRegressive 4D Gaussian Avatarization

AGORA: Adversarial Generation Of Real-time Animatable 3D Gaussian Head Avatars

Splatshot: 3D Face Avatar Generation from a Single Unconstrained Photo