UNICA: A Unified Neural Framework for Controllable 3D Avatars

Jiahe Zhu; Xinyao Wang; Yiyu Zhuang; Yanwen Wang; Jing Tian; Yao Yao; Hao Zhu

cs.CVApr 3, 2026

UNICA: A Unified Neural Framework for Controllable 3D Avatars

Authors: Jiahe Zhu, Xinyao Wang, Yiyu Zhuang, Yanwen Wang, Jing Tian, Yao Yao, Hao Zhu

Organizations: Nanjing University, Nanjing, China State Key Laboratory of Novel Software Technology, China

Abstract

Controllable 3D human avatars have found widespread applications in 3D games, the metaverse, and AR/VR scenarios. The conventional approach to creating such a 3D avatar requires a lengthy, intricate pipeline encompassing appearance modeling, motion planning, rigging, and physical simulation. In this paper, we introduce UNICA (UNIfied neural Controllable Avatar), a skeleton-free generative model that unifies all avatar control components into a single neural framework. Given keyboard inputs akin to video game controls, UNICA generates the next frame of a 3D avatar's geometry through an action-conditioned diffusion model operating on 2D position maps. A point transformer then maps the resulting geometry to 3D Gaussian Splatting for high-fidelity free-view rendering. Our approach naturally captures hair and loose clothing dynamics without manually designed physical simulation, and supports extra-long autoregressive generation. To the best of our knowledge, UNICA is the first model to unify the workflow of "motion planning, rigging, physical simulation, and rendering". Code is released at https://github.com/zjh21/UNICA.

Figures & tables

Figure 1: UNICA is a unified model that generates action-controlled, $360^{\circ}$ -renderable 3D avatars with dynamics. For the first time, UNICA unifies a workflow of " motion planning, rigging, physical simulation, and rendering " within a single model.

$\sqrt{}$ $=$ supported; $\times$ $=$ NOT supported; $\sqrt{}\mkern-9.0mu{\smallsetminus}$ $=$ not fully supported/might be extended.
Method	$360^{\circ}$ Render	Key-Control	Arbitrary Len.	Physics
Animatable 3D Avatar	$\sqrt{}$	$\times$	$\sqrt{}$	$\sqrt{}\mkern-9.0mu{\smallsetminus}$
Motion Generation	$\sqrt{}$	$\sqrt{}\mkern-9.0mu{\smallsetminus}$	$\sqrt{}\mkern-9.0mu{\smallsetminus}$	$\times$
World Model	$\times$	$\sqrt{}$	$\sqrt{}$	$\sqrt{}$
4D Generation	$\sqrt{}$	$\times$	$\sqrt{}\mkern-9.0mu{\smallsetminus}$	$\sqrt{}$
UNICA (Ours)	$\sqrt{}$	$\sqrt{}$	$\sqrt{}$	$\sqrt{}$

Table 1: Comparison between UNICA and prior approaches.

Figure 2: The pipeline of UNICA. UNICA consists of an action-conditioned multi-frame diffusion model for avatar geometry and a point transformer for point-to-3DGS appearance mapping. The diffusion model takes latents of three position maps as context and generates one subsequent position map conditioned on a chosen action embedding. The generated position map is upscaled using bilinear interpolation and combined with other base 3DGS attributes to form a coarse 3DGS. The point transformer then refines the coarse 3DGS into one with avatar-specific appearance details.

Figure 3: The position map rendering process and visualization of a four-frame group. (a) We use an A-Pose mesh of the avatar as geometry and the vertex coordinates of the posed avatar as vertex colors to render position maps. The position maps are rendered from six orthogonal views. (b) We partition the motion sequence into groups of four frames and normalize each group.

Figure 4: Demonstration of progressive 4D inference. During autoregressive inference of UNICA, each round generates a relative movement that is accumulated in 3D space for the actual movement of the 3D avatar. The output frame of round

n

will be renormalized before it is used as input for round

n+1

Figure 5: Animation results of UNICA demonstrating avatar response to key presses. For visualization clarity, we sample one frame every three frames along the trajectory.

Figure 6: FVD scores over a 2000-frame autoregressively generated sequence, computed using a sliding window of 200 frames with a stride of 10. The red dashed line indicates the linear trend, and the gray shaded region denotes $\pm$ 1 standard deviation. The near-flat trend demonstrates stable generation quality over extended rollouts.

Figure 7: Qualitative comparison between UNICA and baseline methods across different frames and viewpoints.

Method	FVD $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	VQS $\uparrow$	PRS $\uparrow$
Ani. Gaussians [ 34 ]	199.93	20.31	0.119	3.47	3.64
Mixamo [ 1 ]	320.06	14.54	0.187	3.39	2.64
UNICA (Ours)	194.87	23.95	0.106	4.36	4.24

Table 2: Quantitative comparison and user study results averaged across five avatars. User study ratings are on a 1–5 scale. Best results in bold . VQS

=

Visual quality score in user study; PRS

=

Physical realism score in user study.

Figure 8: Comparison on the number of context frames. Four consecutive frames are shown for each method.

	FVD $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
w/o PCA	228.31	23.68	0.114
w/ PCA	194.87	23.95	0.106

Table 3: Comparison of results with and without PCA alignment.

Figure 9: Ablation study of position map alignment via PCA reconstruction.

Panel of Table 13 — Table A : Number of frames in each transition sequence from one key press to another. Each entry corresponds to a sequence in which the avatar transitions from the start key (column) to the end key (row).

Figure A : Overview of the five avatars exported from UE. Each avatar exhibits characteristic non-rigid or non-body elements ( e.g . , loose garments, braids, and carried accessories) that go beyond a standard body mesh.

Figure B : Reconstruction error as a function of the number of principal components.

Figure C : Visualization of PCA reconstruction with varying numbers of principal components.

Figure D : Comparison between the feed-forward pipeline and the diffusion pipeline. PCA reconstruction is disabled to show raw model outputs.

Figure E : Comparison of avatar appearance trained under different resolution configurations. (a)

128\times 128

position maps upscaled to

512\times 512

. (b)

128\times 128

position maps upscaled to

1024\times 1024

. (c)

256\times 256

position maps upscaled to

512\times 512

. (d)

256\times 256

position maps upscaled to

1024\times 1024

. All upscaling is performed via foreground-aware bilinear interpolation.

	1 step	2 steps	4 steps	10 steps	20 steps
FVD $\downarrow$	402.88	404.58	248.32	194.87	196.05
PSNR $\uparrow$	19.26	23.02	23.48	23.95	23.90
LPIPS $\downarrow$	0.171	0.118	0.113	0.106	0.109

Table B : Model performance with respect to the number of DDIM steps. Best results are highlighted in bold .

Figure F : Additional qualitative comparisons between UNICA and baseline methods across different frames and viewpoints.

Case	Metric	Ani. Gaussians [ 34 ]	Mixamo [ 1 ]	UNICA (Ours)
Monk	FVD $\downarrow$	136.83	145.97	156.92
	PSNR $\uparrow$	22.45 $\pm$ 3.23	16.63 $\pm$ 1.27	23.21 $\pm$ 2.16
	LPIPS $\downarrow$	0.104 $\pm$ 0.038	0.146 $\pm$ 0.030	0.105 $\pm$ 0.031
Cowgirl	FVD $\downarrow$	186.93	174.64	210.37
	PSNR $\uparrow$	19.17 $\pm$ 3.37	17.14 $\pm$ 1.62	24.48 $\pm$ 2.74
	LPIPS $\downarrow$	0.119 $\pm$ 0.038	0.132 $\pm$ 0.033	0.091 $\pm$ 0.031

Table C : Quantitative comparison across five avatars. Best results are highlighted in bold . PSNR and LPIPS are reported as mean

\pm

standard deviation.

Figure G : Per-avatar user study scores on visual quality and physical realism.

Inference Time (s)				Frames of Turning
Geometry	Appearance	PCA	Others	$180^{\circ}$	$90^{\circ}$
0.414	0.250	$7.23\times 10^{-4}$	$1.85\times 10^{-2}$	13.77	9.84

Table D : Per-frame inference time breakdown and average number of frames required to complete turning events during interactive control.

Figure H : 3DGS artifacts observed in turning poses.

Figure I : Paired key-rendering results of avatar sprint start. Starting from a standing pose, the avatar starts to run forward under the key [ W ]. The sequence is arranged from left to right, top to bottom.

Figure J : Paired key-rendering results of avatar

180^{\circ}

turning, where the key changes from [ W ] to [ S ]. The sequence is arranged from left to right, top to bottom.

Figure K : Paired key-rendering results of avatar

90^{\circ}

turning, where the key changes from [ S ] to [ A ]. The sequence is arranged from left to right, top to bottom.

Figure L : Paired key-rendering results of avatar stopping, where the key changes from [ A ] to [

\emptyset

]. The sequence is arranged from left to right, top to bottom.

Explore similar work

AGORA: Adversarial Generation Of Real-time Animatable 3D Gaussian Head Avatars
Dec 6, 2025Ramazan Fazylov, Sergey Zagoruyko, Aleksandr Parkin +23D Gaussian Splatting 3D Asset Generation
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Polynome AI, Dubai, UAE · MTS AI, Moscow, Russia
LUNA: Learning Universal 3D Human Animation Beyond Skinning
Jun 30, 2026Peng Li, Rawal Khirodkar, Junxuan Li +6Human Motion Generation Human Avatar Animation
The Hong Kong University of Science and Technology · Work was done during an internship at Meta. · Codec Avatars Lab, Meta
HumanNOVA: Photorealistic, Universal and Rapid 3D Human Avatar Modeling from a Single Image
Jun 1, 2026Hezhen Hu, Wangbo Zhao, Lanqing Guo +63D Avatar Generation 3D Shape Generation
University of Texas at Austin · National University of Singapore · Texas A&M University

Dec 6, 2025J/K move · Enter open · S save

AGORA: Adversarial Generation Of Real-time Animatable 3D Gaussian Head Avatars

Ramazan Fazylov, Sergey Zagoruyko, Aleksandr Parkin, Stamatis Lefkimmiatis+1

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Polynome AI, Dubai, UAE · MTS AI, Moscow, Russia

3D Gaussian Splatting 3D Asset Generation 3D Avatar Generation Human Avatar Animation

Open paper AskPDF arXiv

The generation of high-fidelity, animatable 3D human avatars remains a core challenge in computer graphics and vision, with applications in VR, telepresence, and entertainment. Existing approaches based on implicit representations like NeRFs suffer from slow rendering and dynamic inconsistencies, while 3D Gaussian Splatting (3DGS) methods are typically limited to static head generation, lacking dynamic control. We bridge this gap by introducing AGORA, a novel framework that extends 3DGS within a generative adversarial network to produce animatable avatars. Our key contribution is a lightweight, FLAME-conditioned deformation branch that predicts per-Gaussian residuals, enabling identity-preserving, fine-grained expression control while allowing real-time inference. Identity is further preserved through spatial shape conditioning of the identity branch, and expression fidelity is enforced via a dual-discriminator training scheme leveraging synthetic renderings of the parametric mesh. AGORA generates avatars that are not only visually realistic but also precisely controllable. Quantitatively, we outperform state-of-the-art NeRF-based methods on expression accuracy while rendering at 250 FPS on a single GPU and, notably, at

\sim

9 FPS under CPU-only inference -- to our knowledge the first demonstration of CPU-only animatable 3DGS avatar synthesis. This work represents a significant step toward practical, high-performance digital humans. Project website: https://ramazan793.github.io/AGORA/

Abstract

Explore similar work

AGORA: Adversarial Generation Of Real-time Animatable 3D Gaussian Head Avatars

LUNA: Learning Universal 3D Human Animation Beyond Skinning

HumanNOVA: Photorealistic, Universal and Rapid 3D Human Avatar Modeling from a Single Image