AVTR-1: Open Stack for Real-Time Interactive Avatars

Artem Kravtsov; Dmitrii Ziganshin; Vsevolod Poletaev; Gleb Balitskiy; Anastasia Tikhonova; Egor Burkov; Vadim Lebedev

cs.CVSep 19, 2026

AVTR-1: Open Stack for Real-Time Interactive Avatars

Authors: Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev, Gleb Balitskiy, Anastasia Tikhonova, Egor Burkov, Vadim Lebedev

Organizations: Avaturn Live

Abstract

Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an open stack for real-time interactive avatar conversations, built around a compact 153M-parameter autoregressive flow-matching motion generator conditioned on both participants' audio. We adapt its audio encoder for streaming through self-distillation. The stack turns the model's chunk-based generation into a continuous, synchronized audio-video stream driven by an external voice agent, and we analytically derive its contribution to the user-facing latencies and validate the resulting bounds with two commercial voice agents. Further experiments demonstrate that AVTR-1 leads the compared dyadic systems on all reported visual-quality metrics and most conventional listening-motion metrics while remaining competitive in lip synchronization. Its inference runtime operates in real time on data-center and consumer GPUs. However, conventional listening metrics do not establish whether the paired speaker's speech contributes to generated motion. We therefore introduce the Reference-Based Directed Granger Gain (R-DGG), which measures the additional predictive information carried by speaker speech after accounting for listener history and speaker motion. R-DGG finds statistically supported predictive dependence for recorded listeners and all evaluated dyadic systems, but not for talking-head generators without paired audio or mismatched speaker-listener pairs. We release the model weights, renderer, and serving backend under component-specific licenses.

Figures & tables

Property	Value
Parameters	153M
Decoder layers	18
Hidden dimension	512
Attention heads / head dimension	8 / 64
Feed-forward dimension	512
Audio condition encoder layers	2 per stream

Table 1: Configuration of the released AVTR-1 motion model.

GPU	TTFF	Request duration	Real-time factor
NVIDIA L40S	42 ms	71 ms	$2.81\times$
NVIDIA A100	54 ms	91 ms	$2.20\times$
NVIDIA L4	—	202 ms	$0.99\times$
NVIDIA RTX 4060 Ti	99 ms	162 ms	$1.24\times$
NVIDIA RTX 3070	94 ms	180 ms	$1.11\times$
NVIDIA RTX 3060 Ti	—	207 ms	$0.97\times$

Table 2: Per-chunk latency, real-time throughput, and time to first frame (TTFF) across GPUs, measured offline on one five-frame (200 ms) chunk.

Term	RLC	ILC	RLC	ILC	RLC	ILC
	OpenAI Realtime API ( $n=10$ )		Cartesia Line ( $n=10$ )		Predicted
$T_{\mathrm{wait}}$	$86.3$ $[10,187]$	$58.8$ $[6,105]$	$110.3$ $[19,193]$	$89.1$ $[41,169]$	$[0,200)$	$[0,200)$
$T_{\mathrm{pad}}$	$5.0$	$5.0$	$45.0$ $[5,125]$	$5.0$	$[5,205)$	$5$
$T_{\mathrm{present}}$	$200.0$	$200.0$	$200.0$	$200.0$	$200$	$200$
$T_{\mathrm{lead}}$	$99.1$ $[98,100]$	$98.9$ $[98,100]$	$98.9$ $[97,100]$	$99.2$ $[99,100]$	$100$	$100$
Total	$390.4$ $[315,491]$	$362.7$ $[310,409]$	$454.2$ $[426,497]$	$393.3$ $[345,473]$	$[305,705)$	$[305,505)$

Table 3: Measured streamer latency for both voice agents, in milliseconds. Values are the mean followed by

[\min,\max]

, or a single value where constant. The predicted columns are the bounds derived above.

Stage	ms
User $\rightarrow$ streamer (inbound)	114
Microphone capture	40
Opus encoding	21
Network	12
Jitter buffer	40
Opus decoding	1

Table 4: Reference end-to-end response latency budget for an avatar deployment, in milliseconds. The per-stage client and voice-agent numbers are taken from the published voice-to-voice budget of [ 29 ] and grouped into parts, and the bold part totals sum to the end-to-end figure. The two streamer

\leftrightarrow

voice-agent network hops and the streamer RLC are extras introduced by our system: the RLC is the measured mean for Cartesia Line (Table 3 ), and the hops assume the streamer and the voice agent share a physical cloud region.

Table 5: Quantitative comparison with competing methods on the SI-184 subset. Gray marks the best value among the dyadic systems in both panels. For listening metrics without an arrow, values closer to ground truth are better. The pose components of PFD and variance are reported in units of

10^{-2}

. AvatarForcing* denotes the evaluation configuration described in the protocol.

	Paired audio	rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method		Exp	Pose	Exp	Pose	Exp	Pose	Exp	Pose
AvatarForcing*	Yes	0.109	0.163	37.51	7.560	4.749	3.848	1.213	2.099
AVTR-1	Yes	0.083	0.140	25.98	6.417	4.970	3.254	1.313	0.930
DyStream	Yes	0.128	0.141	40.87	7.598	4.497	3.606	1.164	1.731
Ditto	No	0.092	0.168	35.02	6.790	5.006	3.253	1.604	1.386
FLOAT	No	0.117	0.132	34.09	4.753	4.322	2.961	1.007	0.843

Table 6: Comparison using conventional motion-based listening metrics on the VAD-selected listener-silent frames of the SI-184 subset. This extends Table 5(b) with the talking-head generators. For metrics without an arrow, values closer to ground truth are better. The pose components of PFD and variance are reported in units of

10^{-2}

. Dark and light gray mark the best and second-best methods, respectively.

	Paired audio	R-DGG	Bootstrap percentile			$p$
Method			2.5%	50%	97.5%
AvatarForcing*	Yes	0.32	0.07	0.32	0.56	0.008
AVTR-1	Yes	0.57	0.19	0.57	0.98	0.002
DyStream	Yes	0.48	0.06	0.49	0.93	0.011
Ditto	No	0.06	-0.20	0.06	0.31	0.335
FLOAT	No	0.01	-0.33	0.00	0.42	0.491

Table 7: Comparison using R-DGG on the SI-184 subset. Values are reported in units of

10^{-4}

nats. The final column reports

p=\widehat{\Pr}(G_{a\mid m}\leq 0)

. Gray marks estimates whose intervals exclude zero.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Paired audio	rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method		Exp	Rot	Exp	Rot	Exp	Rot	Exp	Rot
AvatarForcing*	Yes	0.138	0.175	1.242	5.784	4.802	3.813	5.200	3.352
AVTR-1	Yes	0.112	0.169	0.671	3.967	4.760	3.152	3.300	0.920
DyStream	Yes	0.151	0.178	1.335	4.392	4.421	3.436	4.300	1.239
Ditto	No	0.109	0.174	0.882	4.106	4.930	3.010	3.800	1.002
FLOAT	No	0.127	0.152	0.878	3.240	4.488	3.061	4.300	0.852

Appendix

Table 8: Comparison using conventional motion-based listening metrics computed from LivePortrait features on the VAD-masked listening portion of the SI-184 subset. For metrics without an arrow, values closer to ground truth are better. Expression PFD and variance are reported in units of

10^{-3}

and

10^{-5}

, respectively. Their rotation components are reported in units of

10^{-2}

. Dark and light gray mark the best and second-best methods, respectively.

		rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method	Paired audio	Exp	Pose	Exp	Pose	Exp	Pose	Exp	Pose
SoulX Lite	No	0.061	0.148	19.94	5.988	5.065	3.419	1.411	0.975
SoulX Pro	No	0.056	0.125	18.75	4.751	5.103	3.499	1.441	0.982
Ditto	No	0.074	0.180	28.82	6.645	4.953	3.459	1.671	1.341
FLOAT	No	0.077	0.128	23.71	3.953	4.816	3.220	1.206	0.811
AvatarForcing*	Yes	0.080	0.145	27.78	5.891	4.887	3.917	1.337	1.800

Appendix

Table 9: Conventional motion-based metrics computed from EMOCA features on all frames of the SI-184 subset. For metrics without an arrow, values closer to ground truth are better. The pose components of PFD and variance are reported in units of

10^{-2}

. Gray marks the best value among dyadic systems.

		rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method	Paired audio	Exp	Rot	Exp	Rot	Exp	Rot	Exp	Rot
SoulX Lite	No	0.092	0.179	0.554	3.867	5.021	3.133	3.400	0.854
SoulX Pro	No	0.091	0.154	0.521	3.089	5.070	3.309	3.500	1.044
Ditto	No	0.123	0.177	0.836	4.300	5.012	3.027	3.700	1.004
FLOAT	No	0.129	0.144	0.686	2.764	4.830	3.252	3.800	0.865
AvatarForcing*	Yes	0.143	0.152	1.043	4.653	4.854	3.830	4.700	2.947

Appendix

Table 10: Conventional motion-based metrics computed from LivePortrait features on all frames of the SI-184 subset. For metrics without an arrow, values closer to ground truth are better. Expression PFD and variance are reported in units of

10^{-3}

and

10^{-5}

, respectively. Their rotation components are reported in units of

10^{-2}

. Gray marks the best value among dyadic systems.

Real-Time Interactive Video Generation

Real-Time Interactive Video Generation creates low-latency video streams that respond continuously to user actions, camera motion, text, or audio.

See all 73 papers

AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars
Jul 27, 2026Hengyuan Zhang, Jingna Sun, Meiguang Jin +1Video Diffusion Models Long-Horizon Video Generation
Alibaba Group, Taobao and Tmall Group
OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
Jul 25, 2026Quanyue Song, Yishan He, Yanbo Ding +4Audio-Video Generation Real-Time Interactive Video Generation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · Shenzhen Key Laboratory of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold
Sep 28, 2026Junjie Chen, Fei Wang, Kun Li +5Talking Head Generation 3D Avatar Generation
Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4

Jul 27, 2026J/K move · Enter open · S save

AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars

Hengyuan Zhang, Jingna Sun, Meiguang Jin, Junfeng Ma

Alibaba Group, Taobao and Tmall Group

Video Diffusion Models Long-Horizon Video Generation Identity-Preserving Video Generation Diffusion Distillation Human Avatar Animation Audio-Driven Facial Animation

Open paper AskPDF arXiv

Production-ready audio-driven avatar generation requires efficient inference without sacrificing fidelity or motion expressiveness. However, existing acceleration methods often compromise quality through restrictive architectural choices, such as causal attention and short temporal horizons, or by reducing model capacity and resolution. Without such compromises, we propose AptAvatar, a 14B-parameter long-form audio-driven avatar generation framework that delivers fast and expressive inference. For efficiency in production-level applications, AptAvatar addresses the extreme two-step generation challenge. To bridge the gap between the multi-step teacher model and the two-step student model, we introduce Endpoint-Anchored Distribution Distillation. It augments vanilla distribution matching with a dedicated Anchor Score Estimator trained on the trajectory-endpoint distribution defined from a frozen pretrained 4-step bridge generator. This provides an attainable endpoint-level anchor for the evolving two-step student. To improve long-horizon consistency, we further introduce Self-Generated History Replay, which reuses cached outputs from earlier generator checkpoints as history conditions during chunk-wise training. This approximates inference-time conditioning on self-generated histories without costly online rollouts, mitigating quality degradation from accumulated history errors. Extensive experiments demonstrate that AptAvatar generates vivid 720p long-form avatar videos with only 2 NFEs, achieving a 60x speedup while preserving visual fidelity and long-horizon identity. Code is available at https://github.com/TaoLiveAIGC/AptAvatar

Abstract

Real-Time Interactive Video Generation

Explore similar work

AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars

OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold