AVTR-1: Open Stack for Real-Time Interactive Avatars

Artem Kravtsov; Dmitrii Ziganshin; Vsevolod Poletaev; Gleb Balitskiy; Anastasia Tikhonova; Egor Burkov; Vadim Lebedev

cs.CVSep 19, 2026

AVTR-1: Open Stack for Real-Time Interactive Avatars

Authors: Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev, Gleb Balitskiy, Anastasia Tikhonova, Egor Burkov, Vadim Lebedev

Organizations: Avaturn Live

Abstract

Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an open stack for real-time interactive avatar conversations, built around a compact 153M-parameter autoregressive flow-matching motion generator conditioned on both participants' audio. We adapt its audio encoder for streaming through self-distillation. The stack turns the model's chunk-based generation into a continuous, synchronized audio-video stream driven by an external voice agent, and we analytically derive its contribution to the user-facing latencies and validate the resulting bounds with two commercial voice agents. Further experiments demonstrate that AVTR-1 leads the compared dyadic systems on all reported visual-quality metrics and most conventional listening-motion metrics while remaining competitive in lip synchronization. Its inference runtime operates in real time on data-center and consumer GPUs. However, conventional listening metrics do not establish whether the paired speaker's speech contributes to generated motion. We therefore introduce the Reference-Based Directed Granger Gain (R-DGG), which measures the additional predictive information carried by speaker speech after accounting for listener history and speaker motion. R-DGG finds statistically supported predictive dependence for recorded listeners and all evaluated dyadic systems, but not for talking-head generators without paired audio or mismatched speaker-listener pairs. We release the model weights, renderer, and serving backend under component-specific licenses.

Figures & tables

Property	Value
Parameters	153M
Decoder layers	18
Hidden dimension	512
Attention heads / head dimension	8 / 64
Feed-forward dimension	512
Audio condition encoder layers	2 per stream

Table 1: Configuration of the released AVTR-1 motion model.

GPU	TTFF	Request duration	Real-time factor
NVIDIA L40S	42 ms	71 ms	$2.81\times$
NVIDIA A100	54 ms	91 ms	$2.20\times$
NVIDIA L4	—	202 ms	$0.99\times$
NVIDIA RTX 4060 Ti	99 ms	162 ms	$1.24\times$
NVIDIA RTX 3070	94 ms	180 ms	$1.11\times$
NVIDIA RTX 3060 Ti	—	207 ms	$0.97\times$

Table 2: Per-chunk latency, real-time throughput, and time to first frame (TTFF) across GPUs, measured offline on one five-frame (200 ms) chunk.

Term	RLC	ILC	RLC	ILC	RLC	ILC
	OpenAI Realtime API ( $n=10$ )		Cartesia Line ( $n=10$ )		Predicted
$T_{\mathrm{wait}}$	$86.3$ $[10,187]$	$58.8$ $[6,105]$	$110.3$ $[19,193]$	$89.1$ $[41,169]$	$[0,200)$	$[0,200)$
$T_{\mathrm{pad}}$	$5.0$	$5.0$	$45.0$ $[5,125]$	$5.0$	$[5,205)$	$5$
$T_{\mathrm{present}}$	$200.0$	$200.0$	$200.0$	$200.0$	$200$	$200$
$T_{\mathrm{lead}}$	$99.1$ $[98,100]$	$98.9$ $[98,100]$	$98.9$ $[97,100]$	$99.2$ $[99,100]$	$100$	$100$
Total	$390.4$ $[315,491]$	$362.7$ $[310,409]$	$454.2$ $[426,497]$	$393.3$ $[345,473]$	$[305,705)$	$[305,505)$

Table 3: Measured streamer latency for both voice agents, in milliseconds. Values are the mean followed by

[\min,\max]

, or a single value where constant. The predicted columns are the bounds derived above.

Stage	ms
User $\rightarrow$ streamer (inbound)	114
Microphone capture	40
Opus encoding	21
Network	12
Jitter buffer	40
Opus decoding	1

Table 4: Reference end-to-end response latency budget for an avatar deployment, in milliseconds. The per-stage client and voice-agent numbers are taken from the published voice-to-voice budget of [ 29 ] and grouped into parts, and the bold part totals sum to the end-to-end figure. The two streamer

\leftrightarrow

voice-agent network hops and the streamer RLC are extras introduced by our system: the RLC is the measured mean for Cartesia Line (Table 3 ), and the hops assume the streamer and the voice agent share a physical cloud region.

Table 5: Quantitative comparison with competing methods on the SI-184 subset. Gray marks the best value among the dyadic systems in both panels. For listening metrics without an arrow, values closer to ground truth are better. The pose components of PFD and variance are reported in units of

10^{-2}

. AvatarForcing* denotes the evaluation configuration described in the protocol.

	Paired audio	rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method		Exp	Pose	Exp	Pose	Exp	Pose	Exp	Pose
AvatarForcing*	Yes	0.109	0.163	37.51	7.560	4.749	3.848	1.213	2.099
AVTR-1	Yes	0.083	0.140	25.98	6.417	4.970	3.254	1.313	0.930
DyStream	Yes	0.128	0.141	40.87	7.598	4.497	3.606	1.164	1.731
Ditto	No	0.092	0.168	35.02	6.790	5.006	3.253	1.604	1.386
FLOAT	No	0.117	0.132	34.09	4.753	4.322	2.961	1.007	0.843

Table 6: Comparison using conventional motion-based listening metrics on the VAD-selected listener-silent frames of the SI-184 subset. This extends Table 5(b) with the talking-head generators. For metrics without an arrow, values closer to ground truth are better. The pose components of PFD and variance are reported in units of

10^{-2}

. Dark and light gray mark the best and second-best methods, respectively.

	Paired audio	R-DGG	Bootstrap percentile			$p$
Method			2.5%	50%	97.5%
AvatarForcing*	Yes	0.32	0.07	0.32	0.56	0.008
AVTR-1	Yes	0.57	0.19	0.57	0.98	0.002
DyStream	Yes	0.48	0.06	0.49	0.93	0.011
Ditto	No	0.06	-0.20	0.06	0.31	0.335
FLOAT	No	0.01	-0.33	0.00	0.42	0.491

Table 7: Comparison using R-DGG on the SI-184 subset. Values are reported in units of

10^{-4}

nats. The final column reports

p=\widehat{\Pr}(G_{a\mid m}\leq 0)

. Gray marks estimates whose intervals exclude zero.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Paired audio	rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method		Exp	Rot	Exp	Rot	Exp	Rot	Exp	Rot
AvatarForcing*	Yes	0.138	0.175	1.242	5.784	4.802	3.813	5.200	3.352
AVTR-1	Yes	0.112	0.169	0.671	3.967	4.760	3.152	3.300	0.920
DyStream	Yes	0.151	0.178	1.335	4.392	4.421	3.436	4.300	1.239
Ditto	No	0.109	0.174	0.882	4.106	4.930	3.010	3.800	1.002
FLOAT	No	0.127	0.152	0.878	3.240	4.488	3.061	4.300	0.852

Appendix

Table 8: Comparison using conventional motion-based listening metrics computed from LivePortrait features on the VAD-masked listening portion of the SI-184 subset. For metrics without an arrow, values closer to ground truth are better. Expression PFD and variance are reported in units of

10^{-3}

and

10^{-5}

, respectively. Their rotation components are reported in units of

10^{-2}

. Dark and light gray mark the best and second-best methods, respectively.

		rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method	Paired audio	Exp	Pose	Exp	Pose	Exp	Pose	Exp	Pose
SoulX Lite	No	0.061	0.148	19.94	5.988	5.065	3.419	1.411	0.975
SoulX Pro	No	0.056	0.125	18.75	4.751	5.103	3.499	1.441	0.982
Ditto	No	0.074	0.180	28.82	6.645	4.953	3.459	1.671	1.341
FLOAT	No	0.077	0.128	23.71	3.953	4.816	3.220	1.206	0.811
AvatarForcing*	Yes	0.080	0.145	27.78	5.891	4.887	3.917	1.337	1.800

Appendix

Table 9: Conventional motion-based metrics computed from EMOCA features on all frames of the SI-184 subset. For metrics without an arrow, values closer to ground truth are better. The pose components of PFD and variance are reported in units of

10^{-2}

. Gray marks the best value among dyadic systems.

		rPCC $\downarrow$		PFD $\downarrow$		SID		Var
Method	Paired audio	Exp	Rot	Exp	Rot	Exp	Rot	Exp	Rot
SoulX Lite	No	0.092	0.179	0.554	3.867	5.021	3.133	3.400	0.854
SoulX Pro	No	0.091	0.154	0.521	3.089	5.070	3.309	3.500	1.044
Ditto	No	0.123	0.177	0.836	4.300	5.012	3.027	3.700	1.004
FLOAT	No	0.129	0.144	0.686	2.764	4.830	3.252	3.800	0.865
AvatarForcing*	Yes	0.143	0.152	1.043	4.653	4.854	3.830	4.700	2.947

Appendix

Table 10: Conventional motion-based metrics computed from LivePortrait features on all frames of the SI-184 subset. For metrics without an arrow, values closer to ground truth are better. Expression PFD and variance are reported in units of

10^{-3}

and

10^{-5}

, respectively. Their rotation components are reported in units of

10^{-2}

. Gray marks the best value among dyadic systems.

Real-Time Interactive Video Generation

Real-Time Interactive Video Generation creates low-latency video streams that respond continuously to user actions, camera motion, text, or audio.

See all 71 papers

Jul 27, 2026

AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars

Production-ready audio-driven avatar generation requires efficient inference without sacrificing fidelity or motion expressiveness. However, existing acceleration methods often compromise quality through restrictive architectural choices, such as causal attention and short temporal horizons, or by reducing model capacity and resolution. Without such compromises, we propose AptAvatar, a 14B-parameter long-form audio-driven avatar generation framework that delivers fast and expressive inference. For efficiency in production-level applications, AptAvatar addresses the extreme two-step generation challenge. To bridge the gap between the multi-step teacher model and the two-step student model, we introduce Endpoint-Anchored Distribution Distillation. It augments vanilla distribution matching with a dedicated Anchor Score Estimator trained on the trajectory-endpoint distribution defined from a frozen pretrained 4-step bridge generator. This provides an attainable endpoint-level anchor for the evolving two-step student. To improve long-horizon consistency, we further introduce Self-Generated History Replay, which reuses cached outputs from earlier generator checkpoints as history conditions during chunk-wise training. This approximates inference-time conditioning on self-generated histories without costly online rollouts, mitigating quality degradation from accumulated history errors. Extensive experiments demonstrate that AptAvatar generates vivid 720p long-form avatar videos with only 2 NFEs, achieving a 60x speedup while preserving visual fidelity and long-horizon identity. Code is available at https://github.com/TaoLiveAIGC/AptAvatar

Hengyuan Zhang, Jingna Sun, Meiguang Jin +1

Alibaba Group, Taobao and Tmall Group

Video Diffusion Models Long-Horizon Video Generation Identity-Preserving Video Generation Diffusion Distillation Human Avatar Animation Audio-Driven Facial Animation

Ask PDF

Jul 25, 2026

OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

Recent advances in diffusion-based generative models have enabled real-time audio-driven avatar generation and unified audio-visual synthesis, providing a promising foundation for interactive avatar systems. However, extending unified audio-visual synthesis to real-time interactive streaming remains challenging, as the generation horizon is unknown in advance and the generated identity may drift over long-term generation. To address these challenges, we propose OmniMate, a unified framework for open-ended real-time interactive audio-visual avatar generation. OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions. To achieve adaptive response progression, we introduce a Generation Progress Controller (GPC) that explicitly models the generation progress of each streaming chunk, allowing the model to complete responses according to the desired progress and achieve seamless transitions between execution and listening states. To preserve long-term cross-modal identity consistency, we propose a Multi-Reference Conditioning Module (MRCM), which leverages multiple reference images and a reference speech segment to provide persistent visual and speaker identity cues throughout long-duration streaming interactions. Extensive experiments on an interaction-oriented adaptation of VerseBench demonstrate that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency. The results further show that OmniMate supports realistic, coherent, and responsive interactive avatar experiences over extended multi-turn conversations.

Quanyue Song, Yishan He, Yanbo Ding +4

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · Shenzhen Key Laboratory of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

Audio-Video Generation Real-Time Interactive Video Generation Identity-Preserving Video Generation Streaming Video Generation Temporal Consistency in Video Generation

Ask PDF

Sep 28, 2026

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold

Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.

Junjie Chen, Fei Wang, Kun Li +5

Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4

Talking Head Generation 3D Avatar Generation Audio-Driven Facial Animation Test-Time Training

Ask PDF

Abstract

Real-Time Interactive Video Generation

Explore similar work

AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars

OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold