Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation

Nikolay Semenov; Artem Riabinin; Sergey Bassenko; Anna Averchenkova; Julia Agafonova; Anastasia Aliaskina; Kirill Kozlov; Lev Novitskiy

cs.CVOct 13, 2026

Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation

Authors: Nikolay Semenov, Artem Riabinin, Sergey Bassenko, Anna Averchenkova, Julia Agafonova, Anastasia Aliaskina, Kirill Kozlov, Lev Novitskiy,

Organizations: Kandinsky Lab

Abstract

We present Kandinsky S2V 1.0, an open family of speech-to-video models for controllable audio-driven human animation, comprising Kandinsky S2V 1.0 Lite (2B parameters) and Kandinsky S2V 1.0 Pro (21B parameters). From a single reference image, a speech waveform, and a text prompt, the models generate long-form 25 FPS video in which lip motion, expression, pose, and gesture follow the speech. The architecture draws text and audio conditioning from one Qwen2.5-Omni model and adds a reference-aware three-scale visual memory. Training starts from the Kandinsky 5.0 Video foundation models [1] and combines mixed reference-memory layouts with an area-adaptive face loss. The training corpus contains 1.17M single-speaker clips selected by active-speaker detection, with captions generated directly in English and Russian. A side-by-side human evaluation against the strongest open-source baseline and two proprietary commercial models shows that Kandinsky S2V 1.0 Pro performs comparably to or better than these models on average and is also preferred for lip synchronization. On common speech to video benchmarks, Kandinsky S2V 1.0 Lite and Kandinsky S2V 1.0 Pro rank among the top performing models according to mean rank over five reported metrics. Beyond the base models, we provide π-Flow distilled variants of both Kandinsky S2V 1.0 Lite and Kandinsky S2V 1.0 Pro. The distilled Kandinsky S2V 1.0 Pro model generates video in 16 sampling steps without classifier-free guidance, achieving a 5.4 times end-to-end speedup over the base model, while matching or exceeding it on nearly all reported benchmark metrics. To accelerate open research and deployment in speech-to-video generation, we release the code, model checkpoints, and diffusers integration under the MIT license.

Figures & tables

Stage	Web collection	TalkVerse (Panda-70M part)
Videos with a detected face, after duration filtering	$>10$ s: $3.8$ M	$>5$ s: $700$ K
Single-speaker, smoothed ASD-score $>2.0$	window mean	max
Q-Align technical score $>4.0$	✓	Not applied
Cluster-based deduplication	✓, over the whole corpus
Exclusion of evaluation data (Section 2.4 )	✓	✓
Clip length used for training	$10$ s	$5$ s and $10$ s

Table 1: Data pipeline and the resulting corpus. Q-Align technical filtering applies only to the web collection. The active-speaker stage and the exclusion of evaluation data are shared, and deduplication runs over the union of the two sources.

Panel of Figure 2 — Figure 2: Overview of the Kandinsky S2V 1.0 architecture. Text features from the frozen Qwen2.5-Omni Thinker and audio features from its frozen audio tower [ 17 ] feed trainable text and audio encoder stacks in the DiT, with $N_{t}$ text blocks and $N_{a}$ audio blocks. The reference image is encoded by a frozen Hunyuan 3D VAE [ 2 ] , and its latent is concatenated (denoted by C) with the noisy target latent $z_{t}$ and a multi-scale FramePack [ 14 ] memory of past frames to form the visual sequence consumed by the DiT. Each DiT block applies self-attention (V2V), cross-attention to the text tokens (V2T), an optional cross-attention to the audio tokens (V2A) injected periodically across the stack, and a feed-forward layer. The denoised visual latent is decoded back to pixel space by the frozen VAE decoder.

Panel of Figure 3 — Figure 3: FramePack memory layout. The $30$ latent frames of the memory clip $V_{\text{mem}}$ are split into three buckets of $24$ , $4$ , and $2$ frames, from oldest to most recent, and patchified with patch sizes $(4,8,8)$ , $(2,4,4)$ , and $(1,2,2)$ , so older frames contribute fewer tokens. The memory tokens are concatenated (C) with the reference and target tokens into one visual sequence for the DiT. The bottom axis shows the temporal RoPE position of each token group and is not drawn to scale.

Setting	2B Lite	21B Pro
Parameters	$2.3$ B	$21.4$ B
Visual blocks $N_{v}$	$32$	$60$
Text blocks $N_{t}$	$2$	$4$
Audio blocks $N_{a}$	$2$	$4$
Model dimension $d_{\text{model}}$	$1792$	$4096$
Feed-forward dimension $d_{\text{ff}}$	$7168$	$16384$

Table 2: Settings that differ between the two variants. Both share the patch size

(1,2,2)

, the

16

-channel VAE latent, the

3584

-dimensional text and audio features from Qwen2.5-Omni, the

768

-dimensional pooled CLIP vector, the reference RoPE position

40

, and the memory frame buckets

(2,4,24)

Panel of Figure 4 — Figure 4: Frame-aligned audio cross-attention (V2A). Audio features from the frozen Qwen2.5-Omni audio tower pass through the trainable audio encoder. In each V2A layer the target video tokens of latent frame $i$ attend only to the audio tokens of the same time span, which gives a block-diagonal attention pattern. Memory and reference tokens skip this layer. The strips at the bottom left mark the visual decoder blocks that contain a V2A layer in the 2B and 21B models.

Setting	2B	21B Pro
Initialization	Kandinsky 5.0 Video T2V, EMA weights
Training steps	$14000$	$9500$
Global batch size	$128$	$128$
Samples per data-parallel rank	$2$	$1$
Nodes $\times$ GPUs	$8\times 8$	$32\times 8$
FSDP shard width	$8$	$64$

Table 3: Training setup of the two runs. Both use AdamW with

\beta=(0.9,0.95)

\epsilon_{\text{Adam}}=10^{-8}

, gradient-norm clipping at

1.0

, an EMA decay of

0.999

, a timestep shift of

s=5

, dense FlashAttention, independent text, audio, and reference dropout at probability

0.1

, and a memory-conditioned step probability of

0.25

Setting	Shared
Student NFE	16
Policy	DX, 5 grid points
Integration substeps (total)	128
Teacher steps per segment	3
Velocity-averaging window	3 substeps
Timestep shift	10

Table 4: Settings of the

\pi

-Flow distillation of Kandinsky S2V 1.0 Pro and Lite. The upper block is shared by both runs; the lower block lists the settings that differ.

Model	Resolution	GPUs	Sampling time per chunk (s)
Kandinsky S2V 1.0 Lite, distilled	SD $512\times 768$	$1/2/4$	$43/27/19$
Kandinsky S2V 1.0 Lite	SD $512\times 768$	$1/2/4$	$220/123/69$
Kandinsky S2V 1.0 Pro, distilled	SD $512\times 768$	$1/2/4/8$	$173/105/70/57$
Kandinsky S2V 1.0 Pro	SD $512\times 768$	$1/2/4/8$	$914/497/272/162$
Kandinsky S2V 1.0 Pro, distilled	HD $1280\times 768$	$1/2/4/8$	$675/375/219/145$
Kandinsky S2V 1.0 Pro	HD $1280\times 768$	$1/2/4/8$	$3854/2017/1060/568$

Table 5: Per-chunk sampling times for Lite and Pro with the released inference code on H100 GPUs. GPU counts and times correspond in order within each row.

Benchmark	Model	Mean rank $\downarrow$	Min-max $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	CSIM $\uparrow$	IQA $\uparrow$	ASE $\uparrow$
EMTD	Kandinsky S2V Pro (dist.)	2.30	0.95	8.03	7.85	0.81	4.81	3.61
	Kandinsky S2V Pro	3.90	0.82	8.05	7.62	0.77	4.73	3.54
	Kandinsky S2V Lite (dist.)	4.90	0.87	7.41	8.24	0.79	4.81	3.58
	Kandinsky S2V Lite	5.10	0.83	7.57	8.03	0.76	4.77	3.58
	LongCat 1.5	2.90	0.88	8.41	7.39	0.75	4.73	3.63
	Wan-S2V	6.40	0.80	7.01	8.47	0.73	4.82	3.59

Table 6: Open-source comparison on EMTD, CelebV-HQ, and HDTF. The base and distilled Kandinsky models share one evaluation pool with the baselines. Mean rank and min-max aggregate the five reported metrics within each benchmark; best values are in bold.

Panel of Figure 5 — Figure 5: Side-by-side human evaluation on the $80$ -item Russian set.

Panel of Figure 6 — Figure 6: Side-by-side human evaluation on the multilingual set.

Panel of Figure 7 — Figure 7: Additional side-by-side human evaluations involving Kandinsky S2V 1.0 Lite on the $80$ -item Russian set. Percentages are reported as whole numbers and sum to $100$ % for each criterion.

Figure 8: Side-by-side human evaluation of distilled Kandinsky S2V 1.0 Pro against base Pro on the

80

-item Russian set, HD.

Figure 9: Side-by-side human evaluation of distilled Kandinsky S2V 1.0 Lite against base Lite on the Russian set, SD.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 155 papers

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
Oct 4, 2026Team Kandinsky, Julia Agafonova, Bulat Akhmatov +85Audio-Video Generation Video Diffusion Models
Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model
Mar 23, 2026SII-GAIR, Sand. ai, : +43Multimodal Transformer Audio-Video Generation
Avatar V: Scaling Video-Reference Avatar Video Generation
Jun 11, 2026Benjamin Liang, Ce Chen, Desmond Lin +20Video-Reference Video Generation Identity-Preserving Video Generation
HeyGen Research

Oct 4, 2026J/K move · Enter open · S save

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

Team Kandinsky, Julia Agafonova, Bulat Akhmatov, Mikhail Aksyutin+84

Audio-Video Generation Video Diffusion Models T2V Generation Audio-Visual Synchronization Video Foundation Models

Open paper AskPDF arXiv

We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (I2AV) modes; a built-in super-resolution model raises the output resolution to Full-HD (1920

\times

1080). Building on the video generation capabilities of Kandinsky 5.0, Kandinsky 6.0 Video employs a dual-stream CrossDiT architecture that connects a pretrained video stream and a newly trained audio stream through bidirectional cross-attention for temporal and semantic alignment. Our continuous pretraining strategy first trains the audio stream from scratch on large-scale audio corpora and then trains both streams jointly on paired audio-video data while preserving unimodal fidelity; pretraining is followed by supervised fine-tuning, reinforcement-learning-based post-training, and distillation. In side-by-side human evaluation, Kandinsky 6.0 Video Pro clearly outperforms its predecessor, Kandinsky 5.0 Video Pro, and remains competitive with leading audio-video generation models, particularly in speech quality. To accelerate open research and deployment in multimedia generation, we release the code, model checkpoints, and diffusers integration under the MIT license.

Abstract

Audio-Video Generation

Explore similar work

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model

Avatar V: Scaling Video-Reference Avatar Video Generation