Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space

Aashish Chandra; Aashutosh A; Abhijit Das

cs.CVFeb 20, 2026

Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space

Authors: Aashish Chandra, Aashutosh A, Abhijit Das

Organizations: Machine Intelligence Group, Department of CS&IS, BITS Pilani, Hyderabad Campus, India · Georgia Institute of Technology, USA

Abstract

We present a novel approach for generating realistic speaking and talking faces by synthesizing a person's voice and facial movements from a static image, a voice profile, and a target text. The model encodes the prompt/driving text, the driving image, and the voice profile of an individual and then combines them to pass them to the multi-entangled latent space to foster key-value pairs and queries for the audio and video modality generation pipeline. The multi-entangled latent space is responsible for establishing the spatiotemporal person-specific features between the modalities. Further, entangled features are passed to the respective decoder of each modality for output audio and video generation.

Figures & tables

Figure 1: Our Network Architecture: Text Prompt-guided joint audio-visual learning representations using dual stream Transformer Encoders and Denoising Diffusion model. The model architecture can be divided into three phases – namely Encoding Phase , Multi-Latent Entanglement , and Decoding Phase . As an output, an audio-visual animation is generated from a single source image, reference audio, and a short text prompt.

Dataset	Model	FID ( $\downarrow$ )	FVD ( $\downarrow$ )	FVMD ( $\downarrow$ )
VoxCeleb	Audio2Head	81.00	90.12	5100.92
	Hallo	67.28	70.69	5703.44
	EAT	85.16	80.38	4878.36
	SadTalker	119.36	112.77	6352.19
	Our Model	42.88	49.78	4192.07
FakeAVCeleb	Audio2Head	93.59	97.85	1329.23

Table 1: Video pipeline evaluation scores across datasets.

Method	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	KVD ( $\downarrow$ )	LPIPS ( $\downarrow$ )	MOS ( $\uparrow$ )
Hallo	33.01	0.68	44.12	0.51	4.10
Audio2Head	29.21	0.51	49.09	0.23	3.51
EAT	24.45	0.52	49.12	0.52	3.57
SadTalker	29.72	0.59	49.61	0.53	3.51
Proposed	35.94	0.73	41.22	0.54	4.22

Table 2: Evaluation of video quality metrics.

Dataset	Model	FAD ( $\downarrow$ )	MCD ( $\downarrow$ )	STOI ( $\uparrow$ )
VoxCeleb	Tortoise	258.54	82.37	0.10
	Your_TTS	199.52	111.79	0.19
	XTTS_v2	249.17	100.80	0.13
	GlowTTS	329.21	103.94	0.15
	Our Model	241.75	75.39	0.17
FakeAVCeleb	Tortoise	871.14	82.12	0.10

Table 3: Audio pipeline evaluation scores across datasets.

Figure 2: Visual comparison on VoxCeleb, in the order: Ground Truth, Ours, Audio2Head, EAT, Hallo, and SadTalker. Columns represent 25s intervals.

Figure 3: Results of our model on FakeAVCeleb, Celeb-HQ and HDTF datasets.

Panel of Figure 4 — Figure 4: Ground Truth vs. Generated Audio Spectrograms for (a) VoxCeleb, (b) CelebV-HQ, (c) FakeAVCeleb and (d) HDTF datasets

Method	ESTOI ( $\uparrow$ )	PESQ ( $\uparrow$ )	WER ( $\downarrow$ )	MOS ( $\uparrow$ )
Tortoise	0.42	3.80	0.21	4.32
Your_TTS	0.31	3.71	0.30	3.61
XTTS_v2	0.33	3.51	0.34	3.21
GlowTTS	0.39	3.83	0.33	3.52
Proposed	0.43	4.01	0.18	4.56

Table 4: Evaluation of audio quality metrics.

Model	LSE-C ( $\uparrow$ )	LSE-D ( $\downarrow$ )
Groundtruth	5.45	8.52
Hallo	3.03	8.71
Audio2Head	2.51	10.34
EAT	4.39	9.35
SadTalker	5.44	10.09
STE	5.71	8.41

Table 5: Evaluation of audio-visual synchronisation.

Components				Video Metrics			Audio Metrics
ETE	STE	DC	EC	FID ( $\downarrow$ )	FVD ( $\downarrow$ )	FVMD ( $\downarrow$ )	FAD ( $\downarrow$ )	MCD ( $\downarrow$ )	STOI ( $\uparrow$ )
		✓	✓	86.70	80.88	5275.89	328.27	95.44	0.07
	✓			68.83	74.19	4412.74	260.91	87.51	0.11
	✓	✓		63.68	71.38	4298.30	250.12	83.96	0.14
	✓	✓	✓	61.44	69.15	2720.41	241.77	81.60	0.17
✓		✓	✓	42.88	49.78	4192.07	241.75	75.39	0.17

Table 6: Ablation study of the transformer and cross-attention components. STE : Shared-TE, ETE : Explicit-TE, DC : Diffusion Cross-Attention, EC : Embedding Cross-Attention.

Ablation Setting	FID ( $\downarrow$ )	FVD ( $\downarrow$ )	FVMD ( $\downarrow$ )	FAD ( $\downarrow$ )	MCD ( $\downarrow$ )	STOI ( $\uparrow$ )
Only Visual Tokens Attended	68.31	78.42	5747.04	304.98	81.17	0.13
Only Audio Tokens Attended	69.02	79.35	6576.85	301.49	80.65	0.13
No audio sequence encoding	85.25	94.28	7483.40	498.33	87.51	0.09
No audio profile encoding	70.10	80.96	5926.64	309.95	89.58	0.11
No visual token encoding	54.38	62.02	5481.36	221.07	63.25	0.12
Proposed Model	42.88	49.78	4192.07	241.75	75.39	0.17

Table 7: Ablation study of the encoder components.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems
Jun 20, 2026Jingjing Jiang, Atsumoto Ohashi, Ryuichiro HigashinakaAudio-Driven Facial Animation Multimodal Generation
Graduate School of Informatics, Nagoya University, Japan
MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
Mar 30, 2026Bharath Krishnamurthy, Ajita RattaniMultimodal Generation Controllable Image Generation
University of North Texas, Denton, TX, USA
AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
May 8, 2026Kai Zheng, Zejian Kang, Rui Mao +4Audio-Driven Facial Animation
Westlake University · Zhejiang University · Tiangong University +2

Jun 20, 2026J/K move · Enter open · S save

Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems

Jingjing Jiang, Atsumoto Ohashi, Ryuichiro Higashinaka

Graduate School of Informatics, Nagoya University, Japan

Audio-Driven Facial Animation Multimodal Generation Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.

Abstract

Audio-Video Generation

Explore similar work

Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems

MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models