We tackle the challenging task of generating complete 3D facial animations for two interacting, co-located participants from a mixed audio stream. While existing methods often produce disembodied "talking heads" akin to a video conference call, our work is the first to explicitly model the dynamic 3D spatial relationship -- including relative position, orientation, and mutual gaze -- that is crucial for realistic in-person dialogues. Our system synthesizes the full performance of both individuals, including precise lip-sync, and uniquely allows their relative head poses to be controlled via textual descriptions. To achieve this, we propose a dual-stream architecture where each stream is responsible for one participant's output. We employ speaker's role embeddings and inter-speaker cross-attention mechanisms designed to disentangle the mixed audio and model the interaction. Furthermore, we introduce a novel eye gaze loss to promote natural, mutual eye contact. To power our data-hungry approach, we introduce a novel pipeline to curate a large-scale conversational dataset consisting of over 2 million dyadic pairs from in-the-wild videos. Our method generates fluid, controllable, and spatially aware dyadic animations suitable for immersive applications in VR and telepresence, significantly outperforming existing baselines in perceived realism and interaction coherence.
Figures & tables
Figure 1: Our method in context. Left: Prior work generates isolated participants, such as Speaker-Only models that lack listener reactions, Listener-Only models that do not model multi-round interactions, or Conversation models that resemble a “video conference” call, failing to model a shared 3D space, spatial layout, or real, physically meaningful eye interaction. Right: Our method takes a single mixed conversational audio stream and a text-based environment prompt to generate the complete, co-located 3D performance of both participants. Our model is the first to explicitly synthesize this crucial spatial relationship, enabling realistic outputs with natural mutual gaze and a controllable spatial layout.
Figure 2: An overview of our dual-stream diffusion architecture. The model employs a shared U-Net backbone to process the noisy input streams for both participants in parallel. It is conditioned on features from the mixed audio, learnable role embeddings, and the speaker probability masks. Dual-speaker cross-attention layers within the decoder allow the two streams to exchange information, modeling the interaction dynamics. The network outputs the predicted 3D animation parameters for expression ( ψ ), rotation ( θ )), and translation ( t ). For conversation data, we utilize losses for all expression, rotation and translation prediction, with auxiliary eye gaze loss on samples with large-scale head movements. For synthetic data, we finetune with losses only on speaker’s lip parameters.
Figure 3: Our Data Curation Pipeline. This figure illustrates our two-pronged approach to dataset creation. (a) Dyadic Conversation Dataset: We process raw conversational videos through various filters to reconstruct 3D facial parameters and extract speaker masks. (b) Synthetic Dubbing Dataset: We generate clean pseudo-conversations by taking single-person videos, applying random cuts, and re-assembling them into new alternating-speaker audio tracks. (c) Dataset Example: A sample of 3D reconstructions overlaid on videos from our curated dyadic dataset.
Figure 4
Methods
FD ↓
P-FD ↓
MSE ↓
vMSE ↓
SID ↑
FULL
FULL
EXP
TRANSL
ROT
EYE
LIP
SPE
LIS
FULL
EXP
ROT
TRANSL
CodeTalker [ 69 ]
47.23
70.54
10.47
-
14.28
3.07
2.95
12.49
6.85
0
0
0
-
SelfTalk [ 48 ]
43.58
53.98
8.21
-
11.59
2.47
2.41
10.98
6.13
1.68
1.27
1.39
-
FaceFormer [ 19 ]
52.66
59.84
13.89
-
12.34
2.96
2.84
10.47
6.44
1.59
0.43
0.86
-
Ours (Single)
19.58
29.03
6.32
-
6.74
1.23
1.14
6.86
5.23
2.23
1.40
1.61
-
Listen-R
63.74
68.75
11.03
8.90
10.93
2.58
2.27
7.35
7.98
1.84
2.39
2.18
2.98
Table 2: Quantitative comparison of our method against baselines. We only compare translation results with retrieval-based methods, as other baselines inherently do not predict translations. Metrics: FD/P-FD (on renders) measure realism/interaction. MSE (on parameters) and vMSE (on vertices) measure accuracy for FULL motion, EXPression, ROTation, TRANSLation, EYE gaze, and LIP motion. SID measures diversity for LIStener and SPEaker roles. Lower is better ( ↓ ) for FD, P-FD, MSE, and vMSE; higher is better ( ↑ ) for SID.
Figure 5: Left: Result of our model compared with baselines, shown as a few representative frames from one sequence. Top two rows feature speaker and listener interaction; Bottom two rows feature our model’s ability to deal with short simultaneous speaking sequences. Right: Run inference with a variety of spatial layout prompts and render the first frame in a shared space. Red squares are used to mark significant speaker lip movements. This could be significantly easier to view as videos in supplementary.
Ablation Setting
FD ↓
P-FD ↓
EXP ↓
TRAN ↓
ROT ↓
Single-Person Only
50.45
50.05
10.01
2.32
3.88
w/o 2nd Stage
60.12
64.44
7.73
1.71
2.94
w/o Speaker Emb.
35.92
35.93
7.18
1.80
2.87
w/o Cross-Attn
30.49
40.87
6.87
1.54
2.98
w/o Gaze Loss
37.46
42.90
7.33
2.59
2.77
Full Model (Ours)
21.71
22.56
5.97
1.50
2.48
Table 3: Ablation study on different model components. Lower FD, P-FD, and MSE (EXPression, TRANslation or ROTation) indicate better performance when evaluated with groundtruth.
Method
Lip
Speaker M
Listener M
Interact
Gaze
SelfTalk
0.9
0.9
1.6
1.6
2.4
DualTalk
3.9
6.3
7.2
5.6
7.9
Ours (S1)
15.9
19.0
18.2
21.4
21.4
Ours
79.3
73.8
73.0
71.4
68.3
Table 4: Human evaluation preference rate (%). Higher is better. Participants selected the best model from four options.
Method
Cycle Error ↓
LPIPS ↓
SSIM ↑
Linear Map
0.684
0.212
0.912
3-Layer MLP
0.327
0.141
0.963
Table 5: Comparison of expression-space mappings. We report cycle-consistency error (lower is better) and perceptual similarity between rendered faces using LPIPS (lower is better) and SSIM (higher is better). Our three-layer MLP significantly outperforms a linear regression baseline.
Method
tMSE ↓
SAS ↑
Zero-Shot
2.90
3.15
k -NN
1.84
2.65
Few-Shot (Ours)
0.72
4.62
Table 6: Quantitative and human evaluation of LLM-based spatial generation. tMSE measures the distance between predicted and groundtruth translations and the lower the better, while SAS measures perceptual quality (1-5 scale).
Method
FD ↓
P-FD ↓
vMSE-S ↓
vMSE-L ↓
DualTalk
34.12
45.20
8.95
9.12
Ours
14.55
31.05
4.92
6.30
Table 7: Performance comparison on the DualTalk OOD Dataset (384 unseen clips). Our method demonstrates superior generalization on unseen identities compared to the baseline trained specifically on the DualTalk domain. Lower is better for all metrics.
Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.
We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (CsF) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalkRT, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.
We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting, formulating the problem as an asymmetric task involving speech-constrained articulation and one-to-many listener reactions. To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure and a stochastic residual that models the remaining one-to-many interaction dynamics. On top of this backbone, Motion Memory acts as a training-only regularizer during brief late-stage fine-tuning to provide local priors for weakly conditioned listening windows, while semantic-group scaling controls residual injection across expression, jaw, and neck. This design balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process. Results from unified, state-wise, and ablation evaluations show that conversational 3D motion benefits from decomposing stable and uncertain components rather than applying stochasticity uniformly. ECHO provides a practical formulation and technical basis for deployable conversational digital humans under strict audio-only conditions.
Zhuoqiang Cai, Yujie Sun, Chaoyue Niu +4
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China