We tackle the challenging task of generating complete 3D facial animations for two interacting, co-located participants from a mixed audio stream. While existing methods often produce disembodied "talking heads" akin to a video conference call, our work is the first to explicitly model the dynamic 3D spatial relationship -- including relative position, orientation, and mutual gaze -- that is crucial for realistic in-person dialogues. Our system synthesizes the full performance of both individuals, including precise lip-sync, and uniquely allows their relative head poses to be controlled via textual descriptions. To achieve this, we propose a dual-stream architecture where each stream is responsible for one participant's output. We employ speaker's role embeddings and inter-speaker cross-attention mechanisms designed to disentangle the mixed audio and model the interaction. Furthermore, we introduce a novel eye gaze loss to promote natural, mutual eye contact. To power our data-hungry approach, we introduce a novel pipeline to curate a large-scale conversational dataset consisting of over 2 million dyadic pairs from in-the-wild videos. Our method generates fluid, controllable, and spatially aware dyadic animations suitable for immersive applications in VR and telepresence, significantly outperforming existing baselines in perceived realism and interaction coherence.
Figures & tables
Figure 1: Our method in context. Left: Prior work generates isolated participants, such as Speaker-Only models that lack listener reactions, Listener-Only models that do not model multi-round interactions, or Conversation models that resemble a “video conference” call, failing to model a shared 3D space, spatial layout, or real, physically meaningful eye interaction. Right: Our method takes a single mixed conversational audio stream and a text-based environment prompt to generate the complete, co-located 3D performance of both participants. Our model is the first to explicitly synthesize this crucial spatial relationship, enabling realistic outputs with natural mutual gaze and a controllable spatial layout.
Figure 2: An overview of our dual-stream diffusion architecture. The model employs a shared U-Net backbone to process the noisy input streams for both participants in parallel. It is conditioned on features from the mixed audio, learnable role embeddings, and the speaker probability masks. Dual-speaker cross-attention layers within the decoder allow the two streams to exchange information, modeling the interaction dynamics. The network outputs the predicted 3D animation parameters for expression ( ψ ), rotation ( θ )), and translation ( t ). For conversation data, we utilize losses for all expression, rotation and translation prediction, with auxiliary eye gaze loss on samples with large-scale head movements. For synthetic data, we finetune with losses only on speaker’s lip parameters.
Figure 3: Our Data Curation Pipeline. This figure illustrates our two-pronged approach to dataset creation. (a) Dyadic Conversation Dataset: We process raw conversational videos through various filters to reconstruct 3D facial parameters and extract speaker masks. (b) Synthetic Dubbing Dataset: We generate clean pseudo-conversations by taking single-person videos, applying random cuts, and re-assembling them into new alternating-speaker audio tracks. (c) Dataset Example: A sample of 3D reconstructions overlaid on videos from our curated dyadic dataset.
Figure 4
Methods
FD ↓
P-FD ↓
MSE ↓
vMSE ↓
SID ↑
FULL
FULL
EXP
TRANSL
ROT
EYE
LIP
SPE
LIS
FULL
EXP
ROT
TRANSL
CodeTalker [ 69 ]
47.23
70.54
10.47
-
14.28
3.07
2.95
12.49
6.85
0
0
0
-
SelfTalk [ 48 ]
43.58
53.98
8.21
-
11.59
2.47
2.41
10.98
6.13
1.68
1.27
1.39
-
FaceFormer [ 19 ]
52.66
59.84
13.89
-
12.34
2.96
2.84
10.47
6.44
1.59
0.43
0.86
-
Ours (Single)
19.58
29.03
6.32
-
6.74
1.23
1.14
6.86
5.23
2.23
1.40
1.61
-
Listen-R
63.74
68.75
11.03
8.90
10.93
2.58
2.27
7.35
7.98
1.84
2.39
2.18
2.98
Table 2: Quantitative comparison of our method against baselines. We only compare translation results with retrieval-based methods, as other baselines inherently do not predict translations. Metrics: FD/P-FD (on renders) measure realism/interaction. MSE (on parameters) and vMSE (on vertices) measure accuracy for FULL motion, EXPression, ROTation, TRANSLation, EYE gaze, and LIP motion. SID measures diversity for LIStener and SPEaker roles. Lower is better ( ↓ ) for FD, P-FD, MSE, and vMSE; higher is better ( ↑ ) for SID.
Figure 5: Left: Result of our model compared with baselines, shown as a few representative frames from one sequence. Top two rows feature speaker and listener interaction; Bottom two rows feature our model’s ability to deal with short simultaneous speaking sequences. Right: Run inference with a variety of spatial layout prompts and render the first frame in a shared space. Red squares are used to mark significant speaker lip movements. This could be significantly easier to view as videos in supplementary.
Ablation Setting
FD ↓
P-FD ↓
EXP ↓
TRAN ↓
ROT ↓
Single-Person Only
50.45
50.05
10.01
2.32
3.88
w/o 2nd Stage
60.12
64.44
7.73
1.71
2.94
w/o Speaker Emb.
35.92
35.93
7.18
1.80
2.87
w/o Cross-Attn
30.49
40.87
6.87
1.54
2.98
w/o Gaze Loss
37.46
42.90
7.33
2.59
2.77
Full Model (Ours)
21.71
22.56
5.97
1.50
2.48
Table 3: Ablation study on different model components. Lower FD, P-FD, and MSE (EXPression, TRANslation or ROTation) indicate better performance when evaluated with groundtruth.
Method
Lip
Speaker M
Listener M
Interact
Gaze
SelfTalk
0.9
0.9
1.6
1.6
2.4
DualTalk
3.9
6.3
7.2
5.6
7.9
Ours (S1)
15.9
19.0
18.2
21.4
21.4
Ours
79.3
73.8
73.0
71.4
68.3
Table 4: Human evaluation preference rate (%). Higher is better. Participants selected the best model from four options.
Method
Cycle Error ↓
LPIPS ↓
SSIM ↑
Linear Map
0.684
0.212
0.912
3-Layer MLP
0.327
0.141
0.963
Table 5: Comparison of expression-space mappings. We report cycle-consistency error (lower is better) and perceptual similarity between rendered faces using LPIPS (lower is better) and SSIM (higher is better). Our three-layer MLP significantly outperforms a linear regression baseline.
Method
tMSE ↓
SAS ↑
Zero-Shot
2.90
3.15
k -NN
1.84
2.65
Few-Shot (Ours)
0.72
4.62
Table 6: Quantitative and human evaluation of LLM-based spatial generation. tMSE measures the distance between predicted and groundtruth translations and the lower the better, while SAS measures perceptual quality (1-5 scale).
Method
FD ↓
P-FD ↓
vMSE-S ↓
vMSE-L ↓
DualTalk
34.12
45.20
8.95
9.12
Ours
14.55
31.05
4.92
6.30
Table 7: Performance comparison on the DualTalk OOD Dataset (384 unseen clips). Our method demonstrates superior generalization on unseen identities compared to the baseline trained specifically on the DualTalk domain. Lower is better for all metrics.