We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting, formulating the problem as an asymmetric task involving speech-constrained articulation and one-to-many listener reactions. To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure and a stochastic residual that models the remaining one-to-many interaction dynamics. On top of this backbone, Motion Memory acts as a training-only regularizer during brief late-stage fine-tuning to provide local priors for weakly conditioned listening windows, while semantic-group scaling controls residual injection across expression, jaw, and neck. This design balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process. Results from unified, state-wise, and ablation evaluations show that conversational 3D motion benefits from decomposing stable and uncertain components rather than applying stochasticity uniformly. ECHO provides a practical formulation and technical basis for deployable conversational digital humans under strict audio-only conditions.
Figures & tables
Figure 1. Teaser of ECHO. Given dual-stream conversational audio, ECHO combines a stable anchor trajectory with stochastic residual refinement. The center curves contrast deterministic, stochastic, and ECHO trajectories.
Figure 2. Overview of ECHO. Given dual-stream audio, ECHO first predicts a deterministic anchor trajectory and then models only the remaining motion with a residual flow branch. During a late fine-tuning stage, Motion Memory regularizes valid listening windows with listener prototypes, while semantic-group residual scaling controls how refinement is distributed across expression, jaw, and neck. Motion Memory is bypassed at inference, so deployment remains strict dual-stream audio-only.
Method
Dynamics & Sync
Global Realism & Coverage
Local FD ↓
Local P-FD ↓
Stochasticity & Speed
MSE ↓
STS ↓
FD ↓
P-FD ↓
rPCC ↓
SID ↑
Exp ↓
Jaw ↓
Neck ↓
Exp ↓
Jaw ↓
Neck ↓
Div ↑
RTF ↓
( ×10−4 )
( ×10−2 )
( ×10−3 )
( ×10−2 )
( ×10−3 )
( ×10−3 )
( ×10−4 )
( ×10−3 )
( ×10−3 )
( ×10−4 )
( ×10−3 )
Deterministic models
UniTalker ( Fan et al., 2024 )
1.78
8.77
9.14
2.51
1.35
2.21
7.58
13.25
14.08
3.90
6.87
6.54
0.00
0.004
DualTalk † ( Peng et al., 2025 )
1.53
8.01
7.93
1.63
1.82
2.03
4.68
6.11
17.09
2.29
3.04
8.24
0.00
0.035
Stochastic models
Table 1. Unified benchmark for dyadic 3D motion generation. Lower is better except SID and Div. DualTalk † is shown as a stronger-input reference since it uses counterpart motion. Bold and underline denote the best and second-best results.
Method
SPEAK
LISTEN
MSE ↓
STS ↓
FD ↓
P-FD ↓
Mode-JSD ↓
Mode-Entropy ↑
Trans-JSD ↓
( ×10−4 )
( ×10−2 )
( ×10−3 )
UniTalker ( Fan et al., 2024 )
1.89
6.33
9.41
2.56
0.165
4.240
0.233
DualTalk † ( Peng et al., 2025 )
1.28
5.00
8.60
1.88
0.204
4.036
0.268
ProbTalk3D ( Wu et al., 2024 )
1.60
5.03
8.57
1.74
0.525
2.175
0.584
DiffPoseTalk ( Sun et al., 2024 )
2.86
7.63
12.70
2.79
0.536
2.093
0.601
Table 2. State-constrained evaluation on ground-truth SPEAK frames and LISTEN segments. Bold and underline denote the best and second-best results.
Variant
Coverage
Global / Paired Realism
Local P-FD
SID ↑
FD ↓
P-FD ↓
Exp ↓
Jaw ↓
Neck ↓
( ×10−2 )
( ×10−3 )
( ×10−3 )
( ×10−3 )
( ×10−4 )
Decomposition
w/o anchor modeling
1.260
13.507
2.872
4.318
6.813
7.03
w/o residual refinement
1.532
5.521
1.264
1.846
2.215
2.14
Conditioning and regularization
Table 3. Ablation study of ECHO. Variants are grouped by decomposition, conditioning/regularization, and sampling. Higher is better for SID; lower is better for all other metrics.
Figure 3. Multi-track temporal comparison with ground-truth states and representative jaw/neck trajectories.
Figure 4. Qualitative comparison on representative speaking and listening moments. The figure highlights lip/jaw fidelity during speech and facial/head responsiveness during listening.
Method
Lip-sync ↑
Interaction realism ↑
Naturalness ↑
UniTalker ( Fan et al., 2024 )
3.03
2.59
2.34
DualTalk ( Peng et al., 2025 )
2.35
1.96
1.71
ProbTalk3D ( Wu et al., 2024 )
1.71
2.11
2.13
DiffPoseTalk ( Sun et al., 2024 )
2.20
1.85
1.77
DIM ( Tran et al., 2024 )
1.96
1.59
1.07
ECHO (Ours)
3.71
3.77
3.82
Table 4. Perceptual user study results (mean MOS, 1–5).
Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.
Junjie Chen, Fei Wang, Kun Li +5
Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4
Dyadic conversational motion generation is essential for realistic interactive digital humans. Existing approaches typically model conversational behaviors within unified dyadic generators. However, such holistic formulations tend to couple self-speech-driven motion with partner-responsive social feedback, leaving the interaction-specific component implicit and underutilizing the speech-motion correspondence already learned by pretrained monologic motion models. We propose Learn2Chat, a unified framework that models dyadic motion as interaction modulation over pretrained monologic motion priors. This design separates intrinsic speech-driven motion from social interaction effects and enables more structured interaction modeling. Specifically, we introduce a Monologic-Anchored Motion Factorization scheme that leverages the semantic motion manifold learned from monologic data to disentangle audio-driven motion dynamics from interaction-induced modulation, yielding clean interaction representations from dyadic sequences. On top of this representation space, a Cross-Attentive Interaction Latent Prediction module maps paired speech signals to interaction latents through cross-branch attention and interaction alignment. During inference, the predicted interaction latents modulate canonical monologic motion to generate coherent and synchronized dyadic behaviors in a data-efficient manner. Extensive experiments on the DualTalk benchmark demonstrate that Learn2Chat achieves state-of-the-art performance across both quantitative metrics and perceptual evaluations. Moreover, the framework is model-agnostic and seamlessly integrates with diverse pretrained monologic motion backbones, highlighting the effectiveness of prior reuse and interaction adaptation for scalable conversational motion generation. More visual results are available on the project page.
Zikai Huang, Siyue Chen, Xuemiao Xu +4
School of Computer Science and Engineering, South China University of Technology, Guangdong, China · School of Design, South China University of Technology, Guangdong, China · Guangdong Engineering Center for Large Model and GenAI Technology +2
Natural human conversation is inherently a real-time interaction in which speech and facial behavior continuously evolve. Enabling such interaction requires a conversational avatar to jointly generate speech and facial motion in real time, prepare facial motion for upcoming speech before the corresponding audio is emitted, and produce non-verbal reactions that reflect the ongoing dialogue. However, existing conversational avatar systems cannot address such requirements: audio-driven methods rely on pre-given speech, while joint streaming generation alone does not ensure anticipatory articulation or semantically appropriate reactions. We propose FacePlex, a unified framework for full-duplex speech-facial motion generation and real-time avatar rendering. FacePlex jointly coordinates speech, facial motion, and Gaussian splatting rendering on a shared streaming timeline. To prepare facial motion for upcoming speech, FacePlex predicts a short speech continuation and uses its future hidden states through asymmetric conditioning and denoising, while continuously updating them as new user audio arrives without observing future user input. For dialogue-grounded non-verbal behavior, we construct SemReact, a dataset aligning dialogue context, reaction semantics, and facial motion, and introduce a semantic behavior router that guides continuous motion during both speaking and listening. Extensive experiments show improved audio-visual synchronization and facial articulation, natural dialogue-grounded non-verbal responses, and low-latency of 122 ms end-to-end avatar interaction.