ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation
Authors: Xiangyu Kong, Xiaoyu Jin, Yihan Pan, Haoqin Sun, Hengde Zhu, Xiaoming Xu, Xiaoming Wei, Lu Liu, Siyang Song
Organizations: University of Exeter, Exeter, UK · Meituan, China · College of Computer Science, Nankai University, China · School of Computer Science and Technology, Tongji University, China
In natural face-to-face interaction, participants seamlessly alternate between speaking and listening, producing facial behaviors (FBs) that are finely informed by long-range context and naturally exhibit contextual appropriateness and emotional rationality. Interactive Head Generation (IHG) aims to synthesize lifelike avatar head video emulating such capabilities. Existing IHG methods typically condition on dual-track signals (i.e., human user's behaviors and pre-defined audio for avatar) within a short temporal window, jointly driving generation of avatar's audio-aligned lip articulation and non-verbal FBs. However, two main challenges persist in these methods: (i) the reliance on short-clip behavioral cues without long-range contextual modeling leads them to produce facial behaviors lacking contextual appropriateness; and (ii) the entangled, role-agnostic fusion of dual-track signals empirically introduces cross-signal interference, potentially compromising lip-region synchronization during speaking. To this end, we propose ECHO, a novel IHG framework comprising two key components: a Long-range Contextual Understanding (LCU) component that facilitates contextual understanding of both behavior-grounded dynamics and linguistic-driven affective semantics to promote contextual appropriateness and emotional rationality of synthesized avatar FBs; and a block-wise Spatial-aware Decoupled Cross-attention Modulation (SDCM) module, that preserves self-audio-driven lip articulation while adaptively integrating user contextual behavioral cues for non-lip facial regions, complemented by our designed two-stage training paradigm, to jointly enhance lip synchronization and visual fidelity. Extensive experiments demonstrate the effectiveness of proposed components and ECHO's superior IHG performance.
Figures & tables
Figure 1: Overview of ECHO and pipeline comparison with existing IHG approaches. (a) Approaches [ 56 , 7 , 40 ] that rely exclusively on dual-track audio signals for avatar FBs generation; (b) Approaches [ 34 , 23 ] that condition avatar FBs generation on short-range-only context modeling via role-agnostic fusion; (c) Our ECHO, which performs long-range contextual understanding and region-wise decoupled cross-attention mechanism, achieving contextually appropriate and precisely lip-synchronized avatar FBs.
Figure 2: The pipeline of our proposed ECHO . The proposed LCU component starts by extracting user’s long-range audio-visual behavioral features ( Ausr1:T and Vusr1:T ) to perform low-level perception encoding along with high-level behavioral context understanding (resulting in perception-understanding representation fpu ). Then, linguistic dialogue context is leveraged to infer avatar’s descriptive emotional state (resulting in embeddings cemo ). Subsequently, two obtained representations are utilized as conditioning inputs for proposed block-wise SDCM module to guide avatar FBs generation. Extended pipeline details of avatar generator are provided in Appendix 2.1 .
Figure 3: Qualitative comparison with open-sourced SOTA methods on IHG. Above two dyadic scenarios including active listening/speaking showcase that our proposed ECHO generates avatar head with more context-consistent and emotionally appropriate facial behaviors, and achieves lip articulation with better audio–visual alignment.
Reactiveness
Motion-Richness
Visual-Quality
Lip-Synchronization
Method
rPCC ↓
SID ↑
Var ↑
FID ↓
FVD ↓
CSIM ↑
LSE-D ↓
LSE-C ↑
GT
0.000
3.972
1.658
-
-
0.796
7.790
6.940
DIM [ 41 ]
0.186
1.083
1.206
26.290
-
0.843
-
-
INFP [ 56 ]
-
2.343
1.638
24.551
159.000
0.867
8.027
6.536
ARIG [ 19 ]
0.125
2.428
2.397
21.640
-
0.901
-
-
AvatarForcing [ 23 ]
-
2.442
1.734
24.328
170.874
0.833
8.060
6.723
Table 1: Quantitative comparison of interactive head generation (IHG) on RealTalk [ 15 ] dataset. w/o LPE removes human user’s long-range visual cues along with multimodal integration module Fint in LCU’s low-level perception encoding. w/o HBCU retains LCU’s perception encoding but discards Φ -guided high-level behavior-grounded context understanding. w/o LAU disables dialogue-derived avatar descriptive affective state for guiding FBs generation. Bold : best; Underline : second-best.
Method
Visual Quality
Lip Synchronization
FID ↓
FVD ↓
CSIM ↑
LSE-D ↓
LSE-C ↑
Hallo [ 45 ]
20.545
173.497
0.860
7.819
6.995
SadTalker [ 49 ]
23.340
203.860
0.821
8.046
7.171
Hallo3 [ 11 ]
20.359
160.838
0.865
8.106
7.252
INFP ∗ [ 56 ]
27.155
187.977
0.840
7.810
7.325
ARIG [ 19 ]
18.320
-
0.876
-
-
Table 2: Comparison with talking head generation models on the HDTF [ 50 ] dataset. ∗ denotes the reproduced version reported in [ 23 ] . Bold : best; Underline : second-best.
Method
FD ↓
rPCC ↓
SID ↑
Var ↑
MSE ↓
Exp
Pose
Exp
Pose
Exp
Pose
Exp
Pose
Exp
Pose
ARIG † [ 19 ]
18.39
0.06
0.05
0.01
4.82
3.94
2.91
0.17
-
-
RLHG [ 52 ]
39.02
0.07
0.08
0.02
3.62
3.17
1.52
0.02
0.86
0.01
L2L [ 31 ]
33.93
0.06
0.06
0.08
2.77
2.66
0.83
0.02
0.93
0.01
DIM [ 41 ]
23.88
0.06
0.06
0.03
3.71
2.35
1.53
0.02
0.70
0.01
DualTalk [ 34 ]
22.27
0.05
-
-
-
-
-
-
0.58
0.01
Table 3: Comparison with listening head generation models on the ViCo [ 52 ] dataset. † denotes metrics calculated via extracted 76-d 3DMM coefficients [ 14 ] , whereas others utilize 56-d FLAME [ 25 ] blendshape, thus presented for reference only. ∗ refers to the reproduced version reported in [ 23 ] . Bold : best; Underline : second-best.
Method
E-Consis ↑
E-Appro ↑
DIM
3.271
3.390
DualTalk
3.422
3.692
Ours
3.798
4.073
Table 4: Quantitative evaluation of emotional expressiveness in generated avatar head videos on RealTalk [ 15 ] dataset. Best results are bold , second best are underlined .
Figure 4: Ablation study: (a) Top-left: Ablation on low-level perception encoding by leveraging long-range visual cues; (b) Bottom Left: Ablation on high-level behavior-grounded context understanding; (c) Top-right: Ablation on SDCM module; (d) Bottom-right: Ablation on linguistic-driven affective understanding.
Figure 5: Qualitative comparison with state-of-the-art Interactive Head Generation (IHG) approaches, including INFP [ 56 ] , ARIG [ 19 ] , and StreamAvatar [ 40 ] , respectively. As all compared SOTA methods are not open-sourced, we follow their established visual comparison protocols and evaluate on two samples from the DyConv dataset introduced by INFP [ 56 ] .
Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.
Junjie Chen, Fei Wang, Kun Li +5
Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4
We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting, formulating the problem as an asymmetric task involving speech-constrained articulation and one-to-many listener reactions. To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure and a stochastic residual that models the remaining one-to-many interaction dynamics. On top of this backbone, Motion Memory acts as a training-only regularizer during brief late-stage fine-tuning to provide local priors for weakly conditioned listening windows, while semantic-group scaling controls residual injection across expression, jaw, and neck. This design balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process. Results from unified, state-wise, and ablation evaluations show that conversational 3D motion benefits from decomposing stable and uncertain components rather than applying stochasticity uniformly. ECHO provides a practical formulation and technical basis for deployable conversational digital humans under strict audio-only conditions.
Zhuoqiang Cai, Yujie Sun, Chaoyue Niu +4
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China
We present EmbodiedHead, a speech-driven talking head framework that equips LLMs with real-time visual avatars for conversation. A practical embodied avatar must achieve real-time generation, unified listening-speaking behavior, and high rendered visual quality simultaneously. Our framework couples the first Rectified-Flow Diffusion Transformer (DiT) for this task with a differentiable renderer, enabling diverse, high-fidelity generation in as few as four sampling steps. Prior listening-speaking methods rely on dual-stream audio, introducing an interlocutor look-ahead dependency incompatible with causal user--LLM interaction. We instead adopt a single-stream interface with explicit per-frame listening-speaking state conditioning and a Streaming Audio Scheduler, suppressing spurious mouth motion during listening while enabling seamless turn-taking. A two-stage training scheme of coefficient-space pretraining and joint image-domain refinement further closes the gap between motion-level supervision and rendered quality. Extensive experiments demonstrate state-of-the-art visual quality and motion fidelity in both speaking and listening scenarios. Project page: https://vpx-ecnu.github.io/EmbodiedHead-Website/.
Yu Zhang, Kaiyuan Shen, Yang Li
School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China