ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation

Xiangyu Kong; Xiaoyu Jin; Yihan Pan; Haoqin Sun; Hengde Zhu; Xiaoming Xu; Xiaoming Wei; Lu Liu

cs.CVMar 18, 2026

ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation

Authors: Xiangyu Kong, Xiaoyu Jin, Yihan Pan, Haoqin Sun, Hengde Zhu, Xiaoming Xu, Xiaoming Wei, Lu Liu,

Organizations: University of Exeter, Exeter, UK · Meituan, China · College of Computer Science, Nankai University, China · School of Computer Science and Technology, Tongji University, China

Abstract

In natural face-to-face interaction, participants seamlessly alternate between speaking and listening, producing facial behaviors (FBs) that are finely informed by long-range context and naturally exhibit contextual appropriateness and emotional rationality. Interactive Head Generation (IHG) aims to synthesize lifelike avatar head video emulating such capabilities. Existing IHG methods typically condition on dual-track signals (i.e., human user's behaviors and pre-defined audio for avatar) within a short temporal window, jointly driving generation of avatar's audio-aligned lip articulation and non-verbal FBs. However, two main challenges persist in these methods: (i) the reliance on short-clip behavioral cues without long-range contextual modeling leads them to produce facial behaviors lacking contextual appropriateness; and (ii) the entangled, role-agnostic fusion of dual-track signals empirically introduces cross-signal interference, potentially compromising lip-region synchronization during speaking. To this end, we propose ECHO, a novel IHG framework comprising two key components: a Long-range Contextual Understanding (LCU) component that facilitates contextual understanding of both behavior-grounded dynamics and linguistic-driven affective semantics to promote contextual appropriateness and emotional rationality of synthesized avatar FBs; and a block-wise Spatial-aware Decoupled Cross-attention Modulation (SDCM) module, that preserves self-audio-driven lip articulation while adaptively integrating user contextual behavioral cues for non-lip facial regions, complemented by our designed two-stage training paradigm, to jointly enhance lip synchronization and visual fidelity. Extensive experiments demonstrate the effectiveness of proposed components and ECHO's superior IHG performance.

Figures & tables

Figure 1: Overview of ECHO and pipeline comparison with existing IHG approaches. (a) Approaches [ 56 , 7 , 40 ] that rely exclusively on dual-track audio signals for avatar FBs generation; (b) Approaches [ 34 , 23 ] that condition avatar FBs generation on short-range-only context modeling via role-agnostic fusion; (c) Our ECHO, which performs long-range contextual understanding and region-wise decoupled cross-attention mechanism, achieving contextually appropriate and precisely lip-synchronized avatar FBs.

Figure 2: The pipeline of our proposed ECHO . The proposed LCU component starts by extracting user’s long-range audio-visual behavioral features (

\mathbf{A}^{1:T}_{\text{usr}}

and

\mathbf{V}^{1:T}_{\text{usr}}

) to perform low-level perception encoding along with high-level behavioral context understanding (resulting in perception-understanding representation

\textbf{f}_{\text{pu}}

). Then, linguistic dialogue context is leveraged to infer avatar’s descriptive emotional state (resulting in embeddings

\mathbf{c}^{\text{emo}}

). Subsequently, two obtained representations are utilized as conditioning inputs for proposed block-wise SDCM module to guide avatar FBs generation. Extended pipeline details of avatar generator are provided in Appendix 2.1 .

Figure 3: Qualitative comparison with open-sourced SOTA methods on IHG. Above two dyadic scenarios including active listening/speaking showcase that our proposed ECHO generates avatar head with more context-consistent and emotionally appropriate facial behaviors, and achieves lip articulation with better audio–visual alignment.

	Reactiveness	Motion-Richness		Visual-Quality			Lip-Synchronization
Method	rPCC $\downarrow$	SID $\uparrow$	Var $\uparrow$	FID $\downarrow$	FVD $\downarrow$	CSIM $\uparrow$	LSE-D $\downarrow$	LSE-C $\uparrow$
GT	0.000	3.972	1.658	-	-	0.796	7.790	6.940
DIM [ 41 ]	0.186	1.083	1.206	26.290	-	0.843	-	-
INFP [ 56 ]	-	2.343	1.638	24.551	159.000	0.867	8.027	6.536
ARIG [ 19 ]	0.125	2.428	2.397	21.640	-	0.901	-	-
AvatarForcing [ 23 ]	-	2.442	1.734	24.328	170.874	0.833	8.060	6.723

Table 1: Quantitative comparison of interactive head generation (IHG) on RealTalk [ 15 ] dataset. w/o LPE removes human user’s long-range visual cues along with multimodal integration module

\mathcal{F}_{\text{int}}

in LCU’s low-level perception encoding. w/o HBCU retains LCU’s perception encoding but discards

\Phi

-guided high-level behavior-grounded context understanding. w/o LAU disables dialogue-derived avatar descriptive affective state for guiding FBs generation. Bold : best; Underline : second-best.

Method	Visual Quality			Lip Synchronization
Method	FID $\downarrow$	FVD $\downarrow$	CSIM $\uparrow$	LSE-D $\downarrow$	LSE-C $\uparrow$
Hallo [ 45 ]	20.545	173.497	0.860	7.819	6.995
SadTalker [ 49 ]	23.340	203.860	0.821	8.046	7.171
Hallo3 [ 11 ]	20.359	160.838	0.865	8.106	7.252
INFP ∗ [ 56 ]	27.155	187.977	0.840	7.810	7.325
ARIG [ 19 ]	18.320	-	0.876	-	-

Table 2: Comparison with talking head generation models on the HDTF [ 50 ] dataset. ∗ denotes the reproduced version reported in [ 23 ] . Bold : best; Underline : second-best.

Method	FD $\downarrow$		rPCC $\downarrow$		SID $\uparrow$		Var $\uparrow$		MSE $\downarrow$
Method	Exp	Pose	Exp	Pose	Exp	Pose	Exp	Pose	Exp	Pose
ARIG † [ 19 ]	18.39	0.06	0.05	0.01	4.82	3.94	2.91	0.17	-	-
RLHG [ 52 ]	39.02	0.07	0.08	0.02	3.62	3.17	1.52	0.02	0.86	0.01
L2L [ 31 ]	33.93	0.06	0.06	0.08	2.77	2.66	0.83	0.02	0.93	0.01
DIM [ 41 ]	23.88	0.06	0.06	0.03	3.71	2.35	1.53	0.02	0.70	0.01
DualTalk [ 34 ]	22.27	0.05	-	-	-	-	-	-	0.58	0.01

Table 3: Comparison with listening head generation models on the ViCo [ 52 ] dataset. † denotes metrics calculated via extracted 76-d 3DMM coefficients [ 14 ] , whereas others utilize 56-d FLAME [ 25 ] blendshape, thus presented for reference only. ∗ refers to the reproduced version reported in [ 23 ] . Bold : best; Underline : second-best.

Method	E-Consis $\uparrow$	E-Appro $\uparrow$
DIM	3.271	3.390
DualTalk	3.422	3.692
Ours	3.798	4.073

Table 4: Quantitative evaluation of emotional expressiveness in generated avatar head videos on RealTalk [ 15 ] dataset. Best results are bold , second best are underlined .

Figure 4: Ablation study: (a) Top-left: Ablation on low-level perception encoding by leveraging long-range visual cues; (b) Bottom Left: Ablation on high-level behavior-grounded context understanding; (c) Top-right: Ablation on SDCM module; (d) Bottom-right: Ablation on linguistic-driven affective understanding.

Figure 5: Qualitative comparison with state-of-the-art Interactive Head Generation (IHG) approaches, including INFP [ 56 ] , ARIG [ 19 ] , and StreamAvatar [ 40 ] , respectively. As all compared SOTA methods are not open-sourced, we follow their established visual comparison protocols and evaluate on two samples from the DyConv dataset introduced by INFP [ 56 ] .

Talking Head Generation

Talking head generation creates animated faces and avatars that speak, listen, and express emotions from audio, text, images, or video.

See all 41 papers

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold
Sep 28, 2026Junjie Chen, Fei Wang, Kun Li +5Talking Head Generation 3D Avatar Generation
Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4
ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction
Aug 29, 2026Zhuoqiang Cai, Yujie Sun, Chaoyue Niu +4Human Avatar Animation Audio-Driven Facial Animation
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China
EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents
Apr 19, 2026Yu Zhang, Kaiyuan Shen, Yang LiStreaming Video Generation Talking Head Generation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China

Sep 28, 2026J/K move · Enter open · S save

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold

Junjie Chen, Fei Wang, Kun Li, Yiqi Nie+4

Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center+4

Talking Head Generation 3D Avatar Generation Audio-Driven Facial Animation Test-Time Training

Open paper AskPDF arXiv

Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.

Abstract

Talking Head Generation

Explore similar work

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold

ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction

EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents