Generating realistic 3D head motion for dyadic interactions is a significant challenge in virtual human synthesis. While recent methods achieve impressive results with speaking heads, they frequently suffer from the `Regression-to-the-Mean' problem in listener motions, collapsing into static faces, and lack the parameter space for complex nonverbal motions. In this paper, we propose GDPO-Listener, a novel framework that achieves highly expressive speaking and listening motion generation. First, we introduce an Auto-Regressive Flow Matching architecture enabling stable supervised learning. Second, to overcome kinematic stillness, we apply the Group reward-Decoupled Policy Optimization (GDPO). By isolating reward normalization across distinct FLAME parameter groups, GDPO explicitly incentivizes high variance expressive generations. Finally, we enable explicit semantic text control for customizable responses. Extensive evaluations across the Seamless Interaction and DualTalk datasets demonstrate superior performance compared to existing baselines on long-term kinematic variance, visual expressivity and semantic controllability.
Figures & tables
Figure 1: GDPO-Listener . Our framework generates expressive speaking and listening head reactions from multimodal dyadic conversational inputs. By utilizing an expanded FLAME parameter space, it naturally supports eye blinking and head nodding. Furthermore, it enables explicit semantic text control to ensure contextually appropriate responses and maintains stable dynamics during long-sequence inference.
Figure 2: Necessity of Our Method . We resolve three critical gaps in previous baselines. (Left) Older methods suffer from static mean-collapse, but we synthesize high-variance expressive reactions. (Middle) Baselines cannot produce complex motions, whereas we enable natural eye blinks and head nods. (Right) Prior models often have no semantic guidance, but our text control ensures contextually appropriate responses.
Figure 3: GDPO-Listener Architecture . Our framework has two training stages. (a) Supervised Learning. Multimodal inputs are encoded as prefix conditions, and an Auto-Regressive Flow Matching model iteratively predicts actor motion latents from noise and history via ODE sampling. (b) Reinforcement Learning. We then post-train the policy model via GDPO. We compute fine-grained, decoupled rewards for distinct FLAME parameters under SDE sampling to explicitly optimize expressiveness.
Speaking
Listening
Methods
LVE ↓
MHD ↓
FDD ↓
PDD ↓
JDD ↓
FDD ↓
PDD ↓
JDD ↓
DiffposeTalk [ 27 ]
9.48
2.96
32.66
7.89
1.40
-
-
-
L2L [ 14 ]
3.10
1.25
53.03
3.52
3.32
25.95
2.18
1.14
ARTalk [ 2 ]
7.46
2.12
31.64
7.66
1.19
-
-
-
ARTalk* [ 2 ]
6.79
2.02
27.41
8.55
0.81
30.62
9.52
1.53
DualTalk [ 18 ]
6.53
1.95
37.46
9.70
1.02
43.58
10.71
2.02
Table 1: Quantitative Analysis on Seamless Interaction Dataset. ∗ means adapting the ARTalk for speak-listen generation by adding additional partner audio input.
All Listening
Expressive Listening
Methods
FDD ↓
PDD ↓
JDD ↓
FDD ↓
PDD ↓
JDD ↓
L2L [ 14 ]
77.04
6.24
4.26
117.57
8.57
6.65
DualTalk [ 18 ]
46.36
5.07
2.83
53.04
5.01
3.32
GDPO-Listener
26.95
3.86
1.62
33.99
3.64
1.94
Table 3: Quantitative Analysis on DualTalk Dataset & Expressive Subset.
Figure 4: Qualitative Comparisons . Other methods have low-expressive speaking and static listening, our method shows better lip sync and highly expressive reactions.
Figure 5: Advanced Generation Capabilities . (Top) Semantic text explicitly controls emotional states. (Middle) We sustain dynamic reactions during long sequences, avoiding baseline static decay. (Bottom) CFG scaling seamlessly modulates expressiveness intensity without retraining.
Methods
Lip Sync
Naturalness
Liveliness
Diversity
L2L
1.9%
5.6%
0.7%
2.2%
DualTalk
27.8%
30.0%
17.0%
20.4%
GDPO-Listener
70.4%
64.4%
82.2%
77.4%
Table 5: User study results with 27 participants from Prolific. Numbers (%) indicate the proportion of users who prefer that method over all methods.
Figure 6: The interface of our user study.
FD ↓
P-FD ↓
MSE ↓
SID ↑
rPCC ↓
Methods
Exp
Jaw
Neck
Exp
Jaw
Neck
Exp
Jaw
Neck
Exp
Jaw
Neck
Exp
Jaw
Neck
Speaking
DualTalk
93.94
7.49
8.28
94.92
7.58
8.44
19.43
2.89
3.15
0.93
1.19
0.87
0.20
0.21
0.55
GDPO-Listener
23.74
3.14
5.86
24.91
3.21
5.98
5.51
1.30
2.22
2.26
1.61
1.07
0.15
0.18
0.48
Listening
DualTalk
94.05
7.16
9.91
95.06
7.28
10.08
19.16
2.63
3.53
0.69
0.68
0.48
0.19
0.28
0.53
Table 6: Additional Quantitative Analysis on Seamless Interaction Dataset. Following DualTalk, for better readability, FD, P-FD and MSE for pose are multiplied by 100; FD, P-FD and MSE for jaw are multiplied by 1000 and MSE for expression is multiplied by 10.
Qwen3-VL System Prompt
Describe the person’s facial expression in a single concise sentence.
1. The specific Emotion: Use precise words like ‘skeptical’, ‘amused’, ‘bored’, ‘attentive’, ‘confused’, ‘happy’, ‘angry’, ‘calm’, or ‘focused’. (Avoid generic ‘neutral’).
2. The specific Facial Geometry and Gaze like ‘narrowed eyes’, ‘raised eyebrows’, ‘head tilted’, ‘looking away’.
Rules:
1. IGNORE mouth movements related to speech.
Table 7: The explicit system prompt provided to Qwen3-VL to extract decoupled expressive text descriptions from our sampled 8-frame clips.
Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.
Junjie Chen, Fei Wang, Kun Li +5
Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4
We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting, formulating the problem as an asymmetric task involving speech-constrained articulation and one-to-many listener reactions. To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure and a stochastic residual that models the remaining one-to-many interaction dynamics. On top of this backbone, Motion Memory acts as a training-only regularizer during brief late-stage fine-tuning to provide local priors for weakly conditioned listening windows, while semantic-group scaling controls residual injection across expression, jaw, and neck. This design balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process. Results from unified, state-wise, and ablation evaluations show that conversational 3D motion benefits from decomposing stable and uncertain components rather than applying stochasticity uniformly. ECHO provides a practical formulation and technical basis for deployable conversational digital humans under strict audio-only conditions.
Zhuoqiang Cai, Yujie Sun, Chaoyue Niu +4
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China
Dyadic conversational motion generation is essential for realistic interactive digital humans. Existing approaches typically model conversational behaviors within unified dyadic generators. However, such holistic formulations tend to couple self-speech-driven motion with partner-responsive social feedback, leaving the interaction-specific component implicit and underutilizing the speech-motion correspondence already learned by pretrained monologic motion models. We propose Learn2Chat, a unified framework that models dyadic motion as interaction modulation over pretrained monologic motion priors. This design separates intrinsic speech-driven motion from social interaction effects and enables more structured interaction modeling. Specifically, we introduce a Monologic-Anchored Motion Factorization scheme that leverages the semantic motion manifold learned from monologic data to disentangle audio-driven motion dynamics from interaction-induced modulation, yielding clean interaction representations from dyadic sequences. On top of this representation space, a Cross-Attentive Interaction Latent Prediction module maps paired speech signals to interaction latents through cross-branch attention and interaction alignment. During inference, the predicted interaction latents modulate canonical monologic motion to generate coherent and synchronized dyadic behaviors in a data-efficient manner. Extensive experiments on the DualTalk benchmark demonstrate that Learn2Chat achieves state-of-the-art performance across both quantitative metrics and perceptual evaluations. Moreover, the framework is model-agnostic and seamlessly integrates with diverse pretrained monologic motion backbones, highlighting the effectiveness of prior reuse and interaction adaptation for scalable conversational motion generation. More visual results are available on the project page.
Zikai Huang, Siyue Chen, Xuemiao Xu +4
School of Computer Science and Engineering, South China University of Technology, Guangdong, China · School of Design, South China University of Technology, Guangdong, China · Guangdong Engineering Center for Large Model and GenAI Technology +2