Generating realistic 3D head motion for dyadic interactions is a significant challenge in virtual human synthesis. While recent methods achieve impressive results with speaking heads, they frequently suffer from the `Regression-to-the-Mean' problem in listener motions, collapsing into static faces, and lack the parameter space for complex nonverbal motions. In this paper, we propose GDPO-Listener, a novel framework that achieves highly expressive speaking and listening motion generation. First, we introduce an Auto-Regressive Flow Matching architecture enabling stable supervised learning. Second, to overcome kinematic stillness, we apply the Group reward-Decoupled Policy Optimization (GDPO). By isolating reward normalization across distinct FLAME parameter groups, GDPO explicitly incentivizes high variance expressive generations. Finally, we enable explicit semantic text control for customizable responses. Extensive evaluations across the Seamless Interaction and DualTalk datasets demonstrate superior performance compared to existing baselines on long-term kinematic variance, visual expressivity and semantic controllability.
Figures & tables
Figure 1: GDPO-Listener . Our framework generates expressive speaking and listening head reactions from multimodal dyadic conversational inputs. By utilizing an expanded FLAME parameter space, it naturally supports eye blinking and head nodding. Furthermore, it enables explicit semantic text control to ensure contextually appropriate responses and maintains stable dynamics during long-sequence inference.
Figure 2: Necessity of Our Method . We resolve three critical gaps in previous baselines. (Left) Older methods suffer from static mean-collapse, but we synthesize high-variance expressive reactions. (Middle) Baselines cannot produce complex motions, whereas we enable natural eye blinks and head nods. (Right) Prior models often have no semantic guidance, but our text control ensures contextually appropriate responses.
Figure 3: GDPO-Listener Architecture . Our framework has two training stages. (a) Supervised Learning. Multimodal inputs are encoded as prefix conditions, and an Auto-Regressive Flow Matching model iteratively predicts actor motion latents from noise and history via ODE sampling. (b) Reinforcement Learning. We then post-train the policy model via GDPO. We compute fine-grained, decoupled rewards for distinct FLAME parameters under SDE sampling to explicitly optimize expressiveness.
Speaking
Listening
Methods
LVE ↓
MHD ↓
FDD ↓
PDD ↓
JDD ↓
FDD ↓
PDD ↓
JDD ↓
DiffposeTalk [ 27 ]
9.48
2.96
32.66
7.89
1.40
-
-
-
L2L [ 14 ]
3.10
1.25
53.03
3.52
3.32
25.95
2.18
1.14
ARTalk [ 2 ]
7.46
2.12
31.64
7.66
1.19
-
-
-
ARTalk* [ 2 ]
6.79
2.02
27.41
8.55
0.81
30.62
9.52
1.53
DualTalk [ 18 ]
6.53
1.95
37.46
9.70
1.02
43.58
10.71
2.02
Table 1: Quantitative Analysis on Seamless Interaction Dataset. ∗ means adapting the ARTalk for speak-listen generation by adding additional partner audio input.
All Listening
Expressive Listening
Methods
FDD ↓
PDD ↓
JDD ↓
FDD ↓
PDD ↓
JDD ↓
L2L [ 14 ]
77.04
6.24
4.26
117.57
8.57
6.65
DualTalk [ 18 ]
46.36
5.07
2.83
53.04
5.01
3.32
GDPO-Listener
26.95
3.86
1.62
33.99
3.64
1.94
Table 3: Quantitative Analysis on DualTalk Dataset & Expressive Subset.
Figure 4: Qualitative Comparisons . Other methods have low-expressive speaking and static listening, our method shows better lip sync and highly expressive reactions.
Figure 5: Advanced Generation Capabilities . (Top) Semantic text explicitly controls emotional states. (Middle) We sustain dynamic reactions during long sequences, avoiding baseline static decay. (Bottom) CFG scaling seamlessly modulates expressiveness intensity without retraining.
Methods
Lip Sync
Naturalness
Liveliness
Diversity
L2L
1.9%
5.6%
0.7%
2.2%
DualTalk
27.8%
30.0%
17.0%
20.4%
GDPO-Listener
70.4%
64.4%
82.2%
77.4%
Table 5: User study results with 27 participants from Prolific. Numbers (%) indicate the proportion of users who prefer that method over all methods.
Figure 6: The interface of our user study.
FD ↓
P-FD ↓
MSE ↓
SID ↑
rPCC ↓
Methods
Exp
Jaw
Neck
Exp
Jaw
Neck
Exp
Jaw
Neck
Exp
Jaw
Neck
Exp
Jaw
Neck
Speaking
DualTalk
93.94
7.49
8.28
94.92
7.58
8.44
19.43
2.89
3.15
0.93
1.19
0.87
0.20
0.21
0.55
GDPO-Listener
23.74
3.14
5.86
24.91
3.21
5.98
5.51
1.30
2.22
2.26
1.61
1.07
0.15
0.18
0.48
Listening
DualTalk
94.05
7.16
9.91
95.06
7.28
10.08
19.16
2.63
3.53
0.69
0.68
0.48
0.19
0.28
0.53
Table 6: Additional Quantitative Analysis on Seamless Interaction Dataset. Following DualTalk, for better readability, FD, P-FD and MSE for pose are multiplied by 100; FD, P-FD and MSE for jaw are multiplied by 1000 and MSE for expression is multiplied by 10.
Qwen3-VL System Prompt
Describe the person’s facial expression in a single concise sentence.
1. The specific Emotion: Use precise words like ‘skeptical’, ‘amused’, ‘bored’, ‘attentive’, ‘confused’, ‘happy’, ‘angry’, ‘calm’, or ‘focused’. (Avoid generic ‘neutral’).
2. The specific Facial Geometry and Gaze like ‘narrowed eyes’, ‘raised eyebrows’, ‘head tilted’, ‘looking away’.
Rules:
1. IGNORE mouth movements related to speech.
Table 7: The explicit system prompt provided to Qwen3-VL to extract decoupled expressive text descriptions from our sampled 8-frame clips.
Hefei University of Technology · EPIC Lab, Shanghai Jiao Tong University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +4
School of Computer Science and Engineering, South China University of Technology, Guangdong, China · School of Design, South China University of Technology, Guangdong, China · Guangdong Engineering Center for Large Model and GenAI Technology +2