Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · SentiPulse · College of Computer Science, Inner Mongolia University
We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at https://sentiavatar.github.io.
Figures & tables
Figure 1: SentiAvatar generates high-quality 3D human motion and expression, which are semantically aligned and frame-level synchronized. The same color indicates the same time step.
Dataset
#Samples
Multi- turn
Role-playing Dialogue
Action Description
Audio
Face
Body Motion
Hands Motion
HumanML3D Guo et al. (2022)
16k
✗
✗
✓
✗
✗
✓
✗
Motion-X Lin et al. (2023)
95k
✗
✗
✓
✗
✗
✓
✓
BEAT Liu et al. (2022a)
2.5k
✗
✗
✓
✓
✓
✓
✗
SnapMoGen Guo et al. (2025)
20k
✓
✗
✓
✗
✗
✓
✗
InterX Xu et al. (2023)
31k
✗
✗
✓
✗
✗
✓
✓
InterHuman Liang et al. (2024)
6k
✗
✗
✓
✗
✗
✓
✗
Table 1: Comparison between SuSuInterActs and existing motion and role-playing dialogue datasets. SuSuInterActs uniquely combines multi-turn dialogue, role-conditioned interactions, and synchronized multimodal signals including text, audio, facial expressions, and full-body motion with hands.
Figure 2: Overview of SuSuInterActs data pipeline. (1) Character design for a consistent persona. (2) LLM script generation with explicit motion/expression tags. (3) Multimodal capture of synchronized speech, full-body motion, and facial expressions.
Statistic
Value
# Total Samples
21,133
Total Duration
36.9 hours
Avg. Duration per Sample
6.3 s
Avg. Characters per Utterance
18.7
# Samples w/ Non-default Expression
9,412
# Samples w/ Non-default Action
14,278
Table 2: Statistics of the SuSuInterActs dataset.
Figure 3: Overview of SentiAvatar. (a) Multi-modal inputs are quantized into tokens via encoders. The body pathway employs a hierarchical generation: (b) LLM planner predicts sparse keyframe tokens to capture high-level dialogue content, followed by (c) an audio-aware Infill Transformer for dense, prosody-driven interpolation to ensure fine-grained temporal synchronization.
Source
# Seqs
Hours
Type
EmbodyAI
84,010
467.7
Conv., Daily
SnapMoGen
20,450
43.7
General
Motion-X
81,084
144.2
Daily, Dance, Sports
Hunyuan Distill.
15,000
20.8
Atomic & Composite
Total
200,544
676.4
—
Table 3: Pre-training data for the Motion Foundation Model. All sequences are retargeted to a unified skeleton and tokenized by the Motion R-VQVAE.
Method
Condition
R@1 ↑
R@2 ↑
R@3 ↑
FID ↓
ESD ↓
Diversity ↑
Real Motion
—
62.20
73.56
78.70
0.000
0.308
22.61
Audio-only methods
EMAGE Liu et al. (2024)
Audio
5.00
9.40
13.32
441.6
0.606
12.92
A2M-GPT †
Audio
8.72
15.96
20.08
13.66
0.477
22.23
Text-only methods
HunYuan-Motion
Text
5.21
8.59
11.9
352.56
0.708
16.92
Table 4: Quantitative comparison on SuSuInterActs. Bold : best; ↑ / ↓ : higher/lower is better. ESD in seconds. “ † ” indicates T2M-GPT variants with token-by-token autoregressive generation.
Method
Condition
FGD ↓
BC ↑
Diversity ↑
DisCo
Audio
9.417
6.439
9.91
CaMN
Audio, Text
6.644
6.769
10.86
TalkSHOW
Audio
6.209
6.947
13.47
EMAGE
Audio, Text
5.512
7.724
13.06
SynTalker
Audio, Text
6.413
7.971
12.72
Language-of-Motion
Audio
5.301
7.780
15.17
Table 5: Comparison on BEATv2. Baseline results from original publications.
Method
Condition
Overall ↑
Semantic ↑
Prosody ↑
HunYuan-Motion
Text
1.88
2.09
2.01
MoMask
Text
2.67
2.79
2.76
EMAGE
Audio
2.48
2.44
2.70
AT2M-GPT
Audio, Text
2.33
2.37
2.45
SentiAvatar (Ours)
Audio, Text
2.99
2.97
3.16
Improvement (%)
+12.0
+6.5
+14.5
Table 6: User study results (1–5 Likert scale, higher is better).
Figure 4: Qualitative comparison of generated motions across methods. Each row shows keyframe sequences for a given motion and speech. Texts and arrows of the same color indicate the same time step. The red arrow indicates an incorrect action.
Variant
R@1 ↑
FID ↓
ESD ↓
Div. ↑
w/o Pre-training
42.56
8.988
0.452
22.42
w/o Infill Transformer
27.52
18.491
0.503
22.36
w/o LLM Planner
28.06
27.567
0.421
22.33
Full pipeline
43.64
8.912
0.456
22.41
Table 7: Architecture ablation on SuSuInterActs.
Variant
R@1 ↑
FID ↓
ESD ↓
Div. ↑
w/o all audio
41.72
9.996
0.523
22.37
w/o audio (Infill)
41.41
10.165
0.497
22.31
w/o audio (LLM)
43.48
9.690
0.517
22.52
Full (both audio)
43.64
8.912
0.456
22.41
Table 8: Ablation on audio conditioning.
Step t
R@1 ↑
FID ↓
ESD ↓
Div. ↑
Token-by-Token
27.52
18.491
0.503
22.36
t=2
34.21
11.773
0.472
22.16
t=4
43.64
8.912
0.456
22.41
t=8
36.44
9.205
0.439
22.63
Table 9: Ablation on keyframe interval (step t ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Attribute
Description
Name
SuSu (user’s cohabiting roommate)
Age
22
Height
165 cm
Education
B.A. in Chinese Language & Literature; strong writing skills
Build
Slender; long limbs, narrow shoulders; movements carry a dance-like fluidity
Appearance
Oval face, almond eyes with naturally curled lashes, straight nose bridge, defined jawline. Fair, cool-toned skin. Expressions shift with a slight “delay,” yielding a soft, natural quality. Gaze conveys layered emotion—calm on the surface, with ripples beneath.
Appendix
Table 10: SuSu: basic attributes.
Aspect
Description
Personality
Quiet and slow to warm up, yet takes initiative once comfortable. Gentle with a playful edge; sweet-tongued without being cloying. Stubborn about small things; occasionally over-serious. Witty and tactful; feigns composure.
Language style
Concise and colloquial. Serious when discussing real matters; slips in dry humor otherwise. Maintains a push-pull dynamic—neither too close nor too distant—conveying affection through indirection, light teasing, and implicit hints rather than explicit statements.
Social habits
Introverted homebody; reluctant to go out, yet naturally warm and approachable in conversation.
Core trait
An inner tension between sensitivity and desire for connection. Uses politeness and emotional distance as self-protection, yet reveals wit, playfulness, and longing for closeness in text and virtual interactions. Interaction pattern: tentative approach then retreat, conveying affection through push-and-pull.
Motion logic
Movements mirror the push-pull core: approach then withdraw, touch then retract. Happy : gentle smile building to laughter, usually a subtle grin. Angry : face goes cold for self-protection, then turns back with a surprise softening once the other party apologizes. Sad : speaks something genuinely moving, then laughs it off as a joke. Every gesture is designed to convey feminine charm and emotional nuance.
Speech rules
Never answers a question head-on; uses metaphor, exaggeration, wordplay, rhetorical questions, or unexpected angles. Each utterance is 10–30 Chinese characters, colloquial and easy to understand. Tone is light and keeps conversations from becoming heavy.
Appendix
Table 11: SuSu: personality and behavioral design. These specifications guide script generation and actor performance to produce diverse yet character-consistent data.
Department of Computing, Imperial College London, UK · Department of Earth Science & Engineering, Imperial College London, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE +8
School of Computer Science and Engineering, South China University of Technology, Guangdong, China · School of Design, South China University of Technology, Guangdong, China · Guangdong Engineering Center for Large Model and GenAI Technology +2