Organizations: Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China. · Shenzhen University, Shenzhen, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Pengcheng Laboratory, Shenzhen, China. · Tongji University, Shanghai, China. · Tsinghua University, Beijing, China. · The Chinese University of Hong Kong, Hong Kong, China. · University of Trento, Trento, Italy. · Huawei, Shenzhen, China.
Visual intelligence seeks to perceive, interpret, and synthesize the visual world and is central to modern computer vision. Human-centered visual intelligence is especially demanding because it studies people as expressive, socially situated subjects whose meaning is rarely conveyed by appearance alone. It couples vision with audio and language across four representative tasks: human emotion recognition, human video generation, human voice cloning, and human video matting. Yet existing resources remain task-specific, providing modalities and annotations for individual problems rather than a shared foundation coordinating understanding and generation. This limits multimodal signal use and broader research. We address this gap with HUG-VIS, a unified benchmark for Human-centered Understanding and Generation in Visual Intelligence. It contains 8,400 seated half-body videos of 30 professional actors, each performing the same 280 emotion-action-prompt assignments under a controlled Mandarin studio protocol, with synchronized video, audio, text, and alpha mattes. We evaluate diverse open- and closed-source models across the four tasks under a unified zero-shot protocol using automatic metrics, criterion-specific mean opinion scores, and multiple cross-task analyses. Results show that (i) linguistic content dominates current emotion recognition, while purely visual affect recognition is weakest; (ii) in video generation and voice cloning, automatic metrics and human judgment agree overall but differ in their top rankings, requiring joint reporting; (iii) boundary fidelity under motion is the main remaining obstacle for human matting; and (iv) task difficulty varies across emotions, models, and metrics, with notable cross-task correlations. The dataset and results are available at https://github.com/GML-MMGroup/HUG-VIS.
Figures & tables
Figure 1: Overview of the HUG-VIS construction and evaluation pipeline. Phases 1–3 establish the shared grid and package temporally synchronized multimodal assets (RGB video, noise-suppressed audio, prompt and transcript, and alpha mattes), while Phase 4 conducts the evaluation of the four understanding and generation tasks together with the cross-task analysis.
Condition
Movement style
Action templates
Illustrative prompt
Happy
Light, relaxed, and expansive, with an open posture
“I finally got a ticket to the concert! I’m thrilled!”
Angry
Fast, forceful, and sustained, with a forward-oriented posture
Table strike; forceful arm sweep; sharp pointing gesture; clenched-fist raise and drop
“Enough! I don’t want to hear another excuse.”
Sad
Slow or paused, with a contracted, downward-oriented posture
Head or neck support with one or both hands; hand on head; self-embrace with hands on upper arms
“Everything I’d hoped for has fallen apart, and now I’m all alone.”
Afraid
Contracted, guarded, and withdrawing, with slight trembling
One- or two-handed guarding gesture; recoil; hands clasped close to the body while leaning backward
“Stay back! Please don’t hurt me!”
Disgusted
Restrained and outwardly rejecting, with a backward lean
Two-handed push-away; one-handed wave or push-away; dismissive wave; shooing gesture
“Get this away from me! I can’t stand looking at it.”
Surprised
Rapid and moderately expansive, ending in an abrupt freeze
Sudden raising of both hands; hands brought behind the head; sudden torso lean; one-hand raise followed by a freeze
“There was nothing behind the curtain. So why is that shadow moving?”
Table 1: Emotion-conditioned performance design in HUG-VIS, showing the intended movement style, action templates, and an illustrative prompt for each emotion. The example prompts are English translations of the Mandarin prompts.
Item
Specification
RGB camera
DJI Action 5 Pro, fixed frontal mount
Background
Uniform green screen for chroma-key matting
Lighting
JHC-2000S LED, fixed color temperature and intensity
Table 2: HUG-VIS acquisition setup and dataset composition.
Figure 2: Samples illustrating the controlled correspondence of the actor-by-assignment grid, with each row showing ordered frames of the rest–expression–rest protocol. The indicated row pairs vary a factor at a time (actor, emotion, action, or all three), so that downstream differences are attributable to the actor, model, or criterion rather than mismatched material.
Figure 3: Acquisition setup of HUG-VIS: Region 1 shows the green-screen environment and equipment, Region 2 the front view configuration and instruction display, and Region 3 the prompt-guided recording with ordered key-gesture frames and synchronized speech. All clips are captured at 1920 × 1080 and 240 FPS (released at 30 FPS) with synchronized audio.
Resource
Emotion annotation
Half-body capture
Alpha supervision
A/V/T alignment
Condition-aligned cross-task analysis
MEAD ( Wang et al., 2020a )
✓
✗
✗
✓
✗
HDTF ( Zhang et al., 2021 )
✗
✗
✗
✗
✗
BEAT ( Liu et al., 2022 )
✓
✓
✗
✗
✗
IEMOCAP ( Busso et al., 2008 )
✓
✓
✗
✓
✗
VCTK ( Yamagishi et al., 2019 )
✗
✗
✗
✗
✗
VideoMatte240K ( Lin et al., 2021 )
✗
✓
✓
✗
✗
Table 3: Comparison of HUG-VIS with representative datasets along five properties: emotion annotation, half-body capture, alpha supervision, audio–video–text (A-V-T) alignment, and condition-aligned cross-task analysis. ✓ denotes that the released resource satisfies the stated criterion, while ✗ denotes that it does not.
Model
Input
Acc. (%) ↑
Image frame
Former-DFER ( Zhao and Liu, 2021 )
Image frame
11.42
Emotion-FAN ( Meng et al., 2019 )
Image frame
13.90
EmotiEffLib ( Sber AI Lab, 2025 ; Savchenko, 2023 )
Image frame
33.37
MMA-DFER ( Chumachenko et al., 2024 )
Image frame
37.05
Video
Table 4: Seven-class emotion recognition accuracy. Bold and underlined values indicate the best and second-best results.
Model
CSIM ↑
Sync-C ↑
Sync-D ↓
EDTalk ( Tan et al., 2024 )
0.710
3.67
8.90
V-Express ( Wang et al., 2024 )
0.786
4.95
8.10
EchoMimicV3 ( Meng et al., 2026 )
0.771
3.15
10.30
AniTalker ( Liu et al., 2024 )
0.744
3.51
9.93
Hallo2 ( Cui et al., 2025 )
0.868
4.65
8.43
Ditto ( Li et al., 2025 )
0.904
3.47
9.31
Table 5: Objective results of the task of audio-driven video generation. Bold and underlined values indicate the best and second-best results.
Model
ID Similarity ↑
Emotion Naturalness ↑
Lip Synchronization ↑
EDTalk ( Tan et al., 2024 )
1.22
1.24
1.33
V-Express ( Wang et al., 2024 )
1.36
1.33
3.40
EchoMimicV3 ( Meng et al., 2026 )
2.93
2.07
2.24
AniTalker ( Liu et al., 2024 )
3.74
2.89
2.78
Hallo2 ( Cui et al., 2025 )
3.69
2.91
2.96
Ditto ( Li et al., 2025 )
4.13
3.02
2.94
Table 6: Subjective results of the task of audio-driven video generation. Bold and underlined values indicate the best and second-best results.
Figure 4: Qualitative comparison of audio-driven video generation for one Happy utterance, sampled at six ordered time points. The top panel shows the source portrait, transcript, and driving audio waveform, and each row shows the temporally aligned outputs of one model.
Model
LPIPS ↓
CSIM ↑
PSNR ↑
SSIM ↑
FID ↓
Open-source
Head Animation
HunyuanPortrait ( Xu et al., 2025b )
0.568
0.868
9.11
0.375
175.71
X-NeMo ( Zhao et al., 2025b )
0.520
0.678
10.15
0.368
150.57
PersonaLive! ( Li et al., 2026 )
0.565
0.834
9.43
0.382
164.94
AniPortrait ( Wei et al., 2024 )
0.565
0.884
9.43
0.383
167.49
Table 7: Objective results of the task of vision-driven video generation. Bold and underlined values indicate the best and second-best results.
Model
ID Similarity ↑
Emotion Naturalness ↑
Motion Naturalness ↑
Open-source
Head Animation
HunyuanPortrait ( Xu et al., 2025b )
4.15
3.55
3.66
X-NeMo ( Zhao et al., 2025b )
4.16
3.68
3.82
PersonaLive! ( Li et al., 2026 )
4.41
3.67
3.63
AniPortrait ( Wei et al., 2024 )
4.04
3.11
3.22
Table 8: Subjective results of the task of vision-driven video generation. Bold and underlined values indicate the best and second-best results.
Figure 5: Qualitative comparison of vision-driven video generation for one Angry clip, with the source frame and driving motion sequence on top and temporally aligned outputs of each model below, grouped into open-source and closed-source.
Figure 6: Fine-grained hand-region failures in vision-driven synthesis, highlighting fused, extra, blurred, distorted, and missing fingers or hands across six systems via enlarged insets.
Model
UTMOS ↑
DNSMOS ↑
Speaker Sim. ↑
Reference
Real audio
1.90
2.84
0.990
Open-source models
GPT-SoVITS v2 ( RVC-Boss, 2024 )
2.26
2.76
0.770
IndexTTS2 ( Zhou et al., 2026a )
2.00
2.94
0.846
CosyVoice 3 ( Du et al., 2025 )
2.30
2.94
0.856
Table 9: Objective results of voice cloning. Bold and underlined values indicate the best and second-best results.
Model
ID Similarity ↑
Emotion Naturalness ↑
Open-source models
GPT-SoVITS v2 ( RVC-Boss, 2024 )
2.99
3.12
IndexTTS2 ( Zhou et al., 2026a )
4.24
4.40
CosyVoice 3 ( Du et al., 2025 )
3.81
4.29
OpenAudio S1 ( Fish Audio, 2025 )
3.64
4.04
Closed-source models
Table 10: Subjective results for voice cloning. Bold and underlined values indicate the best and second-best results.
Model
MAD ↓
MSE ↓
dtSSD ↓
Grad ↓
Conn ↓
VideoMaMa ( Lim et al., 2026 )
34.09
20.43
3.53
26.14
64.38
MODNet ( Ke et al., 2022 )
24.98
20.27
3.84
26.75
47.84
SparseMat ( Sun et al., 2023 )
9.14
4.27
3.19
28.83
14.79
U 2 -Net ( Qin et al., 2020 )
6.78
2.49
2.76
33.11
10.33
InSPyReNet ( Kim et al., 2022 )
4.33
2.52
3.58
31.79
5.77
SAM 3 ( Carion et al., 2026 )
3.92
2.13
3.40
28.01
4.88
Table 11: Objective results of video matting. Bold and underlined values indicate the best and second-best results.
Figure 7: Qualitative comparison of human video matting on one green-screen sequence at seven time points, with rows showing the RGB input, the reference alpha, and the nine model predictions. Red visualization highlights boundary discrepancies relative to the reference alpha.
Figure 8: Emotion difficulty across eight metrics, each averaged over models, oriented so that larger values indicate greater difficulty, and independently min–max normalized over the seven emotions. Values of 0 and 1 denote the easiest and hardest conditions, respectively, and boxes mark column maxima.
Figure 9: Pairwise Spearman rank correlations between the emotion difficulty profiles of the eight metrics. Positive and negative values indicate positive and negative correlations, respectively.
Figure 10: Source-level difficulty of the audio-driven and vision-driven methods on the the common set of valid clips. Hexagon color encodes the number of samples per bin, the dashed line denotes equality, and the red line shows the ordinary least-squares (OLS) fit.
Figure 11: Relationship between motion difficulty and dtSSD, with the line denoting the ordinary least-squares (OLS) fit.