Organizations: Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China. · Shenzhen University, Shenzhen, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Pengcheng Laboratory, Shenzhen, China. · Tongji University, Shanghai, China. · Tsinghua University, Beijing, China. · The Chinese University of Hong Kong, Hong Kong, China. · University of Trento, Trento, Italy. · Huawei, Shenzhen, China.
Visual intelligence seeks to perceive, interpret, and synthesize the visual world and is central to modern computer vision. Human-centered visual intelligence is especially demanding because it studies people as expressive, socially situated subjects whose meaning is rarely conveyed by appearance alone. It couples vision with audio and language across four representative tasks: human emotion recognition, human video generation, human voice cloning, and human video matting. Yet existing resources remain task-specific, providing modalities and annotations for individual problems rather than a shared foundation coordinating understanding and generation. This limits multimodal signal use and broader research. We address this gap with HUG-VIS, a unified benchmark for Human-centered Understanding and Generation in Visual Intelligence. It contains 8,400 seated half-body videos of 30 professional actors, each performing the same 280 emotion-action-prompt assignments under a controlled Mandarin studio protocol, with synchronized video, audio, text, and alpha mattes. We evaluate diverse open- and closed-source models across the four tasks under a unified zero-shot protocol using automatic metrics, criterion-specific mean opinion scores, and multiple cross-task analyses. Results show that (i) linguistic content dominates current emotion recognition, while purely visual affect recognition is weakest; (ii) in video generation and voice cloning, automatic metrics and human judgment agree overall but differ in their top rankings, requiring joint reporting; (iii) boundary fidelity under motion is the main remaining obstacle for human matting; and (iv) task difficulty varies across emotions, models, and metrics, with notable cross-task correlations. The dataset and results are available at https://github.com/GML-MMGroup/HUG-VIS.
Figures & tables
Figure 1: Overview of the HUG-VIS construction and evaluation pipeline. Phases 1–3 establish the shared grid and package temporally synchronized multimodal assets (RGB video, noise-suppressed audio, prompt and transcript, and alpha mattes), while Phase 4 conducts the evaluation of the four understanding and generation tasks together with the cross-task analysis.
Condition
Movement style
Action templates
Illustrative prompt
Happy
Light, relaxed, and expansive, with an open posture
“I finally got a ticket to the concert! I’m thrilled!”
Angry
Fast, forceful, and sustained, with a forward-oriented posture
Table strike; forceful arm sweep; sharp pointing gesture; clenched-fist raise and drop
“Enough! I don’t want to hear another excuse.”
Sad
Slow or paused, with a contracted, downward-oriented posture
Head or neck support with one or both hands; hand on head; self-embrace with hands on upper arms
“Everything I’d hoped for has fallen apart, and now I’m all alone.”
Afraid
Contracted, guarded, and withdrawing, with slight trembling
One- or two-handed guarding gesture; recoil; hands clasped close to the body while leaning backward
“Stay back! Please don’t hurt me!”
Disgusted
Restrained and outwardly rejecting, with a backward lean
Two-handed push-away; one-handed wave or push-away; dismissive wave; shooing gesture
“Get this away from me! I can’t stand looking at it.”
Surprised
Rapid and moderately expansive, ending in an abrupt freeze
Sudden raising of both hands; hands brought behind the head; sudden torso lean; one-hand raise followed by a freeze
“There was nothing behind the curtain. So why is that shadow moving?”
Table 1: Emotion-conditioned performance design in HUG-VIS, showing the intended movement style, action templates, and an illustrative prompt for each emotion. The example prompts are English translations of the Mandarin prompts.
Item
Specification
RGB camera
DJI Action 5 Pro, fixed frontal mount
Background
Uniform green screen for chroma-key matting
Lighting
JHC-2000S LED, fixed color temperature and intensity
Table 2: HUG-VIS acquisition setup and dataset composition.
Figure 2: Samples illustrating the controlled correspondence of the actor-by-assignment grid, with each row showing ordered frames of the rest–expression–rest protocol. The indicated row pairs vary a factor at a time (actor, emotion, action, or all three), so that downstream differences are attributable to the actor, model, or criterion rather than mismatched material.
Figure 3: Acquisition setup of HUG-VIS: Region 1 shows the green-screen environment and equipment, Region 2 the front view configuration and instruction display, and Region 3 the prompt-guided recording with ordered key-gesture frames and synchronized speech. All clips are captured at 1920 × 1080 and 240 FPS (released at 30 FPS) with synchronized audio.
Resource
Emotion annotation
Half-body capture
Alpha supervision
A/V/T alignment
Condition-aligned cross-task analysis
MEAD ( Wang et al., 2020a )
✓
✗
✗
✓
✗
HDTF ( Zhang et al., 2021 )
✗
✗
✗
✗
✗
BEAT ( Liu et al., 2022 )
✓
✓
✗
✗
✗
IEMOCAP ( Busso et al., 2008 )
✓
✓
✗
✓
✗
VCTK ( Yamagishi et al., 2019 )
✗
✗
✗
✗
✗
VideoMatte240K ( Lin et al., 2021 )
✗
✓
✓
✗
✗
Table 3: Comparison of HUG-VIS with representative datasets along five properties: emotion annotation, half-body capture, alpha supervision, audio–video–text (A-V-T) alignment, and condition-aligned cross-task analysis. ✓ denotes that the released resource satisfies the stated criterion, while ✗ denotes that it does not.
Model
Input
Acc. (%) ↑
Image frame
Former-DFER ( Zhao and Liu, 2021 )
Image frame
11.42
Emotion-FAN ( Meng et al., 2019 )
Image frame
13.90
EmotiEffLib ( Sber AI Lab, 2025 ; Savchenko, 2023 )
Image frame
33.37
MMA-DFER ( Chumachenko et al., 2024 )
Image frame
37.05
Video
Table 4: Seven-class emotion recognition accuracy. Bold and underlined values indicate the best and second-best results.
Model
CSIM ↑
Sync-C ↑
Sync-D ↓
EDTalk ( Tan et al., 2024 )
0.710
3.67
8.90
V-Express ( Wang et al., 2024 )
0.786
4.95
8.10
EchoMimicV3 ( Meng et al., 2026 )
0.771
3.15
10.30
AniTalker ( Liu et al., 2024 )
0.744
3.51
9.93
Hallo2 ( Cui et al., 2025 )
0.868
4.65
8.43
Ditto ( Li et al., 2025 )
0.904
3.47
9.31
Table 5: Objective results of the task of audio-driven video generation. Bold and underlined values indicate the best and second-best results.
Model
ID Similarity ↑
Emotion Naturalness ↑
Lip Synchronization ↑
EDTalk ( Tan et al., 2024 )
1.22
1.24
1.33
V-Express ( Wang et al., 2024 )
1.36
1.33
3.40
EchoMimicV3 ( Meng et al., 2026 )
2.93
2.07
2.24
AniTalker ( Liu et al., 2024 )
3.74
2.89
2.78
Hallo2 ( Cui et al., 2025 )
3.69
2.91
2.96
Ditto ( Li et al., 2025 )
4.13
3.02
2.94
Table 6: Subjective results of the task of audio-driven video generation. Bold and underlined values indicate the best and second-best results.
Figure 4: Qualitative comparison of audio-driven video generation for one Happy utterance, sampled at six ordered time points. The top panel shows the source portrait, transcript, and driving audio waveform, and each row shows the temporally aligned outputs of one model.
Model
LPIPS ↓
CSIM ↑
PSNR ↑
SSIM ↑
FID ↓
Open-source
Head Animation
HunyuanPortrait ( Xu et al., 2025b )
0.568
0.868
9.11
0.375
175.71
X-NeMo ( Zhao et al., 2025b )
0.520
0.678
10.15
0.368
150.57
PersonaLive! ( Li et al., 2026 )
0.565
0.834
9.43
0.382
164.94
AniPortrait ( Wei et al., 2024 )
0.565
0.884
9.43
0.383
167.49
Table 7: Objective results of the task of vision-driven video generation. Bold and underlined values indicate the best and second-best results.
Model
ID Similarity ↑
Emotion Naturalness ↑
Motion Naturalness ↑
Open-source
Head Animation
HunyuanPortrait ( Xu et al., 2025b )
4.15
3.55
3.66
X-NeMo ( Zhao et al., 2025b )
4.16
3.68
3.82
PersonaLive! ( Li et al., 2026 )
4.41
3.67
3.63
AniPortrait ( Wei et al., 2024 )
4.04
3.11
3.22
Table 8: Subjective results of the task of vision-driven video generation. Bold and underlined values indicate the best and second-best results.
Figure 5: Qualitative comparison of vision-driven video generation for one Angry clip, with the source frame and driving motion sequence on top and temporally aligned outputs of each model below, grouped into open-source and closed-source.
Figure 6: Fine-grained hand-region failures in vision-driven synthesis, highlighting fused, extra, blurred, distorted, and missing fingers or hands across six systems via enlarged insets.
Model
UTMOS ↑
DNSMOS ↑
Speaker Sim. ↑
Reference
Real audio
1.90
2.84
0.990
Open-source models
GPT-SoVITS v2 ( RVC-Boss, 2024 )
2.26
2.76
0.770
IndexTTS2 ( Zhou et al., 2026a )
2.00
2.94
0.846
CosyVoice 3 ( Du et al., 2025 )
2.30
2.94
0.856
Table 9: Objective results of voice cloning. Bold and underlined values indicate the best and second-best results.
Model
ID Similarity ↑
Emotion Naturalness ↑
Open-source models
GPT-SoVITS v2 ( RVC-Boss, 2024 )
2.99
3.12
IndexTTS2 ( Zhou et al., 2026a )
4.24
4.40
CosyVoice 3 ( Du et al., 2025 )
3.81
4.29
OpenAudio S1 ( Fish Audio, 2025 )
3.64
4.04
Closed-source models
Table 10: Subjective results for voice cloning. Bold and underlined values indicate the best and second-best results.
Model
MAD ↓
MSE ↓
dtSSD ↓
Grad ↓
Conn ↓
VideoMaMa ( Lim et al., 2026 )
34.09
20.43
3.53
26.14
64.38
MODNet ( Ke et al., 2022 )
24.98
20.27
3.84
26.75
47.84
SparseMat ( Sun et al., 2023 )
9.14
4.27
3.19
28.83
14.79
U 2 -Net ( Qin et al., 2020 )
6.78
2.49
2.76
33.11
10.33
InSPyReNet ( Kim et al., 2022 )
4.33
2.52
3.58
31.79
5.77
SAM 3 ( Carion et al., 2026 )
3.92
2.13
3.40
28.01
4.88
Table 11: Objective results of video matting. Bold and underlined values indicate the best and second-best results.
Figure 7: Qualitative comparison of human video matting on one green-screen sequence at seven time points, with rows showing the RGB input, the reference alpha, and the nine model predictions. Red visualization highlights boundary discrepancies relative to the reference alpha.
Figure 8: Emotion difficulty across eight metrics, each averaged over models, oriented so that larger values indicate greater difficulty, and independently min–max normalized over the seven emotions. Values of 0 and 1 denote the easiest and hardest conditions, respectively, and boxes mark column maxima.
Figure 9: Pairwise Spearman rank correlations between the emotion difficulty profiles of the eight metrics. Positive and negative values indicate positive and negative correlations, respectively.
Figure 10: Source-level difficulty of the audio-driven and vision-driven methods on the the common set of valid clips. Hexagon color encodes the number of samples per bin, the dashed line denotes equality, and the red line shows the ordinary least-squares (OLS) fit.
Figure 11: Relationship between motion difficulty and dtSSD, with the line denoting the ordinary least-squares (OLS) fit.
Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/
Xuan He, Cong Wei, Yuhao Cheng +20
University of Illinois Urbana Champaign · University of Waterloo · Vector Institute +6
Recent advancements in audio-video joint generation models have demonstrated impressive capabilities in content creation. However, generating high-fidelity human-centric videos in complex, real-world physical scenes remains a significant challenge. We identify that the root cause lies in the structural deficiencies of existing datasets across three dimensions: limited global scene and camera diversity, sparse interaction modeling (both person-person and person-object), and insufficient individual attribute alignment. To bridge these gaps, we present OmniHuman, a large-scale, multi-scene dataset designed for fine-grained human modeling. OmniHuman provides a hierarchical annotation covering video-level scenes, frame-level interactions, and individual-level attributes. To facilitate this, we develop a fully automated pipeline for high-quality data collection and multi-modal annotation. Complementary to the dataset, we establish the OmniHuman Benchmark (OHBench), a three-level evaluation system that provides a scientific diagnosis for human-centric audio-video synthesis. Crucially, OHBench introduces metrics that are highly consistent with human perception, filling the gaps in existing benchmarks by providing a comprehensive diagnosis across global scenes, relational interactions, and individual attributes.
Lei Zhu, Xing Cai, Yingjie Chen +6
Peking University, Beijing, China · WeChat Lab, Tencent, China · Chinese Academy of Sciences, Beijing, China
In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.
Donghao Zhou, Guisheng Liu, Hao Yang +9
The Chinese University of Hong Kong · ByteDance · Monash University +1