cs.CVAug 18, 2026

Human-Centric Intelligence in the Era of Foundation Models: A Survey

Authors: Yang Chen, Tianqi Wang, Xiaorui Jiang, Yilei Man, Yihua Shao, Mengyuan Liu, Zhi Chen, Xiaofeng Cao, 

Organizations: The Hong Kong Polytechnic University · Peking University · University of Southern Queensland · Tongji University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human-centric intelligence is evolving in the foundation-model era, with growing emphasis on scale, transferability, and general-purpose modeling. Yet it has not fully integrated with foundation models to achieve the comparable progress seen in them. More importantly, recent advances across this broad landscape remain fragmented across tasks, modalities, and research communities, leaving their intrinsic conceptual and methodological connections unclear. To bridge these divides and rethink human-centric intelligence in the foundation-model era, we introduce a full-spectrum human context taxonomy that integrates six interconnected levels by viewing humans as observable subjects through visual appearance and spatial geometry, as dynamic actors through kinematic dynamics and interaction modeling, and as situated agents through world simulation and embodied agency. We next present the methodological foundations of the field, covering human-centric data families, computational architecture paradigms, and representative training and inference optimization strategies. We then systematically review representative methods across these levels and organize the associated datasets, benchmarks, and evaluation metrics. We further discuss open challenges and promising research directions toward human-centric intelligence that is scalable, trustworthy, physically grounded, and deployable, aiming to provide a coherent framework and practical reference for advancing the field. Finally, we provide a systematically organized and continuously updated collection of human-centric AI literature and resources on our project page.

Figures & tables

Figure 1: Outline of the survey. This work focuses on Human-Centric Intelligence in the foundation-model era by characterizing humans as observable subjects , dynamic actors , and situated agents across six human context levels (Sec. 2 ). Built on this taxonomy, we summarize data modalities, computational architectures, and training/inference optimization strategies (Sec. 3 ). We then review main methodologies across six levels (Sec. 4 – 6 ). Finally, we summarize datasets, benchmarks, and metrics (Sec. 7 ), followed by open challenges and future directions (Sec. 8 ).
Figure 2: Full-spectrum human context taxonomy for human-centric intelligence. We organize human-centric intelligence along a progressive expansion of representational scope: from modeling humans as visually observable subjects, to recovering their spatial body structure, capturing their motion over time, reasoning about their interactions with surrounding entities, predicting human-centered world dynamics, and finally transferring human experience into physical execution and agentic intelligence. The images are originally shown in [ 385 , 179 , 86 , 24 , 195 , 221 , 11 , 153 , 325 , 361 , 315 , 60 , 275 , 89 ] .
Figure 3: Representative task families and selected works across the six human context levels. For the complete list of related methods, kindly refer to Sec. 4 , Sec. 5 , and Sec. 6 .
Figure 4: Human-Centric Data and Signal Families. The images are originally shown in [ 225 , 1 , 2 , 284 , 143 , 161 , 13 , 150 , 19 , 41 , 242 , 57 , 20 ] .
Figure 5: Computational architecture paradigms. These paradigms are shared across human contexts and provide the architectural basis for scaling and integrating human-centric capabilities in the foundation-model era.
Figure 6: Overview of visual appearance in human-centric intelligence. We organize it into generalist human perception, discriminative identity understanding, and controllable human generation. The illustrations are adapted from [ 152 , 256 ] .
MethodVenueModalityData ScaleModel SizeTrain.Inf.Arc.ContextFunctionPaperWeb
Sec. 4.1.1 : Generalist Human Perception
Sapiens2 [ 152 ]ICLR’26 1B Images \lx@sectionsign\lx@sectionsign0.4B-5BS D T–SMA Dense Perception  
Sapiens [ 150 ]ECCV’24 300M Images \lx@sectionsign\lx@sectionsign0.3B-2BS T–SMA Dense Perception  
THFM [ 324 ]arXiv’26 20K Videos \lx@paragraphsign\lx@paragraphsign1.3B-14B †{\dagger}F–IGA Dense Perception –
Hulk [ 333 ]TPAMI’25 30M Samples \lx@sectionsign\lx@sectionsign \lx@paragraphsign\lx@paragraphsign113.7M-336.0M †{\dagger}F–SMA Unified Perception  
L-Man [ 453 ]AAAI’25 2.5M Samples \lx@sectionsign\lx@sectionsign \lx@paragraphsign\lx@paragraphsign7BS I–SFA Unified Perception –
Table 1: Representative methods for visual appearance and spatial geometry in human-centric intelligence. ∙\bullet Modalities: Exocentric Image, Exocentric Video, Infrared & Thermal Image, Depth Map, Normal Map, Pose & Skeleton, Mesh & Parametric Body Model, 3D Gaussian Splatting, Text and Audio. ∙\bullet Training: S Scratch-based Training, F Full-Parameter Fine-tuning, P Parameter-Efficient Tuning, I Instruction Tuning, R Reward-based Tuning, D Knowledge Distillation, and T Task-Head Tuning. ∙\bullet Inference: S Semantic Augmentation, G Guided Sampling, and I Iterative Refinement. ∙\bullet Architectures: SMA Single-Step Mapping, SFA Sequential Factorization, IGA Iterative Generative, and HA Hybrid. ∙\bullet Context: Visual Appearance, and Spatial Geometry. ∙\bullet Markers: \lx@sectionsign\lx@sectionsign Real-World, \lx@paragraphsign\lx@paragraphsign Synthesized, †{\dagger} Estimated, and – Not Mentioned.
Figure 7: Overview of spatial geometry in human-centric intelligence. We organize spatial geometry into structured geometry modeling and renderable avatar modeling. The images are originally shown in [ 79 , 385 , 397 , 145 , 179 ] .
MethodVenueModalityData ScaleModel SizePre.Inf.Arc.ContextFunctionPaperWeb
Sec. 5.1.1 : Scalable Motion Modeling
Kimodo [ 283 ]arXiv’26 700 Hours \lx@sectionsign\lx@sectionsign56M-282MS FG IIGA Scalable Motion Generation  
ARDY [ 431 ]TOG’26 700 Hours \lx@sectionsign\lx@sectionsign156MSG IIGA Scalable Motion Generation  
HY-Motion 1.0 [ 346 ]arXiv’25 3K Hours \lx@sectionsign\lx@sectionsign \lx@paragraphsign\lx@paragraphsign0.05B-1BS F RSIGA Scalable Motion Generation  
MotionBricks [ 327 ]TOG’26 700 Hours \lx@sectionsign\lx@sectionsign–SISFA Scalable Motion Generation  
MonoFrill [ 32 ]CVPR’26 2.8K Hours \lx@sectionsign\lx@sectionsign \lx@paragraphsign\lx@paragraphsign8BS F I–SFA Scalable Motion Generation  
Table 2: Representative methods for kinematic dynamics in human-centric intelligence. ∙\bullet Modalities: Exocentric Image, Exocentric Video, Egocentric Video, Pose & Skeleton, Mesh & Parametric Body Model, Inertial Measurement Signal, Action Trajectory, Text and Audio. ∙\bullet Training: S Scratch-based Training, F Full-Parameter Fine-tuning, P Parameter-Efficient Tuning, I Instruction Tuning, R Reward-based Tuning, D Knowledge Distillation, and T Task-Head Tuning. ∙\bullet Inference: S Semantic Augmentation, G Guided Sampling, I Iterative Refinement, P Preference-based Selection, and R Retrieval Augmentation. ∙\bullet Architectures: SMA Single-Step Mapping, SFA Sequential Factorization, IGA Iterative Generative, and HA Hybrid. ∙\bullet Context: Kinematic Dynamics. ∙\bullet Markers: \lx@sectionsign\lx@sectionsign Real-World, \lx@paragraphsign\lx@paragraphsign Synthesized, †{\dagger} Estimated, and – Not Mentioned.
Figure 8: Overview of kinematic dynamics in human-centric intelligence. We organize kinematic dynamics into scalable motion modeling and human video animation. The images are originally shown in [ 283 , 346 ] .
MethodVenueModalityData ScaleModel SizePre.Inf.Arc.ContextFunctionPaperWeb
Sec. 5.2.1 : Human-Object Interaction
ViHOI [ 24 ]CVPR’26 10 Hours \lx@sectionsign\lx@sectionsign3BPGIGA HOI Motion Generation  
Uni-HOI [ 416 ]arXiv’26 10.4K Samples \lx@sectionsign\lx@sectionsign8BP I–SFA HOI Motion Generation –
CoInteract [ 238 ]arXiv’26 12K Clips \lx@sectionsign\lx@sectionsign \lx@paragraphsign\lx@paragraphsign14B †{\dagger}FGIGA HOI Video Generation  
ByteLOOM [ 215 ]arXiv’25 7.8M Samples \lx@sectionsign\lx@sectionsign †{\dagger}1.3B-14B †{\dagger}F–IGA HOI Video Generation  
HOI-PAGE [ 183 ]ICML’26 –––S G IIGA Scalable 4D HOI Generation  
Table 3: Representative methods for interaction modeling in human-centric intelligence. ∙\bullet Modalities: Exocentric Image, Exocentric Video, Pose & Skeleton, Mesh & Parametric Body Model, Gaze Signal, Inertial Measurement Signal, Action Trajectory, Text and Audio. ∙\bullet Training: S Scratch-based Training, F Full-Parameter Fine-tuning, P Parameter-Efficient Tuning, I Instruction Tuning, R Reward-based Tuning, D Knowledge Distillation, and T Task-Head Tuning. ∙\bullet Inference: S Semantic Augmentation, G Guided Sampling, I Iterative Refinement, P Preference-based Selection, and R Retrieval Augmentation. ∙\bullet Architectures: SMA Single-Step Mapping, SFA Sequential Factorization, IGA Iterative Generative, and HA Hybrid. ∙\bullet Context: Interaction Modeling. ∙\bullet Markers: \lx@sectionsign\lx@sectionsign Real-World, \lx@paragraphsign\lx@paragraphsign Synthesized, †{\dagger} Estimated, and – Not Mentioned.
Figure 9: Overview of interaction modeling in human-centric intelligence. We organize interaction modeling into human-object, human-scene, and social interaction. The images are originally shown in [ 238 , 24 , 221 , 335 , 337 , 7 , 250 , 343 ] .
MethodVenueModalityData ScaleModel SizeTrain.Inf.Arc.ContextFunctionPaperWeb
Sec. 6.1.1 : Human-Centered World Generation
Generated Reality [ 361 ]CVPRF’26 5.8K Samples \lx@sectionsign\lx@sectionsign5B-14BP D TGIGA Hand-Guided Interactive Generation  
Hand2World [ 339 ]arXiv’26 –1.3BP DRIGA Hand-Guided Interactive Generation  
PlayerOne [ 316 ]NeurIPS’25 5M Samples \lx@sectionsign\lx@sectionsign1.3BP D–IGA Hand-Guided Interactive Generation  
AnchorWorld [ 199 ]arXiv’26 301K Videos \lx@sectionsign\lx@sectionsign \lx@paragraphsign\lx@paragraphsign5BF T–IGA View-Guided Interactive Generation  
Sec. 6.1.2 : Actionable World Planning
Table 4: Representative methods for world simulation and embodied agency in human-centric intelligence. ∙\bullet Modalities: Exocentric Video, Egocentric Image, Egocentric Video, Pose & Skeleton, Mesh & Parametric Body Model, Action Trajectory, Control Signal, Text and Audio. ∙\bullet Training: S Scratch-based Training, F Full-Parameter Fine-tuning, P Parameter-Efficient Tuning, I Instruction Tuning, R Reward-based Tuning, D Knowledge Distillation, and T Task-Head Tuning. ∙\bullet Inference: S Semantic Augmentation, G Guided Sampling, I Iterative Refinement, P Preference-based Selection, and R Retrieval Augmentation. ∙\bullet Architectures: SMA Single-Step Mapping, SFA Sequential Factorization, IGA Iterative Generative, and HA Hybrid. ∙\bullet Context: World Simulation, and Embodied Agency. ∙\bullet Markers: \lx@sectionsign\lx@sectionsign Real-World, \lx@paragraphsign\lx@paragraphsign Synthesized, †{\dagger} Estimated, and – Not Mentioned.
Figure 10: Overview of world simulation in human-centric intelligence. We organize world simulation into human-centered world generation and actionable world planning. The images are originally shown in [ 361 , 364 ] .
Figure 11: Overview of embodied agency in human-centric intelligence. We organize embodied agency into generalist humanoid control and human-to-agent skill transfer. The images are originally shown in [ 192 , 275 , 89 ] .
Figure 12: Representative human subject resources in human-centric intelligence. From left to right, the columns show StyleGAN-Human [ 83 ] , M4Human [ 69 ] , LLCM [ 425 ] , and MVHumanNet++ [ 174 ] .
Table 5: Representative datasets and benchmarks for renderable human geometry in human-centric intelligence. ∙\bullet Modalities: Exocentric Image, Exocentric Video, Egocentric Image, Egocentric Video, Depth Map, Normal Map, Pose & Skeleton, Point Cloud, Mesh & Parametric Body Model, and Text. ∙\bullet Markers: C Self-Captured, P Public-Curated, W Web-Collected, and S Synthesized.
Figure 13: Representative human dynamics resources in human-centric intelligence. From left to right, the top row shows OpenHumanVid [ 177 ] , VideoNet [ 377 ] , and Motion-X++ [ 424 ] , while the bottom row shows BarbieGait [ 23 ] , M3GYM [ 371 ] , and TripVVT [ 289 ] .
Table 6: Representative datasets and benchmarks for human kinematic motion modeling in human-centric intelligence. ∙\bullet Modalities: Exocentric Video, Depth Map, Pose & Skeleton, Mesh & Parametric Body Model, Control Signal, Text, and Audio. ∙\bullet Markers: C Self-Captured, P Public-Curated, W Web-Collected, and S Synthesized.
Figure 14: Representative human interaction resources in human-centric intelligence. From left to right, the examples show Nymeria [ 242 ] , ParaHome [ 154 ] , TRUMANS [ 138 ] , and Embody 3D [ 245 ] .
Table 7: Representative datasets and benchmarks for egocentric procedural activity modeling in human-centric intelligence. ∙\bullet Modalities: Exocentric Image, Exocentric Video, Egocentric Video, Depth Map, Pose & Skeleton, Gaze Signal, Inertial Measurement Signal, Control Signal, Text, and Audio. ∙\bullet Markers: C Self-Captured, P Public-Curated, W Web-Collected, and S Synthesized.
Figure 15: Representative human embodiment resources in human-centric intelligence. From left to right, the examples show EgoVerse [ 273 ] and PHUMA [ 168 ] .
Table 8: Representative datasets and benchmarks for transferring human experience to embodied agents. ∙\bullet Modalities: Exocentric Video, Egocentric Video, Depth Map, Pose & Skeleton, Control Signal, Tactile Signal, Text, and Audio. ∙\bullet Markers: C Self-Captured, P Public-Curated, W Web-Collected, and S Synthesized.
SymbolMeaning
⋅^\hat{\cdot}Model prediction; the corresponding unhatted variable denotes the reference.
N,T,J,VN,T,J,VNumbers of evaluation units, frames, joints, and vertices.
CCNumber of classes or conditioning groups, depending on the metric.
Kr,Kn,KcK_{r},K_{n},K_{c}Retrieval cutoff, maximum n-gram order, and samples per condition.
Li,LL_{i},LNumbers of reference captions for sample ii and mel-cepstral coefficients.
Ri,Ri\mathcal{R}_{i},R_{i}Relevant-item set for query ii and its cardinality.
Table 9: Unified notation for the evaluation metrics summarized in Table 10 .
MetricBetterFormulaDescription
Articulated Human State
MPJPE↓\downarrow1J∑j=1J∥p^j−pj∥2\frac{1}{J}\sum_{j=1}^{J}\|\hat{\mathbf{p}}_{j}-\mathbf{p}_{j}\|_{2}Mean 3D joint-position error.
PA-MPJPE↓\downarrowmin⁡A∈Sim(3)1J∑j=1J∥Ap^j−pj∥2\min_{\mathbf{A}\in\mathrm{Sim}(3)}\frac{1}{J}\sum_{j=1}^{J}\|\mathbf{A}\hat{\mathbf{p}}_{j}-\mathbf{p}_{j}\|_{2}Joint error after similarity alignment.
MPVPE↓\downarrow1V∑i=1V∥v^i−vi∥2\frac{1}{V}\sum_{i=1}^{V}\|\hat{\mathbf{v}}_{i}-\mathbf{v}_{i}\|_{2}Mean dense vertex-position error.
PCK↑\uparrow1J∑j=1J1[∥p^j−pj∥2<τ]\frac{1}{J}\sum_{j=1}^{J}\mathbf{1}[\|\hat{\mathbf{p}}_{j}-\mathbf{p}_{j}\|_{2}<\tau]Fraction of keypoints within threshold τ\tau .
AUC-PCK↑\uparrow1τmax⁡−τmin⁡∫τmin⁡τmax⁡PCK(τ)dτ\frac{1}{\tau_{\max}-\tau_{\min}}\int_{\tau_{\min}}^{\tau_{\max}}\mathrm{PCK}(\tau)d\tauArea under the PCK-threshold curve.
Table 10: Summary of representative evaluation metrics for human-centric intelligence.
More in this topic

Human-Centered AI

Human-Centered AI designs and evaluates AI around people's goals, abilities, values, relationships, and social contexts.

See all 169 papers

Explore similar work

CardsList

  1. Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer

    Jul 30, 2026Weiquan Lin, Yu Deng, Shiyang Liu +6HOI GenerationHuman-Scene Interaction Reconstruction

    1Xidian University · 2Zhongguancun Academy · 3Microsoft Research Asia +2

  2. HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

    May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic FairnessMultimodal Large Language Models

    Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA

  3. A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws

    Apr 27, 2026Jun Shu, Junxiong Jia, Deyu Meng +1Neural Scaling LawsAI Scaling Laws

    School of Mathematics and Statistics, Xi’an Jiaotong University.