TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction

Ao Li; Yonggen Ling; Yiyang Lin; Yuji Wang; Yong Deng; Yansong Tang

cs.CVApr 10, 2026

TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction

Authors: Ao Li, Yonggen Ling, Yiyang Lin, Yuji Wang, Yong Deng, Yansong Tang

Organizations: Tsinghua Shenzhen International Graduate School · Tencent Robotics X · Futian Laboratory

Abstract

Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception, capable of understanding users' motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.

Figures & tables

Figure 1: We introduce TAIHRI, a task-aware 3D human keypoints localization model designed for close-range human-robot interaction.

Figure 2: The comparison of human pose and shape estimation and human-robot interaction perception. In human-robot interaction perception, the focus is on accurately localizing task-relevant keypoints in camera space from close-range views.

Figure 3: The overview of CloseHRI dataset creation process. We sample various viewpoints around the human subjects in close proximity to obtain the normal maps. Then we utilize WildHuman generator pipeline to synthesize photorealistic images and remove low-quality samples with SAM3 and 2D keypoint reprojection error filtering.

Figure 4: The inference procedure of TAIHRI. Given an egocentric image with the camera intrinsics and a user instruction, we first unify the focal length by image resizing. Then the image and instruction are encoded and fused by the Vision-Language Model backbone. TAIHRI will autoregressively tell the most task-relavant keypoints names, their 2D locations in pixel-level, and their 3D locations in voxel-level in the discretized interaction space. Finally, we decode the voxel-level 3D locations into metric-scale coordinates in camera space. The predicted 3D keypoints can be directly used for human-robot interaction perception or further applied to downstream tasks such as global space human mesh recovery or robot control.

		Harmony4D-Egocentric				EgoBody Test Sample
Methods	Focal	Upper	Lower	L-Upper	R-Upper	Upper	Lower	L-Upper	R-Upper
CameraHMR [ 30 ]	Yes	167.50	165.72	179.22	169.26	94.92	118.96	101.27	99.45
PromptHMR [ 38 ]	Yes	158.70	158.44	158.25	157.18	84.45	127.26	91.82	87.48
3DB-H [ 47 ]	Yes	127.72	129.73	143.41	118.77	88.23	110.58	93.12	94.06
3DB-DINOv3 [ 47 ]	Yes	124.91	127.80	143.13	123.58	89.87	107.75	92.36	93.30
TAIHRI-2B	Yes	97.15	118.82	119.86	103.16	85.62	113.94	93.56	91.37

Table 1: Quantitative comparison on Harmony4D-Egocentric and EgoBody test samples. We report the G-MPJPE in millimeters for different body-part configurations. For SAM 3D Body (3DB), we report the results of ViT backbone 3DB-H and DINOv3 backbone 3DB-DINOv3. Our TAIHRI model outperforms all the compared methods by a large margin across all configurations on both datasets.

Methods	2D	3D	G-MPJPE
* Vision Language Models
GPT-5.2 [ 29 ]	✓	✗	-
Qwen3-VL-235B-A22B-Instruct [ 46 ]	✓	✓	1298.3
Gemini-2.5-Pro [ 7 ]	✓	✓	436.9
* Vision Foundation Models
Rex-Omni [ 13 ]	✓	✗	-

Table 2: Comparison with mainstream VLMs and vision foundation models. VP refers to VitPose [ 45 ] , providing 2D detection results for depth estimation models. The 2D and 3D columns indicate whether the model can predict 2D and 3D keypoints.

Figure 5: The visualization comparison with SOTA methods. We compare our proposed TAIHRI model with several state-of-the-art 3D human pose estimation methods that are capable of estimating the human pose with spatial translation in camera coordinates. Our TAIHRI precisely localizes the task-relevant keypoints, especially for those far from the root joint.

Methods	Upper	Lower	L-Upper	R-Upper
* Camera Int.
w/o Cam Int.	425.13	433.59	429.47	428.11
w Ray Emb.	380.29	423.55	400.09	382.53
* Training
w/o 2D	126.67	138.39	134.50	126.71
w/o RFT	101.82	121.26	110.01	110.79

Table 3: Ablation studies on the injection of camera intrinsics and the training strategy. Results are reported on the Harmony4D-Egocentric test set in terms of G-MPJPE (mm). w and w/o refers to with and without, respectively.

Figure 6: The demonstration of applications. Top: By leveraging the natural language control capability of our method, users can flexibly specify different keypoints for localization based on various interaction tasks. Bottom: Utilizing the accurately localized 3D keypoints as anchor points, we can reposition the human mesh in the global coordinate system.

3D Human Pose Estimation

3D human pose estimation recovers people’s body-joint positions or full-body shape and motion from images, video, and other sensors.

See all 113 papers

SpatialPoint: Spatial-aware Point Prediction for Embodied Localization
Mar 16, 2026Qiming Zhu, Zhirui Fang, Tianming Zhang +3VLMs for Robotics 3D VLMs
Visincept Research · Tsinghua University · International Digital Economy Academy (IDEA)
KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation
Sep 16, 2026Sitong Chen, Fatemeh Zargarbashi, Jin Cheng +2Robotic Control RL for Robotics
ETH Zürich, Switzerland
3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans
Mar 9, 2026Adam Hung, Bardienus Pieter Duisterhof, Jeffrey IchnowskiHuman-to-Robot Motion Imitation Robot Policy Learning
Carnegie Mellon University

Mar 16, 2026J/K move · Enter open · S save

SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

Qiming Zhu, Zhirui Fang, Tianming Zhang, Chuanxiu Liu+2

Visincept Research · Tsinghua University · International Digital Economy Academy (IDEA)

VLMs for Robotics 3D VLMs 3D Spatial Reasoning 3D Visual Grounding

Open paper AskPDF arXiv

Embodied intelligence fundamentally requires a capability to determine where to act in 3D space. We formalize this requirement as embodied localization -- the problem of predicting executable 3D points conditioned on visual observations and language instructions. We instantiate embodied localization with two complementary target types: touchable points, surface-grounded 3D points enabling direct physical interaction, and air points, free-space 3D points specifying placement and navigation goals, directional constraints, or geometric relations. Embodied localization is inherently a problem of embodied 3D spatial reasoning -- yet most existing vision-language systems rely predominantly on RGB inputs, necessitating implicit geometric reconstruction that limits cross-scene generalization, despite the widespread adoption of RGB-D sensors in robotics. To address this gap, we propose SpatialPoint, a spatial-aware vision-language framework with careful design that integrates structured depth into a vision-language model (VLM) and generates camera-frame 3D coordinates. We construct a 2.6M-sample RGB-D dataset covering both touchable and air points QA pairs for training and evaluation. Extensive experiments demonstrate that incorporating depth into VLMs significantly improves embodied localization performance. We further validate SpatialPoint through real-robot deployment across three representative tasks: language-guided robotic arm grasping at specified locations, object placement to target destinations, and mobile robot navigation to goal positions.

Abstract

3D Human Pose Estimation

Explore similar work

SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation

3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans