Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception, capable of understanding users' motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.
Figures & tables
Figure 1: We introduce TAIHRI, a task-aware 3D human keypoints localization model designed for close-range human-robot interaction.
Figure 2: The comparison of human pose and shape estimation and human-robot interaction perception. In human-robot interaction perception, the focus is on accurately localizing task-relevant keypoints in camera space from close-range views.
Figure 3: The overview of CloseHRI dataset creation process. We sample various viewpoints around the human subjects in close proximity to obtain the normal maps. Then we utilize WildHuman generator pipeline to synthesize photorealistic images and remove low-quality samples with SAM3 and 2D keypoint reprojection error filtering.
Figure 4: The inference procedure of TAIHRI. Given an egocentric image with the camera intrinsics and a user instruction, we first unify the focal length by image resizing. Then the image and instruction are encoded and fused by the Vision-Language Model backbone. TAIHRI will autoregressively tell the most task-relavant keypoints names, their 2D locations in pixel-level, and their 3D locations in voxel-level in the discretized interaction space. Finally, we decode the voxel-level 3D locations into metric-scale coordinates in camera space. The predicted 3D keypoints can be directly used for human-robot interaction perception or further applied to downstream tasks such as global space human mesh recovery or robot control.
Harmony4D-Egocentric
EgoBody Test Sample
Methods
Focal
Upper
Lower
L-Upper
R-Upper
Upper
Lower
L-Upper
R-Upper
CameraHMR [ 30 ]
Yes
167.50
165.72
179.22
169.26
94.92
118.96
101.27
99.45
PromptHMR [ 38 ]
Yes
158.70
158.44
158.25
157.18
84.45
127.26
91.82
87.48
3DB-H [ 47 ]
Yes
127.72
129.73
143.41
118.77
88.23
110.58
93.12
94.06
3DB-DINOv3 [ 47 ]
Yes
124.91
127.80
143.13
123.58
89.87
107.75
92.36
93.30
TAIHRI-2B
Yes
97.15
118.82
119.86
103.16
85.62
113.94
93.56
91.37
Table 1: Quantitative comparison on Harmony4D-Egocentric and EgoBody test samples. We report the G-MPJPE in millimeters for different body-part configurations. For SAM 3D Body (3DB), we report the results of ViT backbone 3DB-H and DINOv3 backbone 3DB-DINOv3. Our TAIHRI model outperforms all the compared methods by a large margin across all configurations on both datasets.
Methods
2D
3D
G-MPJPE
* Vision Language Models
GPT-5.2 [ 29 ]
✓
✗
-
Qwen3-VL-235B-A22B-Instruct [ 46 ]
✓
✓
1298.3
Gemini-2.5-Pro [ 7 ]
✓
✓
436.9
* Vision Foundation Models
Rex-Omni [ 13 ]
✓
✗
-
Table 2: Comparison with mainstream VLMs and vision foundation models. VP refers to VitPose [ 45 ] , providing 2D detection results for depth estimation models. The 2D and 3D columns indicate whether the model can predict 2D and 3D keypoints.
Figure 5: The visualization comparison with SOTA methods. We compare our proposed TAIHRI model with several state-of-the-art 3D human pose estimation methods that are capable of estimating the human pose with spatial translation in camera coordinates. Our TAIHRI precisely localizes the task-relevant keypoints, especially for those far from the root joint.
Methods
Upper
Lower
L-Upper
R-Upper
* Camera Int.
w/o Cam Int.
425.13
433.59
429.47
428.11
w Ray Emb.
380.29
423.55
400.09
382.53
* Training
w/o 2D
126.67
138.39
134.50
126.71
w/o RFT
101.82
121.26
110.01
110.79
Table 3: Ablation studies on the injection of camera intrinsics and the training strategy. Results are reported on the Harmony4D-Egocentric test set in terms of G-MPJPE (mm). w and w/o refers to with and without, respectively.
Figure 6: The demonstration of applications. Top: By leveraging the natural language control capability of our method, users can flexibly specify different keypoints for localization based on various interaction tasks. Bottom: Utilizing the accurately localized 3D keypoints as anchor points, we can reposition the human mesh in the global coordinate system.
Embodied intelligence fundamentally requires a capability to determine where to act in 3D space. We formalize this requirement as embodied localization -- the problem of predicting executable 3D points conditioned on visual observations and language instructions. We instantiate embodied localization with two complementary target types: touchable points, surface-grounded 3D points enabling direct physical interaction, and air points, free-space 3D points specifying placement and navigation goals, directional constraints, or geometric relations. Embodied localization is inherently a problem of embodied 3D spatial reasoning -- yet most existing vision-language systems rely predominantly on RGB inputs, necessitating implicit geometric reconstruction that limits cross-scene generalization, despite the widespread adoption of RGB-D sensors in robotics. To address this gap, we propose SpatialPoint, a spatial-aware vision-language framework with careful design that integrates structured depth into a vision-language model (VLM) and generates camera-frame 3D coordinates. We construct a 2.6M-sample RGB-D dataset covering both touchable and air points QA pairs for training and evaluation. Extensive experiments demonstrate that incorporating depth into VLMs significantly improves embodied localization performance. We further validate SpatialPoint through real-robot deployment across three representative tasks: language-guided robotic arm grasping at specified locations, object placement to target destinations, and mobile robot navigation to goal positions.
Qiming Zhu, Zhirui Fang, Tianming Zhang +3
Visincept Research · Tsinghua University · International Digital Economy Academy (IDEA)
Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control. Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals. We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system successfully performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.
Human videos could reduce the reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation enables the policy to extract useful motion cues from human videos. We find this formulation transfers exceptionally well, even when human demonstration motions are infeasible for robots. In controlled simulation and real-world evaluations, 3PoinTr achieves an average success rate 25.0 percentage points higher than the strongest behavior-cloning and video-pretraining baselines on real-world tasks, and an improvement of 29.6 percentage points in simulation. Ablations establish the benefits of our method, including predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and a residual cross-attention mechanism. Our point-track prediction network also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points. Project page: https://adamhung60.github.io/3PoinTr/
Adam Hung, Bardienus Pieter Duisterhof, Jeffrey Ichnowski