Understanding humans from LiDAR point clouds is one of the most critical tasks in autonomous driving due to its close relationships with pedestrian safety, yet it remains challenging in the presence of diverse human-object interactions and cluttered backgrounds. Nevertheless, existing methods largely overlook the potential of leveraging human-object interactions to build robust 3D human pose estimation frameworks. There are two major challenges that motivate the incorporation of human-object interaction. First, human-object interactions introduce spatial ambiguity between human and object points, which often leads to erroneous 3D human keypoint predictions in interaction regions. Second, there exists severe class imbalance in the number of points between interacting and non-interacting body parts, with the interaction-frequent regions such as hand and foot being sparsely observed in LiDAR data. To address these challenges, we propose a Human-Object Interaction Learning (HOIL) framework for robust 3D human pose estimation from LiDAR point clouds. To mitigate the spatial ambiguity issue, we present human-object interaction-aware contrastive learning (HOICL) that effectively enhances feature discrimination between human and object points, particularly in interaction regions. To alleviate the class imbalance issue, we introduce contact-aware part-guided pooling (CPPool) that adaptively reallocates representational capacity by compressing overrepresented points while preserving informative points from interacting body parts. In addition, we present an optional contact-based temporal refinement that refines erroneous per-frame keypoint estimates using contact cues over time. As a result, our HOIL effectively leverages human-object interaction to resolve spatial ambiguity and class imbalance in interaction regions. Codes will be released.
Figures & tables
Figure 1: Main challenges for 3D human pose estimation from LiDAR points. First, due to spatial ambiguity issue of 3D human and object points, previous SOTA method [ 1 ] fails to predict accurate pose. Second, class imbalance issue occurs in frequently interacting body parts ( ↓ ) such as hand and foot, where the number of points are severely low compared to other body parts, object, and background.
Dataset
Body model
# of samples
# of subjects
# of objects
# of instructions
BEHAVE [ 2 ]
SMPL
22K
8
20
6
CHAIRS [ 14 ]
SMPL-X
921K
46
81
32
HODome [ 40 ]
SMPL-H
632K
10
23
–
OMOMO [ 19 ]
SMPL-X
5K
17
15
103
InterCap [ 13 ]
SMPL-X
51K
10
10
–
Table 1: Dataset configuration. We utilize 5 human-object interaction datasets.
Figure 2: Overall pipeline of HOIL. Given an input point cloud P , we first embed it into input features Fp and encode them using a multi-stage encoder with CPPool. The encoded features are then progressively decoded through a multi-stage decoder to produce the final decoder features Fp,dec(0) . At each decoding stage, keypoint queries Q are iteratively updated via multi-stage keypoint decoder with decoder features. Lastly, HOIL predicts point-level segmentation Sp and contact Cp from Fp,dec(0) , and 3D keypoint coordinates K and keypoint-level contact CK from keypoint queries Q .
Figure 3: Human-object interaction-aware contrastive learning. C and NC refer to contact and non-contact, respectively. Others represent other human body parts except hand and foot. For the simplicity of illustration, we do not divide hand and foot for C and NC whereas our HOICL does operate on those cases as well.
Global CL
FIR CL
Contact CL
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
✗
✗
✗
51.16
97.09
98.83
✓
✗
✗
50.05
97.21
98.87
✓
✓
✗
49.91
98.04
98.96
✓
✓
✓
48.83
98.51
99.14
Table 2: Ablation of human-object interaction-aware contrastive learning on Waymo [ 31 ] . CL refers to contrastive learning.
Methods
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
Baseline (CE)
51.16
97.09
98.83
SupCon [ 18 ]
50.07
97.29
98.83
KCL [ 17 ]
50.03
97.31
98.89
BCL [ 44 ]
49.92
97.51
98.81
CBL [ 32 ]
51.13
97.10
98.73
HiMulConE [ 42 ]
49.90
97.35
98.85
Table 3: Quantitative comparison of various contrastive learning techniques on Waymo [ 31 ] . Baseline is a cross-entropy (CE) loss without contrastive learning.
Part-guided
Contact-aware
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
✗
✗
50.78
97.03
98.79
✓
✗
49.99
97.38
98.96
✓
✓
48.83
98.51
99.14
Table 4: Ablation of contact-aware part-guided pooling on Waymo [ 31 ] .
Temporal
Contact
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
✗
✗
38.87
97.52
99.33
✓
✗
37.82
97.61
99.40
✓
✓
36.69
97.91
99.60
Table 5: Ablation of contact-based temporal refinement on InterCap [ 13 ] . Note that temporal refinement is only conducted in this experiment.
Figure 4: Qualitative comparison of 3D human pose estimation with the state-of-the-art method [ 1 ] on Waymo [ 31 ] . Red circles indicate exemplar regions that HOIL outperforms previous methods.
Test dataset
Methods
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
Waymo [ 31 ]
NE [ 39 ]
–
–
–
LPFormer [ 36 ]
61.60
94.52
98.04
PRN [ 10 ]
68.48
93.60
97.87
DAPT [ 1 ]
51.59
97.34
98.98
HOIL (Ours)
48.83
98.51
99.14
SLOPER4D [ 7 ]
NE [ 39 ]
96.80
79.22
90.51
Table 6: Quantitative comparison with SOTA methods of 3D human pose estimation on Waymo [ 31 ] and SLOPER4D [ 7 ] .
Figure S1 : Correlation between segmentation accuracy and human pose estimation error of state-of-the-art method [ 1 ] on Waymo [ 31 ] .
Figure S2 : Error by human body joints of state-of-the-art method [ 1 ] on Waymo [ 31 ] and InterCap [ 13 ] . Red rectangles indicate errors from wrists and ankles which correspond to frequently interacting parts ( e.g., hands and feet).
Methods
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
No refinement
38.87
97.52
99.33
Gaussian filter [ 37 ]
37.30
97.90
99.58
Savitzky-Golay filter [ 28 ]
37.08
97.84
99.53
One-Euro filter [ 3 ]
38.52
97.56
99.36
CTRefine (Ours)
36.69
97.91
99.60
Table S1 : Quantitative comparison of various temporal refinement techniques on InterCap [ 13 ] .
Methods
MPJPE ↓
PCK-3 ↑
PCK-5 ↑
w/o CTRefine
25.43
98.67
99.65
w/ CTRefine (Ours)
25.36
98.69
99.65
Table S2 : Ablation of CTRefine on SLOPER4D [ 7 ] .
Model
Pre-train Memory
Fine-tune Memory
Test Memory
Params.
Speed
GFLOPs
HOIL
33,198MB
20,262MB
8,435MB
53.02M
155.52 Hz
393.89
Table S3 : Computational requirements of HOIL.
Figure S3 : Qualitative comparison of 3D human pose estimation with the state-of-the-art method [ 1 ] on Waymo [ 31 ] . We assort samples where the SOTA method has highest errors from the dataset. Red circles indicate exemplar regions that HOIL outperforms previous methods.
Figure S4 : Qualitative comparison of 3D human pose estimation with the state-of-the-art method [ 1 ] on SLOPER4D [ 7 ] . We assort samples where the SOTA method has highest errors from the dataset. Red circles indicate exemplar regions that HOIL outperforms previous methods.
Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.
Sravan Chittupalli, Ayush Jain, Dong Huang
Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center
Localizing nearby humans and estimating their facing direction are key capabilities for safe navigation and socially aware human-robot interaction. Many pose-estimation pipelines target cameras and 3D LiDAR or assume GPU-class compute, whereas service robots are often equipped only with omnidirectional planar LiDARs and modest onboard processors. We address omnidirectional human detection and relative 2D pose estimation from planar LiDAR sequences with a lightweight network based on Space-Time Blocks, which explicitly separate spatial processing along scan rays from temporal aggregation across scans. Our network processes 360° LiDAR sequences to output per-ray human presence, distance, and relative orientation. We train it via cross-modal self-supervision from a narrow RGB-D body tracker in the sensors' overlap region, removing the need for manual LiDAR labels. Quantitative experiments show that our approach consistently outperforms a parameter-matched baseline model, reducing errors in distance (-38%), position (-28%), and orientation (-15%). We further benchmark on the public FROG dataset, report real-time CPU inference on a service robot, and validate with in-field demonstrations, supporting its suitability for spatial perception on computationally constrained service robots.
Simone Arreghini, Mirko Nava, Nicholas Carlotti +2
Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland
Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.