Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds

Daniel Sungho Jung; Dohee Cho; Kyoung Mu Lee

cs.CVMar 17, 2026

Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds

Authors: Daniel Sungho Jung, Dohee Cho, Kyoung Mu Lee

Organizations: IPAI · Dept. of ECE&ASRI, Seoul National University

Abstract

Understanding humans from LiDAR point clouds is one of the most critical tasks in autonomous driving due to its close relationships with pedestrian safety, yet it remains challenging in the presence of diverse human-object interactions and cluttered backgrounds. Nevertheless, existing methods largely overlook the potential of leveraging human-object interactions to build robust 3D human pose estimation frameworks. There are two major challenges that motivate the incorporation of human-object interaction. First, human-object interactions introduce spatial ambiguity between human and object points, which often leads to erroneous 3D human keypoint predictions in interaction regions. Second, there exists severe class imbalance in the number of points between interacting and non-interacting body parts, with the interaction-frequent regions such as hand and foot being sparsely observed in LiDAR data. To address these challenges, we propose a Human-Object Interaction Learning (HOIL) framework for robust 3D human pose estimation from LiDAR point clouds. To mitigate the spatial ambiguity issue, we present human-object interaction-aware contrastive learning (HOICL) that effectively enhances feature discrimination between human and object points, particularly in interaction regions. To alleviate the class imbalance issue, we introduce contact-aware part-guided pooling (CPPool) that adaptively reallocates representational capacity by compressing overrepresented points while preserving informative points from interacting body parts. In addition, we present an optional contact-based temporal refinement that refines erroneous per-frame keypoint estimates using contact cues over time. As a result, our HOIL effectively leverages human-object interaction to resolve spatial ambiguity and class imbalance in interaction regions. Codes will be released.

Figures & tables

Figure 1: Main challenges for 3D human pose estimation from LiDAR points. First, due to spatial ambiguity issue of 3D human and object points, previous SOTA method [ 1 ] fails to predict accurate pose. Second, class imbalance issue occurs in frequently interacting body parts ( $\downarrow$ ) such as hand and foot, where the number of points are severely low compared to other body parts, object, and background.

Dataset	Body model	# of samples	# of subjects	# of objects	# of instructions
BEHAVE [ 2 ]	SMPL	22K	8	20	6
CHAIRS [ 14 ]	SMPL-X	921K	46	81	32
HODome [ 40 ]	SMPL-H	632K	10	23	–
OMOMO [ 19 ]	SMPL-X	5K	17	15	103
InterCap [ 13 ]	SMPL-X	51K	10	10	–

Table 1: Dataset configuration. We utilize 5 human-object interaction datasets.

Figure 2: Overall pipeline of HOIL. Given an input point cloud $\mathbf{P}$ , we first embed it into input features $\mathbf{F}_{\mathrm{p}}$ and encode them using a multi-stage encoder with CPPool. The encoded features are then progressively decoded through a multi-stage decoder to produce the final decoder features $\mathbf{F}_{\mathrm{p},\mathrm{dec}}^{(0)}$ . At each decoding stage, keypoint queries $\mathbf{Q}$ are iteratively updated via multi-stage keypoint decoder with decoder features. Lastly, HOIL predicts point-level segmentation $\mathbf{S}_{\mathrm{p}}$ and contact $\mathbf{C}_{\mathrm{p}}$ from $\mathbf{F}_{\mathrm{p},\mathrm{dec}}^{(0)}$ , and 3D keypoint coordinates $\mathbf{K}$ and keypoint-level contact $\mathbf{C}_{\mathrm{K}}$ from keypoint queries $\mathbf{Q}$ .

Figure 3: Human-object interaction-aware contrastive learning. C and NC refer to contact and non-contact, respectively. Others represent other human body parts except hand and foot. For the simplicity of illustration, we do not divide hand and foot for C and NC whereas our HOICL does operate on those cases as well.

Global CL	FIR CL	Contact CL	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
✗	✗	✗	51.16	97.09	98.83
✓	✗	✗	50.05	97.21	98.87
✓	✓	✗	49.91	98.04	98.96
✓	✓	✓	48.83	98.51	99.14

Table 2: Ablation of human-object interaction-aware contrastive learning on Waymo [ 31 ] . CL refers to contrastive learning.

Methods	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
Baseline (CE)	51.16	97.09	98.83
SupCon [ 18 ]	50.07	97.29	98.83
KCL [ 17 ]	50.03	97.31	98.89
BCL [ 44 ]	49.92	97.51	98.81
CBL [ 32 ]	51.13	97.10	98.73
HiMulConE [ 42 ]	49.90	97.35	98.85

Table 3: Quantitative comparison of various contrastive learning techniques on Waymo [ 31 ] . Baseline is a cross-entropy (CE) loss without contrastive learning.

Part-guided	Contact-aware	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
✗	✗	50.78	97.03	98.79
✓	✗	49.99	97.38	98.96
✓	✓	48.83	98.51	99.14

Table 4: Ablation of contact-aware part-guided pooling on Waymo [ 31 ] .

Temporal	Contact	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
✗	✗	38.87	97.52	99.33
✓	✗	37.82	97.61	99.40
✓	✓	36.69	97.91	99.60

Table 5: Ablation of contact-based temporal refinement on InterCap [ 13 ] . Note that temporal refinement is only conducted in this experiment.

Figure 4: Qualitative comparison of 3D human pose estimation with the state-of-the-art method [ 1 ] on Waymo [ 31 ] . Red circles indicate exemplar regions that HOIL outperforms previous methods.

Test dataset	Methods	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
Waymo [ 31 ]	NE [ 39 ]	–	–	–
	LPFormer [ 36 ]	61.60	94.52	98.04
	PRN [ 10 ]	68.48	93.60	97.87
	DAPT [ 1 ]	51.59	97.34	98.98
	HOIL (Ours)	48.83	98.51	99.14
SLOPER4D [ 7 ]	NE [ 39 ]	96.80	79.22	90.51

Table 6: Quantitative comparison with SOTA methods of 3D human pose estimation on Waymo [ 31 ] and SLOPER4D [ 7 ] .

Figure S1 : Correlation between segmentation accuracy and human pose estimation error of state-of-the-art method [ 1 ] on Waymo [ 31 ] .

Figure S2 : Error by human body joints of state-of-the-art method [ 1 ] on Waymo [ 31 ] and InterCap [ 13 ] . Red rectangles indicate errors from wrists and ankles which correspond to frequently interacting parts ( e.g., hands and feet).

Methods	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
No refinement	38.87	97.52	99.33
Gaussian filter [ 37 ]	37.30	97.90	99.58
Savitzky-Golay filter [ 28 ]	37.08	97.84	99.53
One-Euro filter [ 3 ]	38.52	97.56	99.36
CTRefine (Ours)	36.69	97.91	99.60

Table S1 : Quantitative comparison of various temporal refinement techniques on InterCap [ 13 ] .

Methods	MPJPE $\downarrow$	PCK-3 $\uparrow$	PCK-5 $\uparrow$
w/o CTRefine	25.43	98.67	99.65
w/ CTRefine (Ours)	25.36	98.69	99.65

Table S2 : Ablation of CTRefine on SLOPER4D [ 7 ] .

Model	Pre-train Memory	Fine-tune Memory	Test Memory	Params.	Speed	GFLOPs
HOIL	33,198MB	20,262MB	8,435MB	53.02M	155.52 Hz	393.89

Table S3 : Computational requirements of HOIL.

Figure S3 : Qualitative comparison of 3D human pose estimation with the state-of-the-art method [ 1 ] on Waymo [ 31 ] . We assort samples where the SOTA method has highest errors from the dataset. Red circles indicate exemplar regions that HOIL outperforms previous methods.

Figure S4 : Qualitative comparison of 3D human pose estimation with the state-of-the-art method [ 1 ] on SLOPER4D [ 7 ] . We assort samples where the SOTA method has highest errors from the dataset. Red circles indicate exemplar regions that HOIL outperforms previous methods.

Explore similar work

Pi-HOC: Pairwise 3D Human-Object Contact Estimation
Apr 14, 2026Sravan Chittupalli, Ayush Jain, Dong HuangHuman-Object Interaction
Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center
Factorized Spatio-Temporal Convolutions for Human Pose Estimation from Planar Lidar
Jul 23, 2026Simone Arreghini, Mirko Nava, Nicholas Carlotti +2Relative Pose Estimation Human Pose Estimation
Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI, Lugano, Switzerland
Reconstructing Humans and Objects in Interaction using Large Reconstruction Models
Aug 27, 2026Agniv Chatterjee, Georgios Pavlakos3D Reconstruction 3D Human Reconstruction
University of Texas at Austin

Apr 14, 2026J/K move · Enter open · S save

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Sravan Chittupalli, Ayush Jain, Dong Huang

Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center

Human-Object Interaction

Open paper AskPDF arXiv

Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.

Abstract

Explore similar work

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Factorized Spatio-Temporal Convolutions for Human Pose Estimation from Planar Lidar

Reconstructing Humans and Objects in Interaction using Large Reconstruction Models