Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Sravan Chittupalli; Ayush Jain; Dong Huang

cs.CVApr 14, 2026

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Authors: Sravan Chittupalli, Ayush Jain, Dong Huang

Organizations: Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center

Abstract

Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.

Figures & tables

Figure 1: Instance-level contact (our Pi-HOC) vs. Category-level contact (InteractVLM). In a scene with concurrent interaction among multiple humans and objects, multiple contacts occur in the circled regions. Left: InteractVLM is unaware of instances of the same categories, and mixed up all concurrent contacts of three categories: human, flower, and wheelchair. Right: Pi-HOC correctly parses contacts over multiple human-object instance pairs, assigning each object contact to the correct person. Pi-HOC is also substantially faster (7.9FPS) than InteractVLM (0.15FPS).

Figure 2: Pi-HOC Architecture. Given an input image $I$ , a frozen DETR detector localizes human and object instances. A human–object pair constructor enumerates candidate pairs and encodes them as HO tokens (orange). InteractionFormer jointly refines HO tokens and image patch tokens (green) over $L$ blocks. A contact-presence MLP first filters non-contact pairs, and a contact decoder then predicts a per-vertex contact mask $\mathbf{m}_{h,o}\in\{0,1\}^{N_{v}}$ for each retained pair, yielding contact instances $\tau(h,o)$ ( $N_{v}$ : number of human-mesh vertices).

Figure 3: InteractionFormer Attention Visualization. Attention maps show that each HO token, $\mathbf{t}_{h,o}$ , attends to the image regions corresponding to its associated human–object pair and also surrounding regions for additional context. The green bounding box indicates the human instance. The red bounding box indicates the object instance in the corresponding HO pair.

Figure 4: Qualitative Comparison. Left: InteractVLM, which is object-category conditioned, predicts contacts (red) but fails to disambiguate multiple people or multiple instances of objects of the same category. Right: Pi-HOC correctly distinguishes individuals and predicts contacts for each human-object pair. Human instance id is shown next to each person, and contacted objects are circled in distinct colors.

Method	F1 $\uparrow$	Prec $\uparrow$	Recall $\uparrow$	Geo. dist $\downarrow$
InteractVLM-Crop ${}_{\text{Pretrained}}$	43.68	39.23	62.13	0.2468
InteractVLM ${}_{\text{Trained}}$	52.02	48.89	69.25	0.084
InteractVLM-Crop ${}_{\text{Trained}}$	54.86	49.91	69.40	0.073
Pi-HOC (Ours)	61.09	58.76	71.98	0.0633

Table 1: MMHOI Semantic Contact Results .

	F1 $\uparrow$	Prec $\uparrow$	Recall $\uparrow$	Geo. dist $\downarrow$
Semantic DECO [ 28 ]	32.72	25.91	71.51	0.59
InteractVLM-w/o-body-parts 1 [ 5 ]	59.48	59.36	68.06	0.095
Pi-HOC (Ours)	63.23	64.49	69.57	0.092

Table 2: DAMON Semantic Contact Results .

# Pairs	1	2	3	5	7	9	11
InteractVLM [ 5 ]	0.30	0.15	0.096	0.057	0.041	0.032	0.026
Pi-HOC (Ours)	8.5	7.9	6.5	5.1	4.0	3.0	2.3

Table 3: Inference Speed Comparison. FPS as a function of the number of human–object pairs per image.

Table 4: Ablation Studies. (a) InteractionFormer initialization. (b) InteractionFormer layer depth. (c) Contact decoder design. (d) Auxiliary losses.

Figure 5: SAM-3D test-time refinement with Pi-HOC contact. Left: input image. Middle: SAM3D Body and Object initialization, which may violate plausible human–object contact (e.g., floating hands). Right: contact-guided refinement with Pi-HOC predictions, yielding more physically consistent interactions.

Figure 6: Referential contact estimation in multi-instance scenes. We compare InteractVLM and Pi-HOC under language queries that refer to specific interaction instances. Rows 1–2: queries target different suitcase instances; InteractVLM predicts nearly identical contact and fails to disambiguate the referred suitcase, while Pi-HOC correctly shifts contact from the right hand (row 1) to the left hand (row 2). Rows 3–4: queries target different people interacting with the same object (wheelchair); InteractVLM mixes contacts across both people, whereas Pi-HOC isolates the correct human instance (sitting vs. standing) and predicts pair-specific contact. Colored circles highlight contact regions for the queried interaction.

	F1 $\uparrow$	Prec $\uparrow$	Recall $\uparrow$	Geo. dist $\downarrow$
InteractVLM (+Part Sup., original eval)	71.2	72.91	76.29	0.075
InteractVLM (+Part Sup., corrected eval)	57.9	59.15	64.24	0.213
InteractVLM-w/o-body-parts	59.48	59.36	68.06	0.095
Pi-HOC (Ours)	63.23	64.49	69.57	0.092

Table 5: DAMON Semantic Contact Results .

Object Category	Semantic DECO [ 28 ]				InteractVLM-w/o-body-parts [ 5 ]				Pi-HOC (Ours)
Object Category	F1 (%) $\uparrow$	Prec (%) $\uparrow$	Rec (%) $\uparrow$	Geo (cm) $\downarrow$	F1 (%) $\uparrow$	Prec (%) $\uparrow$	Rec (%) $\uparrow$	Geo (cm) $\downarrow$	F1 (%) $\uparrow$	Prec (%) $\uparrow$	Rec (%) $\uparrow$	Geo (cm) $\downarrow$
Accessory	40.1	30.7	75.6	21.88	55.44	63.93	63.68	8.08	61.71	65.83	69.35	11.61
Daily Obj	26.5	20.1	52.3	60.34	53.23	53.86	65.05	10.96	58.00	57.85	69.43	10.66
Food	11.7	19.4	12.9	49.61	57.12	54.54	69.72	9.80	58.06	56.45	73.98	11.29
Furniture	24.5	15.8	83.7	29.17	51.09	53.80	49.50	4.49	53.57	52.78	52.96	6.07
Kitchen	27.7	24.7	37.2	52.34	57.07	55.81	67.00	23.79	63.32	62.82	69.78	22.98

Table 6: DAMON category-wise semantic contact results.

Figure 7: Semantic Human Contact Estimation. We show semantic contact estimation on in-the-wild images. Each example shows all objects with which the human is in contact and the corresponding contact regions on the human mesh.

Figure 10: Semantic Human Contact Failure Cases. Each row shows one failure case: (1) interactions that deviate strongly from common interaction patterns, (2) missed DETR detections that prevent human–object pair formation (e.g., row 2: the child–chair interaction in the left image and the woman–handbag contact on the right image), and (3) confusion between left and right hands when they are very close.

Figure 11: SAM3D Refinement Failure Cases. If the SAM3D initialization is severely misaligned, the algorithm can fail because we keep the object fixed and optimize only the human rotation, translation, and scale.

Figure 12: InteractionFormer Attention Maps. Each row starts with the input image, followed by attention maps for individual contacting human–object pairs. The blue box denotes the human instance and the red box denotes the corresponding object instance. The visualizations show that each pair token focuses primarily on interaction-relevant body and object regions, while also attending to limited contextual cues.

Explore similar work

GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems
Mar 19, 2026Xiaojian Lin, Yaomin Shen, Junyuan Ma +73D Human Reconstruction
Tsinghua University, China · XR System Application Research Center, Nanchang Research Institute, Zhejiang University, China · Beijing University of Posts and Telecommunications, China +2
MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation
May 7, 2026Hao Wang, Shiqi Wang, Qi LiuHOI Generation Human Motion Generation
School of Future Technology, South China University of Technology, Guangzhou, Guangdong, China.
Reconstructing Humans and Objects in Interaction using Large Reconstruction Models
Aug 27, 2026Agniv Chatterjee, Georgios Pavlakos3D Reconstruction 3D Human Reconstruction
University of Texas at Austin

Mar 19, 2026J/K move · Enter open · S save

GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems

Xiaojian Lin, Yaomin Shen, Junyuan Ma, Yujie Sun+6

Tsinghua University, China · XR System Application Research Center, Nanchang Research Institute, Zhejiang University, China · Beijing University of Posts and Telecommunications, China+2

3D Human Reconstruction

Open paper AskPDF arXiv

Monocular vertex-level human-scene contact prediction is a fundamental capability for interactive systems such as assistive monitoring, embodied AI, and rehabilitation analysis. In this work, we study this task jointly with single-image 3D human mesh reconstruction, using reconstructed body geometry as a scaffold for contact reasoning. Existing approaches either focus on contact prediction without sufficiently exploiting explicit 3D human priors, or emphasize pose/mesh reconstruction without directly optimizing robust vertex-level contact inference under occlusion and perceptual noise. To address this gap, we propose GraphiContact, a pose-aware framework that transfers complementary human priors from two pretrained Transformer encoders and predicts per-vertex human-scene contact on the reconstructed mesh. To improve robustness in real-world scenarios, we further introduce a Single-Image Multi-Infer Uncertainty (SIMU) training strategy with token-level adaptive routing, which simulates occlusion and noisy observations during training while preserving efficient single-branch inference at test time. Experiments on five benchmark datasets show that GraphiContact achieves consistent gains on both contact prediction and 3D human reconstruction. Our code, based on the GraphiContact method, provides comprehensive 3D human reconstruction and interaction analysis, and will be publicly available at https://github.com/Aveiro-Lin/GraphiContact.

Abstract

Explore similar work

GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems

MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation

Reconstructing Humans and Objects in Interaction using Large Reconstruction Models