Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Sravan Chittupalli; Ayush Jain; Dong Huang

cs.CVApr 14, 2026

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Authors: Sravan Chittupalli, Ayush Jain, Dong Huang

Organizations: Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center

Abstract

Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.

Figures & tables

Figure 1: Instance-level contact (our Pi-HOC) vs. Category-level contact (InteractVLM). In a scene with concurrent interaction among multiple humans and objects, multiple contacts occur in the circled regions. Left: InteractVLM is unaware of instances of the same categories, and mixed up all concurrent contacts of three categories: human, flower, and wheelchair. Right: Pi-HOC correctly parses contacts over multiple human-object instance pairs, assigning each object contact to the correct person. Pi-HOC is also substantially faster (7.9FPS) than InteractVLM (0.15FPS).

Figure 2: Pi-HOC Architecture. Given an input image $I$ , a frozen DETR detector localizes human and object instances. A human–object pair constructor enumerates candidate pairs and encodes them as HO tokens (orange). InteractionFormer jointly refines HO tokens and image patch tokens (green) over $L$ blocks. A contact-presence MLP first filters non-contact pairs, and a contact decoder then predicts a per-vertex contact mask $\mathbf{m}_{h,o}\in\{0,1\}^{N_{v}}$ for each retained pair, yielding contact instances $\tau(h,o)$ ( $N_{v}$ : number of human-mesh vertices).

Figure 3: InteractionFormer Attention Visualization. Attention maps show that each HO token, $\mathbf{t}_{h,o}$ , attends to the image regions corresponding to its associated human–object pair and also surrounding regions for additional context. The green bounding box indicates the human instance. The red bounding box indicates the object instance in the corresponding HO pair.

Figure 4: Qualitative Comparison. Left: InteractVLM, which is object-category conditioned, predicts contacts (red) but fails to disambiguate multiple people or multiple instances of objects of the same category. Right: Pi-HOC correctly distinguishes individuals and predicts contacts for each human-object pair. Human instance id is shown next to each person, and contacted objects are circled in distinct colors.

Method	F1 $\uparrow$	Prec $\uparrow$	Recall $\uparrow$	Geo. dist $\downarrow$
InteractVLM-Crop ${}_{\text{Pretrained}}$	43.68	39.23	62.13	0.2468
InteractVLM ${}_{\text{Trained}}$	52.02	48.89	69.25	0.084
InteractVLM-Crop ${}_{\text{Trained}}$	54.86	49.91	69.40	0.073
Pi-HOC (Ours)	61.09	58.76	71.98	0.0633

Table 1: MMHOI Semantic Contact Results .

	F1 $\uparrow$	Prec $\uparrow$	Recall $\uparrow$	Geo. dist $\downarrow$
Semantic DECO [ 28 ]	32.72	25.91	71.51	0.59
InteractVLM-w/o-body-parts 1 [ 5 ]	59.48	59.36	68.06	0.095
Pi-HOC (Ours)	63.23	64.49	69.57	0.092

Table 2: DAMON Semantic Contact Results .

# Pairs	1	2	3	5	7	9	11
InteractVLM [ 5 ]	0.30	0.15	0.096	0.057	0.041	0.032	0.026
Pi-HOC (Ours)	8.5	7.9	6.5	5.1	4.0	3.0	2.3

Table 3: Inference Speed Comparison. FPS as a function of the number of human–object pairs per image.

Table 4: Ablation Studies. (a) InteractionFormer initialization. (b) InteractionFormer layer depth. (c) Contact decoder design. (d) Auxiliary losses.

Figure 5: SAM-3D test-time refinement with Pi-HOC contact. Left: input image. Middle: SAM3D Body and Object initialization, which may violate plausible human–object contact (e.g., floating hands). Right: contact-guided refinement with Pi-HOC predictions, yielding more physically consistent interactions.

Figure 6: Referential contact estimation in multi-instance scenes. We compare InteractVLM and Pi-HOC under language queries that refer to specific interaction instances. Rows 1–2: queries target different suitcase instances; InteractVLM predicts nearly identical contact and fails to disambiguate the referred suitcase, while Pi-HOC correctly shifts contact from the right hand (row 1) to the left hand (row 2). Rows 3–4: queries target different people interacting with the same object (wheelchair); InteractVLM mixes contacts across both people, whereas Pi-HOC isolates the correct human instance (sitting vs. standing) and predicts pair-specific contact. Colored circles highlight contact regions for the queried interaction.

	F1 $\uparrow$	Prec $\uparrow$	Recall $\uparrow$	Geo. dist $\downarrow$
InteractVLM (+Part Sup., original eval)	71.2	72.91	76.29	0.075
InteractVLM (+Part Sup., corrected eval)	57.9	59.15	64.24	0.213
InteractVLM-w/o-body-parts	59.48	59.36	68.06	0.095
Pi-HOC (Ours)	63.23	64.49	69.57	0.092

Table 5: DAMON Semantic Contact Results .

Object Category	Semantic DECO [ 28 ]				InteractVLM-w/o-body-parts [ 5 ]				Pi-HOC (Ours)
Object Category	F1 (%) $\uparrow$	Prec (%) $\uparrow$	Rec (%) $\uparrow$	Geo (cm) $\downarrow$	F1 (%) $\uparrow$	Prec (%) $\uparrow$	Rec (%) $\uparrow$	Geo (cm) $\downarrow$	F1 (%) $\uparrow$	Prec (%) $\uparrow$	Rec (%) $\uparrow$	Geo (cm) $\downarrow$
Accessory	40.1	30.7	75.6	21.88	55.44	63.93	63.68	8.08	61.71	65.83	69.35	11.61
Daily Obj	26.5	20.1	52.3	60.34	53.23	53.86	65.05	10.96	58.00	57.85	69.43	10.66
Food	11.7	19.4	12.9	49.61	57.12	54.54	69.72	9.80	58.06	56.45	73.98	11.29
Furniture	24.5	15.8	83.7	29.17	51.09	53.80	49.50	4.49	53.57	52.78	52.96	6.07
Kitchen	27.7	24.7	37.2	52.34	57.07	55.81	67.00	23.79	63.32	62.82	69.78	22.98

Table 6: DAMON category-wise semantic contact results.

Figure 7: Semantic Human Contact Estimation. We show semantic contact estimation on in-the-wild images. Each example shows all objects with which the human is in contact and the corresponding contact regions on the human mesh.

Figure 10: Semantic Human Contact Failure Cases. Each row shows one failure case: (1) interactions that deviate strongly from common interaction patterns, (2) missed DETR detections that prevent human–object pair formation (e.g., row 2: the child–chair interaction in the left image and the woman–handbag contact on the right image), and (3) confusion between left and right hands when they are very close.

Figure 11: SAM3D Refinement Failure Cases. If the SAM3D initialization is severely misaligned, the algorithm can fail because we keep the object fixed and optimize only the human rotation, translation, and scale.

Figure 12: InteractionFormer Attention Maps. Each row starts with the input image, followed by attention maps for individual contacting human–object pairs. The blue box denotes the human instance and the red box denotes the corresponding object instance. The visualizations show that each pair token focuses primarily on interaction-relevant body and object regions, while also attending to limited contextual cues.

Explore similar work

Mar 19, 2026

GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems

Monocular vertex-level human-scene contact prediction is a fundamental capability for interactive systems such as assistive monitoring, embodied AI, and rehabilitation analysis. In this work, we study this task jointly with single-image 3D human mesh reconstruction, using reconstructed body geometry as a scaffold for contact reasoning. Existing approaches either focus on contact prediction without sufficiently exploiting explicit 3D human priors, or emphasize pose/mesh reconstruction without directly optimizing robust vertex-level contact inference under occlusion and perceptual noise. To address this gap, we propose GraphiContact, a pose-aware framework that transfers complementary human priors from two pretrained Transformer encoders and predicts per-vertex human-scene contact on the reconstructed mesh. To improve robustness in real-world scenarios, we further introduce a Single-Image Multi-Infer Uncertainty (SIMU) training strategy with token-level adaptive routing, which simulates occlusion and noisy observations during training while preserving efficient single-branch inference at test time. Experiments on five benchmark datasets show that GraphiContact achieves consistent gains on both contact prediction and 3D human reconstruction. Our code, based on the GraphiContact method, provides comprehensive 3D human reconstruction and interaction analysis, and will be publicly available at https://github.com/Aveiro-Lin/GraphiContact.

Xiaojian Lin, Yaomin Shen, Junyuan Ma +7

Tsinghua University, China · XR System Application Research Center, Nanchang Research Institute, Zhejiang University, China · Beijing University of Posts and Telecommunications, China +2

3D Human Reconstruction

Ask PDF

May 7, 2026

MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation

Generating realistic 3D Human-Object Interactions (HOI) is a fundamental task for applications ranging from embodied AI to virtual content creation, which requires harmonizing high-level semantic intent with strict low-level physical constraints. Existing methods excel at semantic alignment, however, they struggle to maintain precise object contact. We reveal a key finding termed Geometric Forgetting: as diffusion model depth increases, semantic feature tend to overshadow object geometry feature, causing the model to lose its perception to object geometry. To address this, we propose MaMi-HOI, a hierarchical framework reconciling Macro-level kinematic fluidity with Micro-level spatial precision. First, to counteract geometric forgetting, we introduce the Geometry-Aware Proximity Adapter (GAPA), which explicitly re-injects dense object details to perform residual snapping corrections for precise contact. Nevertheless, such aggressive local enforcement can disrupt global dynamics, leading to robotic stiffness. In response, we introduce the Kinematic Harmony Adapter (KHA), which proactively aligns whole-body posture with spatial objectives, ensuring the skeleton actively accommodates constraints without compromising naturalness. Extensive experiments validate that MaMi-HOI simultaneously achieves natural motion and precise contact. Crucially, it extends generation capabilities to long-term tasks with complex trajectories, effectively bridging the gap between global navigation and high-fidelity manipulation in 3D scenes. Code is available at https://github.com/DON738110198/MaMi-HOI.git

Hao Wang, Shiqi Wang, Qi Liu

School of Future Technology, South China University of Technology, Guangzhou, Guangdong, China.

HOI Generation Human Motion Generation Diffusion Models Human-Object Interaction

Ask PDF

Aug 27, 2026

Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.

Agniv Chatterjee, Georgios Pavlakos

University of Texas at Austin

3D Reconstruction 3D Human Reconstruction

Ask PDF