GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems

Xiaojian Lin; Yaomin Shen; Junyuan Ma; Yujie Sun; Chengqing Bu; Wenxin Zhang; Zongzheng Zhang; Hao Fei

cs.CVMar 19, 2026

GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems

Authors: Xiaojian Lin, Yaomin Shen, Junyuan Ma, Yujie Sun, Chengqing Bu, Wenxin Zhang, Zongzheng Zhang, Hao Fei, +2 more

Organizations: Tsinghua University, China · XR System Application Research Center, Nanchang Research Institute, Zhejiang University, China · Beijing University of Posts and Telecommunications, China · University of Chinese Academy of Sciences, China · National University of Singapore, Singapore

Abstract

Monocular vertex-level human-scene contact prediction is a fundamental capability for interactive systems such as assistive monitoring, embodied AI, and rehabilitation analysis. In this work, we study this task jointly with single-image 3D human mesh reconstruction, using reconstructed body geometry as a scaffold for contact reasoning. Existing approaches either focus on contact prediction without sufficiently exploiting explicit 3D human priors, or emphasize pose/mesh reconstruction without directly optimizing robust vertex-level contact inference under occlusion and perceptual noise. To address this gap, we propose GraphiContact, a pose-aware framework that transfers complementary human priors from two pretrained Transformer encoders and predicts per-vertex human-scene contact on the reconstructed mesh. To improve robustness in real-world scenarios, we further introduce a Single-Image Multi-Infer Uncertainty (SIMU) training strategy with token-level adaptive routing, which simulates occlusion and noisy observations during training while preserving efficient single-branch inference at test time. Experiments on five benchmark datasets show that GraphiContact achieves consistent gains on both contact prediction and 3D human reconstruction. Our code, based on the GraphiContact method, provides comprehensive 3D human reconstruction and interaction analysis, and will be publicly available at https://github.com/Aveiro-Lin/GraphiContact.

Figures & tables

Fig. 1 : Motivation of GraphiContact. GraphiContact augments reconstructed human geometry with pose-aware priors and robustness training to improve contact prediction over a representative prior method [ 1 ] .

Fig. 2 : Comprehensive workflow of the GraphiContact method for 3D human mesh reconstruction and contact prediction with context integration. (a) The Mesh and Pose Reconstruction Branch processes an input image, which first undergoes SIMU Perturbation Gateway for SIMU Strategy, to extract features using a pre-trained CNN. These features are tokenized and passed through two Transformer Encoders, generating coarse and full mesh tokens. The mesh is refined with the mesh reconstruction loss $\mathcal{L}_{m}$ and contact classification loss $\mathcal{L}_{\text{cls}}$ , determining vertex-environment contact. (b) The Context Integration Branch processes semantic and body-part context to enhance segmentation. It consists of two modules: the Semantic Context Encoder-decoder, guided by $\mathcal{L}_{\text{sem}}$ , and the Body-Part Context Encoder-decoder, guided by $\mathcal{L}_{\text{bp}}$ . These outputs improve contact point prediction in complex human-environment interactions.

Fig. 3 : Overview of the SIMU Strategy for parallelized perturbation learning and contact prediction. It transforms the input tensor, applies multi-path perturbations via the SIMU gateway, and extracts features with the backbone. TARM performs token-wise weighted fusion across paths under multi-task supervision (

\mathcal{L}_{\text{cls}}

\mathcal{L}_{m}

\mathcal{L}_{\text{sem}}

\mathcal{L}_{\text{bp}}

), and the aggregated loss is backpropagated to improve robustness and context-aware contact estimation.

Methods	RICH [ 7 ]				DAMON [ 1 ]				BEHAVE [ 29 ]
Methods	Precision $\kern-3.99994pt\uparrow$	Recall $\kern-3.99994pt\uparrow$	F1 $\kern-3.99994pt\uparrow$	geo. (cm) $\kern-3.99994pt\downarrow$	Precision $\kern-3.99994pt\uparrow$	Recall $\kern-3.99994pt\uparrow$	F1 $\kern-3.99994pt\uparrow$	geo. (cm) $\kern-3.99994pt\downarrow$	Precision $\kern-3.99994pt\uparrow$	Recall $\kern-3.99994pt\uparrow$	F1 $\kern-3.99994pt\uparrow$	geo. (cm) $\kern-3.99994pt\downarrow$
BSTRO [ 7 ]	0.65	0.66	0.63	18.39	0.51	0.53	0.46	38.06	0.13	0.03	0.04	50.45
POSA ${}^{\text{PIXIE}}$ [ 30 , 31 ]	0.31	0.69	0.39	21.16	0.42	0.34	0.31	33.00	0.11	0.07	0.06	54.29
POSA ${}^{\text{GT}}$ [ 30 , 31 ]	0.37	0.76	0.46	19.96	-	-	-	-	0.10	0.09	0.06	55.43
DECO [ 1 ]	0.71	0.76	0.70	17.92	0.64	0.57	0.55	21.32	0.25	0.21	0.18	46.33
DECO $+\mathcal{L}_{pal}^{2D}$ [ 1 ]	0.71	0.74	0.69	10.46	0.65	0.57	0.55	21.88	0.27	0.18	0.18	44.51

TABLE I : Comparison of GraphiContact with SOTA Models on RICH [ 7 ] , DAMON [ 1 ] , and BEHAVE [ 29 ] .

Method	H3.6M	3DPW	TARM	Dropout	Noise	Masking	RICH [ 7 ]				DAMON [ 1 ]				BEHAVE [ 29 ]
Method	H3.6M	3DPW	TARM	Dropout	Noise	Masking	Precision $\kern-3.99994pt\uparrow$	Recall $\kern-3.99994pt\uparrow$	F1 $\kern-3.99994pt\uparrow$	geo. (cm) $\kern-3.99994pt\downarrow$	Precision $\kern-3.99994pt\uparrow$	Recall $\kern-3.99994pt\uparrow$	F1 $\kern-3.99994pt\uparrow$	geo. (cm) $\kern-3.99994pt\downarrow$	Precision $\kern-3.99994pt\uparrow$	Recall $\kern-3.99994pt\uparrow$	F1 $\kern-3.99994pt\uparrow$	geo. (cm) $\kern-3.99994pt\downarrow$
(1)							0.73	0.75	0.74	8.94	0.64	0.59	0.57	21.98	0.32	0.22	0.25	14.58
(2)	✓						0.74	0.76	0.75	8.11	0.64	0.60	0.57	21.72	0.32	0.22	0.25	14.19
(3)	✓	✓					0.76	0.76	0.76	7.19	0.66	0.60	0.58	20.12	0.34	0.23	0.26	13.78
(4)	✓	✓	✓	✓			0.76	0.90	0.83	5.80	0.66	0.59	0.60	17.06	0.29	0.29	0.29	14.42
(5)	✓	✓	✓	✓	✓		0.76	0.94	0.85	3.66	0.69	0.63	0.65	15.91	0.34	0.33	0.33	13.09

TABLE II : Ablation Study of GraphiContact on RICH [ 7 ] , DAMON [ 1 ] , and BEHAVE [ 29 ] .

Methods	3DPW [ 26 ]			Human3.6M [ 27 ]
Methods	MPVE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$		MPJPE $\downarrow$	PA-MPJPE $\downarrow$
GraphCMR [ 8 ]	$-$	$-$	$70.2$		$-$	$50.1$
Mesh Graphormer [ 10 ]	$87.7$	$74.7$	$45.6$		$51.2$	$34.5$
POTTER [ 33 ]	$87.4$	$75.0$	$44.8$		$56.5$	$35.1$
Zolly [ 34 ]	$76.3$	$65.0$	$39.8$		$49.4$	$32.3$
ScoreHypo [ 35 ]	$71.9$	$61.8$	$36.1$		$37.4$	$35.3$

TABLE III : Comparison of GraphiContact with SOTA methods on 3DPW [ 26 ] and Human3.6M [ 27 ] (trained without dual-encoder transfer initialization for fairness).

Model	$\mathcal{L}_{\text{sem}}$	$\mathcal{L}_{\text{bp}}$	Precision(%) $\kern-3.27773pt\uparrow$	Recall(%) $\kern-3.27773pt\uparrow$	F1-Score(%) $\kern-3.27773pt\uparrow$	geo.(cm) $\kern-3.27773pt\downarrow$
(1)			70.8%	57.1%	62.7%	15.33
(2)	✓		71.2%	59.5%	63.2%	14.57
(3)		✓	74.0%	53.2%	61.4%	9.00
GraphiContact (Ours)	✓	✓	70.4%	63.3%	65.2%	15.69

TABLE IV : GraphiContact with Different Losses on the DAMON [ 1 ] .

Fig. 4 : Comparison of 3D human mesh reconstruction and contact prediction performance across DECO [ 1 ] , CONTHO [ 32 ] , and GraphiContact.

Algorithm: SIMU Strategy

Result: Contact probability map

\mathcal{P}

, Reconstructed mesh

\mathcal{M}

Init: RGB image tensor

\mathcal{I}\in\mathbb{R}^{B\times 3\times 224\times 224}

Step 1: Multi-Infer Perturbation: Generate

N

versions of

\mathcal{I}

\bullet

\mathcal{I}^{(1)}

= original input

\bullet

\mathcal{I}^{(2)}

= spatial dropout

\bullet

\mathcal{I}^{(3)}

= embedding noise injection

Table 9

Fig. 5 : Architecture of a Graphormer Encoder. We introduce a transformer encoder augmented with graph convolutions to effectively capture both global and local interactions for 3D human mesh reconstruction. The encoder is composed of a series of four identical blocks.

Methods	Params (M) $\downarrow$	Inference time (ms/img) $\downarrow$
BSTRO [ 7 ]	$25$	$9$
POSA [ 30 ]	2	2
DECO [ 1 ]	$87$	$20$
LEMON [ 15 ]	$29$	$18$
CONTHO [ 32 ]	$141$	$25$
PICO [ 25 ]	$-$	$2498$

TABLE V : Backbone complexity and single-image inference latency for contact prediction on DAMON.

Fig. 6 : Qualitative Comparison of Contact Point Prediction and 3D Mesh Reconstruction (Scenario A)

Fig. 7 : Qualitative Comparison of Contact Point Prediction and 3D Mesh Reconstruction (Scenario B)

Fig. 8 : Qualitative Comparison of Contact Point Prediction and 3D Mesh Reconstruction (Scenario C)

Fig. 9 : Qualitative Comparison of Contact Point Prediction and 3D Mesh Reconstruction (Scenario D)

Explore similar work

Pi-HOC: Pairwise 3D Human-Object Contact Estimation
Apr 14, 2026Sravan Chittupalli, Ayush Jain, Dong HuangHuman-Object Interaction
Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center
UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video
Apr 21, 2026Tanuj Sur, Shashank Tripathi, Nikos Athanasiou +44D Reconstruction Human Mesh Recovery
1National University of Singapore · 2Max Planck Institute for Intelligent Systems, Tübingen, Germany
GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction
Apr 21, 2026Pradyumna YM, Yuxuan Xue, Yue Chen +33D ViTs 3D Reconstruction
University of Tübingen · Tübingen AI Center · Westlake University +1

Apr 14, 2026J/K move · Enter open · S save

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Sravan Chittupalli, Ayush Jain, Dong Huang

Carnegie Mellon University, Robotics Institute · National Robotics Engineering Center

Human-Object Interaction

Open paper AskPDF arXiv

Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.

Abstract

Explore similar work

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video

GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction