SituHOI: Scalable Capture of Scene-Situated Full-Body Human-Object Interaction

Qifan Wang; Qihang Fang; Boao Shi; Yushi Du; Pei Zhou; Siyan Dong; Yanchao Yang

cs.CVOct 13, 2026

SituHOI: Scalable Capture of Scene-Situated Full-Body Human-Object Interaction

Authors: Qifan Wang, Qihang Fang, Boao Shi, Yushi Du, Pei Zhou, Siyan Dong, Yanchao Yang

Organizations: The University of Hong Kong

Abstract

Modeling human-object interaction (HOI) in real environments requires jointly representing whole-body motion, manipulated objects, and scene geometry, yet existing datasets trade off scalable capture against structured, scene-aligned supervision. We introduce SituHOI, a large-scale RGB-D dataset containing 18,429 annotated clips across 16 indoor rooms and approximately 86 hours of quality-screened activity. It pairs recordings from a single moving RGB-D camera with static scene scans, object geometry, language descriptions, and estimated human and object trajectories. Using the scans as persistent references, our pipeline combines camera localization and depth-based metric alignment with human mesh recovery, object tracking, and temporal refinement to register all geometric annotations in a shared floor-aligned frame. These data support FlexHOI-Bench, a goal-conditioned joint motion prediction benchmark with variable observation and prediction durations. Given observed histories, scene and object geometry, and a terminal object pose, models predict future human and object motion. We provide SharedHOI as a reference predictor and evaluate world-space accuracy and human-object coordination. On a MoCap subset, our pipeline reduces human G-MPJPE from 195.4 to 68.3 mm and object translation error from 43.0 to 28.7 mm. SharedHOI leads four adapted baselines on all three aggregate FlexHOI-Bench metrics. Data will be released upon acceptance. An anonymized code repository is available at https://anonymous.4open.science/r/iclr2027-anonymous-code-1DFB.

Figures & tables

Dataset	Capture	Hours	Seq./seg.	Envs.	Full-body 3D	Object pose	Scene geom.	Scene mesh	Semantics	Uninstr.
Full-body human-scene capture
PROX	1 RGB-D	NR	60 seq.	12	✓	✗	✓	✓	✗	✓
RICH	6–8 RGB	NR	142 seq.	5	✓	✗	✓	✓	✓	✓
Full-body interaction capture
GRAB	54-cam MoCap	NR	1,334 seq.	NR	✓	✓	✗	✗	✓	✗
BEHAVE	4 RGB-D	NR	321 seq.	5	✓	✓	✗	✗	✗	✓

Table 1: Comparison of capture setups, hours, sequence/segment counts, environments (Envs.), and annotations. Full-body 3D includes arms and legs; Object pose denotes dynamic-object position and orientation. Scene geom. includes point clouds and layouts; Scene mesh requires environment or aligned static-furniture meshes, not isolated manipulated-object meshes. Semantics covers action labels and language descriptions. Uninstr. excludes body/object tracking attachments, but permits capture cameras and calibration targets. ✓ : documented support; ✗ : criterion unmet or support unestablished; NR: not reported. Definitions and sources: Appendix A.3 .

Figure 1: SituHOI construction pipeline. We collect room scans, RGB-D activity, object meshes, and manual annotations, which support camera, human, and object estimation in a shared scene space. Annotations specify activity intervals, language descriptions, and target-object clicks.

Figure 2: Dataset distributions: (a) object counts; (b) clip durations, normalized per group.

Figure 3: SharedHOI predicts joint motion from history, scene/object geometry, the object goal, and query times. An endpoint-preserving head refines object translation before both trajectories are restored to the metric scene frame.

Pipeline configuration	G-MPJPE /mm $\downarrow$	RA-MPJPE /mm $\downarrow$	Obj. trans. /mm $\downarrow$	Obj. rot. / ${}^{\circ}\downarrow$	HOI-relative /mm $\downarrow$
OSX	195.35	168.84	–	–	–
FoundationPose	–	–	43.0	11.6	–
w/o $E_{D}$	73.6	50.0	28.7	7.5	55.2
w/o $E_{\Delta^{2}x_{t}}$	79.3	55.4	28.7	7.5	61.5
w/o object refinement	68.3	47.6	32.2	8.9	54.8
w/o tracking safeguards	68.3	47.6	48.7	13.2	69.6

Table 2: Accuracy on the 30-minute MoCap subset. OSX ( Lin et al., 2023 ) and FoundationPose ( Wen et al., 2024 ) are the initial human and object estimates. Each “w/o” row removes a component from the full pipeline:

E_{D}

is the depth-based human surface loss, and

E_{\Delta^{2}x_{t}}

denotes all human temporal terms in Eq. 1 ; object refinement is defined in Eq. 3 , and tracking safeguards are described under Object motion in Sec. 3.3 . All configurations use the same calibration and valid samples for each applicable metric. Dashes denote inapplicable or unreported metrics.

Figure 4: Tracking over 1–3 s without (top) and with (bottom) safeguards.

Method	All 11 pairs			$H=\text{3.0}$ s, $F=\text{0.5}$ s			$H=\text{0.85}$ s, $F=\text{1.6}$ s			$H=\text{0.25}$ s, $F=\text{3.75}$ s
	Root/mm $\downarrow$	Obj./mm $\downarrow$	F1(%) $\uparrow$	Root/mm $\downarrow$	Obj./mm $\downarrow$	F1(%) $\uparrow$	Root/mm $\downarrow$	Obj./mm $\downarrow$	F1(%) $\uparrow$	Root/mm $\downarrow$	Obj./mm $\downarrow$	F1(%) $\uparrow$
CHOIS	244.7±5.4	65.8±0.04	68.2±0.6	126.4±1.9	21.8±0.06	80.9±0.9	248.0±3.4	69.9±0.05	65.2±0.2	457.7±9.6	143.5±0.02	48.6±0.6
DecHOI	190.8±0.5	65.8±0.02	72.1±0.2	98.5±0.2	21.8±0.00	83.1±0.5	188.6±0.8	69.7±0.02	70.0±0.2	362.2±1.3	143.6±0.01	54.4±0.6
InterDiff	265.8±0.6	83.5±0.33	66.7±0.3	137.2±4.2	42.8±2.21	79.8±0.6	263.3±1.8	87.8±0.30	63.8±0.9	507.6±2.4	158.8±0.56	45.7±0.1
OMOMO	322.3±4.8	65.4±0.00	60.9±0.6	149.1±1.4	21.4±0.00	78.0±0.9	334.4±6.1	69.3±0.00	55.6±1.1	628.5±25.1	143.1±0.00	37.7±0.7
SharedHOI	187.2±3.3	61.0±0.22	74.9±0.4	96.3±3.0	19.7±0.02	85.3±0.4	186.9±3.2	64.8±0.22	73.1±0.4	355.0±4.1	132.9±0.66	58.1±0.7

Table 3: FlexHOI-Bench results: 6,948 testing windows, equal-weight averages over 11 pairs, and representative short, training-excluded, and long forecasts.

		6k total steps		12k total steps
Method	Metric (mm $\downarrow$ )	BEHAVE only	+SituHOI pretrain	BEHAVE only	+SituHOI pretrain
InterDiff	Object ADE	157.39±4.76	141.32±4.25	155.24±4.32	139.36±4.68
HOIMotion	Body MPJPE	67.93±0.34	56.59±0.47	67.73±0.18	55.43±0.49

Table 4: Transfer to BEHAVE under matched 6k- and 12k-step training budgets. For each budget, the control trains entirely on BEHAVE, while the transfer variant trains for the first 3k steps on SituHOI and the remaining steps on BEHAVE. All models are evaluated on BEHAVE. InterDiff reports eight DDIM sampling seeds and HOIMotion three runs; values are means

\pm

sample SD. Bold denotes the better mean within each comparison.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Full semantic corpus	Dynamic-object subset
Distinct indoor rooms	16	16
Semantic intervals	18,429	10,335
Annotation-covered time (h)	85.98	48.64
Dynamic object types	—	13
Median interval span (s)	15.80	16.03
Interval-span interquartile range (s)	12.07–20.43	12.47–20.35

Appendix

Table 5: Corpus inventory. Hours count unions of annotation spans within recordings; the dynamic-object subset is contained in the full corpus.

Figure 5: SituHOI’s 13 dynamic object types, independently scaled.

Figure 6: One continuous 684-frame interaction, with six real moments spanning 22.77 s. Saved SMPL-X/object estimates, original interval text, scene, depth, masks, mesh, and height traces share the source timeline. Phase/state labels are visual interpretations, not extra annotations. Insets use $t=12.63$ s; depth spans 0.4–5 m, with invalid pixels gray.

Component / setting	$N$	Metric	Ablated / control	Full	Paired benefit
Temporal terms	390	Body $\Delta^{\text{2}}$ p95 (mm/f 2 ) $\downarrow$	91.61	11.88	86.41% $\downarrow$
Joint second order	390	Body $\Delta^{\text{2}}$ p95 (mm/f 2 ) $\downarrow$	17.78	11.88	28.91% $\downarrow$
Temporal terms (long)	130	Body $\Delta^{\text{2}}$ p95 (mm/f 2 ) $\downarrow$	85.44	11.08	87.03% $\downarrow$
Depth-scan loss	390	Scan distance (mm) $\downarrow$	61.82	60.64	1.94 mm $\downarrow$
Keypoint loss	390	Scan distance (mm) $\downarrow$	62.62	60.64	2.03 mm $\downarrow$
Keypoint loss	390	Person IoU $\uparrow$	0.6812	0.6930	+0.0113

Appendix

Table 6: Effects of human fitting, object smoothing, and tracking safeguards. Each row compares the full pipeline with a component removed or a control setting on the same samples. For continuous metrics, Ablated/control and Full report median clip scores; Paired benefit reports the median improvement calculated clip by clip. Strict pass counts clips passing geometric checks.

N

: clips (acceptance guard: decisions); pp: percentage points.

Figure 7: Pipeline ablations corresponding to Table 6 : (a) human temporal p95 and (b) median paired scan-distance reductions on 390 clips; (c) temporal ratios on 104 object clips; (d) strict passes on 140 tracking clips; (e) IoU regressions over 900 candidate decisions; and (f) matched-budget lexical variation. Boxes in (a) span the 25th–75th percentiles, with 10th–90th percentile whiskers; dark bars in (a,c) mark medians.

Fixed short-clip cohort
Configuration	$N$	Body p95 $\downarrow$	Scan $\downarrow$	IoU $\uparrow$	2D $\downarrow$
Full fitting	390	11.88	60.64	0.6930	20.48
No second-order terms	390	91.61	53.47	0.6774	14.18
No joint second-order term	390	17.78	54.25	0.6781	20.91
Quarter temporal weight	390	19.05	53.41	0.6644	21.41
Half temporal weight	390	14.77	61.95	0.6804	26.11

Appendix

Table 7: Human-fitting component and weight ablations on 390 short clips and 130 long clips: cohort medians of per-clip metrics. Temporal p95 is in mm/frame 2 , scan distance in mm, and keypoint residual in pixels.

$H$ (s)	$F$ (s)	Root (mm)	Object (mm)	Rotation (deg)	F1 (%)
0.25	0.50	102.09	21.43±0.01	5.54	85.75
0.50	1.00	145.45	45.13±0.06	9.39	79.55
0.50	3.50	331.21	123.83±0.70	23.36	58.90
1.00	2.00	217.63	76.68±0.26	15.93	69.29
2.00	1.00	153.42	43.07±0.05	9.19	78.25
3.00	0.50	96.28	19.68±0.02	5.23	85.30

Appendix

Table 8: SharedHOI across all 11 duration pairs on all 6,948 HOLDOUT windows. Entries are three-seed means; object ADE also shows sample SD. Daggers mark pairs inside excluded training-duration regions.

Method	Context	Metric	Updates	Full	Removed	Shuffled
TMR	Text	R@1 (%) $\uparrow$	6k	8.91±1.21	3.11±0.47	3.42±0.94
SceneDiffuser	Scene	MPJPE (mm) $\downarrow$	12k	792.40±11.15	803.74±25.11	804.93±20.24
	Scene	MPJPE (mm) $\downarrow$	50k	760.11±18.05	809.53±31.61	802.52±22.29

Appendix

Table 9: Language and scene-context ablations on SituHOI HOLDOUT. Values are means

\pm

sample SD over three runs; bold denotes the best mean within each row.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 111 papers

MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances
Oct 8, 2026Mingyuan Lei, Yoonchang Sung, Tat-Jen ChamHOI Generation
College of Computing and Data Science Nanyang Technological University, Singapore
Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
May 21, 2026Jiahe Chen, ZiRui Wang, Feiyu Jia +7Robot Motion Generation Robotic Manipulation
Zhejiang University · Shanghai AI Laboratory · The Chinese University of Hong Kong
MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow
Oct 12, 2026Qiang He, Yonghao Zhang, Yinda Zhang +7HOI Generation One-Step Generative Modeling
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google

Oct 8, 2026J/K move · Enter open · S save

MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

Mingyuan Lei, Yoonchang Sung, Tat-Jen Cham

College of Computing and Data Science Nanyang Technological University, Singapore

HOI Generation

Open paper AskPDF arXiv

Generating realistic human-object interactions (HOI) in complex 3D scenes requires two complementary capabilities: reasoning about interaction feasibility in the environment and synthesizing realistic human-object motion. However, supervision for these capabilities is rarely available jointly at scale. Human-scene datasets provide rich information about environment-aware motion, while human-object datasets capture detailed interaction dynamics, yet paired human-object-scene data remain scarce. We present MAMHOI, an affordance-mediated factorization for scene-aware human-object interaction generation. MAMHOI factorizes scene-aware HOI generation through an explicit motion-affordance interface between scene understanding and motion synthesis: a scene-conditioned model first predicts where and how an interaction can be feasibly executed, and an affordance-conditioned HOI model then generates the corresponding human-object motion. This factorization allows scene understanding and interaction dynamics to be learned from complementary sources of supervision without requiring paired human-object-scene data. Experiments in complex indoor environments show that MAMHOI reduces object--scene penetration while better preserving human--object interaction quality, yielding more realistic and physically feasible scene-aware interactions. Project page: https://leimingyuan.github.io/MAMHOI-project-page/

Abstract

Human-Object Interaction

Explore similar work

MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow